I ran your algo with a pseudo-random universe of stocks instead of your optimized list. The results aren't pretty. In 2008 the algo starts borrowing heavily just before the credit crunch and never recovers.

Hi Dennis C,

thanks for the interest. You're absolutely right about the ETF basket chosen. Choosing stocks at random will undoubtedly find LOTS of 'baskets' which won't perform at all. That's why you need to optimize. As I mentioned, I used zipline to do this offline, as there's really no way to do it in Quantopian. The result was a combination of ETFs, RS weightings, lookback parameters and no. of ETFs to hold.

With long term investing in mind, I would venture that portfolio optimization is a must - mean-variance, efficient frontier or whatever. Clearly, this hardly applies (if at all) to day trading. If Quantopian is also interested in attracting long-term investors (like me), then optimization will be a must. Of course there is always zipline offline.

rgds,
DaveG

What period did you optimize for? Was it the same years as the backtest you posted?

I'd be interested in seeing your basket optimized for 2007 and backtested on 2008. Same for 08/09, 09/10, etc.

The optimization I've done is very rudimentary, using different start dates until the present. I deliberately included 2007-2008 in the example to show the performance before, during and after the 2008 melt-down. What you are asking can easily be achieved by porting the code to zipline and looping through all the combinations you're interested in (you could also do the same online manually, of course :) ). I'm not suggesting that the optimization process is easy - I'm doing it by stupid brute force, which is computation-heavy and time consuming. Now having someone weigh in on this conversation with much better knowledge on this kind of optimization would be great! Especially the choice of the 'optimum' basket to use for different market conditions.

Thanks Dave. That's totally understandable.

But the reason I was asking was because of optimization bias / overfitting. You didn't answer whether your optimization overlapped the dates you are showing above.

Edit: Sorry, you did answer: "using different start dates until the present". But that is a concerning answer since it completely overlaps.

This is an interesting thread and very similar to the thread I was going to post. I've seen this issue referred to as "data mining". Now, I've heard of the phrase "data mining" used in a positive sense in biomolecular engineering but I see it used in a negative sense with regard to trading algorithms, and for good reason. Essentially, by "optimizing" performance to a given set of data and then testing on that same data set you bias the results toward the positive.

I am having much difficulty with this issue now. Furthermore, I've tried to go about this in another way as well. Essentially I calculate the "ETF replay algorithm" (which I think btw may be (0.4*63 day return) + (0.3*21 day return) MINUS (0.3*21 day volatility) ) but rather than doing a backtesting, I plot the value of the equation on the x-axis versus the 63 day FUTURE performance. I see no correlation. Is it wrong to assume a correlation must exist for a trading strategy to be more than dumb luck?

@Daniel, I think you are on the right track. However I am by no means an expert on this.

My take is that data mining can easily result in overfitting. And when that happens the optimization is unlikely to be useful on out of sample data (e.g. a time period outside of your optimization). And you won't notice if you test on the same time period.

In other words if you optimize and test on the same time period then during the testing you have a "time travel problem". This is because your optimization used data from the future relative to the "current date" within the backtest. This is also known as "look ahead bias".

To find a cure for this problem we must slay two issues: overfitting and time travel.

For overfitting I believe the answer is to use simple models (linear models are good) to reduce the chance of curve fitting. And don't spend a lot of time tweaking. Even better let your algorithm find "natural" ratios instead of putting in magic numbers. So an algorithm that responds to a 200 day trailing window is better than an algorithm based on "optimization" of years of historical data.

For time travel (aka look ahead bias) you should always validate your algorithm on previously unseen time periods. This includes all tinkering and development you did to choose your algo and tweaks you made to it. For instance you could do all of your testing on 2009-2010 time periods (hopefully randomizing within that) and then validate on 2011-2012.

If you have been developing on all time periods (2002-2013) and build a great "optimized" algo then you will have no choice but to start live paper trading to validate it. That is because the only "unseen" time period is strictly the future. (That is the great secret of "walk forward" testing since you are guaranteed to get unseen data). The downside is that your validation test will take days or weeks!

I agree on all fronts Dennis. I guess we'll just have to keep these issues in mind as we move forward. I just wish there was a more systematic way of proceeding.

Where did you learn to code like that bro? Sort the lists using functions etc?
I just discovered quantopian the other day.. made an account.. wrote my first bot this weekend. Just realized it's very similar to yours, etf replay style :)

Each month, buy the top 2 relative strength out of this basket: VTI, VEU, TLT, VNQ, GLD (Ivy5)
it has a 16.2% CAGR since 2003 with 12.6% drawdown (on etf replay).
I wonder why these RS strategies are all lagging in the 2012-2015 period though - maybe too many RS strategies so RS-killers were developed?

My coding is horrible - i'm basically keeping the "max" and "maxposition" inside a while loop - not even using sort :(
I was wondering if you had any tips on the quickest, fastest way to code better :)

thanks a lot!

Hi Ben - Do you know any other programming languages? If you are new to programming, I would recommend you google a Tutorial. I find other languages, like C++, more ubiquitous and so I usually recommend starting there. However, there is nothing wrong with starting with Python. If you are familiar with programming - i.e. the concept of loops, conditions, etc. - then I'd search for a language reference and just keep at it. Sharing parts of your code can bring helpful insight from others.