Very impressive strategy!! I can’t really offer any improvement suggestions, but I’m curious what precautions you took to minimise overfitting the strategy on this particular market period?

Hi Joakim,
Thanks.

Regarding overfitting: In my opinion, if you start from a point of stock analysis, economic rationale and financial ideas, you are already two thirds the way there avoiding overfitting. While if you start from a point of view of data mining and technical indicators you'll risk overfitting way more. Both ways can lead to something useful, but if you take ideas and stock picking approaches developed in the 1950s or 1980s and they still work in the 2010s over a multi year period, you're probably on to something. What is the probability that a random factor (or combination of random factors) would have produced good results, let's say a sharp above 1 with the adverse conditions of costs and slippage? Very low I would say, so just the fact that it seems to work based on an idea, is a very big step in the right direction. The other way around, e.g. looking at 1000s of factors and slimming down, will produce a lot of spurious correlations and easily leads to overfitting in my opinion. As I said earlier, both can lead to good results and in the real world you'd need a bit of both, but starting with an idea instead of starting with data is--at least for me-- way better.

With this algorithm I'm very confident that it isn't overfit as it is based on an algorithm, i.e. exact same universe and exact same factors (this one being better risk managed), I made +6 months ago that has shown very good, in-line out of sample results. Since Quantopian seems to believe that every algorithm is overfit from birth, it's up to the algorithm to prove to Quantopian it isn't. If an algorithm after some months true out of sample is showing results that are in line with earlier behavior, I would say it's very unlikely that it was overfit.

Let me know what your thoughts are.

Jade Horse

Hi Bjarke (Danish? I'm Swedish as you could probably tell :)),

Very good points; I can't say I disagree with any of it.

One thing I might say though, and I could definitely be wrong on this, but I believe it's very difficult to completely eliminate the possibility of overfitting a strategy. In other words, most strategies are more or less overfit to some extent(?), and we can only try to reduce the likelihood of (unconsciously perhaps) overfitting(?).

Anyway, great strategy!

With the risk of maybe 'stating the obvious' (and perhaps again increase the risk of overfitting, haha ;)) one feedback I might add would be to try to find an alpha factor that's ideally uncorrected with your existing factors, and combine it with your other factors. This might improve your 'longish' drawdowns in 2016 for example, or perhaps improve your profit 'hit rate' and/or 'profit factor' of your Shorts (though Jess' feedback on my algo was not to worry too much about this - see the webinar)

Yes, it could be good suggestion to try adding an uncorrelated factor. However, I kind of feel like it's like the cartoon where the character in a boat tries to plug a leak, only to have it pop up somewhere else. And to me, it's kind of an "overfitter mentality" to worry too much short periods of falls that every (realistic) algorithm will face eventually. I think our time is probably better used trying to minimize risks; making the algorithms scalable and possibly most importantly to think about what made those factors/universe work in the past and if the same conditions are likely to be present in the future.
That said I might give it a go :) hehe

Yes I'm Danish :) and yes I had already figured out that you're Swedish already :)

Skål

Hi Bjarke,

Is the performance the same in earlier periods till 2004 or as far back as the Q data goes (how about the 2008 downturn).

I have only seen two sub 2% MDDs on Quantopian. Joakim and Bjarke.

Must be something in the water up there Denmark/Sweden, lol :)

-Leo

Must be something in the water up there Denmark/Sweden, lol :)

Um, there's also my 0%DD, not to start an international incident as we prefer to negotiate peace but my genes are Swiss (can also store your money for you).

Bjarke,

My feedback to you is borrowing from Jess's ideas in the "Live Tearsheet webinar".

a) The market has predominantly gone up since 2011. Try periods of economic downturns (maybe going back to 2004).
c) Try a different universe (maybe 500 equities from the lower half of Q Tradeable Universe)

With the hindsight that the markets have only gone up in the period you evaluated while the results look extremely good it is difficult to say if performance will be same if you switch the universe or include periods of economic downturns.

-Leo

@Olive Coyote
Thanks and thanks for the feedback. I'll look into that.

@Leo M
It's definitely doing worse before around 2010, but it's not an off-the-cliff, it's slightly up, still very little volatility, though a bit more. Its max draw down is barely 4%. So yes, it's definitely a better fit for the last few years than the year during and preceding the great financial crisis, but I personally would prefer that as I think that it's more important with good recent performance, and well combined with a good idea that the its worst case scenario not being all that horrible in order to control risk.

I can't really switch the universe as the factors work well for a particular subset of the QTU. Of course if the market change its dynamics this algorithm will not have as good a performance, but since it's a version of the algorithm that I used in contest 38 I feel pretty confident that it wasn't overfit.

Not all factors would work equally well for all. Just see the difference between many financial ratios between e.g. a bank and a biotech company.
What do you think?

Bjarke, would you have a tear sheet for the period 2004 to 2012. I'd like to compare both the tear sheets for further feedback.