A correlation matrix in Returns or holdings space for all contest strategies would be quite interesting and helpful to have on an ongoing basis.

+1

Since Q no doubt takes correlation into account when licensing new strategies, I don't see why they don't apply it to the contest as well. If the most novel strategies bubbled to the top, it would encourage more novel strategies.

In general strategies cannot be uncorrelated. Pretty much the idea is that everybody is trying to find a function f such that

returns = f(some inputs) + shocks.

The power of shocks is very strong compared to the power of signal f even when everybody uses different inputs or different f, hence the low singal to noise ratio in markets. Thus returns are very correlated with the shocks rather than they are correlated with f. Since all the competitors are experiencing pretty much the same shocks your returns will be highly correlated.

@Fly Juice, not every algorithm is a mean reversion algorithm, because not every market inefficiency is a "shock." To illustrate, I have a momentum algorithm that's scoring 1.2 in the contest (not making it onto the leaderboard...yet! but pretty close). It works by specifically filtering the universe down to the stocks that exhibit the least amount of "shocks."

I would expect my momentum algorithm to have fairly low correlation to Jamie's mean reversion algorithm. They likely operate on entirely different stocks.

Interestingly, I have another momentum-based algorithm that is also scoring >1.0 in the contest. So it seems we've entered a market regime that favors technical quant strategies generally -- not necessarily just mean reversion strategies.

Also interesting to me is that some of the all-time best performing entrants are scoring horrendously low at the moment. A critique I've presented before is that the contest format favors strategies that exhibit intermittent winning streaks over algorithms that are more consistent. Because negative scores aren't penalized, a strategy that is down -20% one month, is flat for the next month, and then up +20% on the third month will likely score much higher than a strategy that consistently gains +1% every month month, though the latter strategy is obviously much better.

@Viridian Hawk I don't know how you translated what I wrote to "mean reversion". Momentum or mean reversion can both be represented by the same formula. It is the definition of "f" that determines the type. Least amount of shocks simply translates to low beta. Everybody experiences the market shocks based on their beta, and simply a reflection of how much market risk they would like to take.

I don't know how you translated what I wrote to "mean reversion".

I assumed by "shocks" you were referring to volatility. "Shocks" is not a technical term, nor have I ever seen it used in the context you are using it.

returns are very correlated with the shocks rather than they are correlated with f.
Everybody experiences the market shocks based on their beta, and simply a reflection of how much market risk they would like to take.

You're thinking about this wrong. These are dollar-neutral strategies hedged against common market risks, including against market beta. If you want to use the signal-to-noise analogy, these strategies are like XLR cables that carry the signal on one wire and the inverse signal on another. Any "shock" or noise gets cancelled out when the inverted signal is inverted again and summed back together with the original signal. These strategies do not have high exposure to noise (if you want to call market beta "noise") compared to signal as you suggest, and market beta does not explain their correlation. In hedged strategies returns do not derive from market exposure.

In general strategies cannot be uncorrelated.

Of course they can.

The post that hurts.

I've spent years of study and research, 2843 backtests (thanks for stats in profile, Q).
And my top performing algos either exceed exposure to short term reversal, or after tweaks pass but fail to make make it to top 10 (even to top 100).

I look at the scores of top 10 and wonder how. Are you applying magic outside Hogwarts?
If my algo earns twice SP500 and is graded in the end of ranking what do yours earn?
Is it possible at all on equity markets?
Or alternatively is your volatility so tiny it makes the score so high?

Seriously, the score of 2+ means the algo makes at least 0.06% daily yet experiencing low exposures to the risk factors and low volatility.
And even if the returns are distributed equally across the stocks selected the maximum position appears 0.048 corresponding to at least 21 stocks with such a great performance.
Ten people have found ways to daily select no less than 21 stocks from 8k+ which return on average 0.06% daily in balanced long/short positions while fitting the risk model.

Why am I so dumb. When I've finished crying I'll return to the analysis with a cold head and hopefully write a post like this one upon a time.

@Dmitry You can download the detailed stats for the contest in CSV format. You'll see that based on max position concentration, Byzantium Cheetah's portfolio holds at a minimum 200-500 positions, and likely far more. That's how s/he achieves such low volatility. High returns don't matter so much as low volatility. The top-performing entries have just over 10% total returns.

With enough entries, one would expect that some would have phenomenal streaks of outperformance just by chance. Don't take it too hard. The current winners will eventually slide off the leaderboard, just as previous winners have done.

I am in second place and what I can say is that I developed a very complex strategy employing various ML techniques combined.

@Emiliano, congratulations, well done!

If you don’t mind sharing, what’s your average daily turnover for that strategy, as well as average volatility?

Sorry Joakim but I can't share these infos 🙊

@Emiliano I don't think you have anything to fear revealing your average turnover rate and volatility...

@Joakim his volatility is somewhere between the min/max values of 0.035 and 0.064, so pretty low but not super low.

A funny thing I just noticed while looking the contest scores is how many different people are trading the exact SAME strategies. I assume these are cases of people cloning other people's work off of the forum and submitting it unaltered as their own. Hrmph! And they're scoring fairly high!

Something about that feels dishonest, or at least not in the spirit that algorithms are shared here -- especially since actual cash prizes are at stake. I've posted various algos on these forums for the sake of helping others learn, or on the belief if my ideas inspired somebody those people would return the goodwill and share the improvements back.

@Dmitry, how does it feel that despite all your hard work, people are scoring much higher than you (in the 50s) without having to put forth any effort or thought at all?

For that matter, these lazy entrants are placing almost 200 places ahead of Maroon Hippopotamus, who still leads the all-time board. What does it all mean?

Thanks @Viridian. I was more interested in the average daily turnover actually. My guess is that it’s quite high, perhaps just under 65% (for most strategies in the top 10), but maybe I’m wrong. I didn’t realise these were proprietary ‘secret sauce’ figures. Good thing I didn’t ask for a full tear sheet haha.

@Joakim: Have you seen the video "A Data Science Approach to Managing Crowd-Sourced Systematic Trading Strategies” by Dr. Jess Stauth? It seems like most higher ranked algorithms have lower turnover rates in the 2018 dataset [1]. Do you think this is different now in the current contest?

[1] -- https://youtu.be/CmuRTj0BOSE?t=1547

Hi Enigma,

Thanks for pointing that out! Maybe I'm wrong (hopefully I'm wrong), but I remain skeptical. If anyone consistently in the top 10 recently could disclose their average daily turnover I'd be very grateful!

@Joakim,

max turnover, min turnover are in columns BD, BE of daily leader board spread sheet.

daily-leaderboard-2019-08-12

rank score max_turnover min_turnover
1 2.094230931 1.001927801 0.040599459
2 1.924747724 0.987601091 0
3 1.853173381 0.854318567 0
4 1.73472803 1.818938532 0.008935463
5 1.65657798 1.000594268 0.027647264
6 1.620420368 0.796574515 0
7 1.553676969 1.262850272 0
8 1.519812058 0.906366232 0
9 1.506548923 0.952612843 0.057455081
10 1.50547302 0.9526956 0

@Vladimir,

Thanks, I'm aware of this. I'm interested in the average daily turnover though, not the min/max.

Well, I'm pretty much in the @Dmitry camp...getting "good scores" whilst meeting the constraints has eluded me!

@Joakim: I'm assuming you want to know the average daily turnover to see how close the algo is to an HFT strategy...just on a daily basis
This, to me, seems one of the new-big-deals in algo trading...the ability to trade massive amounts of names and shares on a short-term basis.
I don't believe that this was feasible in the past except for the largest of traders.
We know that HFT worked...now trying to find strategies where daily-HFT works.
I think how it relates to the contest is that to get >K, over a 63-day window, you need to keep AvgVol very low(0.02-0.04), and the AvgDailyReturns > K*0.03 %

Comments welcome!

It appears the min column for turnover in the CSV is broken. Most people's min turnover is presented as 0. Considering that most algos run the optimizer daily, it seems nearly impossible that no capital would be transacted on any single day for the vast majority of participants.

@Joakim: My Causal Inference Algorithm is currently in the top 10, so I can confirm that there is at least one algorithm currently doing (a little too) well with a turnover of ~0.075 per day.

@Enogma,

Nice, thank you, really appreciate that! I concede to being wrong on this. Also, congratulations on creating a strategy that’s working great on live data! For me, back to the drawing board.

Thank you! But you do have this beautiful VALUE Factors composite algorithm, I'm looking forward to see how it will perform in the contest.

Thanks @Enigma. However, I can tell you that that one did really poorly during the 1 year holdout period (it uses FactSet), to the point that I’m almost declaring it overfit. I will let it run for at least 3 months in the contest though first. That’s my own rule for anything I submit to the contest. :)

@Viridian, I wonder if the 0 min turnover strategies are set not to trade on half days, and that’s why they don’t have any turnover on those days?

Value has been doing terribly in 2018 but especially since Dec 2018. The value to growth differential was at a 30 year low even before Aug 1st and it has only gotten worser. So I wouldn't necessarily say overfit, just that the market conditions are way too unfavorable for value investing.

Thanks @Leo, appreciated! I thought about that too, but then my Value factors aren’t very correlated with traditional Value, so I thought maybe I’m just fooling myself? Anyway, once I have a Growth factors and a Quality factors composites in the contest as well I’m hoping they’ll be fairly uncorrelated so at least one of them will do ok at any given time? I’d like to enter a Reversal factor too but I’m just not very good at that so I might try a Low Vola / Low Beta (BAB) one instead.

Low Vola / Low Beta

what would that look like in the long-short format?

Long the above, short the opposite. Might correlate with my Quality factors though, so I might include it in that composite instead.

This post got me taking a closer look at the contest, and I noticed that out of the top-placing 112 entries (for the day I was looking at), 45 of them are duplicates, representing only 16 unique entries. How does this happen? -- Simple: somebody shares some code on the forum for the purpose of getting help, soliciting collaboration, or generously educating, and other people simply clone the person's code and submit it unaltered into the contest as their own.

Jamie's critique turned to a fairly high-level, ascribing the risk model to a high correlation among submissions. My critique is pretty low level: entries are similar because a huge percent are literally copying other people's ideas, often byte for byte.

There are just under 400 contest entrants total. Roughly the bottom 50 have a score of 0. Of the remaining ~350 positive entries, roughly the top 100 is 40% clones, which doesn't even capture entries that are slightly modified clones. Some portion of entries is total rubbish -- I'm not going to draw the line, but we all know it is there somewhere. What's clear though is that the top is heavily populated by unoriginal ideas, which skews everything. There are far fewer unique entrants than would appear, and of the remaining, it's skews heavily towards the sub-par.

To me this raises serious questions about the integrity and efficacy of the contest. Also, it seems there's an element of the platform and contest's design that encourages unoriginality.

What's up with people? Why is this behavior so ubiquitous?

I'm pretty sure of those 16 top-performing cloner strategies at least one must be one of my strategies, since I just checked and the OOS has been really great and there were a number of "clones" of it. I'm running a variation of the same strategy and have been scoring >1.3. Is there a way I can calculate what my contest score would be so I can see which users are scoring with my code? What happens if somebody beats me onto the leaderboard with my own strategy? Do they get paid for submitting my work? What happens if they -- gasp -- submit it for an allocation?! Ok, so maybe I'm a dumb-dumb for reciprocating and showing my appreciation for the help I've received by posting publicly what I did with that help. Learned my lesson.

What's the purpose of limiting each person to 5 contest strategies if we have essentially unlimited submissions (via cloners)? If the 5-limit is to encourage quality, well there's no shortage of rubbish in the sample from other users, so why limit the most talented users? Of course Quantopian is up against the infinite monkey theorem, which the 5-submission limit helps combat, but it is wholly inadequate.

The platform needs some metric to score entries by originality, and it needs to be designed to favor the originator. This would both make it more fair for us, and also encourage the development of more quality, novel strategies for the Q Fund.

And a caveat -- I once racked up a bunch of contest winnings with a strategy that I later noticed had a typo in an equation that made the whole thing complete nonsense. In other words, thanks to the magic of chance, a random walk can place you for an extended stay in the top 10. My nonsense strategy was no doubt highly uncorrelated to the rest of the contest participants.

@Viridian, you make good points I think, which I largely agree with (coming from someone at the bottom of the rankings).

@Viridian, I think your perception of the contest standing might be totally wrong.

You have 400 strategies that have been submitted. For one thing, we, or any of us, cannot see the inside of any of those strategies, and neither should Q, for that matter. All anyone can view is the outcome of those strategies and some of the metrics gathered by Q in order to differentiate the strategies based on Q's most desired traits.

Like I do not participate in the contests for the simple reason I do not design strategies for the contest and therefore should have no say as to the composition of those strategies. However, I am pretty good at analysis and numbers.

Look at the rules of the contest per se. You have to use the Optimizer API. You have to be market-neutral, sector-neutral, beta-neutral with near full exposure in a market that has been going up for the past 10 years. Not only that, but you also need a relatively high daily turnover (even if it should not be necessary). You have to follow made-up style constraints which will tend to obliterate variance. And as a consequence will be forced to trade 200+ stocks to mitigate the impact of drawdowns and volatility while smashing your average net profit per trade to double digits or even less.

And then, you would want all those strategies to be all different when in reality they tend to be variations on the same theme using about the same stock selection criteria with outdated factors, fundamentals and what have you that seem to operate over the fumes of randomness, more or less.

What would you expect?

I find that using such trading techniques under those constraints and barely achieving a 10% CAGR should be considered as high marks. You smash \(\Delta p\) to peanuts and then wonder why you need hundreds of thousands of trades to make a strategy worthwhile? I would suggest: wake up to the data, the structure of the imposed game you have to play and under which conditions you have to play it.

The only thing that matters in that contest are the top 10. Everything else that it be cloned from whatever or whoever is not really relevant.

My suggestion is: design to be on top using whatever is at your disposal and let all the other see you from far behind. None of it has any impact on what you do if your strategy is right up there at the top.

Even if someone “cloned” one of your strategies, it is not by the output that you will be able to say that it was your strategy that they cloned. It is only by seeing the code that you would be able to do that. And that as a premise is not available to any of us.

I would add: let anyone clone whatever you want to show as a trading strategy, just do not show your best effort. It is as simple as that.

@Guy if every single metric down the line is identical down to the last floating point decimal place, there is no question it is the exact same, unaltered strategy. There is too much noise in the markets to accidentally have the same drawdown, same volatility, same return, same sharpe, same sector exposure, etc. just by chance. Maybe by chance one could be the same one day, but if they are all the same day after day, it's a clone.

@Guy I do agree I should not be too concerned with getting ripped off. You make a good point -- The copycats do not have the spark of originality, and so they'll never get very far. My strategies and implementations are always evolving, so I have nothing to worry about really. My winnings so far are in excess of $1000, but I would like that to be more. On principle I disagree with people who just take the fruits of others' hard work for their own benefit. It's discouraging. But half my critique is not for my own sake, but for Quantopian's sake. Something is wrong when the platform so heavily encourages grifters to the point where they are as common as the serious participants. They are not the users who will develop the alpha Quantopian needs.