The short answer is - I made a mistake in the contest rules.

The longer answer: When an algorithm hasn't traded yet, there are several metrics that can't be calculated. They are effectively division by zero. When we wrote the contest rules, we had to decide what to do with these undetermined metrics.

I really want the contest to be a good experience, particularly for new members of the community. New algorithms often don't have trades. If you enter the contest and you are ranked dead last, it's a big downer. So, I decided to make several of the undetermined numbers rank highly. That way new contestants get a warm fuzzy feeling. Then, as their algorithm starts trading, the undetermined numbers become determined, and a more accurate calculation kicks in.

All decisions have unintended consequences. In this case the unintended consequence is that a no-trade entry ends up looking a lot better than it really is. The good news is that every day the paper trading score is a higher fraction of the total score, and over time, that #2 will be #3 and lower and lower. It's not going to win the contest, and it certainly won't get picked for the hedge fund.

The even better news is that I've been working all week on the revisions to the rules for the June contest. They're not final yet and I haven't shared them, but I assure you that June's leaderboard won't have any highly-ranked no-trade algorithms!

Thanks for the response.

I can imagine the contest rules going through many iterations.

But like you wrote, "All decisions have unintended consequences." Wouldn't it be better if there was a discretionary aspect to the contest?

Ultimately, the biggest issues is overfitting...algos that do well in the backtest often do much worse in live performance. It's the reason I believe that the past winners have not been performing that well.

Check out this video for some perspective on backtest overfitting. You can find the corresponding paper here.

Dan, I hope the new rules will tilt more toward longer-term paper-trading performance. That would help counter the ongoing backtest inflation. Every month the backtests get better, and much of that improvement may represent overfitting. That won't translate into good long-term future performance, but with the current rules, it seems like a new algo with an incredible backtest and a few lucky one-day trades in its first month could win the contest.

Perhaps the metrics could include a statistical significance test of daily returns, excluding days with zero returns. For example, a one-sample sign test would favor a long series of small steady returns rather than a few lucky days.

Jonathan, there absolutely is a discretionary element to the contest (specific quote below). The discretion, though only extends to disqualification. We very intentionally don't have a human thumb on scale to change the ranking. We want the computation of the leaderboard to be transparent. (And, of course, we want it to reflect the same methods we're using to choose algorithms for the hedge fund).

I actually don't think that overfitting was a problem with the winners of the contest so far. I think the February's winner's flaw was that it had too high of a beta to the market, and the market tanked and took the winner with it. I think March's algorithm is doing just fine. The jury is still out on April's winner, but I am concerned that we still don't have beta weighted properly in the scoring. I'm worried that will bite us on April.

One of the changes we made to prevent an overfitted algorithm from doing well in the contest was by imposing the consistency score. If your backtest doesn't look like your paper trade, then your score is going to get beaten down. In practice, an overfitted algorithm's in- and out-of-sample performance are going to be inconsistent. This is really where my mistake in the rules lies: consistency is one of those numbers that can't be calculated when there are no trades, and I said, "oh, what the heck, let's just define it as 1.0 in this case." If I had said, instead, "no trades is clearly inconsistent with some trades, so make it 0.5" then we wouldn't have the #2 algorithm that we do.

I have read that paper, but the video is new to me, I'll check it out.

Michael, I think that the consistency metrics is essentially what you're asking for. Quoting the rules update I put up last month: "We're computing the consistency score using a kernel-density estimate using Gaussian kernels found in the Python scipy package. Both the backtest daily returns and the paper trade daily returns are each pushed through the function to fit them to a distribution separately. The difference between the areas of each of the distribution curves is used for the consistency score." Mea culpa for not applying consistency properly in the specific case where there are no trades at all.

"Additionally, we reserve the right to disqualify any entry at our sole discretion. . . . if we conclude that the algorithm is not suitable or financially prudent to trade with real money, we would disqualify the entry." https://www.quantopian.com/open/rules

How is the consistency calculated?

Is this the same as:

Stability of Return: This measures how consistently an algorithm generates its profits over time. (Mathematically, the R-squared of the linear regression line drawn through the algorithm's equity curve based on log-returns).

Scratch that. I got it.

https://www.quantopian.com/posts/scoring-changes-for-the-april-quantopian-open