Vladimir - we think that consistency is important no matter what. If a live algo is out-performing the backtest, the question is immediately, "why is that happening?" If it's not something that was predictable, then it's probably some form of luck, and luck flows both ways. We're valuing predictability very highly. I don't think of it as "punishing someone for doing better" because "better" is defined as "consistent." That said, if there is a reason why the performance changed, that's interesting and we might be able to learn from it.
I'm glad you looked at the April results. We put that CSV there for exactly that reason - so people can study it!
When an algo does absolutely nothing, makes no trades, then the consistency can't be calculated (division by zero, effectively). An incalculable factor for the other 7 metrics in the contest aren't so big of a deal because they are ranked. Consistency, though, is multiplied, so we need to have something for a score. We could have chosen 0, or .5, or 1, or anything. In the end we went with 1 because consitency is applied gradually over the first few days of trading while the paper trading record is very volatile, and is fully applied at 20 days of trading. For the algorithms that still haven't traded after 20 days, like the one that you reference, it's definitely an anomaly. I'm simply not worried about it because I don't think a no-trade algorithm will ever challenge for the lead.
Taking a step back, I think it's a healthy thing to see how consistency is affecting the contest. In the chart below I plot consistency against the paper-traded Sharpe rank. That means that the highest Sharpe algos are on the left, ordered left-to-right. The higher dots are are the most consistent algorithms. The color is the final contest rank - dark green are top-ranked, gray is middle of the pack, and red is bottom of the pack.
What you see is that, roughly, it goes from green to red, left-to-right. You see algos that are on the left side that aren't well ranked in the contest because they were pushed down by consistency. This is, more or less, what we were hoping to see. I'm sure we'll be doing other revisions later.
Also note: in the upper-right corner are the zero-trade algos, all stacked on top of each other. They are statistically a little weird and that shines through.
Finally, taking an even bigger step back, it's important to remember that this contest isn't just about the contest - it's about the Quantopian hedge fund. When we launch the fund we're going to have more than a couple algorithm writers. In time, we're going to have dozens. The algorithm that comes in 10th, or 30th, or even 100th might end up being a better fit for the fund than the one that came in 2nd. Algorithms that don't trade won't get picked, nor will inconsistent algorithms. Overall, if you write your algorithm to meet the scoring package you see here, it can get you a bigger win than just the contest.
