Well, according to Dan here,

"One month from now, dozens in the fund".

I wonder if they will use the same criteria to select algorithms for inclusion in the fund?

Lucas, it's an interesting question that I think about every day.

One thing to hold in mind is that the ranking technically only needs to be correct at the end of the month. Of course, I'd love it to be correct every day. But the nature of the game is that we're initially ranking people with a tiny sample size. When those small sample sizes are ranked, we get some wild numbers until the samples get larger.

One change that I am thinking of is introducing a filter sort of like the beta filter where you have to have at least 10 or 15 days of live trading. That would mean that all algos are ranked, but the top n algorithms would all have a certain amount of stability. And, all entries would get the requisite number of days before the prize is awarded. Basically, it would keep you from ranking highly until enough days have passed.

I'm not sure that we know enough about an "ideal Sharpe" to draw a curve like you describe. You can download the spreadsheet and draft your own scoring system - let us know if you see something you like.

As for the weight of the forward testing, it's articulated in the contest rules.

The Participant's overall contest score will be a combination of the
two, with greater weight given to the live trading score as time goes
on. At submission, the backtest score is 70% of the total score. The
backtest score’s weight linearly decreases to 0% over 60 days of live
trading; thus on day 61, the total score is equal to the live trading
score.

I've looked at a few different backtesting changes. If we give a random backtest, people who are trying to game the system will just resubmit over and over - lots of noise, lots of churn, and it doesn't solve the problem.

Beginner, you changed my quote! The actual quote was: "One a month for now, dozens per month in the fund." I wasn't committing to dozens in the fund next month - though that sure would be nice!

That said, we are shaping the fund the same way we are shaping the contest. I think you should see that the contest rules are saying the same things as the fund information. The contest will continue to evolve to match the goals of the fund.

the backtest score is 70% of the total score. The backtest score’s
weight linearly decreases to 0% over 60 days of live trading; thus on
day 61, the total score is equal to the live trading score.
This is a good method to remove the noisy backtests but I am not sure if it is a good way to select the algorithm.
If we assume all backtests are honest, the paper trading would be used to validate the backtesting and have a fraction of its score. The current score is favouring lucky algorithms that do well for 1month. The competition lasts 30 days, If someone submit the algorithm 1 day before the deadline the backtest that could be dishonest will weight 35% of the score and the paper-trading 65%.
I think 65% for the result for the result one month is way too much. I am not sure giving all the weight to the paper-trading is the way to go.

Maybe Quantopian should extend the paper-trading period to 3 months, after 1 month filter the top X and repeat the process.
And still with three months of paper trading 65% might be too much weight. Extending time might help to improve the selection but then waiting 3 months for the result might demotivate some, but that is another issue

The only way to discourage fitting is NOT relying on in sample performance for scoring at all. Use in sample only as minimum filtering, and increase the required out of sample length.

Is there really a need to penalize higher Sharpe? I don't think so. While Sharpe is not the best measure of risks and rewards, a higher Sharpe is not a bad thing to have. Stat arb strategies tend to have a higher sharpe. That's by design.

That said, I completely agree with Dan. A beta as close to zero as possible is almost a prerequisite if you are looking to generate absolute returns.

Lastly, a high weightage for backtest does incentivize the business of 'trying to fit' somehow. But one or 2 months of live trading may not be enough for strategies that don't do a lot of trades in this period. A better way of doing that may be to measure strategies over 30 (or any statistically significant number) trades that it takes. Also, Quantopian can look at dividing the historical data into training data and test data randomly to discourage fitting.

One more suggestion: if draw down in out sample > x times draw down in sample => eliminate the algo. If the algo have .2 draw down in 2 years in sample and 1% draw down in 1 week out sample, there is obviously something wrong with it.

To sum up my suggestions:
1. Use in sample as minimum filtering only
2. Increase minimum out of sample length
3. eliminate algos that has out of sample draw down > x times in sample down down

I have to figure that the art of subtlety biasing a backtest would be welcome by Quantopian, in prospective managers. They'll be marketing their hedge fund with in sample and out-of-sample results, I'd imagine, so the contest needs to include some backtest component. Otherwise, they might get a bunch of complete straight arrows going up against Wall Street. I know, another whacky perspective from yours truly, but being able to spin just the right amount is expected in some professions.