Well here is the offical rules page https://www.quantopian.com/open/rules.

Edited original post for clarity.

More clarity:

• Is there equal weighting to each quantifiable metric when comparing entries? (I.e. are returns = beta = sharpe = drawdown, etc, or are returns weighted most, etc.)
  
• What are the qualitative steps in the judgment process?
  
• Should we expect out-of-sample historical tests? I.e. Should we optimize over a decade of data as well as focus on making something that works going forward?
  

Hello Alexander,

It's been confusing, for sure. However, the basic idea is that Quantopian is in the process of building a hedge fund, https://www.quantopian.com/fund. So, over the past year or so, there's been a sausage-making development process that has played out as they've learned how to get what they need from the crowd of users participating in the contest and how to manage winners' algos trading real money.

A few changes I'd anticipate:

• The contest now has a 6-month out-of-sample paper trading period. Q will need to sort out how to weight the out-of-sample results against the 2-year backtest. Perhaps the pyfolio tool will be formally incorporated into the contest (https://www.quantopian.com/posts/new-feature-comprehensive-backtest-analysis)?
  
• Some encouragement/incentive for contestants to look back further than 2 years might be implemented. I think the problem is that some algos might not be back-testable beyond 2 years (e.g. for a fixed, small stock universe), so it is not clear how this would work as a rule. However, it is now clear that if your algo stinks going back 10 years, it'll be rejected (although if you can explain recent good performance, then maybe not).
  
• Note that on https://www.quantopian.com/fund, "Allocations will range between $1 and $25 million per algorithm." so the $100K level for the contest is 1-2 orders of magnitude off. One might expect that Q would start looking at capacity, as well (although my sense is that it gets tricky, since each algo might require a custom slippage model to be developed based on the details of the algo).

Reading between the lines, Q has an opportunity to line up some initial funding (not from institutions, but from private investors, most likely). To get it, they need to put together a sales pitch, showing that they've done their due diligence. So, the more you can do to arm them with evidence that your algo is the super-duper, the better your chances of getting money from their funders--there are no rules in capitalism. I would think of the contest as a first cut. If you submit an algo that does well in the contest, then you might have a fundable algo, but you'll need to provide more support (even sending them the code).

The other note is that although Fawce tries to claim otherwise on https://www.quantopian.com/posts/full-winners-returns-data-now-available, the contest winners are the fund at this point, and will continue to be part of the fund, at a seed capital level. Q wants the real-money trading results so that they can use them in their sales pitch to investors. So, if you do win, you are best to have predictable returns for 6 months that will give investors a warm fuzzy to ramp up to $1M or more in capital. If your real-money returns suddenly jump up then caveat emptor will start flashing in their minds, if no explanation is provided.

Grant

Hello Alexander, welcome to Quantopian. I'll try to add to the previous answers.

The contest rules have evolved over the last 10 months, and I know that has caused some confusion. The good news is that every contest started and finished with the same rules. We didn't start a contest and then retroactively change the rules. (The exceptions to that are things that changed to the winner's benefit, like when we started paying winners monthly instead of after 6 months). So, what you see is what you get, month-to-month.

Also, the pace of the rules changes have slowed down. We announced significant changes in May and June, and the 6-month duration change in July. August and September didn't have significant announcements, and I don't forsee any major ones this month. If there are any changes for the next contest (as I said, only minor ones anticipated) I will make every effort to announce them next week.

As for the disqualification this month, I'm hopeful that the 6 months of out-of-sample testing will make that a very rare occurrence in the future. The problem with the 1-month contest is that it was just too easy to be lucky and grab the win. But when one looked at the algorithm from a broader time frame, it was doubtful that the results were repeatable. That was what forced the disqualification. Going forward, it is much harder to have 6 lucky months in a row. I expect the winners of the 6-month contest aren't going to have large discrepancies in performance for in- and out-of-sample. There will be a lot of entries that have differences, but they won't be the winners.

If you have a particular entry you're looking at getting feedback on, please run the backtest, create a tearsheet, and then share the tearsheet in the community asking for feedback. That tearsheet doesn't include your source code and we can review the results. Tearsheets have the option of suppressing positions, too, if you want to hide what you're trading in. You can also email us privately at [email protected] if you don't want to share the tearseet.

Some of the other questions you asked have different answers for judging the contest v. for analysis for the fund. Within the contest we are constrained by the rules we create - we can't change them retroactively, and we hew to them closely. We have to pick a winner every month, on the calendar. With the fund, we have discretion, we learn faster, and we can pick winners when they are seasoned, regardless of the calendar. With the fund we're not using a strict calculation; we're using some broad metrics to screen algorithms, waiting for the out-of-sample data come in, and then evaluating each algorithm on its own merits.

Dan,

You speak of the fund in the present tense, is there actually a fund up and running? Why would you use different metrics to select investments for the fund and for the contest? How have you tested these different metrics? Curious and anxious to hear about the actual Quantopian crowd-sourced hedge fund.

Thanks, Dan. That's just what I was looking for.