Our next edition of the Quantopian Open kicks off on August 3rd at 9:30 AM EDT, and it's time to update the rules - and the prizes.
Last month we made a couple of significant changes: we stopped scoring the backtest, and we required algorithms to be hedged. We've been very pleased with how those rules worked. The algorithms sitting on the top of the leaderboard this month have been far and away the best-quality set of algorithms we've seen yet. We only have a few minor tweaks to leaderboard scoring which you can read about below.
The biggest change to the Quantopian Open will come later in the month. On August 17th, at 9:30AM EDT, we are kicking off a 6-month contest. The scoring system and prize will be the same, but instead of being scored on just one month out of sample data, you will be scored on six months of out of sample data. We see this as the best way to ensure that the best algorithms are rising to the top of the leaderboard. We knew from the beginning that there was a tension between wanting a contest that turns around quickly and the need to gather more out-of-sample data. We feel like we've got a scoring system that is reasonably solid and we're pleased with the flow of new entries coming in every month. We think the time is right to add a longer time frame to encourage algorithms that are also looking at a longer time frame.
We will kick off a new 6-month contest every month, mid-month, for the foreseeable future. The first 6-month contest prize will be awarded after the market closes on February 12, 2016, and monthly thereafter.
We are continuing to run the 1-month contest as we have been, at least through the end of the year. We haven't decided what we'll do next year once the 6-month contest results come in and we start giving prizes to those winners. We may choose to continue both contests indefinitely, or we may change the 1-month prize in a way that reduces our capital commitment. It seems unlikely that we' d end the 1-month contest entirely; it's too interesting to remove, even if it has a small out-of-sample size.
All contest entries are eligible for both the 1- and 6-month versions of the contest. Once you submit your entry, it's automatically entered to all contests going forward, unless you withdraw (stop) your entry.
New Prize: Consulting for Contest Winners
For the first few contest winners, we started them trading the very next day after declaring the winner. When we picked our first algorithm for the hedge fund we did something different. We reached out to the individual and offered them some advice on how to make the algorithm more robust. The changes we suggested were about how to handle algorithm stops and starts better, how to order more wisely using Interactive Brokers (as opposed to paper trading), and things like that. The algo author made a few changes, then we tested the algorithm rigorously. The result is a better algorithm than the original. We did the same thing with July's contest winner, and we think the algorithm that resulted was much more robust.
We're going to offer the same process for all future winners. This is going to introduce a few days delay between the contest winner being declared and the start of the prize period. We believe this will greatly improve the payouts for the winners.
Leaderboard Changes
The changes are relatively small, and affect a handful of the contest entries.
- The consistency factor is removed entirely. We've developed other, better methods to identify people who are "gaming" the contest. The consistency factor is now simply adding noise to the results without any benefit, so it is removed.
- The hedge filter is relaxed a little bit. If your backtest has a few days where it is either long or short at the end of the day, it can now get the "hedged" badge. There are only four current contest entries affected by this, so it's only a small change.
- We swapped the Sortino ratio in for the Calmar ratio in the scoring. Most algorithms rank the same on these two metrics, but there are a few that see significant ranking changes.
- Previous winners of the Quantopian Open may re-enter the contest, provided they enter using a different strategy.
Looking Forward
Last month we made the rules changes at the last minute, just a couple days before the deadline. We're doing better this month, 12 days before the contest kicks off. We promise to keep getting better at this, and will announce future rules changes at least two weeks in advance.
There are two other rules changes that we've been considering but haven't formalized yet that I'd like to share.
- One rule change we're considering is to permit more entries per person. On one hand, we want to encourage people to make thoughtful entries. We're worried that if we permit more entries per person, we're going to see more low-quality algorithms that are just wild bets. On the other hand, if someone has a few algorithms performing decently, we'd like that person to be able to make more entries without having to decide which entry to turn off. We're interested in what the community thinks about the current limit.
- We evaluated several different measures of algorithm turnover. Our fund is seeking actively trading algorithms. We've seen a few algorithms that open positions and then make no other trades. These algorithms are not interesting for the fund, and they aren't good entries for the contest. While we haven't written a specific rule to exclude these algorithms, we would most likely find such an algorithm to be not suitable for trading with real money.
As always, feedback is desired. Thank you for your past contributions to the contest.
Good luck to all competitors! We hope you're as excited about the 6-month contest as we are.
