Interesting to note that few of the leaders participate in the forum...
Something like: "Those that can, do. And those who can't -- write about it." (I'm afraid I must place myself in this last group.)
Rather like the difference between CEOs and economists.

I have an alternate take.

In (social) internet culture, the passive viewers vastly outnumber active participants. It's just more likely the (silent) majority will outnumber the active forum participants.

@Josh, good point. Law of Large Numbers kind of thing. It is interesting though that the leaders required no help, nor offered none, from/to the verbose crowd. A notion foreign to me. But it makes a certain sense, what fool shares the crown jewels with the masses? (Oh, yeah, this fool.)

Hello Dan & Josh,

Is there a sure-fire way to match my algos entered into the Open to their leaderboard stats? I noticed that a hexidecimal ID is assigned to each Open algo (see Excel spreadsheet, https://www.quantopian.com/leaderboard/csv). When I compare the assigned ID to the hexidecimal ID associated with my algo, it appears that they differ by a decimal value of 1 (where the Open algo ID has the higher value). Will this relationship always hold? In other words, will I always be able to add a decimal 1 to my algo ID to obtain the Open ID?

Grant

Hi Grant,

We're soon going to be adding something on the drill-down page of your Open algorithm that links you back to the source algorithm.

The relationship between the Open algorithm ID and your source algorithm ID is not guaranteed to hold (and in fact, will certainly not always hold).

thanks for the feedback,
Jean

The creation date is the gross way to tell them apart right now. We will do a better differentiation later.

Dan

Hi Dan,

Please clarify the weighting applied to backtest results versus live trading. The rules state:

The Participant's overall contest score will be a combination of the two, with greater weight given to the live trading score as time goes on. At submission, the backtest score is 70% of the total score. The backtest score’s weight linearly decreases to 0% over 60 days of live trading; thus on day 61, the total score is equal to the live trading score.

To me, this says that if I submit an algo 60 days prior to the end of the contest, when my algo gets judged, you will use only my live trading results. And if I submit my algo right at the deadline, after 30 days, you'll still count my backtest results in the judging (presumably with a 35% weighting applied to the backtest). This would say that if I have solid backtest results, I might want to wait until the deadline to submit?

Regarding differentiating algos in the Open, I found that I could compare the auto-run backtest results with the ones on the leaderboard.

Grant

Way to go @Grant -- #1 (for now).
If there's anyone who deserves to win this contest -- it would be you. Your, what, years? of continued participation is unmatched. And as has been discussed, a striving young quant like yourself could find no better venue to advance his Wall Street career. Power on!

Grant - your analysis of how this will play out is correct. (One nit is that everything in here is trading days, not calendar days, but that doesn't change your basic conclusion).

The key question is, on the day the prize is awarded (market close on 2/27), will our calculation be a good/reasonable/fair way to determine the winner? On that day, some entries' scores will be 53% backtest, and some will be 69% backtest, and others everywhere in between (days traded will range from 20 to 33 on 2/27). I think that's going to play out appropriately, but we will see.

The nice part of the backtest component is that it's a longer date range, so a lot of the judging metrics have settled out. The bad part about the backtest is that it's entirely "in sample" and one can tune (curve fit) the algorithm somewhat to get a better score.

The good part about the paper trading component is that it's entirely out of sample and can't be over-fitted. The bad part is that we don't have the patience to wait years to determine a contest winner.

We hope that this formula we are using is a smart blend of the two. We readily admit that this is a bit of an experiment. If we get to February 27 and we don't like the way the leaderboard looks, we will tweak the rules and try again starting on March 2.

So far I'm liking what I'm seeing. It's really early going. The paper trading scores are super noisy - they don't have a lot of days traded (in many cases, just 1 day). Those noisy scores make the papertrading component of limited value. But more days traded means a more valid score, and the weight of them will increase, too.

I will note that looking at last night's numbers, 57 people had a higher rank in paper trading than their backtest, 56 people had a better backtest rank, and 4 people had the same rank in both. It's not clear to me that waiting to the last minute is a good strategy. I think it's more of a roll of the dice. If it turns out that tactics like choosing the right day to enter is impacting the final results, then we definitely will try to adjust the rules to prevent that.

Finally - I second Market's thoughts. I love your contributions.

@ Dan, Thanks for the clarification and elaboration.

@ Market Tech, It's just a hobby, made more fun by the Open contest.

What sort of Sharpe ratio is require to get into the top 10?

You can download a .csv file with all the leaderboard stats from the icon just above the score column on the leaderboard.

John von Neumann's elephant and its wiggling trunk.

Not only is the elephant wiggling its trunk, I think I can hear the orchestra conducted by this elephant...

What is the only rational explanation for some of the back test results we're seeing in the Q's Open leaderboard? I think von Neumann would know. But I think there's a simple solution to alleviate this discrepancy.

The problem with the backtest part of the evaluation is that it's all "in sample". All the data, all the instruments are all available for in sample testing. That's a problem. Of course a quant can optimize to the 9's through parametric and instrument selection optimization. But where's the test of skill in that? Where's the measure of algorithmic aptitude? The real-time paper trading may or may not be adequate, in the end, in helping distinguish the over optimized from the truly talented. Luck of submission timing seems to play a large part in this factor. So, what modification of the contest could be enacted to simulate some OOS, some out-of-sample evaluation?

Simple really: "set_test_universe()"

During development backtesting set_test_universe() will return a security selection, fixed for everyone; perhaps 10% of the S&P, 50 securities, for the life of the monthly run up. These are the instruments that must be used to build and test and perfect one's strategy. Now, you could optimize all you want on this fixed set. You could even reverse engineer this list and determine which stocks performed best for your algo and only select them for your testing. But...

When run during evaluation mode (by the back end system in judging the algorithms) set_test_universe() will deliver a different set of instruments. The same number and of the same general behavior, but completely unknown to the quant developer. Only by providing a semblance of OOS can the merits of the actual algorithm be truly considered.

An algorithm designed to handle whatever instrument selection was handed to it, under whatever market conditions, for whatever duration, handle all of this and deliver performance and low risk? That will be an investment worthy strategy.

@Market Tech, you might be correct about people over-fitting their algos on purpose in order to get into the upper tiers of the leaderboard, but that's a really short sighted way to go about it. Here's why,

• Any winnings will be based on out of sample performance
  
• As time rolls on, the backtest becomes less significant in the overall score
  
• The longer running algos will get taken more seriously
  
• It's a rolling competition, this is just the first one.

It just doesn't make sense to put an algo out there that you don't really have confidence in, you might shoot yourself in the foot. You could also get lucky, but long term consistency makes for a much more intelligent bet than short term luck.

Hello Market Tech,

My two cents.

Personally, I'm trying to come up with an algo that would both win and make money with the $100,000 of Quantopian capital. So there is incentive not to have an algo that will blow up and hit the $90,000 limit (although admittedly, I don't yet have code to avoid it). For the algo I've been working on for the Open, I recently switched from about 10 securities to roughly 20 securities in my portfolio, and found that I was able to get better backtest results. The securities were picked by applying some simple, general criteria--nothing fancy (although undoubtedly there is bias). So, my experience so far is that diversification helps. I think that my algo will do best if the current bull market continues, but it might actually do o.k. or outperform if there is a sharp, volatile downturn (e.g. a repeat of 2008-2009), based on backtesting. I also figure that Quantopian may adjust the contest rules on a rolling basis. If in 3-4 months they end up with a bunch of winners each sitting on $90,000 in cash, then there will be a head-scratch over how winners are selected. And if anyone trips the $90,000 limit, a new algo would just take its place.

Grant

I disagree with the set_universe proposal - it rules out all spread trading and ETF structural arbitrage. (which just so happen to be my interests!)

Testing strategies with a random universe is not really feasible because we don't know what the strategies are doing. A large subset of strategies depend on the assets they are trading, for example, if somebody is hedging ETN's to play the volatility yield curve, throwing equities in there would give meaningless results.

Perhaps if the cutoff date occurs followed by a two week burn-in period, any ultra high backtest weighted strategies would be subjected to this burn-in, a time decay of their high, single sided score. But if an algo is picked within 24 hours of the last submission, a performant selection would be problematic. In the burn-in scenario, next month's selection would allow quants to submit strategies for a two week window, which also then gets sealed, and burned-in. Without some tempering period I would posit the selection may be an unbalanced one.

Alas, I see that the selection will not entail a burn-in period:
...end on Friday, February 27, 2015 at 4:00 PM ETS, ... winner will be announced on or before midnight, EST, February 28, 2015.

[Also, the set_test_universe(), would of course be built into each strategy, developed and tested with the strategy. No arbitrary insertion of extra symbols. Of course hard-coded symbols, for actual trading and not supporting analysis, would be forbidden. This is the only way to simulate OOS for a backtest (in this environment). Otherwise you will get elephants playing violins.]

[Case in point: Charlie Brown's pineapple]

Market tech, I still oppose the universe idea. I understand your point, but it effectively limits the contest to cross-sectional factor models and "technical analysis". I don't have a good solution, but perhaps what is needed are overnight batch processing hooks to properly do whole-universe analysis and ranking with which to build the trading universe. However, it still would not address my algo needs, which pretty much require a hardcoded list of ETF's that have baked in interrelationships that I want to exploit.

There might be a fundamental problem with including backtest results, since the backtest could be run on an algo that is effectively completely different from the one submitted. Presumably, this would run afoul of the rules, and Quantopian could apply:

Additionally, we reserve the right to disqualify any entry at our sole discretion. For instance, if we believe that an entry is made in bad faith with the intent to "game" the contest, or if we conclude that the algorithm is not suitable or financially prudent to trade with real money, we would disqualify the entry.

But how would Quantopian detect trickery? Let's say that a competitor does an offline optimization and creates a look-up table that is used within the algo to dynamically allocate a portfolio (potentially a very large one) to maximize their backtest score. And also within the code, he has a completely valid algo, well-suited to run under simulated live trading for a month. Via code logic, just before the submission deadline, the fake algo gets disabled and the valid one runs. My hunch is that Quantopian is assuming that the backtest algo and the live trading algo will be the same, but as I see it, the backtest algo could be wildly biased to "game" the competition (or perhaps just biased enough to give the algo an edge, without being suspected as a cheat).

Solution? Base competition on live trading only? Automated inspection of algo code, that would detect "gaming" of the competition? Allow a set of trusted judges to inspect code, yet keep it private (these would have to be industry outsiders and not be involved in retail algorithmic trading, as well)?

Another potential problem is hackability of Quantopian code, in such a fashion that backtests and simulated live trading results could be manipulated. This is certainly feasible via a custom slippage model, which has been excluded from usage in the competition. This would be a worse problem, because then one could "fit" the backtest, apply the hack to the live trading portion of the competition, and then switch to a valid algo for real money trading. Other than a custom slippage model, I don't know of other tricks, but the fact that it could be done via slippage raises concern that some undetectable hacks exist. Paper trading at Interactive Brokers (IB) would eliminate the risk of hacking, since performance statistics could be pulled from the IB paper trading account.

Hello Grant
I am running one of my Q-open algos in parallel using my IB paper account to compare results. Based on trades last week, it appears to me that IB's fill model is not as sophisticated as Q. I have a very limited sample, but one of my trades took a few bars to complete in Q's paper test that I expected this, because it is a small-cap. But in IB, it looked to me like and immediate fill. I ended up with a cost_basis a dime or so more in Q then what IB gave me which I agree with. While the IB paper account is useful, I am not so sure I would use it over Q. (Confession: I am a heavy zipline user).

Maybe it is the kind of quant I am, or the type of work I attract from my clients, but the main constraint I hit against with Quantopian is the 200 universe limit. This leads to the situation that Market Trader warns about where I am pushed towards off-line search/optimization (or a one-off #Fundamentals algo) just to pick the universe before I can put hands to keyboard on the real algo.

Richard Prokopyshen
http://quant-coder.prokopyshen.com

Thanks Richard,

Interesting. Perhaps IB's paper trading is more for debugging? I would think if they were serious about simulations, it'd be very accurate, since they have all of the trade data, etc. with which to build a model.

Grant

Great post Grant. I think you're probably right that the competition would be fairest (and easier for Quantopian to judge) if it was only based on live trading results.

Fairest, but risky. No fund manager would ever bet any money on a strategy that had not run for a agonizingly long time. Proof of long term performance is what they all require. (Or the nearest facsimile thereof.)

It's an interesting quandary. And rather like scientific validation. As a scientist you can write all the white papers you want, but if other colleagues cannot reproduce your results... [loud buzzer sound] no one will fund your next research grant.

Market Tech,

Gotta start somewhere. And if your model is accurate then simulations are nearly as good as the real thing. It's a matter of identifying specific risks and mitigating them. With respect to the Open, $100,000 is about the bare minimum for successful retail trading of U.S. securities, as I understand (I've heard figures more like $250,000 minimum, actually). So, in analogy to engineering a product, this is a prototyping effort, on Quantopian's dime. And I have to imagine that they are looking at this as a proving ground for their Fund, which as an institutional-level enterprise, will be dealing in 10x-1000x more capital. In actuality, it's not a contest, but rather an R&D effort, with unpaid Quantopian users as substitutes for the paid staff that a hedge fund would have in-house. We're cheap and potentially more clever and effective than a staff. Also, one would have to think that winners (and maybe some losers) would be approached by Quantopian for inclusion in the Fund.

The contest could also be considered a marketing effort to attract users who would eventually pay service fees. But my read is that Quantopian's business plan is to be a hedge fund, rather than trying to charge for their services at a low profit margin.

Grant

To assume an algorithm must trade blindly is not a proper solution. It is the same as assuming every stock behaves the same have same correlations etc..