Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Quantopian Open, Final June Leaderboard is Delayed

We've identified a bug in the leaderboard calculation, and we're delaying the final June leaderboard until we've fixed and verified the calculation. We will then announce the winner of the June edition of the Quantopian Open. We are very sorry about the delay, and we apologize to all of the contestants and other interested watchers for the delay and inconvenience.

I can't promise when it will be fixed. However, if the fix we're working on implementing does in fact solve the problem as we hope, we should be able to make the announcement on the afternoon of July 1st. If not fixed, I certainly will at least give further updates on our progress.

I'd like to thank both Michael and Vladimir for pointing out that something wasn't right. Their posts got us digging into the data and the code.

What we've discovered tonight is that the root of the problem isn't the risk-free rate. The risk-free rate is being chosen in a reasonable way (though there are reasonable disagreements about whether it's the best way). The actual problem is in the annual return that is being returned and displayed in the leaderboard. We appear to be mis-counting some dividend payments. Once the annual returns are corrected, then the reported sharpes are harmonized with the risk-free rate and the volatility. This conclusion is tentative until we've verified that the fix corrects the problem.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

7 responses

Can we still expect an announcement this afternoon?

Hello all -

We aren't ready to post the leaderboard today. We have increased confidence that we've correctly identified the bug. We have a drafted fix. That fix has not been tested to our satisfaction. Our plan is to test it further this evening and into tomorrow.

My hope is to be able to post the leaderboard tomorrow. If not. . . . I'll have another update. But I'm feeling pretty good that we're going to have a winner.

We apologize again, sincerely, for the difficulty and distress.

The bug was part of the whole competition (June) and people developed according to it, the rank was public. Like in a real trading system one cannot untrade I think the rank should stay as it is and the bug be fixed for July.

The final June leaderboard is now published. June is currently missing the correlation scores, those will fill in later. July and August leaderboards will be posted shortly, using the updated rules.

Congratulations to Michael VK on pulling out the win!

The bug was indeed in the dividend accounting. The dividend cash transaction was happening after the closing portfolio value was computed, but before next day's opening portfolio value was computed. Some of our risk and measurement statistics, like overall returns, are computed by comparing one day's closing value to the next day's closing value. These statistics were not affected. However, some risk statistics were calculated by comparing the day's opening value to the day's closing value. Those statistics couldn't "see" the dividend because it was already paid before the opening value. Volatility and sharpe were affected. This bug affected paper trading, but does not affect backtests.

The underlying bug in zipline is still present, and that fix is still in progress. What we've done today is fix the leaderboard.

We found that the bug has been there since the very beginning of paper trading! We appreciate the feedback from the community that found this bug. Every time we dig one of these up, the platform gets better for everyone. The thousands of eyes checking the backtester and the results is making the tools more robust.

To summarize: algorithms that had dividend events during their paper trading had incorrect sharpe and volatility calculations, which affected the leaderboard's scoring. The leaderboard has been corrected.

Lucas - I'd like to clarify the principle we are using here.

If the bug had resulted in things that cannot be undone, like trades, then we would not have held the leaderboard and revised it.

However, in this case, the actual trades, dividends, etc. were all done correctly. The error was in the derived math that happens separate from the "real world" events. When that type of error is found in a timely fashion, there's no reason to stick with the bad calculations.

Hi Dan,

I am not quite clear on why such a bug existed. All prices should be in consistent state (relatively) before or after dividend adjustment.

When does dividend adjustments happen? It seems to me what you saying is that the risk stat computations take place after close price get adjusted, but before open price get adjusted. Is that correct?

Hi Dan;

I do not find my algo in the June leaderboard. . .

Algo is live trading in contest since 15th June 2015 and was seen on leaderboard
before the bug was detected. . .

Same algo is currently ranking at 128 in July leaderboard