Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
June leaderboard scoring bug?
Michael Van Kleeck
posted

After last night’s June leaderboard update, I noticed an odd thing for two algorithms other than mine: Their backtest, paper trading, and consistency scores all went up, but their overall scores went down. This seems odd.

Paper trading gets more weight as time goes on, but these algos have higher paper trading scores than backtest scores. So even if nothing else had changed during Monday’s trading, their overall scores should have increased.

The consistency score gets progressively applied over time, with full application at 20 days. But before Monday, both algorithms had already been trading at least 20 days. So increased application of consistency doesn’t explain why the scores would go down.

Before Monday, I could use the June rules to calculate scores from the components, and the results pretty closely matched the leaderboard. After the Monday night update, the results no longer match — the scores on the leaderboard for these two algos seem too low.

I tried before-and-after calculations on a few other algos. Some seem too high. Mine seems low. I’ve emailed Dan and Alisa to ask for clarification. Maybe some changes for July accidentally got into the June scoring? Does anyone have other ideas?

7 responses
Michael Van Kleeck

Today's midday update fixed the discrepancies.

Dan Dunn

The leaderboard we published last night, June 29th, had a calculation error. It was a significant error that shuffled the ranking. The error has been corrected, and a new, updated leaderboard is now available. We apologize for the distress and inconvenience that this has caused.

Some details for the curious: We made a change under the hood for the "days traded" calculation - one of those things that you don't expect to change the leaderboard. Also, we did expect the leaderboard to change yesterday because of the spike in market volatility. So, when the leaderboard had a new leader and a shuffled top-20, we weren't surprised. Upon further investigation this morning we found that the "days traded" calculation used to return a float, and it now returns an integer. Our leaderboard calculation didn't handle that change, and the calculations were incorrect.

Again, we apologize for the error.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Vladimir

There are a lot of incorrect sharpe ratio calculations.
If annualized returns and volatility calculated correctly than the only risk free return affect the results.
It should be near zero now but it vary from -5.5%-pravin to 3.26% -csaba

Simon Thornington

Upon further investigation this morning we found that the "days traded" calculation used to return a float, and it now returns an integer. Our leaderboard calculation didn't handle that change, and the calculations were incorrect.

This is my biggest pet peeve with Python! This is a trivial error that any static type system could have caught. The problem only gets exponentially worse as dynamically typed code bases grow - it becomes an inordinate amount of work to find out what a function actually returns, and what you can do with it.

Thanks for the update.

Michael Van Kleeck

Thanks Dan for the fix and the explanation.

André Pajak

Simon: it's actually worse - you could say "what a function returns when":

def f(x): return 0.0 if x else 0

returns a float or an int.

André Pajak

Michael: Things like this, and the Sharpe ratio calculations, make me seriously wonder about the quality of the whole Q code base. The lack of response to Vladimir's numerous posts do even more.

About Quantopian
Careers
Community
Events
Documentation
Academia
Lectures
Status
Facebook
Twitter
LinkedIn
YouTube
Terms of Use
Privacy Policy

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.