Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Is there a BUG in calculating Annualized Volatility results in 34x of max possible move affecting Contest 27 ranking?
Willie Trader
posted
Contest Statistics

In Contest 27 Sky Blue Hawk moved from rank 11 to rank 1 in 2 days (from 8/25/2017 to 8/29/2017)

Reviewing Sky Blue Hawk performance metrics in Leaderboard csv files for the above dates one metric stands out as improbable - annVol_pt decreased from 0.059188114 to 0.042750392.

According to formula to calculate Annualized Volatility theoretically max possible volatility change in 2 days could only be -0.8% vs Quantopian calculated -27.8% or a difference of ~34 times. This bug also triggered excessive blow up of Sharpe and Sortino Ratios, and as a result, drastically adjusted leaderboard ranking - ie my own algo moved from 2nd to 4th place.

I believe this was not done intentionally!

Please fix this bug ASAP as we only have 1 day left in Contest 27

8 responses
Dan Dunn

The leaderboard adjustment last night was intentional, not a bug.

When we changed the rules for Contest 32, we made some changes in the leaderboard calculation code. As sometimes happens with software, we introduced a bug. The bug caused a number of contest entries to be calculated without the full algorithm live-trading history. Since August 1st, a handful of entries were getting incomplete scores. We fixed that last night, and it the leaderboard standings were re-ordered. The leaderboard is now correct, to the best of our abilities and knowledge.

I'm sorry that you were on the wrong end of that bug fix. I'm sure you understand that we want the contest to be fair. It's important that every entry get the same data, the same measurement, and the same rules. In this case, I believe we got to the right place. We found the problem and applied the fix fairly. In retrospect, I wish we'd gotten the fix out faster, so there wouldn't be last-minute turnover like this. I apologize for that, and for the original bug in the first place.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Vladimir

Dan,

The leaderboard adjustment last night was intentional, not a bug.
I'm sure you understand that we want the contest to be fair.

At least you want the contest to be fair.

But what you are doing is unfair:

It is unfair intentionally manually manipulate leaderbord metrics.(Like you have done yesterday).
It is unfair intentionally to make changers to contest rules in 1-2 days before finish.
It is unfair intentionally to promote algo of one participant for expense of other.
There is no apologize for those.

If you like so much Douglas Staple algo - fund it, but not show that you can intentionally make him the 27 contest winner, because it is not a fair way.

Willie Trader,
You should be aware that such discretionary activity happens every September in 2015, 2016 , 2017

Willie Trader

Hi Dan

Your answer is NOT satisfactory and this practice of changing/adjusting the rules during live contest shall be stopped!
Vladimir has a point - what you did is NOT FAIR!

Please return the leaderboard calculation code to pre-adjusted state - as it was on August 27, 2017

If you want to improve the system please do it ONLY for the contests that have NOT started yet.

You have investors to answer to, and these kind of on-the-fly corrections are not good, who knows how many more bugs are there, it shall not require you to readjust the leaderboard calculation code EVERY time you find one.

Alex Temesov

It is disturbing to see how unprofessional and clumsy Q is trying to reshuffle leaderboard in the last days of the competition.

I support Willie Trader's request to return the leaderboard calculation code to pre-adjusted state - as it was before August 28, 2017.

It is not just Willie who is on the "wrong end of the bug fix", it affects EVERYONE, every participant in EVERY currently running contest.

Changes/improvements/bug fixes must only be made to the contests that have not started yet.

Let's hear from other participants and from management and investors.

Willie Trader

I have looKed at the original leaderboard files:1488350517, 1502776314, 1502862626,

1502948978, 1503035479, 1503037469, 1503381075, 1503467491,

1503553958, 1503640449, 1503726994, 1504072428, 1504158770.

And found that

Douglas Staple (Sky Blue Hawk) at the begining of contest 27 had only one contest entry (submission_id 58b38f426dad21000a1d9d6b).

From 8/17/2017 he has two algos with submition ids 58b57a5c98847226134617c6 and 58b588bf83494e7b1ed586f9

The score metrics of the last one was artificially inflated on 8/29/2017 with lame explanation from Dan Dunn about some bug fix (see above).
It looks like manipulation of contest metrics in order to award first place to the algo that was no even there at the start of the contest.

Also, why such a drastic adjustment was made by Dan Dunn's team without publicly disclosing it?

Victoria

It is much easier to fight for justice in the contests with this support from Willie , Vladimir , Alex .

Vladimir

Victoria,

Welcome to the club.

Willie Trader

Thank you all for your feedback on this important matter, the rules shall NOT be changed DURING LIVE contests, otherwise manipulation could very easily be implied.

Also, any significant changes to the future contests shall be made PUBLIC BEFOREHAND, not accidentally discovered by participants.

May the CLEAN competition and professionalism (thank you Doug for pointing it out) prevail!

About Quantopian
Careers
Community
Events
Documentation
Academia
Lectures
Status
Facebook
Twitter
LinkedIn
YouTube
Terms of Use
Privacy Policy

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.