Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Contest hacking / 'cheating'
Joakim Arvidsson (Cream Mongoose)
edited May 4, 2018
Contest

Hi all folks at Quantopian,

I have two questions related to Contest hacking (or cheating rather):

  1. Are all contest entries running with the same amount if capital in the live sim market, regardless of how much capital was used in the >2 year backtest that was used to submit the algo? I've noticed how different amount of capital deployed in the backtest can significantly affect the sharpe ratio (mostly returns and not so much volatility I believe - it's easier to get 100% fills and less slippage with less capital), so unless all entries are running with the same amount of capital in the contest, it's not really a level playing field in my view.

    If that's the case, in my opinion at least, an additional contest requirement should be that the >2 year backtest need to be run with a certain amount of minimum capital (e.g. 10 Mil.).

  2. Since the volatility portion of the score is based on the past rolling 63 days, it would be fairly easy to find what the lowest volatile long/short portfolio in the past 63 days is that also meets the contest criteria/constraints. One could then just schedule the algo to hold those stocks in the past 63 days (in a manner that is similar to how the real algo behaves) in order to minimise the volatility of the algo and potentially spike the score from day 1. The effect would depreciate over time of course as the 'real' volatility would catch up, but still...

    So my question then is, do you have anything in place to check for entries with statistically significantly higher volatility from day 1 in the contest, compared to the most recent 63 days (or the full backtest for that matter). In short, something to check for volatility spoofing? What's the consequence for anyone caught doing this? If nothing, aren't you indirectly encouraging and incentivising this type of algo design?

Lastly, thanks for a great site and platform. I love Quantopian!

Joakim

9 responses
Jamie Veitch (Azure Hawk)
May 4, 2018

  1. I believe they're always run with $10M

  2. More of a problem, I believe they disqualified a some people for gaming the contest a few months ago

Grant Kiehne
May 4, 2018

Regarding the volatility, there is a minimum volatility used in the scoring. It is defined as the larger of the trailing 63 day volatility and 0.02 (see https://www.quantopian.com/contest/overview). As it was explained, this is in place to preclude a certain type of contest gaming. So, supposing one could construct a long-short portfolio with artificially low trailing volatility and decent out-of-sample return, meeting all of the other contest criteria, there is a guard in place in the scoring, clamping the volatility at 0.02. I don't recall any in-depth explanation of why 0.02 was chosen. Presumably, someone at Q has a sense of the range of volatility one would expect for the kind of long-short equity algos they need for the Q fund; anything less than 0.02 would be craziness.

Joakim Arvidsson (Cream Mongoose)
May 4, 2018

Thanks guys. Yes, I knew about the minimum of 0.02 volatility, but still, if you have a high risk/return algo that normally returns 0.20 alpha with 0.10 volatility, getting 0.02 volatility from day 1 would artificially, and in my view, unfairly inflate the score quite a bit and for quite a while until the real volatility has caught up.

I would like to hear from someone at Quantopian regarding this. If this is considered 'fair game' then shouldn't all entries start out with 0.02 volatility from day 1?

Joakim Arvidsson (Cream Mongoose)
May 4, 2018

just increasing the number of bets (and thereby decreasing each individual bet-size) in the last 63 days to 1000 longs and 1000 shorts ought to bring down volatility to less than the minimum, regardless of what alpha factors the algo uses. If not, setting all Risk constraints to 0 should definitely do it...

Jamie McCorriston
May 4, 2018

@Joakim, The 0.02 volatility rule is the only technical guard against #2. In general, we spot check the leaderboard before it gets published every day and if we notice odd behavior or big jumps in score from one day to the next, we investigate. I think it's also worth noting that the in-sample volatility has a diminishing effect from day-to-day. Now that the competition has been running for almost 3 months, it's unlikely that it will have an impact on the top 10. That said, I'll keep an eye out!

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

James Villa
May 4, 2018

@Jamie,

Is it possible to add cummulative returns and trailing 63 day volatility seperately in the scoring dashboard of the daily contest to give us a better indication of performance. Just the score and rank doesn't tell you much. Thanks.

Joakim Arvidsson (Cream Mongoose)
May 5, 2018

Thanks Jamie!

Ok, I'll live with it (and I won't cheat). As for my first question, could you confirm that all contest entries are running with 10 mil., regardless of how much capital was used in the backtest?

Grant Kiehne
May 5, 2018

Hi Joakim,

Q has official rules:

https://www.quantopian.com/contest/rules

There, it says;

The backtest will simulate deployment of $US 10 million.

Combined with the leverage requirement, $10M is invested (versus an allowance to hold a portion in cash).

Joakim Arvidsson (Cream Mongoose)
May 5, 2018

Thanks Grant, appreciated!

About Quantopian
Careers
Community
Events
Documentation
Academia
Lectures
Status
Facebook
Twitter
LinkedIn
YouTube
Terms of Use
Privacy Policy

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.