Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Quantopian Open - variable backtest length, backtest length as an additional metric
Lewis Liu
edited Jan 26, 2015

I believe the backtest portion of Quantopian Open should use the maximum backtest length possible for each algorithm (i.e. for as long as data exists for the securities being traded in the algorithm), and that backtest length should be added as an additional performance metric to influence the overall score of an algorithm. The minimum backtest length to qualify for the Quantopian Open can stay at 2 years, but should result in a lower score modifier than an algorithm with, say, 10 years of backtest data.

The more backtesting data that exists for an algorithm, the more confidence we can have in the backtest results being a good representation of the algorithm's performance in all market conditions. I'm not exaggerating when I say that I feel this is one of the most important metrics you can measure for any algorithm.

I realize that this is not trivial to implement, and may be too late for the Quantopian Open in this current round, but there's always next time. =)

Any thoughts?

7 responses
Jorge
Jan 26, 2015

+1 I totally agree with your viewpoint

Market Tech
Jan 26, 2015

... their game, their rules.

(Already pointed out the risk of their parameters. But again, we're not the ones with a spare $1/2M to play with. Our only choice, check our guns and our opinions at the door.)

Deleted User
Jan 26, 2015

Lewis,
This is definitely on our radar for future contests. I particularly like your idea of possibly giving "bonus points" to strategies that backtest over longer periods. For the first contest we decided to standardize on 2-years, but we are not wedded to this lookback period for future contests. We definitely want to seek out strategies which achieve stable performance over varying market cycles and regime shifts.

Thanks for starting this thread and sharing your thoughts,
Justin

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Grant Kiehne
Jan 27, 2015

Note this in the rules (https://www.quantopian.com/open/rules):

At submission, the backtest score is 70% of the total score. The backtest score’s weight linearly decreases to 0% over 60 days of live trading; thus on day 61, the total score is equal to the live trading score.

So, looking ahead, if an algo is on the leaderboard now, by the next round, the backtest won't matter anyway. But maybe there is some way to work the system, and time the backtest submission within the 60-day window, to improve the chances of winning? In this case, maybe it would make sense to consider different backtesting scenarios.

Market Tech
Jan 27, 2015

It would appear that there are conflicting statements in that quoted paragraph "...decrease to 0%.." and "..equal to the live trading score."

I would have to bet that that first should read "... decrease to 50% of influence over 60 days of trading."

As far as suggesting updates, "2 years of back test + minimum of X weeks of live to qualify" might be appropriate.

Grant Kiehne
Jan 27, 2015

Hmm, so you think the backtest score goes from 70% weight to 50% weight over 60 days of live trading? This would make sense, but it sure doesn't read that way. --Grant

Market Tech
Jan 27, 2015

True, the verbiage is misleading. But abandoning 2 years of results in favor of 2 months -- illogical. In fact, even the 50/50 resulting ratio is strangely weighted. "Here swim swans of a coal like color..."

About Quantopian
Careers
Community
Events
Documentation
Academia
Lectures
Status
Facebook
Twitter
LinkedIn
YouTube
Terms of Use
Privacy Policy

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.