Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Inconsistent backtest results with identical code.
Yuyang zheng
edited

I've run 3 backtests on the same piece of code on the same dates. Returns range from 10% to 200%. I'm not sure what is causing the discrepancies since the algorithm is quite simple and does not rely imported data from outside quantopian that can be updated/edited between backtests. I am just using price data to track the volatility of GLD and trade based on the price returns of VXX.

The code contains some variables and such that aren't used, as I was still experimenting with the code to find any possible leads. In addition, there is a large block of code commented out at the bottom from an example algo that I was using as a template for machine learning. These parts are identical across all 3 backtests, but I just wanted to clarify in case someone got confused with my code and what I was planning.

5 responses
Yuyang zheng

Here is a different backtest

Yuyang zheng

And finally a third. Haven't run anymore backtests, so I'm not sure if doing so will provide another variation.

Georges Bilan

You're using a random forest which will give you different results each time if you don't set a seed. Try this one.

clf = RandomForestClassifier(random_state=33)

Alex S

RandomForestClassifier is using different random numbers each time and so the model it fits and thus the trades you make are different each time. On a related note, your model is pretty fragile if different random seeds change things so much.

(On preview, what Georges said)

Mikko M

It should also be noted that sklearns random forest has extremely low default ensemble size (10 classifiers!) so that increases variability even more.

About Quantopian
Careers
Community
Events
Documentation
Academia
Lectures
Status
Facebook
Twitter
LinkedIn
YouTube
Terms of Use
Privacy Policy

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.