What change has been made that could effect my algorithm's returns?

Quantopian's community platform is shutting down. Please read this post for more information and download your code.

Back to Community

What change has been made that could effect my algorithm's returns?

posted Mar 5, 2018

I took some time off between July 22, 2017 and Dec 11, 2017 from quantopian. In between that time something changed in the system to where a algorithm I was backtesting which used to get 1642.37% now gets 677.19%. I am not quite ready to post my algorithm but here are the things I checked:

1) Code is the same. (I like the new back test compare buy the way).
2) Date range for back test is the same. Jan 03 2003 - Jul 21 2017
3) I have been trying to find a post that would suggest data or api changes that I need to make. Only I thing I found was faster Fundamental API for accessing morningstar data. I changed to Fundamental API and still get 677.19% returns.

Without being able to add more logging to the algorithm and run on the data back in July 2017, I am at a loss. So I am looking for anything people know has changed I should focus on. Thank you in advance.

6 responses

Jacob,

I saw a similar phenomenon that seems to be related to the change in fundamental data. I actually saw the returns go the other way for me so I was pleasantly surprised (but still frustrated because I was live trading it with real money). It was the same thing where the code was the same, dates the same, initial capital the same etc. The only thought I had was that the fundamental database changed a bit with their Fundamental API because my algorithms were/are heavily based in fundamentals.

Have a look at the specifics of the backtest though because, for me, the difference in performance seemed limited to only the most recent few years (which is again frustrating considering thats when I was using real money).

Not sure there is much to do about it other than acknowledge that we both need to be careful of overfitting. I definitely did some overfitting when I first joined Q and probably still have some of that going on, it's hard not to. But when I first started I was probably being a little egregious, I'd keep changing fundamental filters and re-running backtests trying to optimize the returns. I went back and deleted backtests when I looked at my profile in Quantopian and saw that I had thousands of backtests. Again, it's hard not to have some overfitting but I guess we should both try and limit it as much as possible.

Hopefully, the new fundamentals API will not change too much in the future!

Hope this helps a bit,
Steve

Default slippage model in the backtester has changed since Jan 2018. It is now fixed basis points slippage.

Stephen,

Thank you that gives me some ideas. I am pretty sure its fundamental data, I will definitely compare output and see which years are effected. I was definitely watching out for over fitting and using nice round numbers for my filters but who knows. Really hate to think what I came up with had 1000% in luck but its possible.

Leo - Thanks but my back tests exclude Jan 2018. So that isn't it.

@Jacob I am not saying you are backtesting Jan 2018. I am saying an event happened in Jan 2018 that affects all backtests (of any period) that is run on Quantopian.

Hello Jacob,

In addition to the slippage model update mentioned by Leo, the default commissions model was updated on November 7th, 2017. You can learn more in the official announcement we made back then.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

You've successfully submitted a support ticket.

Our support team will be in touch soon.

Need help? Contact support.

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

About Quantopian

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.