Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
I'm "winning" the latest two contects and unsure of the validity of the backtests
Quanty McQuant
posted

Hi All!,
I just wanted to see if anyone else was running into problems with backtest vs paper trade in these contests. I'm "winning" the last two expiring contests ( Feb 1st and March 1st ) if the backtesting Sharpe section isn't taken into account. (see screenshot at the end )
I'm running an algo that has multiple alphas blended together from an alpha generation system that doesn't use quantopian data. I built it off of a different data source for equities and I originally submitted this algo to see how well quantopian tracked my system. Surprisingly when I generated the long/short subsector hedged alphas they gave numbers very close to my out of sample results in the contest but did not match at all with the quantopian backtest.
My backtest in my alpha generation system ( June 1st 2015 - June 1st 2017 ): Sharpe 4.1
Quantopians backtest in the algo submission: Sharpe -0.008
My paper trade after algo submission in contest : Sharpe 3.85

I submitted the algo to the contest originally to see why there was such discrepancy between my system and quantopian's backtester but it seems the "live" paper trading is matching pretty closely to my system out of sample. I get different numbers from my algo in the quantopian backtest during the time period of "live" trading vs contest "live" trading as well. Any insight into this would be much appreciated from the quantopian team. maybe bump my backtest up 0.003 ;) joking...
Thanks for your time!
Screenshot of Feb 1st contest:
https://imgur.com/a/RjYeu? (copy and paste the question mark doesn't link )

2 responses
Jamie McCorriston

The two explanations I can think of for different backtest results are 1) the backtester is different and/or 2) the input data is different.

1) Is the backtest you are referring to with 4.1 Sharpe run off the platform? If so, are you using Zipline, or your own backtester?

2) If you're running the backtest on Quantopian, are you using Fetcher to import your signal, and then replacing it with data from Quantopian when you submit to the contest? If you're using the Quantopian backtester, I'd recommend doing a comparison of your data against the Quantopian version that you're using.

Separately, have you tried testing to see if your algorithm meets the criteria of the upcoming new contest (after Contest 38)? I'm wondering if a new submission might push you above the 0-returns line.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Quanty McQuant

Hi,
Thanks for the reply! I'm not using fetcher just the built in pipeline and pandas functions. My backtester is one I wrote for a HFT firm which I own the rights to that was used for live trading for many years. ( I was a Financial Engineer there for 3 years )
I ran the exact same code for the exact same dates in the backtester and it loses money (sharpe -1.44 vs sharpe 2.87 in the live test ). Please see attached. I'm using set_commission(commission.PerShare(cost=0.001, min_trade_cost=0.0)). Maybe that's treated differently between live testing and backtesting? Are the data sources the same?
contest screenshot: https://imgur.com/QEQgTkF
Thanks!

About Quantopian
Careers
Community
Events
Documentation
Academia
Lectures
Status
Facebook
Twitter
LinkedIn
YouTube
Terms of Use
Privacy Policy

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.