Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Algorithm at a disadvantage
Daniel Sandberg
edited Jul 16, 2013

Hi All -

So I've posted a few things on the forum regarding an algorithm involving moving averages. Because the N-day moving average is not valid until N-days into the backtest, I wanted to effectively hold trading until the N-th day. I coded an algorithm that buys SPY and tracks the time. The algorithm correctly identifies when it is time to apply the algorithm (as opposed to staying in SPY) BUT during the holding period, where it is holding SPY, the algorithm significantly underperforms the benchmark. Is the benchmark not SPY? Is that the reason why there is a discrepancy? It seems the algorithm is a fixed amount below the benchmark. I'm just trying to track down the discrepancy between the algorithm and benchmark in this code.

15 responses
Rich Frank
Jul 16, 2013

Thanks for bringing this up Daniel. We're currently looking into it.

-Rich
Quantopian

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Daniel Sandberg
Jul 16, 2013

Thanks Rich - No problem.

I coded my own software to do these sort of backtests (it doesn't run via the web and isn't as sophisticated) but I ran into a similar issue. For me, it was a rounding error stemming from the fact that I permitted my benchmark to own a non-integer number of shares (so as to avoid holding any cash position in the benchmark portfolio) but forcing the algorithm to integer number of shares. Just throwing that out there....

Gus Gordon
Jul 16, 2013

Daniel, I just dug into this, that sounds like it could be part of the problem. Another part might be that when you send a buy command it takes a minute (or a day, depending on which mode) to go through. So if the price goes up in the lag time, the benchmark may have already bought before that. Now the benchmark has a larger portfolio value, and that portfolio subsequently increases in value more quickly as the S&P's price rises since the benchmark will have always effectively bet more. Also, in your algo, you are using round(), which can round up which is probably unwanted. Not saying any of these things are definitely the cause, since I can't figure it out, just things I noticed. Interesting problem though.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Daniel Sandberg
Jul 16, 2013

@Gus - thanks for the comments. I did change round() to math.floor(), which is what I actually wanted. You make some excellent points but I'm a little skeptical that the problem could be caused by the delay. Off the bat, the difference is 1.6% and the price change in SPY at the beginning of the simulation is minimal.

Dennis C
Jul 16, 2013

I'm pretty sure this is related to the benchmark change Dan announced a while back.

https://www.quantopian.com/posts/upcoming-features-backtester-changes

After that time I noticed that the benchmark would sometimes 'start' at a positive or negative return (instead of zero like the algorithm).

Jean Bredeche
Jul 16, 2013

Hi everyone:

Apologies for the confusion. Our benchmark for backtesting is the S&P 500, and our benchmark for paper trading (walk-forward trading with live data) is SPY. We switched to SPY when building live trading, and haven't backported that change to backtesting yet. It's on our near/medium-term roadmap.

Obviously, the S&P 500 and SPY should behave nearly identically. On Jan 3, 2012, the S&P 500 opened at 1258.86 and closed at 1277.06, for a 1.4% gain. Meanwhile, SPY opened at 127.75 and closed at 127.49, for a -0.2% return. This is strange, and we're still investigating why this happened (ETF rebalancing maybe).

Besides this one-time ~1.6% discrepancy, for the rest of the backtest, the two instruments behave almost identically.

If we find out anything about why that discrepancy exists, we'll be sure to update you.

thanks and sorry for the confusion.
Jean, Quantopian

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Daniel Sandberg
Jul 16, 2013

@Jean - thanks for responding so quickly. No need to investigate the discrepancy between SPY and S&P 500 - that is hardly a Quantopian issue unless you folks are affiliated with State Street Global Advisors (who run SPDRs). I just wanted to make sure there wasn't something else going on here.

Ben McCann
Jul 17, 2013

Interesting. They both rose 1.6% from the previous day's close. It's just the opens that are wonky. I'd guess if you started the backtest from any other day that you would not see this.

Dan Dunn
Jul 25, 2013

This was bugging me, and I kept digging. There are a couple other factors at work that make the algo perform worse than the benchmark.

  1. commissions
  2. slippage. For right or wrong, our slippage model thinks that if you buy $1,000,000 in one minute that you affect the price

Throw in these two lines, and everything starts to match up closer:

    set_commission(commission.PerShare(cost=0.00))  
    set_slippage(slippage.FixedSlippage(spread=0.00))
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Daniel Sandberg
Jul 25, 2013

Thanks so much for looking into this Dan. These are small issues but precision is important. Thanks again.

Daniel Sandberg
Jul 30, 2013

Dan - I tried amending my "pause" function and got an unknown error. Any ideas?

def pause(nth_Day, mycontext, mydata):  
    if mycontext.timestep == 1:  
        set_commission(commission.PerShare(cost=0.00))  
        set_slippage(slippage.FixedSlippage(spread=0.00))  
        nshares=math.floor (mycontext.portfolio.cash / mydata[mycontext.SPY].price)  
        order(mycontext.SPY,+nshares)  
#        log.info("First timestep and paused. Ordering "+str(nshares)+" of SPY.")  
    if mycontext.timestep < nth_Day:  
        return(0)  
    elif mycontext.timestep == nth_Day:  
        #Reset defaults  
        set_commission(commission.PerShare(cost=0.03))  
        set_slippage(slippage.VolumeShareSlippage(volume_limit=0.25, price_impact=0.1))  
        #Liquidate position  
        liquidate_position(mycontext.SPY,mycontext)  
        return(1)  
    else:  
        return(1)
Dan Dunn
Jul 30, 2013

set_commission and set_slippage can only be set in intialize().

We should give you a better error, so that is a problem.

Daniel Sandberg
Jul 30, 2013

Hmmm.... Ok. So pause function would require no commission or slippage for the whole simulation. I guess soon the backtester will have "warmed up" data on day 1 so I won't need this function anymore!

Dan Dunn
Jul 30, 2013

I haven't quite put the full warmup on the roadmap yet - I can't claim that we're going to make the change "soon" just yet. I'm pretty convinced it's something we should do, but there are a few other higher priorities on deck before we do that one.

Daniel Sandberg
Jul 31, 2013

That's understandable. It isn't impossible to work around the issue to vet algorithms.

About Quantopian
Careers
Community
Events
Documentation
Academia
Lectures
Status
Facebook
Twitter
LinkedIn
YouTube
Terms of Use
Privacy Policy

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.