Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
how will algo backtest results differ from trading results

Hi Quantopians!

I am just getting started and have a few questions.

  1. How will my algo backtest results differ from trading results? I ask this because simply cloaning a sample algo and running a backtest gives 43+ percent returns in just over a year time frame. I highly doubt this will happen when I use the same algo in real trading.

  2. backtest result logs show the first 9 trades are selling the stock, how is this possible? Not a single buy but 9 sell orders?

Would appreciate if someone helps me understand this.

Thanks!

5 responses
  1. There are a couple of ways in which real performance will diverge from the backtest. Some are mechanical (the costs of trading, including the bid-ask spread may be worse than the backtest assumes, when you sell short, you'll be charged a borrow fee that Quantopian doesn't include, etc) but more important is generalization error. What that means is that this algo may have worked well in the past, and in fact, if you had run it live in the backtest period, it may have been successful but that's no guarantee of future performance. The reason is that the main reason you're considering this algo is that it backtested well, so there's a lot of overfitting of data. IE, an 'algo' which just bought Apple in 2000 and held on since would also do great, but in some sense, you only know that after the fact.

  2. I'm not sure what you're talking about -- when I run the backtest, the first transaction is a buy, second is a sell, third is a buy.

If you're asking how true backtests results are compared to real time results, that is, when backtests are re-run over periods where one has accumulated real time results, the two, apparently, tend to match up fairly closely. That is, according to the techs here at Quantopian. Alisa has confirmed that executions (bt vs rt) align on their monthly ETF rebalance strategy. Grant H. also confirmed that his currently trading strategy, when backtested over the same periods as the real time results showed considerable alignment.

Bottom line, bt vs rt results (when tested over the same periods) do not diverge appreciably.

If you're asking about predicting the future based on backtest results, you're in a world of prognostication here. I'm sure you've seen the caveats plastered over all of the "theoretical" results for every fund for every strategy ever created "Past results do not predict future returns."

Thank's Alex!

I understand there will be costs associated with each buy and sell, will look into incorporating that in my back tests somehow.

Not sure how bid-ask spreads would affect backtest results vs live trading? If I am buying and selling at market price in my algo than how would the bid-ask spreads be any different than they would have been while trading in that period back then?

So the algo which I attached would have performed exactly the same during trading in the same time-frame if there were no transaction costs and the bid-ask spreads would not have affected?

Hope I am able convey my point :)

I would like to know how to mirror an algo performance in backtest and when trading in the same time frame.

Appreciate your help!

Thanks! Market Tech,

Yes, I am not expecting algo to produce similar returns based on how it performs in past. I wanted to know if actual returns would mirror backtest results if the same algo was used to trade in the same time frame excluding transaction costs.

Also, below are the first few orders from the log for the algo I posted. First few transactions are selling Apple without any buy orders. First buy orders happen on 1/14/2014. Dont understand why sell orders are placed before any buy orders.

1/2/2014 handle_data:60INFO Selling AAPL
1/3/2014 handle_data:60INFO Selling AAPL
1/6/2014 handle_data:60INFO Selling AAPL
1/7/2014 handle_data:60INFO Selling AAPL
1/8/2014 handle_data:60INFO Selling AAPL
1/9/2014 handle_data:60INFO Selling AAPL
1/10/2014 handle_data:60INFO Selling AAPL
1/13/2014 handle_data:60INFO Selling AAPL
1/14/2014 handle_data:53INFO Buying AAPL
1/17/2014 handle_data:60INFO Selling AAPL
1/21/2014 handle_data:60INFO Selling AAPL
1/23/2014 handle_data:53INFO Buying AAPL
1/24/2014 handle_data:60INFO Selling AAPL
1/27/2014 handle_data:60INFO Selling AAPL
1/28/2014 handle_data:60INFO Selling AAPL
1/29/2014 handle_data:60INFO Selling AAPL
1/30/2014 handle_data:60INFO Selling AAPL
1/31/2014 handle_data:60INFO Selling AAPL
2/3/2014 handle_data:60INFO Selling AAPL
2/4/2014 handle_data:53INFO Buying AAPL

You'll realize that you're selling (closing your position) regardless if you HAVE a position or not. Try this:

    if (context.portfolio.positions[context.security].account > 0):  
        order_target(context.security, 0)  
        log.info("Selling %s" % (context.security.symbol))