Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Frustrating Experience with switching to minute backtests

Hi all, recently I've been trying to convert my mean reversion algo from daily to minute, and I can't figure out what documentation to reference. The standard functions automatically convert their time periods to minutes instead of days, which isn't very helpful. This is what I have, which slows the backtest to a crawl, and frankly I'm not even convinced it's working properly. Any help would be appreciated.

            price_history_10 = history(bar_count=10, frequency='1d', field='price')  
            price_history_20 = history(bar_count=20, frequency='1d', field='price')  
            price_history_40 = history(bar_count=40, frequency='1d', field='price')  
            stock_history_10 = price_history_10[stock]  
            stock_history_20 = price_history_20[stock]  
            stock_history_40 = price_history_40[stock]  
            mavg_10_a = talib.EMA(stock_history_10)  
            stdev_10_a = talib.STDDEV(stock_history_10)  
            mavg_20_a = talib.EMA(stock_history_20)  
            stdev_20_a = talib.STDDEV(stock_history_20)  
            mavg_40_a = talib.EMA(stock_history_40)  
            stdev_40_a = talib.STDDEV(stock_history_40)  
            mavg_10 = mavg_10_a[0]  
            stdev_10 = stdev_10_a[0]  
            mavg_20 = mavg_20_a[0]  
            stdev_20 = stdev_20_a[0]  
            mavg_40 = mavg_40_a[0]  
            stdev_40 = stdev_40_a[0]

8 responses

Hello Ken,

As a start, you might try paring down the number of calls to the history API. I think you should be able to use just one:

price_history = history(bar_count=40, frequency='1d', field='price')  

Then, you can use indexing (http://pandas.pydata.org/pandas-docs/stable/indexing.html) to pick the timeframe of interest.

Also, I'd have a look at the efficiency of talib, compared to pandas (or numpy/scipy) for computing the statistics.

Finally, keep in mind that one would expect a minutely backtest to be ~ 390 times slower than a daily one. If you only need to trade once a day, or within a time window, then you can speed it up by applying logic to basically skip blocks of code within handle data for large portions of the trading day. But this will only go so far, since there is still overhead.

Grant

Hello Ken/Grant,

I ran the algo for 2013 with just 2 SIDS giving a run time of 10 mins. 29 secs. Amending the algo to calculate only at 09:31 EST / 14: 31 GMT reduced the run time to 2 mins. 7 secs.

I then used one call to 'history' and the run time was unchanged at 2 mins. 8 secs. (see attached). Maybe Grant can 'pandafy' the algo?

P.

Nice. Yes, I forgot to mention that keeping the number of sids to a minimum is a consideration, as well. I think that history will grab all of the sids referenced in the algo (not commented out), so if any are just hanging around, I'd remove them. I won't get to the "Pythonic pandafication" now. Gotta run. --Grant

Thanks! Will try it out.

The results keep coming up with nan for some reason, am I missing something? I'm running this on Apple and SPY.

Hello Ken,

Specify a time period to TA-Lib i.e.

mavg_10_a = talib.EMA(stock_history_10, timeperiod = 10)  
stdev_10_a = talib.STDDEV(stock_history_10, timeperiod = 10)

P.

Still getting nan. This is my current code:

price_history_40 = history(bar_count=40, frequency='1d', field='price')  
    exchange_time = pd.Timestamp(get_datetime()).tz_convert('US/Eastern')  
    if exchange_time.hour == 9 and exchange_time.minute == 31:  
        for stock in context.stocks:  
            #Universal variables  
            price = data[stock].price  
            order_ticket = 0.0  
            #Mean reversion variables  
            stock_history_10 = price_history_40[30:40][stock]  
            stock_history_20 = price_history_40[20:40][stock]  
            stock_history_40 = price_history_40[stock]  
            mavg_10_a = talib.EMA(stock_history_10, timeperiod = 10)  
            stdev_10_a = talib.STDDEV(stock_history_10, timeperiod = 10)  
            mavg_20_a = talib.EMA(stock_history_20, timeperiod = 20)  
            stdev_20_a = talib.STDDEV(stock_history_20, timeperiod = 20)  
            mavg_40_a = talib.EMA(stock_history_40, timeperiod = 40)  
            stdev_40_a = talib.STDDEV(stock_history_40, timeperiod = 40)  
            mavg_10 = mavg_10_a[0]  
            stdev_10 = stdev_10_a[0]  
            mavg_20 = mavg_20_a[0]  
            stdev_20 = stdev_20_a[0]  
            log.info(mavg_10)  
            log.info(stdev_10)  

Never mind, I got it working with -1 instead of 0 here:

mavg_10 = mavg_10_a[0]  

Thanks for the help!