Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
How to "chunk" minutely data into 5, 15, 78 minute bars?

Hi All,

New to Quantopian and trying to figure out how to create say 5 minute bars from the minute based DataFrames returned by history(). So if I wanted to create a 20 period moving average of 5 minute bars, how would I do that?

I tried using the Pandas resample() function but I get a lot of NaN entries which causes runtime errors when I try using talib functions on the resampled data.

8 responses

here is a small code example the produces the NaN runtime error:



import talib  
import math

def initialize(context):  
context.stock = symbol('SPY')  
context.pct_invested = 0  
Will be called on every trade event for the securities you specify.

def handle_data(context, data):

# prices is a pandas dataframe with several built-in transformations  
prices = history(200, '1d', 'price')  
prices_minute = history(500, '1m', 'price')  


# Pandas built-in re-sampling function  

weekly = prices.resample('1W')  
monthly = prices.resample('1M')  

#try to calc 20 period weighted moving average of 5 minute bars  
min5bars = prices_minute.resample('5Min')  
WMA = talib.MA(min5bars[context.stock], timeperiod=50, matype=2)  

#now try something more complicated like a 16 period HULL moving average of the 5 minute bars  
wma_length = 16  
WMA1 = 2*talib.MA(min5bars[context.stock], timeperiod = (wma_length/2), matype=2)  
WMA2 = talib.MA(min5bars[context.stock], timeperiod = wma_length, matype=2)  
HullMA = talib.MA(WMA1-WMA2, timeperiod = round(math.sqrt(wma_length)), matype=2) #RUNTIME ERROR HERE  


log.info(weekly.tail()) # .tail just displays the 5 most recent entries  
log.info(monthly.tail())  
log.info(min5bars.tail())  

record(weekly=weekly.mean()[context.stock],  
       monthly=monthly.mean()[context.stock])  

Hi Rakesh,

If you are just trying to compute a moving average, I don't see the advantage of creating custom bars. What's your thinking? Are you trying to replicate results from another backtester?

Grant

Hi Grant,

What I am trying to do is similar to understanding what is going on in daily/weekly/monthly charts but at the intraday level.

In general, I would like to apply talib-like functions (not just moving averages) across multiple intraday timeframes (not just 1 minute bars).

If I can start to simultaneously compare what is happening on 15min/60 min/240min timeframes for example, I can generate "stronger" buy and sell signals when all the timeframes are "in sync."

Thanks Rakesh,

Since you are working intraday, you might be interested in the Pandas pivot table example I posted recently:

https://www.quantopian.com/posts/minutely-history-frame-interval-slicing

Scott also provides some Pandas groupby guidance there.

I don't yet know how to do the intraday OHLCV bars at an arbitrary sampling period you're looking for, but it should be feasible.

Grant

Thanks for the link Grant. I actually looked it over yesterday but it is not what I need.

The intraday OHLCV at arbitrary sampling periods is exactly what I am looking for (although having the accumulated Volume for the sample period is not critical at this time)

Hello Rakesh,

Rather than creating intraday OHLCV bars at an arbitrary sampling period, you might consider analysis on a minute-by-minute rolling basis, with a variable trailing window length (e.g. your 15 min/60 min/240 min timeframes). I think if you basically smooth the minute bars provided by Quantopian on a rolling basis, with varying degrees of smoothing, and then apply your functions, you'll be able to compare the various smoothing timeframes.

If you generate bars, it seems you'll be throwing away information. If you want to take this route, note that Pandas supports rolling stats (see http://pandas.pydata.org/pandas-docs/stable/computation.html), including:

rolling_sum
rolling_min
rolling_max

So these can be used to find the V, L, & H values, respectively, for the trailing window. To find the O & C values, you just need to grab the first and last values in the trailing window. Then, you can construct the OHLCV bar in a format you can feed to your functions.

Hope this helps. If you arrive at a solution, it'd be great if you posted an outline or code here (obviously generalized to remove any confidential information).

Grant

Hey Grant,

Thanks for the rolling stats link. I will definitely look at and report back (if it makes sense).

Hi everyone, I have the same problem as Rakesh.
I thought that it is not necessary to resample the 1 minute bars into 5, 10, 15, 30min bars etc, especially since for each bar I only use the close (and not the coplete bar).
To me it would be enough from a list of closes of 1 minute bars to take a value every 5, 10, 15 etc.
The rolling function on a list, however, does not do the right one? in your opinion how can i do it?