Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Problem resampling 1 minute bars into 60 minute bars
Harry Ziegler
posted
Seeking Help Getting Started Using Data Tools and Tips numpy/pandas

Hi all,

I have some experience building algorithms, but I'm very new when it comes to Quantopian and Python. Right now, I'm trying to port all of my algorithms from TradingView to Quantopian because this platform is much more powerful and provides me with more historical data. However, I'm running into some issues with creating 60 minute bars from the 1 minute data that Quantopian provides. I tried to resample the data following these two posts (https://www.quantopian.com/posts/writing-algos-for-time-frames-other-than-1m-and-1d and https://www.quantopian.com/posts/custom-intraday-bars) as closely as possible, but I can't quite get it to work. I printed the resampled DataFrame 'prices_60m' to troubleshoot, and I noticed something very weird. Some timestamps give me the correct price every hour, but most give me NaN. Could I get some help with this?

Here is a snippet of the logs I'm talking about:
2014-03-07 09:30 PRINT prices_60m: 2013-12-30 14:00:00+00:00 3.800
2013-12-30 15:00:00+00:00 3.795
2013-12-30 16:00:00+00:00 3.878
2013-12-30 17:00:00+00:00 3.870
2013-12-30 18:00:00+00:00 3.850
2013-12-30 19:00:00+00:00 3.840
2013-12-30 20:00:00+00:00 3.850
2013-12-30 21:00:00+00:00 3.840
2013-12-30 22:00:00+00:00 NaN
2013-12-30 23:00:00+00:00 NaN
2013-12-31 00:00:00+00:00 NaN
2013-12-31 01:00:00+00:00 NaN
2013-12-31 02:00:00+00:00 NaN
2013-12-31 03:00:00+00:00 NaN
2013-12-31 04:00:00+00:00 NaN
2013-12-31 05:00:00+00:00 NaN
2013-12-31 06:00:00+00:00 NaN
2013-12-31 07:00:00+00:00 NaN
2013-12-31 08:00:00+00:00 NaN
2013-12-31 09:00:00+00:00 NaN
2013-12-31 10:00:00+00:00 NaN
2013-12-31 11:00:00+00:00 NaN
2013-12-31 12:00:00+00:00 NaN
2013-12-31 13:00:00+00:00 NaN
2013-12-31 14:00:00+00:00 3.865
2013-12-31 15:00:00+00:00 3.840
2013-12-31 16:00:00+00:00 3.845
2013-12-31 17:00:00+00:00 3.860
2013-12-31 18:00:00+00:00 3.8...

5 responses
Vladimir

Harry,
Try to change this line: 

prices_60m = data.history(context.stock, 'close', 18000, '1m').resample('60T').last().bfill()
Harry Ziegler

Hi Vladimir,

Thanks for the help! I put that in my code and it worked! I have one question though: is backfill a proper solution, or more of a quick fix? Shouldn’t none of the values be NaN?

Vladimir

Harry Ziegler.

To me bfill() is better solution than ffill() or dropna().

Harry Ziegler

Hi again,

It seems that I have a gap in understanding with regards to how the data.history DataFrame works. Does the 1 minute DataFrame return NaN values before/after trading hours? If not, then why is the resampled DataFrame doing that?

Thanks,
Harry

Harry Ziegler

Hey guys,

I just wanted to give you a little update on this issue. I figured it out by poring over previous Quantopian posts (https://www.quantopian.com/posts/how-to-use-the-resample-correctly, https://www.quantopian.com/posts/resampling-other-timeframes, and https://www.quantopian.com/posts/algorithm-takes-time-to-warm-up). Much thanks to Dan Whitnable for all the answers! Here's the code I ended up with for calculating 1 hour bars:

prices_60m = data.history(context.stock, 'close', 18000, '1m').resample('60T', label = 'right', closed = 'right').mean().dropna()
About Quantopian
Careers
Community
Events
Documentation
Academia
Lectures
Status
Facebook
Twitter
LinkedIn
YouTube
Terms of Use
Privacy Policy

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.