Hello Po,

The answer depends on whether you want to accumulate the data as a backtest runs, or if you would like the data at the start of a backtest. If the data are needed at the backtest start, then it is a question of how much minute-level data the backtester can provide and process, without exceeding memory and/or timeout limits. For the latter approach, you would use history to pull up two year's worth of minute bars (using field='high') and then find the max for each day.

Grant

Hi Grant,

Great idea! In my case I need to pull them at the start of a backtest. I am new to Python and Quantopian. Just curious, how would you store the pulled historical minute data (time and value) for running other calculations?

Po,

Not yet sure if it'll work. I'm trying:

def initialize(context):  
    context.stock = sid(24)

def handle_data(context, data):  
    highs = history(196560,'1m','high')  

The backtest seems to just hang.

You might need to do this on a rolling basis, which means that your backtest would get delayed for ~2 years.

By the way, what are you trying to accomplish?

Grant

Finally got an error, so the 'history' approach may be a non-starter:

There was a runtime error.
MemoryError
Algorithm used too much memory. Need to optimize your code for better performance. Learn More

Too bad to hear that... Trying to analyze stock intraday trends with these data. Thank you though!

I sent a help request to Quantopian to find out if hitting the memory limit is expected, or if it is a bug.

If you are just needing to do analysis and not simulate trading, then accumulating the data might work. Each day, you'd analyze 390 data points, and pick the absolute high and its datetime stamp. After two years of running the backtest, you could then do your analysis on the daily highs. Let me know if this would be of interest. There should be some Pandas tools that make it easy.

I'ts very easy to get the intraday high for every day of the last two years. Just use history in 1d mode, and look at the high price.

To get the intraday high, and the minute that high happened, will take more work. The method that Grant posted is a brute-force method that is going to hit a memory limit. It's expensive to maintain a datapanel with every minute's worth of data for 2 years. However, what you could do is run through every minute of a given day and store the high and the minute in your own datapanel. The resulting data panel will be 1/390th of the size, and will be very manageable.

Po, here's a backtest that tracks the minute high price and stores the value if its the highest price of the day. I also added logging to see the price changes and the time it happens.

Is this what you were looking for? If you'd like additional help we can expand this further depending on your strategy.

Hi Po,

It is also possible to store (i.e. log) the daily high prices and datetime stamps (e.g. as a Pandas DataFrame), if that would be useful. However, as Dan points out, presently there is no way to have the backtest 'warmed up' at the start with the 2-year long dataset you are requesting.

Grant

Grant, Dan and Alisa,

Thank you all for the great ideas! I'll try them out!