Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Rookie Q: how do I store data for consecutive ticks?

Hi there,

This is my first time using python (although I'm experienced in MATLAB), so I'm assuming the solution here is something I'm missing in basic python..

Can I store a variable containing a numerical value calculated in one iteration of the code (i.e. in one bar), for use in the next? In other words, how do I create a variable that will store price data for a number of consecutive minutes? I tried using 'history' to call past prices, but this would only pull the daily closing price; not the prices for previous minutes.

Any help much appreciated. Thank you!

8 responses

Hey Romi,

You can use the context object to store data in one bar and use it in the next. context is passed to handle_data on every iteration.

Best,
Ryan

Thanks a lot Ryan!

Hi Romi,

I just posted a similar example code snippet to another thread and realized it was applicable here as well. You are correct that history() currently only supports grabbing daily trailing prices. Here is one way to store up 1 trailing day (390 bars) of minutely data. Do note that while history will populate the historical window with data back prior to the backtest start time, this method is an accumulator and will start off empty, adding bars each minute until the maximum length of 390 minutes is reached.

Hope this is helpful. Best wishes, Jess

from collections import defaultdict, deque  
from functools import partial  
import numpy as np


def initialize(context):  
    context.stock = sid(24)  
    context.prices  = defaultdict(partial(deque, maxlen=390)) 

# Will be called on every trade event for the securities you specify.  
def handle_data(context, data):  

    price = data[context.stock].price  
    context.prices[context.stock].append(price)  
    stock_series = np.array(context.prices[context.stock])  
    record(number_of_points_stored = len(stock_series))  
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Hi Jessica - thank you very much for sharing this example. That's exactly what I'm looking to do. Very helpful; much appreciated!

@Romi (and Jessica) keep in mind that You must limit the size of your array via something like

del stock_series[0:maxLength]  

Otherwise the Quantopian simulation run will crash over a multiple year run. Run out of memory I'm guessing. (This happened to me in my framework until I added the truncation logic)
you can see my (complex) framework here: https://github.com/Novaleaf/QuantShim

EDIT: oh i see that the 'defaultdict' line has that occur automatically. Just keep in mind that you always do this for normal arrays too!!!

This seems to work and you get a DataFrame to work with. Deque is probably more efficient, but I'm not really sure.

class DataCache:  
    def __init__(self, bars=390):  
        self.bars = bars  
        self._data = {}  
        self.df = None  
    def append(self, data):  
        dt = get_datetime()  
        bar = pd.Series({i: data[i].price for i in data.keys()})  
        self._data[dt] = bar  
        self.df = pd.DataFrame(self._data).T.tail(self.bars)  
        self._data = self.df.T.to_dict()

Here's another approach that uses the minute-bar data returned by history to build up a dataframe:

import pandas as pd

window = 5

def initialize(context):  
    context.stocks = [ sid(19662), # XLY Consumer Discrectionary SPDR Fund  
                       sid(19656), # XLF Financial SPDR Fund  
                       sid(19658), # XLK Technology SPDR Fund  
                       sid(19655), # XLE Energy SPDR Fund  
                       sid(19661), # XLV Health Care SPRD Fund  
                       sid(19657), # XLI Industrial SPDR Fund  
                       sid(19659), # XLP Consumer Staples SPDR Fund  
                       sid(19654), # XLB Materials SPDR Fund  
                       sid(19660)] # XLU Utilities SPRD Fund  
    context.prices = pd.DataFrame()  
def handle_data(context, data):  
    prices = history(1,'1d','price')  
    context.prices = context.prices.append(prices)  
    context.prices = context.prices.tail(window)  
    if len(context.prices) < window:  
        return  
    print context.prices  

[EDIT] Please note the help page documentation on history if adapting this code to include volumes (e.g. computation of VWAP). Although history returns price data for the current minute, the volume is cumulative over the current trading day.

I saw you are new to Python so I thought I'd toss it in an example so you can see how it works. This one keeps just the last 25 bars.