Richard,
Thanks for sharing, fetcher/history issues are something I've run into as well. I'm not 100% sure if this will apply to your specific case, but if you would like to do historical calculations on fetched data, there is one other option I have found.

If you add a pre_func argument to the fetch_csv function, you will have access to the entire dataframe while in that function. That means you can apply rolling calculations on the data and append the results as new columns. This example here adds a moving average and standard deviation field to the VIX data. You will have to be careful not to introduce look ahead bias though.

David

David
Seong Lee suggested this to me first, but I wanted something a bit more generic, because I have a lot of code and I wanted the flexibility of having the array of values to manipulate. But it's good that you mentioned it, because for many people it's probably a better solution.

Rich

I just discovered the need for this, but I think I'll have to do the packed string method - I need to calculate historical spreads between various fetched time-series. Thanks, I look forward to a better way of doing this one day though!

Hi dAVid... what is the difference... between... te first chart and your chart...?? are you using... vix as buy and sell signal... or its already in the algo... that when the vix spikes... automatic buy some stocks... puzzled....

@David - Don't the mean and std seem to be going right to left in your chart? Have in mind that Yahoo historical data comes in reverse-chronological order.

Just realized fetch_csv does not work in live mode. I am stunned that the fetch_csv function was even introduced without proper support in live execution. Now my algo is stuck in backtest mode without this brittle hack.

It works for me, you just can't enter them into the contest. What is the issue you are seeing?

https://www.quantopian.com/posts/function-like-history-but-for-fetch-csv-data

This is the unpleasant hack I use to get access to some number of historical rows of data from fetch_csv, in a way that avoids look-ahead bias. It's been working well enough for a couple of months of live trading though.

EDIT: And all due credit to the above Richard Diehl and Seong Lee for the original idea!

Thank you Simon, I will have a look!

The main issue as I see with these workarounds is that they increase risk of bugs in live algorithms, where you'd least want increased risk. I'd be much happier Quantopian built a consistent API where the difference between live and backtesting was minimal.

Yeah I absolutely agree, but I think fetch_csv in general is something that Quantopian has less interest in supporting these days, so I think it's basically take-it-or-leave-it. Don't want to speak for them though.

Good motivation to keep alternative options on the back burner!

Yeah, the new pivot seems to be selling data so the motivation for providing methods for people to get their own is diminishing I guess!

@Klon - I'd be much happier Quantopian did not release tools that allow you to calculate moving averages based on future data; and when the bug was discovered (see July 25, 2015, above), admit and correct it, rather than just quietly disallowing it in the contest.

Yeah, fetch_csv can be tricky to use properly.

There is no simple way. To do it correctly, you must do it the complicated way.

Perhaps try printing the specific value you are interested in, the same way you are recording it.

data.current('vix', 'Adj Close')

I posted a simple version of saving the data to an array:

https://www.quantopian.com/posts/fetcher-history-calculate-moving-average

Very good idea about adding a column of data...that really just skips the need to capture the array. Thinking about it now, I would just process the dataframe and then replace it with the values I need on a daily basis.

Hi Winston ,

See if this works for you. Just make sure to spot check a few values to make sure my logic is right.

from collections import deque

def initialize(context):

    context.vix_fields = ['Adj Close', 'High', 'Low', 'Open']  
    context.vix = {}  
    for field in context.vix_fields:  
        context.vix[field] = deque([], 5)

    url = 'http://ichart.finance.yahoo.com/'  
    url += 'table.csv?s=%5EVIX&d=0&e=1&f=2017&g=d&a=0&b=1&c=2010&ignore=.csv"'  
    fetch_csv(url,  
              date_column='Date',  
              date_format='%Y-%m-%d',  
              symbol='vix',  
              usecols=context.vix_fields) 

def before_trading_start(context, data):

    for field in context.vix_fields:  
        val = data.current('vix', field)  
        context.vix[field].appendleft(val)

    # today's open  
    print 'open - %.2f' % context.vix['Open'][0]

    # yesterday's high/low (check that we have more than one day's data)  
    if len(context.vix['Open']) > 1:  
        print 'yesterday high/low - %.2f/%.2f' % (context.vix['High'][1], context.vix['Low'][1])  

Hi Winston and thanks :-)

Here, I will give you the cliff notes:

initialize:
- create array of csv fields at context.vix_fields
- create dictionary at context.vix
- initialize dictionary with deques (see https://docs.python.org/2/library/collections.html#deque-objects)
- make call for csv file

before_trading_starts:
- loop through field names, get data value for each field and add to context.vix dict

context.vix contains last five days of data. You can change deque([], 5) to some other number.

Hi Winston,

this works:

    if len(context.vix['Adj Close']) > 1:  
        vix_close_yesterday = context.vix['Adj Close'][1]  
        print vix_close_yesterday  

basically, what is happening is we are creating the data structure in the initialization function. The data structure looks like this:

context.vix = {  
    'Open': deque([], 5),  
    'High': deque([], 5),  
    'Low': deque([], 5),  
    'Adj Close': deque([], 5)  
}

For reference:
https://docs.python.org/2/tutorial/datastructures.html#dictionaries
https://docs.python.org/2/library/collections.html#deque-objects

The deque is initialized with an empty list as a container and will be max length of 5 elements. Then in the before_trading_start function, which is called everyday, we are filling in the data. We get an index error if we try to access an element before the data has been filled in. Once we get the five elements, going on older values are dropped and new values are added.

So, at the start of the backtest, if we want to access context.vix['High'][1] first we need to test that context.vix['High'] has a length greater than one (since lists are indexed starting at zero). To access context.vix['High'][2], test that the deque has a length greater than two, etc. up to context.vix['High'][4] which needs a length greater than four because we want the fifth element.

If you need more than five days, change the deque number in the initialization function.

Hope that helps...it is harder to explain than it is to code :-)

Less fancy, which is sometimes better!

from collections import deque

def initialize(context):  
    context.vix = {  
        'Open': deque([], 5),  
        'High': deque([], 5),  
        'Low': deque([], 5),  
        'Adj Close': deque([], 5)  
    }  
    url = 'http://ichart.finance.yahoo.com/table.csv'  
    url += '?s=%5EVIX&d=0&e=1&f=2017&g=d&a=0&b=1&c=2010&ignore=.csv"'  
    fetch_csv(url, date_column='Date', symbol='vix')  

def before_trading_start(context, data):  
    for field in ['Open', 'High', 'Low', 'Adj Close']:  
        val = data.current('vix', field)  
        context.vix[field].appendleft(val)  

    print 'open - %.2f' % context.vix['Open'][0]  

    if len(context.vix['Open']) > 1:  
        print 'high - %.2f' % context.vix['High'][1]  
        print 'low - %.2f' % context.vix['Low'][1]  
        print 'close - %.2f' % context.vix['Adj Close'][1]  

Note that %.2f rounds the float. See:
https://docs.python.org/2/library/string.html#formatspec

Trouble with these methods is they take time to "warm up" in live trading. That becomes more tedious when you need 30+ days of history.

True, Simon. Winston only needs five days of data. Of course, if you need 30 days of data, you can do something like this:

def shift(df):  
    vals = df['Close'].shift(-30)  
    df['Last Month Close'] = vals  
    return df

def initialize(context):  
    url = 'http://ichart.finance.yahoo.com/table.csv'  
    url += '?s=%5EVIX&d=0&e=1&f=2017&g=d&a=0&b=1&c=2010&ignore=.csv"'  
    fetch_csv(url, date_column='Date', symbol='vix', pre_func=shift)  

def before_trading_start(context, data):  
    for field in ['Close', 'Last Month Close']:  
        val = data.current('vix', field)  
        print '%s - %.2f' % (field, val)  

There's more than one way to skin a cat :-)

Extend that to an arbitrary number of days in the past, and you might get a solution similar to the one I posted last year. :)

haha...I guess there is always someone reinventing the wheel :-)

Hey, thanks too, Winston...I also figured out a bunch of stuff working through this with you!