Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Difficulty understanding fetcher_csv

Hey all,

Trying to get a .csv to work correctly with no avail. Essentially, the .csv has a set of symbols, dates, and a value of an indicator (called percentage_variance) that create signals for the algorithm. If a value is present, I'd like to be able to purchase the underlying symbol, but I'm having a hard time iterating one a date-by-date basis (the values seem to be conglomerated together). If anyone could point me in the right direction of being able to purchase based on the above critera, I would greatly appreciate it.

Thanks!

3 responses

An example of the conglomeration when attempted to iterate is attached to this post (the percentages are not correct, as much as I would like them to be)

Since the code is having issues populating in the algorithm pane, here it is:

from datetime import date as dt  
import numpy as np  
import pandas as pd  
# url is public  
url = 'https://docs.google.com/spreadsheets/d/1EpBxF1eu45UWGze5g9pJSZR2wuympC0pUSARh0it-mg/export?format=csv&id=1EpBxF1eu45UWGze5g9pJSZR2wuympC0pUSARh0it-mg'

def preview(df):  
    print df.tail(10)  
    return df

def forward_fill(df):  
    preview(df)  
    # grab latest value for each security  
    df2 = df.groupby('symbol').last().reset_index()  
    # get today's date  
    today = dt.today()  
    # create index for today's entries on DataFrame  
    index = np.repeat(today, len(df2))  
    # reset index to today's date  
    df2.index = index  
    # concatenate previous dates and today's date  
    df = pd.concat([df, df2])  
    preview(df)  
    return df

def initialize(context):  
    """  
    Called once at the start of the algorithm.  
    """  
    fetch_csv(url,  
              pre_func=forward_fill,  
              date_column='date')  
def before_trading_start(context, data):  
    currently_listed = [x for x in data.fetcher_assets if data.current(x, 'percentage_variance') > 0]  
    print len(currently_listed)  

Here's the last work that I did on Fetch_CSV https://www.quantopian.com/posts/dynamic-tspp-list

There might be some clues in there that can help you, I basically had to take a list of stocks with dates and NOT buy them. The behavior in the backtest is different than live trading. In live trading I needed to make all the dates today's date because that's what the live trading looks for. In the backtester it uses each stock from it's date going forwards. The backtester is incapable of looking at stocks with dates further in the future than the current day to prevent look-ahead bias.

I will tell you that what helped me understand it was when I learned that fetch_csv() is just quantopian's wrapper around pandas.read_csv() https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html and the **kwargs is a dictionary that is passed directly to the pandas.read_csv() function. This can be used to help put the CSV into a format that quantopian's fetch_csv() can understand.

Now, here' the catch:

When Fetcher is used in Live Trading or Paper Trading, the fetch_csv() command is invoked once per trading day, when the algorithm warms up, before market open. It's important that the fetched data with dates in the past be maintained so that warm up can be performed properly; Quantopian does not keep a copy of your fetched data, and algorithm warmup will not work properly if past data is changed or removed.  
Note that when using this feature in live trading, it is important that all historical data in your fetched data be accessible and unchanged. We do not keep a copy of fetched data; it is reloaded at the start of every trading day. If historical data in the fetched file is altered or removed, the algorithm will not run properly. New data should always be appended to (not overwritten over) your existing Fetcher .csv source file. In addition, appended rows cannot contain dates in the past. For example, on May 3rd, 2015, a row with the date May 2nd, 2015, could not be appended in live trading.  

For some reason, probably related to the statement's above, I could not get fetch_csv() to work reliably in live trading for longer than a few days. It would randomly fail to fetch at least once a week and cause the algorithm to error out and be manually restarted. Even when the data was always perfectly availible and unchanging (I tried dropbox, google drive, and directly from the FINRA website. I eventually had to give up because it was such a HUGE pain in the ass.

My recommendation: DO NOT LIVE TRADE WITH FETCH_CSV()