Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Way to correctly fetch 1 min intervals csv data?

Hello all,

I am very new to Quantopian, heard a lot of good things about it and would like to evaluate it here.

Many of my trading strategies involve 5 min and some times 1 min custom indicators. I do want to import/fetch them to build my strategies and backtests here.

I wrote a simple code to try to fetch the csv and print to to verify.

def initialize(context):

    csv_url = 'https://dl.dropboxusercontent.com/u/1049138/For_Cx/aazz.csv'

    fetch_csv(csv_url , symbol='goods', date_column = 'datetime', date_format='%Y%m%d %H:%M:%S', timezone='US/Eastern')


def handle_data(context, data):  
    a = data.current('goods', 'indicator')  
    print(a)

the csv file contain two columns
i) a date time column at 1 min resolution. and
ii) a indicator column.

The file contains 1 full day of indicator as show below. (Note: I have truncated many rows of data to avoid making this post too long.)

datetime,indicator  
20151117 09:31:00,114.74  
20151117 09:32:00,114.88  
20151117 09:33:00,114.7  
20151117 09:34:00,114.64  
20151117 09:35:00,114.55  
20151117 09:36:00,114.61  
20151117 09:37:00,114.65  
20151117 09:38:00,114.45  
20151117 09:39:00,114.41  
20151117 09:40:00,114.39  
...
...
...
20151117 15:50:00,113.5  
20151117 15:51:00,113.65  
20151117 15:52:00,113.54  
20151117 15:53:00,113.57  
20151117 15:54:00,113.7  
20151117 15:55:00,113.64  
20151117 15:56:00,113.57  
20151117 15:57:00,113.63  
20151117 15:58:00,113.66  
20151117 15:59:00,113.69  
20151117 16:00:00,113.8  

I then run a backtest with start and end time 11/17/2015 to 11/18/2015

and below is the result from the 'Log' tab

2015-11-17 22:31  PRINT 113.8  
2015-11-17 22:32  PRINT 113.8  
2015-11-17 22:33  PRINT 113.8  
2015-11-17 22:34  PRINT 113.8  
2015-11-17 22:35  PRINT 113.8  
2015-11-17 22:36  PRINT 113.8  
2015-11-17 22:37  PRINT 113.8  
2015-11-17 22:38  PRINT 113.8  
2015-11-17 22:39  PRINT 113.8  
2015-11-17 22:40  PRINT 113.8  
...
...
...
2015-11-19 04:53  PRINT 113.8  
2015-11-19 04:54  PRINT 113.8  
2015-11-19 04:55  PRINT 113.8  
2015-11-19 04:56  PRINT 113.8  
2015-11-19 04:57  PRINT 113.8  
2015-11-19 04:58  PRINT 113.8  
2015-11-19 04:59  PRINT 113.8  
2015-11-19 05:00  PRINT 113.8

As you can see, it seems the fetcher only retain value of the last row of indicator '113.8' and discarded the rest of them which obviously is not what I was expecting.
Note : I am operating at UTC+8 time zone and hence, I guess, the timestamps at the beginning of each printout row are shown in local time.

I have tried to play around the 'fetch_csv' method parameter such as changing timezone to 'US/Central' or eliminate the timezone parameter completely. Same result obtained.

I don't know what else I can do to make it works. So I hope some experts here could help me out.

Sorry for asking a maybe overly simple question, please forgive me and thanks in advanced.

Thanks
Kow

2 responses

Fetcher only runs once in the morning, and I think you can only really get the most recent value up to and including today's date. If you need the complete history, there are hacks to make that work, but you still only get the values as-of around midnight.

Hello Simon,

Many thanks.

Originally I thought that only History function cannot be used on imported custom data. I was assuming that after csv data is imported, the backtester will spit out the csv data based on the minute interval timestamp specified.

But according to your response, it isn't true, only one daily data will be returned.

I believe support of imported data portion of Quantopian should greatly improved to make the platform more useful.