Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Difficulty building custom CSV file
Pumplerod
posted Jan 27, 2014

I'm trying to use fetch_csv to import data into my algo which is not really time series data. However I did add the columns needed (I thought) to pretend that it could be.

The file is on a google-drive but this is all it is...

date,symbol,structure,w1,w2,w3,w4,w5,w6,w7,w8,w9,w10,w11,w12,w13,w14,w15,w16,w17,w18,w19,w20,w21,w22,w23,w24,w25,w26,w27,w28,w29,w30,w31,w32,w33,w34,w35,w36,w37  
2014-1-27,network,[[100100010][100101100][111111000]][[[1111][0011][1001][0100][1110]][[1][1][1]][[-]],3.96489983052,0.256267928294,2.58264955512,-3.02993251828,-0.612409291844,2.86787389432,-1.89348562646,-3.13479725416,-1.01000592679,0.977270740461,-0.872816523585,0.663956467671,2.92114547775,1.50545732285,-2.44931115262,0.760518144019,-2.51452218298,-0.744949743825,-1.55683875615,-0.648191368613,0.0172223126075,-0.779474079962,1.4852870671,-0.837721859841,1.81820255163,1.97897433171,0.831748710153,0.0882299791464,-0.103895015628,0.331294397245,1.08311485906,0.137883248438,0.517014392756,-0.27755271172,1.9031025167,-0.985003230088,-1.8757665386

When I try to build the algo I get an error: "KeyError: u'no item named date'"

what have I done wrong?

Here is the code I'm using to test to make sure I can import the data...

def initialize(context):  
    context.STOCKS = [sid(8551)]  
    fetch_csv('https://drive.google.com/uc?id=0B7H0VnE16WlORWhLcEtPdDkwb1k',symbol='network',date_column= 'date',date_format = '%y-%m-%d')  
    pass


def handle_data(context, data):  
    pass
14 responses
Alisa Deychman
Jan 27, 2014

I took a look and your format was correct. The problem is importing files from Google Drive, which throws an error. I've made a note of the issue, thanks for bringing it to our attention! In the meantime, try uploading the data into your public folder on Dropbox and using the public link in Fetcher. For more info, see here.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Pumplerod
Jan 27, 2014

Just for people following...

I did try moving the file over to dropbox and the link that they provided me was: http://www.dropbox.com/s/qa6uipsrznpxmpl/testNetwork.csv
I then added the "?dl=1" so that the file would be directly downloadable. This did not work.

However if I changed the URL to: http://dl.dropboxusercontent.com/s/qa6uipsrznpxmpl/testNetwork.csv then I no longer get an error. Now I'm stuck trying to figure out how to access the data. Since it isn't a time series, it's something I want to use during the code initialization. Is there a way to do this?

Jessica Stauth
Jan 28, 2014

I think your best bet might be to use the pre/post function capabilities that are part of Fetcher - they let you interact with the imported data structure all at once vs the time sliced acces in handle_data().

I'm not completely clear on the process you are looking for, do you need this data to be lined up with the OHLCV data?

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Pumplerod
Jan 28, 2014

I my case my data doesn't need to be lined up with anything. I'm simply trying to pull in an artificial neural network and associated weights to later be used to filter data through.

I can have all the code to generate the network within my methods, however, what I'm looking for is a process I can use to pull in a network that I have trained elsewhere.

Jessica Stauth
Jan 28, 2014

So here's an example of pulling the weights in (I'm not sure how to pull in the structure object, you'll have to explain a bit more about how you want to use that).

The way this needs to work with Fetcher is to give a dummy date as to when you'd like the weights to be available to your algo. Fetcher tries to protect you from lookahead by not making data available to you in handle_data() from the 'future'. So I've assigned your weights vector a date of 1/1/2013 (you had 1/27/2014 in there, which won't work with trying to access the values on any dates earlier).

While you only feed in one weight vector our system will forward fill the values (or use the 'last available' value depending on how you interpret it) and so you'll have access to the w1 ... wN values in handle data.

Now, if you'd like to have these weights be dynamic you can also just point Fetcher to a document that updates and the new weights will be available to your algo based on the dates you assign.

In this example I've just plotted the first 5 weight values using the record() function for inspection.
Let me know if this is useful and if there's anything else I can help with.

Cheers,
Jess

Pablo Ríos
May 5, 2014

Hi Alisa, are you planning to fix the issue of not being able to fetch csv files from Google Drive ? If so, what's the ETA for it ? Are you publicly tracking user found defects / user enhancements requests on Quantopian somewhere in this site, so users can check the status of them ? I'm a Google Drive user and I really don't want to have a separate drive for this.
Thank you !

Alisa Deychman
May 5, 2014

Hi Pablo,

We're starting to look into the logistics of importing files from Google Drive via Fetcher, but I'm not sure of the delivery date. At present, we support Dropbox, Copy, and other storage providers that pass simple CSV files. Unfortunately, when Google Drive passes the file, it adds additional formatting that cannot be interpreted by the backtester.

We don't publicize our list of bugs and feature requests, but when we fix something or add a new feature that's been mentioned in the forums, we make sure to share the news!

Jason Swearingen
May 7, 2014

fyi google drive works fine, but you need to set up a public webshare, and access it via that public web url. accessing the link directly from google drive does NOT work, as it's not actually a link to the file but rather the page hosting the file

edit: instructions here: https://support.google.com/drive/answer/2881970?hl=en

Pablo Ríos
May 8, 2014

Thanks Jason, it worked. However, I was not using the https://drive.google.com/file/d/0B0bc4zStzXO1dWtQSkFJbUFaYkk/edit?usp=sharing URL syntax, but the https://drive.google.com/uc?export=download&id=0B0bc4zStzXO1dWtQSkFJbUFaYkk URL syntax which is actually a link to the file, which can be retrieved by any HTTP client.

Dan Dunn
May 9, 2014

Pablo, that's pretty interesting. I spent a few minutes trying to find that second link, the one that worked for you, but I couldn't figure it out. Do you mind walking me through how you got that good link?

Thanks.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Jason Swearingen
May 10, 2014

@Dan, he followed the instructions I posted.

Pablo Ríos
May 10, 2014

Hi @Dan
The second link worked for me with several HTTP client libraries (ex.: Python 2 urllib2 or Python 3 urllib.request) but not with Quantopian's fetch_csv(), so I think you're (purposefully ?) not agnostic to the HTTP source. As Jason said what worked for me are the instructions he posted, which are also described here

Dan Dunn
May 12, 2014

The thing I've been struggling with is how to find the URL of the form "export=" rather than "usp=sharing". The former works, the latter does. I wonder if we're on different versions of Google Drive or something.

We're definitely picky in that we need the CSV, and not the "pretty" HTML wrapped around the CSV on some pages.

Pablo Ríos
May 12, 2014

I found the second URL syntax in a post at Google Product Forums public group, so I don't know how much support from Google it has.
I wasn't able to get this URL from Google Drive web UI, I needed to create it manually by taking the file id from the 1st URL ('...usp=sharing').
Anyway, I believe the recommended way to support Google Drive in fetch_csv() would be using Google Drive REST API (https://developers.google.com/drive/v2/reference/)
Regards

About Quantopian
Careers
Community
Events
Documentation
Academia
Lectures
Status
Facebook
Twitter
LinkedIn
YouTube
Terms of Use
Privacy Policy

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.