Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Using CSV fetcher to set universe

Hello,

I'm still trying to get the hang of quantopian and have a few questions about using external csv files. I've attached the following backtest containing my source code.

Simply, I'm trying to make the following:

  1. I import a list of stocks in a csv formatted like this:
symbol,start_date,rank  
FXCM,01/01/2015,1  
IRC,01/01/2015,1  
FCS,01/01/2015,1  
HSEB,01/01/2015,1  
CXW,01/01/2015,1  
YOKU,01/01/2015,1  
BIN,01/01/2015,1  
ADMS,01/01/2015,1  
EDE,01/01/2015,1  
HR,01/01/2015,1  
ROVI,01/01/2015,1  
AFFX,01/01/2015,1  
SNH,01/01/2015,1  
BDBD,01/01/2015,1  
PBY,01/01/2015,1  
KING,01/01/2015,1  
WPP,01/01/2015,1  
BMR,01/01/2015,1

From what I've read, you need to define a date field, so in real life use, I would have a PHP script that updates that csv every day with the new stocks and the current date.

  1. Set these stocks to my current universe
  2. Loop through them and buy them based on some criteria

I would think something like this would be simple. I've been reading for hours, and am stuck. The confusing thing is that the code that I currently have works only sometimes. If I build the algorithm 20 times, maybe 4 of the times it will come back and show some logs of the csv data, but I can't ever get it to advance to my handle_data method to print out the stocks. Can someone provide some insight as to what I may be missing here?

Thanks a ton!
Bryce

EDIT: It looks like you can't see the backtest source code if there are build errors. Here's the code:


import datetime  
import pandas as pd  
import numpy as np

# my_universe returns a set of securities that define your universe.  
def my_universe(context, fetcher_data):  
    # fetcher_data is the data resulting from the CSV file from fetcher.  
    # set my_stocks to be every security in the fetcher_data  
    my_stocks = set(fetcher_data['sid'])  
    # log the size of the universe for debugging  
    context.count = len(my_stocks)  
    # print 'total universe size: {c}'.format(c=context.count)

    # return the securities we identified earlier  
    return my_stocks

# see a snapshot of your CSV for debugging  
def preview(df):  
    log.info(' %s ' % df.head())  
    return df

def initialize(context):  
    log.info("starting")  
    # import the custom CSV data file  
    fetch_csv("http://some_csv.csv",  
              pre_func=preview,  
              post_func=preview,  
              date_column='start_date',  
              universe_func=(my_universe))

def handle_data(context,data):  
    log.info("now handling data")  
    for stock in data:  
        log.info(data[stock])

13 responses

Your data starts at 1/1/2015 and your backtester is starting at 1/5/2015. Set the backtester's start date to 1/1/2015.

Jack,

Thank you for your reply. I may be wrong, but according to the docs if your csv contains a previous date and no current date is specified, then it will use the previous stocks as the universe until new ones are added to the csv.
Either way, changing the backtesting date unfortunately has no effect on the results.

Bryce - Jack is right, and the documentation isn't helping in this case. The previous stocks in the universe will indeed be used - but 1/5/2015 is the first day of the universe. There are no stocks to carry forward (yet).

Can you put up a version with a "real" CSV, even if it's not the real data? It's hard to troubleshoot Fetcher cases without the CSV.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Dan,

Thanks for the help. Here I've uploaded an example csv.

-Bryce

What's also complicating the matter is your data is starting on 01/01/2015 which is a day that the market is closed and is a day the backtester skips. This fails to initialize a universe (in minutely mode; however, it does get initiated in daily mode for some odd reason).

Try starting your data at 1/2/2015.

Hey! Thanks Jack! I really appreciate it. That seems to have fixed my problem!

-Bryce

Hey,

So I've started trying to live trading my algorithm, just trying a really simple version of this. But it errors out as soon as I start it! Can someone see maybe what's wrong with my code that causes it to do this?

I'm just trying to load in a CSV every day before trading starts formatted like so, then purchase all the stocks on the list at the target percentages given a few minutes after trading open, and sell them again at trading close.

It backtests without errors, so I know it's not a syntax issue. I'm just not really sure how to debug that.

symbol,start_date,rank  
WRE,02/23/2016,0.059  
WERN,02/23/2016,0.101  
NUAN,02/23/2016,0.091  
IIVI,02/23/2016,0.14  
G,02/23/2016,0.11  
FR,02/23/2016,0.068  
EZCH,02/23/2016,0.10  
COT,02/23/2016,0.099  
BRKR,02/23/2016,0.063  
BRC,02/23/2016,0.158  
import datetime  
import pandas as pd  
import numpy as np  
import talib  
import math

# my_universe returns a set of securities that define your universe.  
def my_universe(context, fetcher_data):  
    # fetcher_data is the data resulting from the CSV file from fetcher.  
    # set my_stocks to be every security in the fetcher_data  
    my_stocks = set(fetcher_data['sid'])  
    # log the size of the universe for debugging  
    context.count = len(my_stocks)  
    # print 'total universe size: {c}'.format(c=context.count)

    # return the securities we identified earlier  
    return my_stocks

# see a snapshot of your CSV for debugging  
def preview(df):  
    log.info(' %s ' % df.head())  
    return df

def initialize(context):  
    log.info("starting")  
    set_commission(commission.PerTrade(cost=0.00))  
    set_long_only()  
    context.money = context.portfolio.cash  
    # import the custom CSV data file  
    fetch_csv("http://somwhere.com/stocks/csv/stocklist.csv",  
              pre_func=preview,  
              post_func=preview,  
              date_column='start_date',  
              universe_func=(my_universe))  
    schedule_function(portfolio_buy,date_rules.every_day(), time_rules.market_open(hours=0,minutes=5))  
    schedule_function(portfolio_sell,date_rules.every_day(), time_rules.market_close(hours=0,minutes=5))  
def portfolio_sell(context, data):  
    for sec in data:  
        order_target_percent(sec, 0)  

def portfolio_buy(context, data):  
    for sec in data:  
        percent_rank = data[sec]['rank']  
        order_target_percent(sec, percent_rank)  
def handle_data(context,data):  
    print('handling data')  
    pass  

What is the error?

Thanks Simon,

Error:
There was a problem loading your live algorithm, please try again or contact feedback.

It might want the CSV to have today's date. I've never used fetch_csv for live trading universe selection though...

Hmm, I did think it could be that at first, but the PHP page that serves it is always set to use the current days date.

Strange, sorry I can't help more.

Hello,
it is happening the same to me. My algo works in backtesting without any problem, but it doesn't work in live trading in IB, because it doesn't load the CSV in live trading.
The csv is like this

date,symbol,,  
22/08/16,IVV,,  
22/08/16,VOO,,