Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Setting Universe to S&P 500 stocks using Fetcher

Hello,

I am trying to get the list of all S&P 500 stocks using Fetcher. There are two problems:
1) The algorithm only works in daily mode. When I try to use it in minute mode, I get an error, but there isn't any description of it: "Something went wrong. Sorry for the inconvenience. Try using the built-in debugger to analyze your code. If you would like help, send us an email." I need to use the algorithm in minute mode when I actually do transactions.

2) When I use the algorithm in daily mode, it works, but on the logs, it says: "1970-01-01 null:null WARN Dropped 18 rows from fetched csv." Also, it only prints the total universe as being 455, even though my csv file contains 500 symbols.


import pandas as pd  
import numpy as np

def initialize(context):  
    fetch_csv("https://dl.dropboxusercontent.com/u/77898006/constituents.csv", pre_func = preview, post_func = preview, universe_func = my_universe)  
def preview(df):  
    log.info(' \n %s ' % df.head())  
    return df

def my_universe(context, fetcher_data):  
    sids = set(fetcher_data['sid'])  
    context.count = len(sids)  
    print 'total universe size: {c}'.format(c=context.count)  
    return sids

def handle_data(context, data):  
    log.info("Executed handle_data")  

Any help in resolving this issue would be greatly appreciated.

8 responses

The problem is the data in your CSV is not entirely correct. There are 18 securities that are not resolving
in the Quantopian universe.

For example, if you type

 context.viab = sid(  

after the open parenthese, start to type the text "VIACOM" and you will see securities start to appear.

Here are 2 examples of missing securities: VIA-B (should be VIAB), and BRK-B (should be (BRK_B)

Take a look at the attached backtest.

Note that the securities in the Quantopian universe may not be exact. I have found problems with
securities such as Comcast (CMCSA) which is recognized as CMCS_A

context.cmcsa = sid(1637) # CMCS_A (Comcast)

Click here to take a look at the SP500 securities

Why people should look for third party list ?

There should be few such popular lists maintained by Quantopian itself.

Yes, that's what I thought too. One should be able to, say:

set_universe(sp_500)  
set_universe(dj)  

etc, for major indices.

I agree with Yagnesh and Rai. Why can't Quantopian provide the popular indices like sp500 and nasdaq 100?

Hi Chaoyu, we definitely agree it would be nice to have those available on the site. Unfortunately, they're quite expensive to license them from their proprietors. We've created the Q500 and Q1500 which might be helpful for your purposes: https://www.quantopian.com/posts/the-q500us-and-q1500us

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.