What is the date convention here? Is it dd/mm/yy? Is it possible to specify how the date string is passed?

This looks awesome! I'm trying to do the same for stocks that hit a 52 week low on the NYSE, AMEX and NASDAQ.

This sort of data http://www.quandl.com/URC/AMEX_52W_LO-AMEX-Number-of-Stocks-Making-52-Week-Lows

@jessica How would I set that universe?

Hi Simon,

See an example algo attached that pulls this quandl time series in and plots it. I'm pretty sure this isn't exactly what you're going for though - that data series is simple a time-series of the number of stocks making 52 week lows, rather than a listing of which stocks are at 52 week lows each day.

Do you have another data source with the ticker, date pairs of the stocks you want in the universe?

Is it possible to add this to the "Help & API Docs" as this thread is not longer on the front page.

If there are any other things that you have rolled out and has not being put into the doces is it possible to update this so that the docs section would be a on stop place to learn about ALL of Quantopian functionality.

Is it possible to explain what this piece of code does in the above example.

symbols = [s.symbol for s in sids]  

Also will the following have the same effect.
context.count = len(sids)

Also is it possible to explain set a bit more. As I understand this takes pandas series (will other data structure be legal?) either as symbols or sids and return a list of security objects.

Also what is are the date formats acceptable in the data column by default. Can different date for formats be used as in the Fecher Docs?

Hi Suminda,

The documentation is now available in the Fetcher Section under "Using Fetcher to create a custom universe" and includes another example. The help docs are now up-to-date with our new features!

To answer your other question, this line of code

symbols = [s.symbol for s in sids]  

Is saying iterate through the list of sids and convert it to a list of symbols. So for example, [Security(24), Security(300)..] will become ["AAPL", "AA", ...] using Python list comprehension.

The next line,
context.count = len(symbols) is counting the number of symbols in the list and printing the value. This helps identify how many stocks are in the universe during each bar of the backtest.

You can specify the date format of your data using the date_format parameter in fetch_csv.

Best,
Alisa

Hi,

Is there a way to disable the forward filling while using Fetcher to create a custom universe?
Right now, universe_func will forward fill if no new data occurs on a new day. Is it possible to disable this behavior and instead have no universe / not trade the new day if there is no data on that day. I am writing an algorithm to daytrade a list of stocks that I import from Fetcher, and right now if there is no stocks on a trading day, it will trade the same stocks from the previous day for two consecutive days, which is what I do not want.

Thanks

Hey TingTing,
There are a couple ways. First, you can add rows to the CSV you are importing with the missing dates. With those missing dates, add a stock you don't care about then set up a line to ignore that stock when you are going through the data variable containing all the stocks for that day.

You can also use your existing CSV and add a pre_func to the fetch_csv function that fills in the missing dates with a stock you don't care about. Then similar to the first method, you ignore that stock in handle_data. Here's an example of that. I used the example from "Using Fetcher to create a custom universe" here as a starting point. You can see in the code there is a pre_func where I fill in the missing dates with AAPL, then ignore AAPL in handle_data. You will probably have to slightly adjust the stop_forward_filling function I made for your own use.

Does this work for you?

Jessica, it'd be great if you could share the quandl search criteria or code with us. I can't seem to figure out how to get data for the stocks and not the S&P index.

Hi Thomas,

I grabbed the SP500 index constituent from this page: http://www.quandl.com/resources/useful-lists

Scroll down to the Stock Index Constituents header - they have a static snapshot as of 10/10/2013

Hope this is useful!
Best wishes,
Jess

Hi Jessica, I was trying to use your example of defining a custom universe using fetcher but I seem to be stuck out of the gate.

Looks like fetcher is correctly loading my symbols file from dropbox since the preview function returns the right results. However, it never seems to call the my_universe function nor the handle_data function. Any ideas? Thanks!

Hi Claus,

Thanks for posting!

What's happening is that fetcher looks for the 'date' column in your CSV file and fills your universe with the stocks in your CSV starting from that date. So in this case, because your CSV contained the date '1/1/02' and the backtester date was set to start sometime at 2013, it never had the chance to hit '1/1/02' so your universe was never filled with stocks. In addition, the earliest trading date we have is '1/3/02' so you'd have to change '1/1/02' to '1/3/02' and either start your backtest on that date or sometime before.

I've done that here:

import datetime  
import pandas as pd  
import numpy as np

def preview(df):  
    log.info(' \n %s ' % df.head(n=10))  
    df['date'] = '1/3/02'  
    return df

# Function for returning a set of SIDs from fetcher_data  
def my_universe(context, fetcher_data):  
    log.debug("in my_universe")  
    # Grab just the SIDs for the one sector:  
    #market = fetcher_data[fetcher_data['market_group'] == 'Equities']  
    #sids = set(market['sid'])  
    sids = set(fetcher_data['sid'])


    symbols = [s.symbol for s in sids]  
    context.count = len(symbols)  
    print "total universe size: {c}".format(c=context.count)  
    log.info("total universe size: {c}".format(c=context.count))  
    context.target_weight = 1.0/context.count  
    return sids

def initialize(context):  
    fetch_csv(  
       "https://dl.dropboxusercontent.com/u/12406727/symbols.csv",  
        pre_func=preview,  
        date_column='date',  
        date_format = '%m/%d/%y',  
        universe_func=my_universe)

    context.target_weight = 0.01  
    log.debug("done with initialize")  
def handle_data(context,data):  
    log.debug("we're in handle_data")  
    for stock in data:  
        print stock

        if 'price' in data[stock]:  
            order_target_percent(stock,context.target_weight)  
            print stock  
        else:  
            log.warn("No price for {s}".format(s=stock))  

I just changed your 'date' column to be '1/3/02' instead of '1/1/02' and set my backtester date to start at 1/3/02. Try it out and let us know how things work out!

Seong

Hi Seong, thanks for quick reply. Looks like I got this to work, although not until I ran a full backtest (vs just building the script).

Thanks again!

Claus

Could @TingTIng solve his problem also by comparing data[stock]['Date'] to the current (simulation) date and process only when that's true?

Hi, I tried to run this and seems like it's looping through all the stocks in the fetcher csv, not just the "Financials". How can I make sure the def my_universe function runs properly?
When I changed the handle_data to "print stock", I got a list of all the stocks.

Thanks,
Nikki

bump, im seeing the same behavior