Hello Ahz,

You can use update_universe() in before_trading_start() to add up to 500 symbols per day. Is this not working for you?

Grant

Hi Grant,

I'm using update_universe() in before_trading_start() with equities retrieved from the get_fundamentals(). When I run update_universe() again, the universe seems to get over written with what's in the list.

I think I've figured out the issue.

Calling update_universe() a second time with the symbols from my list was over writing the universe that was previously set. The solution was to add my list of equity objects to list of equity objects returned by get_fundamentals().

example:
newlist = np.concatenate((context.fundamental_df.columns.values, [s for s in context.stocklist]), axis=0)
update_universe(newlist)

Calling update_universe() a second time with the symbols from my list was over writing the universe that was previously set

My understanding of update_universe() is that it adds securities to the data object. You should not see dropping of securities from the data object.

Here is my test code.

I'm logging the length of the data object and the length of my list for every period.

The length of data is the same as the size my list after the first period. If you uncomment line 21 you will start to see the length of data increase.

import numpy as np

# Put any initialization logic here.  The context object will be passed to  
# the other methods in your algorithm.  
def initialize(context):  
    context.stocklist = []  
def before_trading_start(context, data):  
    context.fundamental_df = get_fundamentals(  
        query(fundamentals.valuation.market_cap)  
        .filter(fundamentals.asset_classification.morningstar_sector_code == 206)  
        .filter(fundamentals.valuation.market_cap / fundamentals.valuation.shares_outstanding >= 2.0)  
        .filter(fundamentals.valuation.market_cap / fundamentals.valuation.shares_outstanding <= 10.0)  
        .order_by(fundamentals.valuation.market_cap.desc())  
        .limit(10)  
    )  
    update_universe(context.fundamental_df.columns.values)  
    if(len(context.stocklist) > 0):  
        update_universe(context.stocklist)  
        #update_universe(np.concatenate((context.fundamental_df.columns.values, context.stocklist), axis=0))  
# Will be called on every trade event for the securities you specify.  
def handle_data(context, data):  
    for stock in data:  
        ma50 = data[stock].mavg(50)  
        ma200 = data[stock].mavg(200)  
        if not(stock in context.stocklist):  
            if ma50 < ma200 :  
                context.stocklist.append(stock)  
    log.info("Data Length: " + str(len(data)))  
    log.info("StockList Length: " + str(len(context.stocklist)))  

Have a look at this code, and see if it helps. If I understand correctly, you are simply wanting to trade your point-in-time context.stocklist. So, if a stock is in data, but not in context.stocklist, you need close out any position.

Grant, thanks for the code and quick responses. It definitely helped me connect some of the dots.

I want to use context.stocklist as a watchlist I have no positions for. It could be that in the next period the symbol(s) in the watchlist are not returned by the fundamental query and since I have no open positions for them, data will not contain those symbols. I'd like them added into data so I can continue "watching" them.

Once you add symbols to the data object, they should remain. As far as a watchlist, if you want it to persist you'll need to store it in context.

I'm sorry for the constant back and forth and appreciate your willingness to help, but I might not be getting something.

The following test shows the data is overwritten by update_universe().

This is the output to the console. Notice "VVUS" is in the first period but not in the subsequent period.

2015-01-05 PRINT [u'ATRS' u'CTIC' u'CUR' u'SNTA' u'VVUS']
2015-01-06 PRINT [u'ATRS' u'BPTH' u'CTIC' u'CUR' u'SNTA']
2015-01-07 PRINT [u'ATRS' u'BPTH' u'CTIC' u'CUR' u'SNTA']

If I hold a position in "VVUS", it will then remain in data in subsequent periods

2015-01-05 PRINT [u'ATRS' u'CTIC' u'CUR' u'SNTA' u'VVUS']
2015-01-06 PRINT [u'ATRS' u'BPTH' u'CTIC' u'CUR' u'SNTA' u'VVUS']
2015-01-07 PRINT [u'ATRS' u'BPTH' u'CTIC' u'CUR' u'SNTA' u'VVUS']

Test Setting and Code:

From 2015-01-04 to 2015-01-07 with $5,000 initial capital (daily data)

Hold no positions:

import numpy as np

# Put any initialization logic here.  The context object will be passed to  
# the other methods in your algorithm.  
def initialize(context):  
    pass

def before_trading_start(context, data):  
    fundamental_df = get_fundamentals(  
        query(fundamentals.valuation.market_cap)  
        .filter(fundamentals.asset_classification.morningstar_sector_code == 206)  
        .filter(fundamentals.valuation.market_cap / fundamentals.valuation.shares_outstanding >= 2.0)  
        .filter(fundamentals.valuation.market_cap / fundamentals.valuation.shares_outstanding <= 2.10)  
        .order_by(fundamentals.valuation.market_cap.desc())  
        .limit(5)  
    )  
    context.stocklist = list(fundamental_df.columns.values)  
    update_universe(context.stocklist)  
# Will be called on every trade event for the securities you specify.  
def handle_data(context, data):  
    print(np.sort([x.symbol for x in data]))  
    for s in data:  
        if s.symbol == 'VVUS':  
            pass  

Hold a position:

import numpy as np

# Put any initialization logic here.  The context object will be passed to  
# the other methods in your algorithm.  
def initialize(context):  
    pass

def before_trading_start(context, data):  
    fundamental_df = get_fundamentals(  
        query(fundamentals.valuation.market_cap)  
        .filter(fundamentals.asset_classification.morningstar_sector_code == 206)  
        .filter(fundamentals.valuation.market_cap / fundamentals.valuation.shares_outstanding >= 2.0)  
        .filter(fundamentals.valuation.market_cap / fundamentals.valuation.shares_outstanding <= 2.10)  
        .order_by(fundamentals.valuation.market_cap.desc())  
        .limit(5)  
    )  
    context.stocklist = list(fundamental_df.columns.values)  
    update_universe(context.stocklist)  
# Will be called on every trade event for the securities you specify.  
def handle_data(context, data):  
    print(np.sort([x.symbol for x in data]))  
    for s in data:  
        if s.symbol == 'VVUS':  
            order(s, 1)  

I'll have to get back to this later. Maybe you can get a hand from Q support tomorrow. Hope you sort it out. --Grant

Ahz/Grant you're on the right track. Every morning before market opens, the method before_trading_start() is called. It runs the query within get_fundamentals() to calculate the current list of securities that meet your criteria. The function update_universe is then used to update your security universe to trade the current list.

Securities will be added/removed from update_universe() as they newly meet or no longer adhere to the query requirements.

If you want to track the list of securities that your algo ever traded, you'll need to create a custom list and append the symbols.

Alisa,

I don't follow. Say I hold a position in a stock, and then it gets removed from the data object. How would I sell it?

Grant

This backtest shows that if I order a stock, it remains in data.

However, if I don't place any orders, the length of data does not grow. Kinda confusing. I would think that update_universe() would add stocks to data regardless of whether they are ordered or not. Or am I missing something?

@Grant, I can see in my bactests that data objects always holds stocks set via update_universe() PLUS stocks from context.positions if those are not part of the universe you have set.

@Grant - @Luca is exactly right: data includes assets from your most recent call to update_universe as well as your portfolio's positions. Previous calls to update_universe are not considered, but it may look like they are if those assets are now in your portfolio.

@ Rich,

Thanks, but management of data is still confusing. Presumably, if I have a sid called out explicitly anywhere in my code (e.g. sid(8554)), then it will always be in data regardless of calls to update_universe. Or does update_universe do a complete replacement of sids (except that held positions are not dropped)?

Also, if I hold a position in a security, then update_universe won't drop it from data. But what if I held a position, but then closed it out? Could it then be dropped by a call to update_universe? Or would the security be considered part of my portfolio, but with no present position?

What if update_universe were called twice in before_trading_start? Would the second call override the first call?

Also, if I'm not mistaken, you changed update_universe so that it throws an exception if a security does not exist in your database. My prior experience was that it would simply ignore the security. Was there a change?

Does update_universe take into account the start and end dates of the securities, and the current time of the backtest? Or will it add to data regardless?

Grant

Hi Grant, Great questions -

That's right - an explicit mention will keep an asset in data, no matter your calls to update_universe or your portfolio. And yet another vector for inclusion in data is set_universe.

If you close out a position, then it won't appear in data anymore (at least not because of the position). An open order will keep it in data though, even if unfilled.

Only the most recent call to update_universe will affect data, so yes, the second (or later) call in before_trading_start is the only one that matters.

I don't know of any recent changes to the validation in update_universe, but maybe there's some new interaction I'm not aware of - do you have an algo you could share?

Regarding the start and end dates of assets, I'm not sure. If your algo hasn't reached an asset's start date (or has passed its end date) when you pass it to update_universe, then I don't think you'll see it in data until the algo then reaches the asset's start date. I could be mistaken though.

Thanks Rich,

The code below used to run on daily bars, 3/6/2014 - 3/6/2015. Now I get an error such as:

SidNotFound: Asset with sid '1830' was not found.
There was a runtime error on line 10.

So, I'm wondering if something changed either in your database or the way update_universe works.

My guess is that in order to see an added security in data, not only does the asset need to have reached its start date, but also its first trade for that day. Correct?

import numpy as np

def initialize(context):  
    pass

def before_trading_start(context):  
    sid_list = np.random.randint(1,20000,200)  
    update_universe([sid(s) for s in sid_list])

def handle_data(context, data):  
    num_stocks = len(data)  
    record(num_stocks = num_stocks)  
    for stock in data:  
        order_target_percent(stock,1.0/num_stocks)  

Grant - From my testing, it looks like a regression in the behavior of the sid function that's also called on line 10. Will keep looking and hopefully get a fix through soon...

Hi Rich,

Any luck with this? If the fix is in, I can re-test.

Grant

Hi Grant,

After investigating further, I've realized that we didn't intend to support the ability to use sid() that way. It used to silently return a "dummy Equity" with that ID, but that has implications in downstream code which are hard to manage. In short, though it worked, we didn't intend it to.

Perhaps there is another way to do what you're looking for. Can you share more about why you're looking for a stock at random?

We have a separate project in the works that, I believe, will solve most use cases for random by letting you look up pricing data for any stock without having to specify a universe first.

-Rich

Thanks Rich,

No problem. I don't actually use the code above for anything other than some testing I did awhile back. It was odd that the code worked, but more recently it failed. No matter, though. If it isn't the intended use of sid(), then so be it.

Grant

Hi Rich,

in the project in the work you are mentioning, will it be then possible to dynamically include stocks in your universe during trading hours?
Will the concept of "universe" be removed, letting users to look up/trade any stock?
Is volume included in the pricing data too?

Hi Nicola,

That's a use case we're certainly thinking about, but unfortunately I don't have the design details for you yet.

Thanks,
Rich