Hi Lucas,

This code line won't work in the IDE: order_target_percent(symbol(get_current_symbol_str()), 1.0)

The reason is because symbol() needs to take a string literal. It can't be changed dynamically within the algorithm. When the algorithm compiles, we run a static analysis of the code to retrieve the data needed for the strategy for faster processing. So you will need to format the code in a manner like this:

order_target_percent(symbol('AAPL'), 1.0)

Cheers,
Alsa

Is there any plan in removing this limitation?
If we want to trade based on a market scanner it might be hard right?

Say take top 5 movers from last trading day and do some mean reversion algorithm on them?

Is the static analysis mainly checking if we have less then 200 symbols in our universe?

Thanks,
Lucas

Hello Alisa,

Looking over the documentation, it appears that the absolute limit is 200 securities per algo (see https://www.quantopian.com/help#ide-sid-lookup, "Each method accepts up to 200 securities."). If I understand correctly, it is possible to add securities during algo execution but not to drop them, up to a limit of 200? Is it feasible that one could periodically provide a list of securities via fetcher, to be added to the algo's universe? Then, once the limit of 200 securities is reached, the algo could be stopped, to flush out the universe? This would allow offline screening of a much larger pool of securities.

Also, is there fundamentally any reason you couldn't allow dynamic add/drop of securities available to an algo? For a given security, if there are no open positions, and no open orders, why couldn't it be dropped? Is it a book-keeping issue?

Grant

I think it is probably a performance Issue, since get_history will get history of all symbols even the non-active ones. 200 symbols might be pretty heavy already. but the Idea from Grant would solve it. Or probably move the check from static to dynamic, so we can set an "Active" flag in the symbols and if the number of actives exceed 200 we get a run time exception.

Quantopian does some magic in the background any time symbols are entered. For example, this compiles:

def initialize(context):  
    symbol('A')  
    symbols('AA', 'AAPL')

def handle_data(context, data):  
    pass  

Notice the securities are not assigned to any variable. Yet they will all be available in data (for any bar/frame where they happen to be trading).

I have settled into this model:

def initialize(context):  
    symbols('AA', 'AAPL')

def handle_data(context, data):  
    for sec in data:  
        sym = sec.symbol  
        # keep track of sym, like 'AAPL', in own dictionary, things like indicator compilations, prv vs now etc  
        # order based on sec, the security object  

I don't like what comes next here, just for illustration to assist in understanding, one could do ...

def initialize(context):  
    context.str_to_symbols = {  
        'MUX'  : symbol('MUX'),  
        'NFLX' : symbol('NFLX'),  
    }

def handle_data(context, data):  
    # If MUX looks buyable or sellable at the moment  
    order_target_percent(context.str_to_symbols['MUX'], .5)  

I did that but after 40 securities I started to lose motivation to create a table with 200 symbols. Maybe I managed 60.

Quantopian's magic makes it easier:

def initialize(context):  
    symbols('AA', 'AAPL')

def handle_data(context, data):  
    for sec in data:  
        sym = sec.symbol  
        keep track of sym, like 'AAPL', in own dictionary, things like indicator compilations, prv vs now etc  
        order based on sec, the security object  

More than 200 can be held and processed. This processes a high of 306 securities in the universe at one point (during just one year). The last line of logging output shown here includes the number contained in data (241) compared to the total now (276), and same for those in positions. I noticed over 1400 securities processed overall in one long-running algo by someone (although I wasn't keeping track of the max during any single bar/frame at that time).

2015-03-02 handle_data:55 INFO max 306 data 241/276 pos's 149/276 nonzero_pos's 86  

I haven't tested this so I might be missing something but I think you could do a fundamentals query to get the security object associated with the symbol. e.g.

sym_list = ['MUX', 'NFLX']
df = get_fundamentals( query(fundamentals.company_reference.primary_symbol).filter(fundamentals.company_reference.primary_symbol.in_(sym_list)) )

Or maybe: fundamentals.share_class_reference.symbol.in_

The key of each data frame entry is the security obj. e.g.

for security_obj,q_series in df.iteritems():

Then later on you could place an order for the security obj:

order_value(security_obj, money)

Hi Gary,

I haven't kept up with the get_fundamentals functionality. Is it possible to have more than 200 sids in data with it? Does update_universe just keep loading them in, without limit?

Grant

Sure looks like this algo takes a position in more than 200 securities...

Recommended for anyone curious, clone and make some changes, often be able to add even more insights.

Here's another example. In the end, over 2000 securities in the portfolio, but it appears that only 200 can be added per day. --Grant

For reference, one can remove and then re-add securities to the data object:

    # Test to see if we can remove stocks from data  
    removeThese = []  
    if (context.Counter < 200 or context.Counter > 250):  
        for stock in data:  
            if (stock.symbol == "AA"):  
                removeThese.append(stock)  
        for stock in removeThese:  
            del data[stock]  
    else:  
        if ('AA' not in [stock.symbol for stock in data]):  
            newSid = symbol("AA")  
            newSIDData = zipline.protocol.SIDData(newSid)  
            data[newSid] = newSIDData

Seems like removing stocks from the universe could be a problem. For example, what if there is a position in the stock? Would you ever be able to sell it once it is removed from data?

Also, I'm wondering how the 'history' API deals with deletions? In the next bar, will it update the trailing window, dropping the security? Or will there be confusion?

It would seem to be a good practice to have only those securities required at any given time in the universe, but it would be good to hear from the Quantopian staff experts before applying the add/drop code above. From the standpoint of memory management and efficiency, I can see a use case. What if an algo runs for 10 years without stopping and accumulates so many securities that it implodes?

I'm confused how update_universe() works. I get an error when I try:

update_universe(symbols('SPY'))  

Also, it seems that update_universe requires a list data structure, since an error results if I do:

update_universe(sid(8554))  

Shouldn't update_universe be smart enough to handle a single value, without having to put it in a single-element list?

The attached backtest highlights the confusing errors I'm getting.

Grant

A universe of one!? Maybe that's what they thought. And I wouldn't blame them. A universe of many is the common use of the API. But, a little work under the covers would accommodate your wishes: Python method overloading

Per the removal/addition of SIDData items to data, it was only for reference and discovery. My personal feelings on maintaining two separate lists of securities that are traded are that if I can use just the master (data[]), then I don't have to worry about inconsistencies. Alas, Dan shot down the whole dynamic property on data[stock] which allowed a single source of security management. But the context.S[stock].DynamicProperty was a workable fix. Still, that mechanism uses data[] as the master. Therefore, adding / removing securities from data might be useful if not necessary down the road.

The problem still exists that this thread was dedicated to: there is no "newSID = createSidFromVarSymbol(ticker)" which I believe they will have to change. Forcing symbol("abc") or symbols("abc","dev") in order to preoptimize the data retrieval is bunk now with the ability to add thousands of symbols through the fundamental API.

We've proven that SIDs added through update_universe immediately have their data made available. I'll test the technique with the add/remove SID from data[], but my bet is that it works just fine there too.

Additional (unexpected) behavior discovery.

Although del data[stock] "appears" to work in the current state of a single bar event's processing, on the subsequent bar, the deleted SID is back within data. So, one can NOT remove a security from data. One can try, and succeed for the duration of your event. But on the next event it will be back. This code prints "Removed: AA" over and over. We never actually delete the AA SID.

#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  
def ReconcileData(context, data):  
    # Test to see if we can remove stocks from data  
    removeThese = []  
    for stock in data:  
        if (stock.symbol == "AA"):  
            removeThese.append(stock)  
    for stock in removeThese:  
        del data[stock]  
        print("Removed: {0}".format(stock.symbol))  

Yeah, I haven't sorted out a way to use update_universe() in combination with a list of symbols to add securities dynamically (although it appears that if one, ahead of time, has found the corresponding sid id's, then dynamic adding of securities is possible, but no dropping, as you have discovered).

With the 200 new securities per day limit, I think that Q is just hoping that's a strict enough limit so users don't do something that would cause handle_data to time-out, which will cause the the algo to crash (and as far as I know, there is no sure-fire way to catch the error). Also, of course, there could be additional slippage within the trading minute, if the algo gets bogged down, but doesn't crash.