New (experimental) feature: easier symbol lookup

Hello Thomas,

So, I tried:

def initialize(context):  
    context.stocks = [sid(8554),sid(33652),sid(19920)]  
def handle_data(context, data):  
    for s in ['SPY','BND','QQQ']:  
        order(symbol(s), 100)

I get the error "non-string argument given to symbol()". Shouldn't the code work?

Grant

Hi Grant,

Thanks for checking it out. The correct way would be to use it instead of the sid() calls; i.e.:

def initialize(context):  
    context.stocks = [symbol('SPY'), symbol('BND'), symbol('QQQ')]  
def handle_data(context, data):  
    for s in context.stocks  
        order(s, 100)

symbol() has the same limitations that sid() does -- you can't use it dynamically.

Disclaimer

That's kinda awkward, in my opinion. It seems like folks are gonna want to just paste a list of symbols into the editor.

For example:

context.stocks = security(SPY,BND,QQQ)

When the backtest is started, security would automatically generate the list you show above:

context.stocks = [symbol('SPY'), symbol('BND'), symbol('QQQ')]

Otherwise, users will have to write an offline script to convert from SPY,BND,QQQ to symbol('SPY'),symbol('BND'),symbol('QQQ'), and then paste into the editor.

For the Nasdaq 100, it would be:

context.Nasdaq_100 = security(ATVI, ADBE,   AKAM,   ALXN,   ALTR,   AMZN,   AMGN,   ADI,    AAPL,   AMAT,   ADSK,   ADP,    AVGO,   BIDU,   BBBY,   BIIB,   BRCM,   CHRW,   CA, CTRX,   CELG,   CERN,   CHTR,   CHKP,   CSCO,   CTXS,   CTSH,   CMCSA,  COST,   DTV,    DISCA,  DISH,   DLTR,   EBAY,   EQIX,   EXPE,   EXPD,   ESRX,   FFIV,   FB, FAST,   FISV,   GRMN,   GILD,   GOOG,   HSIC,   ILMN,   INTC,   INTU,   ISRG,   GMCR,   KLAC,   KRFT,   LBTYA,  LINTA,  LMCA,   LLTC,   MAR,    MAT,    MXIM,   MU, MSFT,   MDLZ,   MNST,   MYL,    NTAP,   NFLX,   NVDA,   NXPI,   ORLY,   PCAR,   PAYX,   PCLN,   QCOM,   REGN,   ROST,   SNDK,   SBAC,   STX,    SIAL,   SIRI,   SPLS,   SBUX,   SRCL,   SYMC,   TSLA,   TXN,    TSCO,   TRIP,   FOXA,   VRSK,   VRTX,   VIAB,   VIP,    VOD,    WDC,    WFM,    WYNN,   XLNX,   YHOO)

Grant

Grant, that's a neat idea. Could definitely build that on top of symbol()!

Disclaimer

Jason Swearingen

++ on grants proposed extension.

Thomas,

Integration with fetcher is another consideration. Ideally, users could also load a list (ideally, several lists) of symbols via fetcher.

Grant

Suminda Dharmasena

Can you add the exchange to symbol also. In small companies the same symbol refers to different companies in different exchanges.

Suminda,

Yes, that is planned. Currently Quantopian only supports NYSE so that what it defaults to.

Thomas

Disclaimer

Colin Alexander

In theory, you should be able to use python's list comprehension to accomplish this.

    tickers = ['SPY', 'BND', 'QQQ']  
    context.portfolio = list([symbol(t) for t in tickers])

The Quantopian implementation of symbol, however, will only accept a string literal (not a variable referencing a string). This would presumably be a problem when using fetcher to retrieve a list of tickers for a given portfolio.

Colin Alexander

oops, I forgot portfolio is a reserved word. I should have used something else like stocks.

    tickers = ['SPY', 'BND', 'QQQ']  
    context.stocks = list([symbol(t) for t in tickers])

Colin, this is not possible. We do the symbol-lookup before running the algorithm via static analysis.

Disclaimer

John Fawcett

Maybe symbol could accept a list of string literals?
symbol('bnd','qqq','spy') could return [security(bnd),security(qqq),security(spy)]

Disclaimer

Suminda Dharmasena

Also for sids perhaps you can have sids(xxx, yyy, zzz) which would return a list.

Also a bind_sec method which would add the securities so you can directly asses then as spy, aapl in the code. E.g. bind_sec(symbol('qqq', 'spy', 'aapl') + sids(111, 222)) would mean you access spy variable which will hold the security.

Pynance IO

I like where this is heading, I think it will be more intuitive for people getting used to the platform. I do see how it can be a pain in the backend so thanks for putting this together.

I would vote for a plural initialization of a list of securities though. e.g. symbol('MMM') and symbols('MMM', 'DD'), where symbols is [symbol(x) for x in symbols]

Hi Thomas,

I realize it's maybe out of scope, but could you write out the entire database of securities (excluding prices & volumes) to a file every night, and make it available (e.g. via ftp)? Then users could filter on the list, and upload the filtered list via fetcher (or some other upload tool). This would avoid the copy-and-paste of potentially mismatched symbols into the editor.

Grant

I don't have anything new to add to the meat of the discussion, but I just wanted to comment briefly on this:

   tickers = ['SPY', 'BND', 'QQQ']  
   context.stocks = list([symbol(t) for t in tickers])

Note that wrapping list() around a list comprehension is redundant and incurs an unnecessary (extremely slight) performance penalty. The list comprehension already returns a list; you don't need to convert it into another list. I.e., the second line above should just be this:

context.stocks = [symbol(t) for t in tickers]

(This is leaving aside for the moment that as other people have noted you can't do this right now in Quantopian because symbol(), like sid(), requires hard-coded strings that Quantopian can statically analysis to arrive at a list of SIDs.)

Disclaimer

While I'm all for us finding ways to make the API more user-friendly, I find it hard to believe that someone who is competent enough to do stock filtering / selection outside of Quantopian, and who is also competent enough to write a trading algorithm in Python within Quantopian, is not also competent enough to know how to do a string replace on a list of symbols to turn it into the format [symbol('A'), symbol('B'), symbol('C'), ...] to cut and paste directly into Quantopian. Adding syntactic sugar to make the code for specifying static list of symbols cleaner is a perfectly fine idea, but it's just that, syntactic sugar, which doesn't add any new actual functionality to the platform.

I'd rather see effort focused on, e.g., allowing the universe to be defined by the stocks referenced in a Fetcher file rather than hard-coded in the algorithm.

Disclaimer

David Edwards

I agree with Jonathan on his last point, I think that working on dynamic sid generation from fetcher files would be time well spent. I like the new addition, it is a step in the right direction but the workarounds required to make it happen are mostly to facilitate backtests. For live trading that happens in real time, the ticker is the ticker and should map 1 to 1 onto its current sid.

Dynamic universe generation would take the possibilities in Quantopian to new level. I would love to have stock selection programs run on another server and save its selection for the fetcher to grab in the morning.

Suminda Dharmasena

Why not give the facility to run stock selection in Quantopian itself.

Why not give the facility to run stock selection in Quantopian itself.

Stock selection tools would certainly be a powerful addition to Quantopian, and we know that, and it's on our queue of features we intend to add. The queue is long and has many important features on it, and we do not have infinite development capacity. We cannot simply wave our hands and presto! everything on it is finished; we have to prioritize.

Disclaimer

Hello Jonathan,

Narrowing the scope a bit (your point about prioritization is valid), in the discussion above I got the sense that Thomas' proposal/experiment might be more effective if users had access to the securities database, with the OHLCV data removed. Even if fetcher upload of securities had to be tabled, access to the database would allow users to manually review selections prior to a copy-and-paste operation. This would eliminate mismatches between symbol and the security referenced in a backtest. It would also be kinda handy to have direct access to the range of dates over which securities were trading.

Would it be possible to publish the securities database? Or is it restricted, just like the OHLCV data?

On your other point, I agree that many users will be more than capable of writing an offline script to generate the required string of up to 100 securities in the format security('XYZ'), but it is a matter of making it easy and transparent. For example, somebody new to the site might have a spare hour to give it a try, and they just want to copy-and-paste some symbols into the code editor from a web search (as I was able to do very quickly for the Nasdaq 100 example I posted). You want the newbie to be focused on getting the code up and running, not on how to find securities and get them specified properly in the code.

Grant

Grant,

I think we all understand the advantages of publishing a list of the securities we support. There are both legal questions -- are we allowed to redistribute the data? -- and prioritization questions -- is this the best use of our time, in terms of implementing features that will most benefit our users? -- to be resolved before we can do such a thing.

On the prioritization question, one factor to consider is whether there is already another way to accomplish much the same thing. And hey, look, there is.

Also, as I've noted before, it is not incredibly helpful for the same person to repeat the same feature requests over and over. It is certainly not helpful to do so less than a day apart, in the same thread. Just sayin'.

Disclaimer