Grant, I'm not sure how the backtest you posted illustrates your point.

I just tested it, and as far as I can tell, before_trading_start() is, in fact, called on the first day of the backtest.

I did discover during my testing that there appears to be an issue where if a backtest generates fewer than three log messages, the log messages don't show up in the UI. Perhaps that's what was causing you to believe before_trading_start() wasn't being called on the first day. I've filed an internal defect report about this issue. Hopefully we'll be able to figure out what's going on and fix it soon.

Thanks Jonathan,

When I run the code above, the output is:

2015-10-08 PRINT AAPL
2015-10-08 PRINT QQQ
2015-10-08 PRINT SPY
2015-10-09 PRINT AAPL
2015-10-09 PRINT QQQ
2015-10-09 PRINT SPY

Shouldn't it start printing on 2015-10-07, when the backtest starts?

This code shows that before_trading_start does get called the first day of a backtest. However, I'm still confused, since it appears that data is empty until the second day, since this loop has no output the first day:

for stock in data:  
        print stock.symbol  

Shouldn't the universe get updated upon the first call to update_universe and since I have context.aapl = sid(24) in initialize, even if update_universe doesn't take effect immediately, shouldn't AAPL show up in data?

Backtest log output:

2015-10-07 PRINT 2015-10-07 00:00:00+00:00
2015-10-08 PRINT 2015-10-08 00:00:00+00:00
2015-10-08 PRINT AAPL
2015-10-08 PRINT QQQ
2015-10-08 PRINT SPY
2015-10-09 PRINT 2015-10-09 00:00:00+00:00
2015-10-09 PRINT AAPL
2015-10-09 PRINT QQQ
2015-10-09 PRINT SPY

Hey Grant,
The issue is that on the first day of the backtest, before_trading_start doesn't know what your universe is, so it doesn't know what SIDs to get data for. Assuming you update your universe on that first day, by the second day it has data. This is something we will be fixing, for now it's just something you have to work around if you want to use data in BTS.

Thanks Karen,

I don't understand how you'll fix this behavior. If I understand correctly, a stock doesn't actually get added to data until the first trade is encountered for the day. So, if there is at least one trade over the first minute of the market, a stock will show up immediately in data. However, there can be a delay for stocks that are thinly traded. Do I understand this behavior correctly?

This goes back to a dated post, https://www.quantopian.com/posts/thinly-traded-stocks-why-no-gaps, which discusses how you forward fill. The problem I see is that you can forward fill from the prior trade day to have data available in before_trading_start. However, if the call to before_trading_start is on the start_date for a given security, then what will you do? You can't backward fill, since it would introduce a look-ahead bias, right? Or would you backward fill with NaNs?

This would seem to be relevant to pipeline, since I think where you want to land is that a coarse filter could be applied using pipeline using daily data, and then it could be refined using minutely data, by calling history, after the call to update_universe. If my understanding is correct, you'd have a problem, since within before_trading_start, you'd have one set of stocks, and as the trading day went on, you'd have another. I suppose this is no show-stopper, just a pitfall, but users just have to know to always check if a stock in in data before using it as a key.

Under live trading, how do you handle all of this? I'm guessing that you just get a stream of trades, from which you create minute bars, and as soon as you can create a bar, you add the security to data. It is the same behavior as for the backtester. You have no knowledge of whether or not a given stock is tradeable on a given day.

On a separate but related topic, are you planning to support order submission from within before_trading_start? It would be a natural progression. Filter stocks using pipeline. Refine the selection using history. Place orders to be filled when the market opens at 9:30 am.

I have run into difficulties trying to place orders first thing in the morning for illiquid stocks. Checking if a stock is in data before placing an order for it is one of the most frustrating things in Quantopian, since there is absolutely no reason one shouldn't be able to place an order for something that hasn't yet traded today.

Simon,

I think the problem may be that Q just gets trades from Nanex, but I could be wrong. If it is only trades, then the injestor spits out only bars. Until the first bar is received for a given stock, there's no way of knowing that the stock is even listed for trading. At least that's my working hypothesis at this point. But maybe Q knows ahead of time which stocks could possibly trade on a given day? Murky at this point.

It does seem whacky. Why not just fill the bars with NaNs and put the order on ice (backtesting) or send it on its merry way to IB (live trading). One implication, I think, is that using Quantopian, one could never be the first to buy a stock when it is first issued.

Do you see the problem only when you try to place orders on the start_date for the stock? Or is the problem unrelated to the start_date?

Grant

I feel like I have to have that check in all the time, or I'll eventually get an exception. Sometimes I move my schedule_function to 10:30am to minimize it, but it's still a problem that shouldn't exist...

I think you hit the nail on the head. The ability to submit and manage orders should not depend on whether or not the data is available. It should only depend on the trading status of the security. If it is open for trading, then Q should support ordering it.