Hi Karen,

Since the theme here is speed, you run pipeline every day, when it is only needed the day of rebalancing. I attached an example of how to run pipeline only when necessary.

It is a little clunky, since I need to schedule a function one day before rebalancing, to set a context flag, which is then used to trigger the run of pipeline the day of the scheduled rebalancing. The code would be cleaner if before_trading_start could be called as a scheduled function. Would it be possible to change the API so that if before_trading_start is scheduled, then it would run per the schedule, but otherwise (if not scheduled), it would run every day?

@Grant: Unfortunately, there's currently no way to prevent pipeline from running on each day of a simulation. Even if you don't call pipeline_output, it still gets run every day. This is something that's on our list to change, but for now, the condition in your before_trading_start function won't prevent pipeline from running.

O.K. Sorta looked like this line was a call to pipeline:

context.output = pipeline_output('mean_reversion_example')  

But I guess it is just grabbing data that is there already?

Is there a reason pipeline runs every day? Is it accumulating data, or something? Mysterious. Pointless compute cycles, it would seem, for algos that don't trade every day. Or is there some overall optimization in having it run every day by design?

Hi Karen/Jamie,

Rather than pulling data for 8000+ securities from the database, say I wanted to limit it to a subset? For example, can I provide a list of stocks to be analyzed? Or limit to Nasdaq stocks only? It sounds like this masking business happens after the data are loaded into working memory, and only applies to the computation?

The help states "All other stocks for which the Filter returned False will be filled with missing values." So, it sounds like the stocks are still returned, but the data are presumably None (not sure what is meant by "missing values")? So is it then necessary to add code to drop stocks whose data are all None? In the code you posted above, this must be happening, but I don't see it explicitly. Are you dropping the stocks that don't pass the mask filter implicitly somewhere in the code?

I see that you are using if data.can_trade(stock):. Is this necessary? It would seem to run counter to the guidance on https://www.quantopian.com/posts/when-to-use-data-dot-can-trade, where it is stated "You don't need to use data.can_trade when you are working with today's output of pipeline or get_fundamentals". In practice, is it still required (e.g. to guard against data errors)?

Grant, I have been running timing tests to see how long it takes the pipeline to run in different scenarios. What I found is that the pipeline is only really doing something that takes time about twice per year of backtest. If I'm doing something with a bunch fundamental data, I'll see it pause for up to 80 or 90 seconds once in January, and once in July of each year (assuming my start date is in January). The rest of the time it may be doing something, but it isn't anything time consuming.

I would love to limit pipeline data by exchange. I see that as one of the biggest missing features of pipelines.

@Karen, do you mean that pipeline.set_screen() method doesn't exclude factor calculation for filtered out securities? If not, why not fixing that instead of this solution?