Anyone has the same issue?

Because pipeline can only be initialized in initialize(), at the time there is nothing in the portfolio so you cannot specifically include any of your portfolio stocks in pipeline over time.

My problem is say on the first day, pipeline screened a list of stocks among which I bought some; then next day pipeline screened another list of stocks that do not include the stocks I bought the day before so I have no updated data for my portfolio stocks.

My workaround is to create another function to get the data I need for the portfolio stocks and merge with the pipeline data.

Anyone has a better or easier solution?
Thanks.

Hi Wei,

As you mentioned, defining a pipeline is done upon initialization of an algorithm, so you can't include position information in it. That being said, I think you can get what you're asking for. Instead of specifying a particular filter to be the screenon your pipeline, you can instead add that filter as another column on the pipeline, and select stocks to trade based on that column in before_trading_start. That way, you still get data for all stocks in before_trading_start, and you still have a way to pick the new stocks that you want to trade.

So instead of having something like this:

my_pipe = Pipeline(  
    columns={  
        'my_column': my_column,  
    },  
    screen=my_final_filter  
)

Add your screen as a column instead:

my_pipe = Pipeline(  
    columns={  
        'my_column': my_column,  
        'my_universe': my_final_filter  
    },  
)

And then in before_trading_start, do something like:

def before_trading_start(context, data):  
    context.pipeline_data = pipeline_output('my_pipeline')  
    context.my_universe = context.pipeline_data[context.pipeline_data['my_universe']]

    # This will have my_column results for all stocks since we removed the screen.  
    context.column_data = context.pipeline_data['my_column']  

Does this help?

Thanks much Jamie for your idea and sample code.

I thought about this but I was concerned about the pipeline time.
I normally screen ~100 stocks out of at least 3000 and the list of ~100 stocks could change everyday.
So if I bring data for 3000 stocks vs. 100 through pipeline, how does this affect the time spend on pipeline?
I haven't tested it but will there be a big running time jump? Time is also a concern to me.

Thanks.
Wei

The final screen on pipeline doesn't affect computation time. It just determines which rows to output. All of the computations are done before that. If you have computationally expensive factors (like a regression term), then you can provide a mask to that term to help save time, but a screen won't help. So you should be ok to remove it!

That's good to know.
I tested it and it works that way.
Very helpful. Thanks again.