Hi Pravin,

I gather that you don't want to delay the start of your backtest/trading for 60 days. Otherwise, you could just store them in context.

I'm curious...maybe you could do everything in pipeline? Couldn't you operate on a trailing window of data within pipeline on a rolling basis, to generate the factors versus time, and then do whatever else you need to do in pipeline, and spit out the result every day?

Thanks Grant. I guess I could do it as a custom factor within pipeline. Trouble is I don't know how to code a custom factor that combines all other factor values using some technique. The idea is that different stocks have different exposures to factors. Say Oil stocks might be more exposed to one factor compared to a tech stock. That is why, I want to change Thomas's ML algorithm to regress data only on related stocks and not on the entire universe. I guess I will have to play around with pipeline to get a hang of it. It is painfully slow to run and I don't like it.

There is a bit of a learning curve, and I'm no expert at all. However, I was advised that pretty much anything can be done within pipeline computationally, that could be done outside of it. For example, I was able to get CVXPY to run within pipeline.

See:

https://www.quantopian.com/posts/limit-pipeline-factor-to-specific-list-of-stocks
https://www.quantopian.com/posts/using-a-specific-list-of-securities-in-pipeline

Pravin,

CustomFactors are the right approach. You can pass pipeline terms as inputs to other factors as long as they are 'window safe'. A pipeline term is window_safe it is comparable across split- or -dividend adjustments. For more on window safety, see this notebook.

My recommendation would be to work on your pipeline in research. It's much easier to iterate on a pipeline in research, and you get more detailed error messages, so it's quicker to debug.

For speed, you will want to provide a mask to your factors that have any significant computation step. As a starting point, you could start with a StaticAssets filter to work with a hard-coded set of stocks to test your factors, and then move to a dynamic universe. I suggest this because you could start with a small test group so that your development cycle is faster. The mask is also very important for cutting down compute time in the CustomFactor. It won't speed up the step of reading data (the CustomFactor will always ask for data for all 8000+ securities), but it will restrict any computations after that to being computed over the subset of securities that pass the mask.

At the end of the day, it sounds like you should end up with a CustomFactor that takes your other factors as input, and performs some sort of reduction on the arrays of data. Just in case you haven't seen it, the CustomFactor lesson in the Pipeline Tutorial has some detail that might help you to work on the reduction step (e.g. argument types and shapes).

I hope this helps.

Thanks Jamie. That definitely helps.