Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Pipeline in Research - Build, Test and Visualize your Factors & Filters
Karen Rubin
edited
Q Updates Research Pipeline

Today, just in time for the holiday, we have launched pipeline into the research platform. This will allow you to build, test and plot your factors and filters.

Attached is a research notebook which shows how to construct a pipeline and view the output. It also includes a new method, show_graph, which allows you to graph how the pipeline is being calculated.

The example included is the same ranked universe, long-short equity strategy that we have used to explain the pipeline functionality in the forums.

(Please excuse the bug which messes up the display of the hierarchical dataframe that is the pipeline output. This only happens in the forums, if you clone it, you will see the correct view of the dataframe in research.)

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

17 responses
Anthony FJ Garner

Stunning Karen, absolutely stunning. Having spent a week groping in the dark coding Kaufman's Efficiency Ratio in the back tester I can immediately appreciate what a vast improvement this facility represents.

Giuseppe Moriconi

Thank you very much for this new feature, i have a queston:
when you say 'run_pipeline will produce the output of your pipeline': since this is a multindex dataframe if i want to work on it exactly in the way i work in the algorithm i have to specifically refer to 'today'.
So if this is the code 

p = run_pipeline(pipe, start_date='2015-11-01', end_date='2015-11-25')

the output i have in the algorithm is p BUT indexed for that specific day.. for example
p.ix['2015-11-25']

i am asking this because i am not very famliar with multiindex dataframes and - for example - in your code if i want to select the 100 biggest stocks for liquidiy i cannot just make
p.sort('liquidity').iloc[:100] as i would do in a normal algo, but here i have to make some sort of loop to run trough al the dates from start to end, am i right?
Thank you

Karen Rubin

Anthony,
Glad you like it. I agree that it will make building factors and understanding the output much simpler!

Giuseppe,
You want to index into the dataframe using .xs. (Here is the Pandas reference.) To get an individual day, your code looks like,

output.xs('2015-11-09').sort('liquidity')

Which returns the pipeline for just that date sorted by liquidity.

Hope this helps.

Giuseppe Moriconi

yes! thank you Karen

James Jack

impressive -- I like the automatic graph output!

Costantino

Great new feature!

Giuseppe Moriconi

i have a problem (see attachment) when i use pipeline to retrieve sector codes

And if i change the sector query like this:
inputs = [ morningstar.sector_code ]
i have another error: when i istantiate: 

sector = Sector()  
AttributeError: type object 'sector_code' has no attribute 'atomic'
Scott Sanderson

@Giuseppe

The errors you noticed in your post are a bug introduced by an update that was made last night. We're working on a fix. Thanks for your patience as we work out the kinks in the new system.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Michael D

Could you clarify what this part is doing?

weeklyreturn.eq(weeklyreturn)

Previously you had:
# Set a screen to ensure that only the top 2000 companies by market cap # with a momentum factor greater than 0 are returned 

    pipe.set_screen(top_2000 & (momentum>0))
Karen Rubin

That is one approach to removing NaNs. As this example has evolved, I realized I want the companies with negative momentum still to be included in my pipeline. Including (momentum > 0) is removing too much. Because NaN is never equal to itself, including (momentum.eq(momentum)) effectively removes all the NaN values, while not removing companies with negative momentum.

Julien D

Karen, I am wondering what would be the best way to add forward returns to the pipeline dataframe in order to do analysis similar to the example in the Notebook for "Quantopian Lecture Series: Ranking Universes by Factors."

Ravi Aggarwal

I was wondering the same, it would be great if you could allow negative window length for EquityPricing in the Research Mode to get forward prices or something similar to achieve the same

Serge d'Adesky

Karen.

I know when you first announced pipeline it was not yet possible to use this in live trading. Has this restriction now been removed?

Thanks

Karen Rubin

Julien D and Ravi,
You both make valid points for features that would be helpful now that this is available in research. What you are asking for isn't possible yet, but I think it's a great suggestion and will add it to the list.

Serge,
You can paper trade pipeline algos, but we are still working on resolving a few things and testing the API before we make full live trading available.

Julien D

Thanks Karen, in the meantime what would be the best work around to use get_pricing to add those forward return to the pipeline dataframe?

Jeppe Udesen

When I try to rerun your notebook Karen, I get the following error. Any suggestions why that is?

stock pluse

Hi Jeppe Udesen ,

Its not only you please see this post https://www.quantopian.com/posts/keyerror-on-pipeline-tutorial-lesson-5.

Cheers
Stock Pluse

About Quantopian
Careers
Community
Events
Documentation
Academia
Lectures
Status
Facebook
Twitter
LinkedIn
YouTube
Terms of Use
Privacy Policy

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.