Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
code performance review
michelle carter
edited Jan 10, 2020
Research Pipeline numpy/pandas

So over 50% of my Pipeline generation time is spent generating prior returns in a custom factor, via the code below. I'm populating and using my own prices DataFrame by design. I did some crude timing tests and the vast majority (by ~ 2 orders of magnitude) of time is spent selecting the relevant asset prices via prices[[symbols(sid) for sid in assets]]. Can the performance of this code be improved?

def compute(self, today, assets, out, close):  
    todays_price_index = prices.index.get_loc(today)  
    prices2 = prices[[symbols(sid) for sid in assets]].as_matrix()  
    out[:] = (prices2[todays_price_index] - prices2[(todays_price_index - self.window_length)]) / \  
              prices2[(todays_price_index - self.window_length)]
3 responses
Marlowe Quart
Jan 23, 2020

Dictionarys are faster than dataframes. Using a dictionary may make that part of your code faster, but it may be better to use the slower dataframe depending on what else you need to do in your code: https://stackoverflow.com/questions/22084338/pandas-dataframe-performance

Jamie Veitch (Azure Hawk)
Jan 23, 2020

Why not just use the pre-built Returns() factor Quantopian provides?

Dan Whitnable
Jan 23, 2020

A few questions...

  1. What does the 'prices' dataframe look like? Maybe include the code which creates it.
  2. I assume 'prices' is defined at a scope making it available to the compute function. Is that true?

Also, a few comments. This pipeline will only work in the notebook environment (which you probably already knew). Just wanted to verify. Pipelines are run 'asynchronously' in an algo and cannot effectively access data outside of the custom factor class scope. If the reason you are using 'prices' in this manner is to get access to 'forward adjusted' prices then be very careful. Lookahead bias abounds. However, if all you are calculating is returns, then these will be the same regardless of how prices are adjusted. Might as well not go through all this effort and simply use the built in returns method that @Jamie Veitch mentioned.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

About Quantopian
Careers
Community
Events
Documentation
Academia
Lectures
Status
Facebook
Twitter
LinkedIn
YouTube
Terms of Use
Privacy Policy

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.