Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Research Memory Limitation
Joakim Arvidsson (Cream Mongoose)
edited Dec 12, 2018
Seeking Help Using Data Show Q Research Alphalens

Hi @Jean, @Josh, @Jamie,

[I started off writing this as a response to the latest comment in this thread, but then thought it might warrant a separate post.]

Thank you for the continued effort to improve effective use of limited resources in Research.

Here's my current experience in Research from a single user perspective:

  1. From a fresh start (killing all active notebooks (NB)), just listing my NBs in Research (I have quite a few) uses up 1% of my allocated memory (leading me to think I don't get much memory to start with).
  2. Opening up a single NB, without running any cells, uses up another 4-5% of memory. (see attached NB as an example). Now memory utilization is at 5-6% (without actually running anything).
  3. Having only the attached NB open, and trying to run 'all cells' from the top, results in memory maxing out, and kernel restarting, about midway through the NB.
  4. There's no way for me to tell if a particular cell will result in maxing out memory, other than just 'trial and error,' which can be quite time-consuming.

The attached NB is only slightly modified from Thomas W's excellent NB. Yes, it runs over 10 years instead of 7, and uses 63 day SMA instead of 5 days, but it also only uses a single field from Morningstar, rather than something from a Premium Dataset, which I believe uses more resources.

Maybe I have unrealistic expectations, but since Q offers data back to 2002, I would have expected to be able to research this data in full, using a reasonably complex factor, including reasonable 'smoothing' etc., without running into the memory limit?

As I see it, the only option I have is to use 'trial and error' to find out how long a period I'm able to perform research on for a specific factor (period length may depend heavily on factor complexity), which can be quite time-consuming and in my view is very unfortunate. Or maybe I need to learn how to write more 'memory effective' code (admittedly I'm not the best python programmer)?

In your capacity planning, how was the current memory limit determined? Does memory 'scale-up' as more users are starting to use the Q platform, or do we each get allocated a smaller piece of the same memory-pie, as more users are added? Is memory allocated per user 'fixed' or 'dynamically' allocated depending on current/projected usage during the day?

I would be quite happy to pay a reasonable 'subscription fee' for a 'premium Q service' if it meant I could have more memory to do more effective research. I'm just a single person, but if Quantopian Enterprise (QE) offers more memory in Research at a reasonable price, I might look into that. The QE FAQ page doesn't mention anything about more memory however, and I actually prefer to not have access to the FactSet holdback period (so I can test for overfitting in the Contest), so I don't really want to pay for access to the 'holdback period.'

If you or anyone else have any ideas or suggestions on how I can make better use of memory in Research, I'd be all ears. E.g. maybe there's a time of day when I'm more likely to be allocated more memory?

Thank you for an otherwise excellent platform, and for your continued effort to improve the Research experience!

Joakim

PS: I did see the Alphalens 'pull request' in Github to Speed up compute_forward_returns and get_clean_factor which is great, and might help with memory utilization as well?

4 responses
Saba Nejad
Dec 18, 2018

Hi @Joakim,

Thank you for posting this and sharing your feedback.

We understand your frustration and the problem you're facing. It's one that's been on our radar for a while. We often bump into the limit ourselves when we write sample algorithms or community posts. Our current priority is adding in more data to the platform. We anticipate that the increasing use of the added data will further stress the memory problem. With that in mind, we've started doing research on ways we can fix this issue and we hope to tackle it in some fashion this coming year.

Unfortunately, I don't have a better answer at the moment but I can assure you that we're looking into this.

Best,
Saba

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Zenothestoic
Dec 18, 2018

Hmmm....

Joakim Arvidsson (Cream Mongoose)
Dec 19, 2018

Hi Saba,

Thank you. I really appreciate your response and update!

I'm quite happy with this answer, and I think I'm probably (massively) underestimating the enormous challenge of providing a free Research environment to hundreds of thousands of people. I'm glad it's on your radar, and I'm looking forward to the new datasets (more quality stuff from FactSet I'm guessing), which is AWESOME!!

For now, (just like everyone else) I'll just have to work with the current limitations. The recent speed improvement update to two of the Alphalens methods helps a ton as well in my research! Thanks again!

Pieter-Jan
Dec 19, 2018

Hi Saba,

Really appreciate your honest answer! Thanks for the update.

About Quantopian
Careers
Community
Events
Documentation
Academia
Lectures
Status
Facebook
Twitter
LinkedIn
YouTube
Terms of Use
Privacy Policy

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.