Any idea why I got this error?
OperationFailure: database error: Runner error: Overflow sort stage buffered data usage of 33559486 bytes exceeds internal limit of 33554432 bytes
There was a runtime error on line 218.
Line 218:
prices = history(20*390,'1m','price',ffill='True') Grant, you've hit a bug that we've seen before but we don't strictly know how to reproduce it. It's happening about a dozen times per month. Your code sample is interesting because that's the first time I've seen the customer code that triggers it. We were pretty sure the problem was in history() and your example would appear to confirm that.
That's a lot of history that you're loading! 7800 prices per equity, multiplied by the size of your trading universe. For reasons unknown, that attempt to load the historical data from the db failed.
I don't have a cause or a fix for this at this point. There is an optimization branch that we're working on that will ship this fall, and it seems likely that branch will fix this because it will change the way history loads data.
Sorry for the inconvenience, and sorry I don't have a better answer.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.
Thanks Dan,
I expect there are at least 200 securities, so that's 1.56M price values. It doesn't seem crazy, since you support up to 500 securities, and it is only a 20-day trailing window.
Would you expect the error to be only in live trading? Or should it show up in backtesting, too?
Grant
I'm idly curious what you're doing with those 1.56M price values, it's not a use case I have a ready understanding of. "Only" 7800 data points per security is quite a few.
I don't have enough understanding of the root cause to say if it's only in live trading or not.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.
Each security has a 20 day trailing price window, which is smoothed, and then sampled. This avoids using daily data, which I think are too coarse, and noisy. Is there a way I can request less data from history(), but sample throughout the day (e.g. every 10 minutes)?
Not at this time. There are some changes coming this fall for how history data is loaded, and I expect those will resolve the strain of this query.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.
Thanks Dan,
I'll try daily prices, and see how things play out:
prices = history(20,'1d','price',ffill='True') It should reduce the load by 390X, but I'm not confident it'll work for this case.
Hi Dan,
Here's an algo that might help you track down the error. It trades about 500 securities every day, and has:
price = history(20*390,'1m','price') I just launched it into live trading (https://www.quantopian.com/live_algorithms/55e95a6073f4c42af40002d0).
If the error is as I suppose, the algo should crash 1 hour after the market opens.
It's no big deal to me that the error is fixed quickly, but I figure you should be able to induce the error so that when you do upgrades, you can make sure it gets fixed then.
Grant
For the algo posted immediately above, I got the error under live trading:
TimeoutException: Too much time spent in handle_data call
There was a runtime error on line 29.
Line 29 is:
price = history(20*390,'1m','price') So whatever is going on under the hood to pull data ain't working.
Since the platform supports up to 500 securities, it seems a 20 day minutely price history for all 500 should be supported. A bug, yes?
Hi,
is this bug fixed?
I've just had an error in a live trading algo with history(60*4*30,'1m','price') in order to resample it to get a 4 hour data. And there were only 20 securities in the universe
Any response pleassse?
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.
In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.
