Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Log length
Vlatko Primorac
posted

Is there a way one can get reasonably long logs? Mine get cut off just then the things get interesting, I understand that server should not be brought down by 5G individual logs, but say 1M would be quite welcome. Below is an example of a log cutoff, just as I was hoping to log the reason for each trade. 

1969-12-31initialize:30INFOTrading ETFs:  
1969-12-31initialize:32INFOSPY SPDR S&P 500 ETF TRUST  
1969-12-31initialize:32INFOIYR ISHARES US REAL ESTATE ETF  
1969-12-31initialize:32INFOIYC ISHARES U.S. CONSUMER SERVICES  
1969-12-31initialize:32INFOIYG ISHARES U.S. FINANCIAL SERVICE  
1969-12-31initialize:32INFOIYJ ISHARES U.S. INDUSTRIALS ETF  
1969-12-31initialize:32INFOIYK ISHARES U.S. CONSUMER GOODS ET  
1969-12-31initialize:32INFODIA SPDR DOW JONES INDUSTRIAL AVER  
1969-12-31initialize:32INFOEWW ISHARES MSCI MEXICO CAPPED ETF  
1969-12-31initialize:32INFOEWC ISHARES MSCI CANADA ETF  
1969-12-31initialize:32INFOEWD ISHARES MSCI SWEDEN ETF  
1969-12-31initialize:32INFOEWA ISHARES MSCI AUSTRALIA ETF  
1969-12-31initialize:32INFOEWI ISHARES MSCI ITALY CAPPED ETF  
1969-12-31initialize:32INFOEWJ ISHARES MSCI JAPAN ETF  
1969-12-31initialize:32INFOICF ISHARES COHEN & STEERS REIT ET  
1969-12-31initialize:32INFOIYE ISHARES U.S. ENERGY ETF  
1969-12-31initialize:32INFOIVE ISHARES S&P 500 VALUE ETF  
1969-12-31initialize:32INFOIJJ ISHARES S&P MID-CAP 400 VALUE  
2013-01-02handle_data:40DEBUGCash at 100000.0  
2013-01-02handle_data:46DEBUGOrder 100 shares of Security(14517 [EWC])  
2013-01-02handle_data:40DEBUGCash at 100000.0  
2013-01-02handle_data:46DEBUGOrder 100 shares of Security(14530 [EWW])  
2013-01-02handle_data:40DEBUGCash at 100000.0  
2013-01-02handle_data:46DEBUGOrder 100 shares of Security(22446 [ICF])  
2013-01-02undefined:undefinedWARNLogging limit exceeded; some messages discarded  
End of logs.
6 responses
Dennis C

I had to experiment a bit. I think there's a different cutoff for max number of messages per event and the character length of each message.

Vlatko Primorac

Rules of log length available are thus:

  1. Log length is 30 lines, maybe shorter.
  2. There may be a way to allow for longer logs.
  3. Maybe not.
Jonathan Kamens

Individual log entries are limited to 1024 characters and are truncated if they exceed that.

The number of log entries a backtest is allowed to generate is limited as described in the API documentation.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Dan Dunn

We tried to build the limits in a balance of usefulness and performance and cost. If the limits are keeping your from getting the information you need, let us know, and we can modify them in the future. Feedback is most welcome.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Vlatko Primorac

Back of the envelope calculation would make a desired log be something like:

  • One log line per trade or signal, about 100 characters
  • 100 trades or events before something interesting happens

That would be 100x100 = just about 10k of logs. 1k is restricting enough to make logs all but useless. But don;t take my word for it. Look at how many algorithms published are using logs in any meaningful way.

Judging from those I have read, probably none. If it happens nobody needs logs these days anyways, increasing the limit will have no ill effect on the performance. If on the other hand increasing the log size even temporarily results in logs actually being used for debugging and analysis, the hit on server performance may be worth it.

Dan Dunn

Vlatko, there may be some misunderstanding, The 1024 character limit is per line, not for overall logging.

The limits are based on each call of handle_data. Say you put all of your signal information into one line, and all of your trade information into a 2nd line - in that scenario you can log almost indefinitely. The overall, per-person limit is several megabytes and it takes a long while to reach that limit.

Our theory is that there is "basic" logging ongoing, and then there is a need for "bursty" logging when something exciting happens. That drove the current implementation. It looks like your logging, above, is pretty verbose the whole way through - it wasn't a usage pattern that we thought was sustainable for us. For algos that look at hundreds of stocks, for instance, verbose logging could lead to thousands of log lines per handle_data, and then performance becomes an issue at scale.

I completely agree that we need to experiment with log sizes. We essentially had no limits for months and we watched logging behavior, and we put some limits in this spring, and we're watching logging behavior. Feedback and usage tracking will drive the next iteration.

About Quantopian
Careers
Community
Events
Documentation
Academia
Lectures
Status
Facebook
Twitter
LinkedIn
YouTube
Terms of Use
Privacy Policy

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.