Contest criteria checker Notebook with compact output and more detail

Back to Community

edited

This is a Notebook algo checker producing a little more detail and yet fairly compact output, for saving metrics like this with the algo itself.
Example output ...

Results: 2011-01-04 to 2014-11-21  
Score                     4.5065  Constraints met 8/9  
Returns                   109.8%  PASS: Positive  
Positions              4.47|5.43  PASS: Max position concentration 4.47% <= 5.0%  
Leverage     0.74|0.94|1.06|1.17  PASS: Leverage range 0.94x-1.06x between 0.8x-1.1x  
Turnover         2.6|2.9|5.7|6.7  FAIL: 2nd percentile turnover 2.9% < 5.0x  
Net exposure             2.6|4.9  PASS: Net exposure (absolute value) 2.6% <= 10.0%  
Beta-to-SPY            0.12|0.14  PASS: Beta 0.12 between +/-0.30  
Sectors                0.18|0.18  PASS: All sector exposures between +/-0.20  
Style                  0.37|0.37  PASS: All style exposures between +/-0.40  
Tradable                  96|100  PASS: Investment in QTradableStocksUS >= 95.0%

Most of the changes that matter are in check_constraints().

Wish List
Q announced the new full Improved Backtest Analysis and for efficiency perhaps it could offer a text output like this at some point.
In the Notebook, does anyone know how to pull Alpha, Beta, Sharpe, Drawdown and so on to add to those?

6 responses

vladimir prelovac

Brilliant, just brilliant.

Delaney Mackenzie

I just wanted to say huge thanks for doing this, it's this kind of work that keeps the community going. I haven't had time to check this for correctness, but if other people are using it it would be helpful to post if it lines up with what we have in the new full backtester.

In general moving more and more of the workflow to research, such as how you're doing here, can greatly reduce iteration time and speed up work.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Bala Vignesh

Thank you Blue Seahawk, great stuff!

Josh Payne

Blue, I like your Wish List idea. One thing I've personally been contemplating is how we can provide better sample code and sample notebooks in the advanced tab of the new backtest screen to augment the simple pre-built code that runs get_backtest for the backtest in question and runs a full tear sheet. The tearsheet is a generalized assessment. This works great as a specific assessment for the Quantopian Contest.

Regardless, great contribution to the community here and for experienced folks, this notebook looks to be very useful.

Thanks
Josh

Disclaimer

Blue Seahawk

Updated the notebook to flexibly handle vertical alignment with any sets of numbers.

Here's a test from today and I'll discuss those numbers a little bit.
Individual values like 3.9|4.3|8.3|8.6 along with Q's summary line are stored and then formatted in columns when all set.

Results: 2016-05-02 to 2018-05-14  
Score                     0.0792  Constraints met 8/9  
Returns                     7.9%  PASS: Positive  
Positions              4.88|5.17  PASS: Max position concentration 4.88% <= 5.0%  
Leverage     0.95|0.97|1.03|1.06  PASS: Leverage range 0.97x-1.03x between 0.8x-1.1x  
Turnover         3.9|4.3|8.3|8.6  FAIL: 2nd percentile turnover 4.3% < 5.0x  
Net exposure             1.7|2.1  PASS: Net exposure (absolute value) 1.7% <= 10.0%  
Beta-to-SPY            0.24|0.28  PASS: Beta 0.24 between +/-0.30  
Sectors                0.08|0.08  PASS: All sector exposures between +/-0.20  
Style                  0.28|0.29  PASS: All style exposures between +/-0.40  
Tradable                  96|100  PASS: Investment in QTradableStocksUS >= 95.0%

Since I haven't tested extensively, please reach out to someone if you run into anything off.

Helps to see how close to the edges results are, for example ...
Leverage is pretty comfortable, net exposure has a lot of headroom. I think the key figure on beta is .24 and that's a concern. Sectors and style are in good shape. I wasn't sure about tradable stocks until looking in the code. 96 is the outlier figure I think. it was more often 100% so that seem to be fine.

You can see in the summary "FAIL: 2nd percentile turnover 4.3% < 5.0x" that 4.3 was the disqualifier rather than the low, 3.9, an outlier. I could benefit from a pointer to a description how those work, for example position concentration is close to the limit with 4.88 as the 98% percentile high, meanwhile 5.17, being over 5 was ok because that's the 100% percentile limit, and looking in the notebook code I see that limit is double, allowing for quite a bit of wiggle room like at the beginning, or unusual brief conditions throughout the run I think.

You may want to uncomment

#bt.create_full_tear_sheet()

... and by the way I hear that a lot of folks are fans of the round_trips option for the extra information it provides:

bt.create_full_tear_sheet( round_trips=True )

Info on round_trips

Blue Seahawk

Do you, the reader, know notebooks well? Please consider taking a few minutes to make a more presentable version of this, thanks in advance. I'll be in the engine room with a wrench if you need me.

You've successfully submitted a support ticket.

Our support team will be in touch soon.