Major variance between I.S. and O.O.S. results from model that uses supervised machine learning

Quantopian's community platform is shutting down. Please read this post for more information and download your code.

Back to Community

Major variance between I.S. and O.O.S. results from model that uses supervised machine learning

edited Sep 4, 2018

Has anyone ever had such a MASSIVE discrepancy between in sample and out of sample results? (click on view notebook)

Annual Returns: ALL = 669%, IS = 531.6% OOS = 3,463.7%

This is a dollar/market neutral strategy that returns 12% - 18% annually un-levered. It is designed for substantial capacity in excess of several yards, and leverage more reflective of what hedge funds typically get for this type of portfolio construction.

I checked stocks that may or may not have been available to verify data_can_trade screen.

This one's a real head scratcher. The purpose of validating using out of sample data, specifically as it applies to machine learning, is a sort of sanity check. With this model, the AI dynamically tunes factor exposures (among other things), based on a forward prediction. So for me, because I use supervised machine learning, validation can be accomplished by taking a slice of the data set and not allowing the AI to use it for training and model development. Alternatively, validation can be achieved by running a live forward test. In this specific case, I have used both methods. The forward test (simulation results) are in line with the OOS results.

I see some discrepancy sometimes, but generally, I'm looking for results b/w IS and OOS to be somewhat close, NOT on different planets.

4 responses

@Pej H,

I suggest you do a longer multiyear run and hopefully you do not encounter the compute timeout error. This will give you a better gauge of in sample and OOS consistency.

It is designed for substantial capacity in excess of several yards, and leverage more reflective of what hedge funds typically get for this type of portfolio construction.

Your gross leverage is 18.95! Is this even possible or realistic?

Delaney Mackenzie

I would double check that you're not leaking information about the future into your model in some way. Also any deviation, up or down, from in-sample results when running an out of sample test is worrying. It means there's an effect you don't understand. Here I'd say that your out of sample returns are in our Bayesian cone, so it seems fine.

I would also be interested to see what type of results you get with leverage constrained. You're correct that generally funds take out more leverage, but 19x is high. The amount of leverage also depends on the strategy and the estimated risk. Starting from 1x and then showing what happens to returns as you increase to 2x, 3x, ... will provide a lot of information.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

@James VIlla

Of course it's possible. Perhaps the most significant incentive behind developing market and beta neutral strategies (not always dollar neutral especially in a sustained bear market because of how correlations affect individual stock betas), is that while the "net returns" are not glamorous, the strategy offers substantial capacity with uncorrelated returns that exhibit very low vola.

@Pej H,

I don't know of any financial institution that will lend you 19x your capital base but perhaps you know something that I don't. I am very curious as to how this can be done for a market/beta neutral strategy. What would be the compelling reasons for a lender to take on such exposure?

You've successfully submitted a support ticket.

Our support team will be in touch soon.

Need help? Contact support.

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

About Quantopian

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.