pattern recognition based on zlib

Quantopian's community platform is shutting down. Please read this post for more information and download your code.

Back to Community

pattern recognition based on zlib

posted Feb 28, 2013

Here is a simple example of a pattern recognition algorithm based on the text string compression routine zlib. Thanks to James Jack for useful discussions and coding the NCD and CDM functions, and to Quantopian for enabling zlib.

For those interested in doing an intellectual "deep dive" into the topic, a starting point is:

Ming Li; Xin Chen; Xin Li; Bin Ma; Vitanyi, P.M.B.; , "The similarity metric,"
Information Theory, IEEE Transactions on , vol.50, no.12, pp. 3250- 3264, Dec. 2004
http://homepages.cwi.nl/~paulv/papers/similarity.pdf

2 responses

I ran the algorithm on SPY & SH (S&P 500 ETF & S&P 500 short ETF, respectively). Sample output, with X & Y representing the coded prices (relative to their respective moving averages) over a 30-day trailing window:

2012-02-23handle_data:53INFO----------------------------------  
2012-02-23handle_data:54INFO X: 000000000000000011111111111111  
2012-02-23handle_data:55INFO Y: 111111111111111100000000000000  
2012-02-23handle_data:56INFO----------------------------------  
2012-02-23handle_data:57INFO NCD: 0.142857142857  
2012-02-23handle_data:58INFO CDM: 0.571428571429  
2012-02-23handle_data:59INFO----------------------------------

One might have expected NCD & CDM both to be ~ 1 (indicating a high degree of dissimilarity). Instead, both indicate a relatively high similarity between SPY & SH (NCD << 1 & CDM ~ 0.5). The interpretation, I think, is that SPY & SH (as coded) have the same information content. For example, if I am given the SPY time series, I can predict the SH time series (so long as I know that it moves in the opposite direction). I obtain a similar result for the pair SPY & IVV, which move in the same direction.

Jason Swearingen

Hi Grant, I think I better start with RSquared, as it's something I should be able to understand easily, and is pretty straight forward. For example, SH vs SPY has a RSquared above 0.9, which is consistent with them being mirrors of eachother.

You've successfully submitted a support ticket.

Our support team will be in touch soon.

Need help? Contact support.

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

About Quantopian

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.