Calculation of correlation in pipeline

Quantopian's community platform is shutting down. Please read this post for more information and download your code.

Back to Community

Calculation of correlation in pipeline

Francesco Cricchio

posted May 21, 2016

Hello,
what could be the good way to calculate the correlation between two quantities in the quantopian pipeline ?

I was trying

class PriceFactCorr(CustomFactor):
inputs = [alphaone.article_sentiment, USEquityPricing.open]
window_length = 100

def compute(self, today, assets, out, sentiment, open_price):  
    out[:] = np.corrcoef(sentiment, open_price)[0, 1]

but it gives all Nans

Equity(693 [AZO]) NaN
Equity(1374 [CDE]) NaN
Equity(1582 [CL]) NaN
......

Did someone found a good solutions?

Thanks

Francesco

4 responses

Hey Francesco,

Wondering what the [0,1] is for?

I use the following code and it always works well:

Correlation = np.corrcoef(Factor_1, Factor_2)

Does your code create a pipeline class that is supposed to calculate the correlation between sentiment 100 days ago and open price 99 days ago? If so, I think (and I welcome comments if I am incorrect) the code might work if you do this instead:
out[:] = np.corrcoef(sentiment[0], open_price[1])

If you are trying to get the correlation between sentiment yesterday and open_price today, I think you would want to decrease the window length. Or maybe better to try this:

out[:] = np.corrcoef(sentiment[-2], open_price[-1])

I didn't test it, but something to try.

Francesco Cricchio

Thanks for the useful suggestion Frank, I will try those!
Regarding your suggestion

Out[]=np.corrcoeff(sent[-2]...

I do not agree, I believe that correlation should be calculated on a time series and not only two points since it is the cov(x,y) / sigma_x sigma_y and should include some time points I guess
https://en.m.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient

[0,1] then refers to off diagonal elements of the correlation matrix returned by numpy --> (X,Y) [0, 0] would have been (X,X)

Francesco,

My mistake. I forgot that factor_1, and factor_2 from my basic example are lists of said factors. I wonder if something like this would do the trick:

np.corrcoeff(sentiment[0:99], open_price[1:100])

could you share your notebook?

You've successfully submitted a support ticket.

Our support team will be in touch soon.

Need help? Contact support.

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

About Quantopian

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.