Quantopian Lecture Series: The Good, the Bad, and the Correlated

Back to Community

edited

Correlation analysis is ubiquitous in many fields of quantitative study. In the field of quantitative finance, it is especially useful for diversifying your portfolio by minimizing correlation between your assets' return streams. We will provide a primer on correlation analysis and discuss its relation to covariance.

In addition, we have released a notebook on Spearman Rank Correlation here. Spearman Rank Correlation is robust to differing scales in the underlying data and non-normal distributions.

This is part of Quantopian’s Summer Lecture Series. We are currently developing a quant finance curriculum and will be releasing clone-able notebooks and algorithms to teach key concepts. This notebook will be presented at this meetup. Stay tuned for more.

Credit for the notebooks goes to Evgenia 'Jenny' Nitishinskaya, and credit for the algorithms goes to David Edwards.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

23 responses

Delaney Mackenzie

We implemented a correlation reduction algorithm from this paper. The idea behind correlation reduction is that it helps diversify your returns streams and implicitly reduce risk exposure. Here is a backtest from an long-short momentum strategy that uses correlation reduction as a filter before its final buy/sell orders.

Disclaimer

Delaney Mackenzie

Here, as a control, is the same strategy without the correlation reduction filter. Notice how the sharpe ratio is lower.

Disclaimer

Kelly Elmstrom

Hi everyone,

We will be running a live webinar on this content next Thursday, July 16th at 12pm EDT. You can register here: http://bit.ly/thegoodthebadthecorrelatedwebinar. Once the webinar is wrapped up, we will share the recording here.

Hope you can join us!

Disclaimer

Kelly Elmstrom

Here is the recording from our live webinar today on "The Good, the Bad, and The Correlated": http://bit.ly/thegoodbadcorrelatedvideo. We hope you enjoy it!

Disclaimer

Tim Vidmar

When I clone and back-test the algorithm I get a runtime error message.

There was a runtime error.
TypeError: 'float' object is not iterable
... USER ALGORITHM:57, in trade
lower, upper = ranks.quantile([.05, .95])

Could you please investigate?

Many thanks!

Tim Vidmar

I take it back, it seems to be working (again). My apologies.

Tim Vidmar

Inspite of my blunder, allow me another burning question, please.

If I change the trading frequency to every_day, i.e., if I replace

schedule_function(trade,  
                  date_rule=date_rules.month_start(),  
                  time_rule=time_rules.market_open(minutes=20))

with

schedule_function(trade,  
                  date_rule=date_rules.every_day(),  
                  time_rule=time_rules.market_open(minutes=20))

the leverage and the exposure inevitably go haywire at some point.

How is this possible, especially when dealing with the algorithm version which does not perform correlation reduction,
because in this case the following lines get executed:

    longs = longs.abs()  
    longs /= longs.sum()  

    shorts = shorts.abs()  
    shorts /= shorts.sum()

followed by

for stock in data:  
    if stock in context.ignores:  
        continue  
    try:  
        if stock in shorts.index:  
            order_target_percent(stock,  
                                 context.shortleverage * shorts[stock])  
        elif stock in longs.index:  
            order_target_percent(stock,  
                                 context.longleverage * longs[stock])  
        else:  
            order_target(stock, 0)  
    except:  
        log.warn("[Failed Order] stock = %s"%stock.symbol)

where

context.longleverage = 0.5  
context.shortleverage = -0.5

Shouldn't this guarantee that the leverage remains (almost) exactly unity under any circumstances?

David Edwards

Hi Tim,
I know exactly what you are talking about, the issue is that the universe goes haywire through the financial crisis period. I don't have an exact reason for you, but my hunch is that it has to do with the massive number of bankruptcies, mergers, and other corporate events.

This sort of thing becomes a problem in backtesting whenever you are dealing with a dynamic universe. If I recall the leverage/exposure goes nuts even at a monthly frequency. Unfortunately I don't have a simple solution. You will have to write some helper functions to make sure all of the stocks you are buying/selling are still being actively traded. Maybe looking at their volumes is a good place to start. Sorry I don't have an easy answer, hope this helps though.

David

Tim Vidmar

Many thanks for your explanation, David

In the meantime I have applied the following change, which seems to help quite a bit:

    try:  
        if get_open_orders(sid = stock):  
            continue  
        elif stock in shorts.index:  
            order_target_percent(stock,  
                                 context.shortleverage * shorts[stock])  
        elif stock in longs.index:  
            order_target_percent(stock,  
                                 context.longleverage * longs[stock])  
        else:  
            order_target(stock, 0)  
    except:  
        log.warn("[Failed Order] stock = %s"%stock.symbol)

instead of the original

    try:  
        if stock in shorts.index:  
            order_target_percent(stock,  
                                 context.shortleverage * shorts[stock])  
        elif stock in longs.index:  
            order_target_percent(stock,  
                                 context.longleverage * longs[stock])  
        else:  
            order_target(stock, 0)  
    except:  
        log.warn("[Failed Order] stock = %s"%stock.symbol)

Delaney Mackenzie

The open orders check is a great one, Tim. Alisa covered it in this list of ways to make your algorithm more robust: https://www.quantopian.com/posts/tips-for-writing-robust-algorithms-for-the-hedge-fund

My personal recommendation is that any algorithm you submit to the contest or live trade should be run through her checklist.

Disclaimer

Giuseppe Moriconi

when i try to run this algo i get this error:

AttributeError: 'float' object has no attribute 'rank'
... USER ALGORITHM:105, in get_reduced_correlation_weightsGo to IDE
ranks = initial_weights.rank()

Delaney Mackenzie

Hi Giuseppe,

It works for me, any chance you're running in daily mode? It needs to be minutely as far as I know.

Thanks,
Delaney

Disclaimer

Yachi Wang

I tried to increase the leverage to 1, which I supposed the exposure should still be zero (someone please correct me if I was wrong), But at some point of backtesting, the exposure changed to -1 over night, what could be the case?

David Edwards

@ Giuseppe, is it possible that you are limiting the percentile of selected stocks so that only one stock is in the long or short baskets? If so a length 1 series will coerce to a float and you would get the error you're describing.

@ Yachi, your assumption that the exposure should still be 0 is correct. I am not sure what happened in your situation (I'll take a closer look when I find a minute). There are some issues with stocks being delisted around the 08 debacle that cause similar problems. You will have to hone in on the trades the algo made that caused it to go nutty. I find it interesting that the allocations never went back to normal after another rebalance.

Giuseppe Moriconi

Thanks David and Delaney, i actually was running the algo in daily mode... this was the issue

Yachi Wang

@David, Thank you for your explanation, I take into consideration of delisting in my code this time by adding a "if get_open_orders(sid = stock): /n continue "statement, and it did alleviates the frenzy of leverage. However, it did not explain the abruptly decreasing of leverage. I also tried out the code in different period, interestingly, I found out the leverage changing is not consistent in different timeframe, I tried to figured out if it is caused by the algo... For example, if took timeframe from 2008-05-01 to 2008-12-31, the deleveraging seemed to occurred periodically

Yachi Wang

What confused me is if I extend the timeframe from 2008-02-01 to 2009-04-30, the leveraging only occurred once, on Jun,1 2008,
what could be the cause of this situation?

Blue Seahawk

At the leverage drop from 1.86 to 1.08, you can examine the trades at that time by adding track_orders(). You might want to set a start date in its options for convenience and to avoid overwhelming the logging window with a boatload of output, no stop date required, just let 'er rip and see what happens.

Edit: I had a bug in PvR code now fixed where it wasn't always accounting for all shorting as risk and have had to drop some of what I said here previously about this code. Note this code only spends ~1/4 of initial capital however it shorts beyond initial capital ($2.3M) and that's extra risk. The beta at way below zero is often (or always, I'm not sure) a sign of copious shorting. If you adjust to go long more, to use more of that starting capital, and short less, the mountain during the '08 crash will be less and I think the long-term separation above SPY will be correspondingly higher. Consider/try increasing longleverage and decreasing shortleverage and see what the differences would be thru 2016. That'll take hours. :/ :) You can run two or more at the same time.

I was seeing a flatline for about a year, all in cash, to look into, a bit beyond the range of the test above.

Shawn Emhe II

I have a question about the use of log prices in this algo. Because the prices are converted to log on line 50, shouldn't they be subtracted rather than divided on line 52?
Replacing:

    prices = np.log(history(context.lookback, '1d', 'price').dropna(axis=1))  
    R = (prices / prices.shift(context.return_window)).dropna()

with:

    prices = np.log(history(context.lookback, '1d', 'price').dropna(axis=1))  
    R = prices.diff(context.return_window).dropna()

The pct_change calculation on line 63 would also need a similar change.

Shawn Emhe II

I played around in the research environment and found the answer to my own question. By dividing the log prices the results are normalized. Subtracting them gives a ranking that's weighted toward higher priced stocks, which may have a higher log return but lower percent return. But that brings me to the question, why use log prices? What advantage does it offer over just dividing the normal prices?

Delaney Mackenzie

The algorithms in this thread may cease working briefly as we're doing maintenance during the switch to Q2. Please let me know if this causes any issues.

Disclaimer

kay star

Do you by chance have a fresh link to the paper?

Delaney Mackenzie

Looks like you can get it here: https://www.scribd.com/document/106570475/CSS-Analytics-Minimum-Correlation-Algorithm#from_embed

Disclaimer

You've successfully submitted a support ticket.

Our support team will be in touch soon.