Quantopian Lecture Series: The Good, the Bad, and the Correlated

Jul 8, 2015

We implemented a correlation reduction algorithm from this paper. The idea behind correlation reduction is that it helps diversify your returns streams and implicitly reduce risk exposure. Here is a backtest from an long-short momentum strategy that uses correlation reduction as a filter before its final buy/sell orders.

Disclaimer

Jul 8, 2015

Here, as a control, is the same strategy without the correlation reduction filter. Notice how the sharpe ratio is lower.

Disclaimer

Kelly Elmstrom

Jul 10, 2015

Hi everyone,

We will be running a live webinar on this content next Thursday, July 16th at 12pm EDT. You can register here: http://bit.ly/thegoodthebadthecorrelatedwebinar. Once the webinar is wrapped up, we will share the recording here.

Hope you can join us!

Disclaimer

Kelly Elmstrom

Jul 16, 2015

Here is the recording from our live webinar today on "The Good, the Bad, and The Correlated": http://bit.ly/thegoodbadcorrelatedvideo. We hope you enjoy it!

Disclaimer

When I clone and back-test the algorithm I get a runtime error message.

There was a runtime error.
TypeError: 'float' object is not iterable
... USER ALGORITHM:57, in trade
lower, upper = ranks.quantile([.05, .95])

Could you please investigate?

Many thanks!

I take it back, it seems to be working (again). My apologies.

Inspite of my blunder, allow me another burning question, please.

If I change the trading frequency to every_day, i.e., if I replace

schedule_function(trade,  
                  date_rule=date_rules.month_start(),  
                  time_rule=time_rules.market_open(minutes=20))

with

schedule_function(trade,  
                  date_rule=date_rules.every_day(),  
                  time_rule=time_rules.market_open(minutes=20))

the leverage and the exposure inevitably go haywire at some point.

How is this possible, especially when dealing with the algorithm version which does not perform correlation reduction,
because in this case the following lines get executed:

    longs = longs.abs()  
    longs /= longs.sum()  

    shorts = shorts.abs()  
    shorts /= shorts.sum()

followed by

for stock in data:  
    if stock in context.ignores:  
        continue  
    try:  
        if stock in shorts.index:  
            order_target_percent(stock,  
                                 context.shortleverage * shorts[stock])  
        elif stock in longs.index:  
            order_target_percent(stock,  
                                 context.longleverage * longs[stock])  
        else:  
            order_target(stock, 0)  
    except:  
        log.warn("[Failed Order] stock = %s"%stock.symbol)

where

context.longleverage = 0.5  
context.shortleverage = -0.5

Shouldn't this guarantee that the leverage remains (almost) exactly unity under any circumstances?

David Edwards

Hi Tim,
I know exactly what you are talking about, the issue is that the universe goes haywire through the financial crisis period. I don't have an exact reason for you, but my hunch is that it has to do with the massive number of bankruptcies, mergers, and other corporate events.

This sort of thing becomes a problem in backtesting whenever you are dealing with a dynamic universe. If I recall the leverage/exposure goes nuts even at a monthly frequency. Unfortunately I don't have a simple solution. You will have to write some helper functions to make sure all of the stocks you are buying/selling are still being actively traded. Maybe looking at their volumes is a good place to start. Sorry I don't have an easy answer, hope this helps though.

David

Many thanks for your explanation, David

In the meantime I have applied the following change, which seems to help quite a bit:

    try:  
        if get_open_orders(sid = stock):  
            continue  
        elif stock in shorts.index:  
            order_target_percent(stock,  
                                 context.shortleverage * shorts[stock])  
        elif stock in longs.index:  
            order_target_percent(stock,  
                                 context.longleverage * longs[stock])  
        else:  
            order_target(stock, 0)  
    except:  
        log.warn("[Failed Order] stock = %s"%stock.symbol)

instead of the original

    try:  
        if stock in shorts.index:  
            order_target_percent(stock,  
                                 context.shortleverage * shorts[stock])  
        elif stock in longs.index:  
            order_target_percent(stock,  
                                 context.longleverage * longs[stock])  
        else:  
            order_target(stock, 0)  
    except:  
        log.warn("[Failed Order] stock = %s"%stock.symbol)

The open orders check is a great one, Tim. Alisa covered it in this list of ways to make your algorithm more robust: https://www.quantopian.com/posts/tips-for-writing-robust-algorithms-for-the-hedge-fund

My personal recommendation is that any algorithm you submit to the contest or live trade should be run through her checklist.

Disclaimer

Giuseppe Moriconi

Jan 5, 2016

when i try to run this algo i get this error:

AttributeError: 'float' object has no attribute 'rank'
... USER ALGORITHM:105, in get_reduced_correlation_weightsGo to IDE
ranks = initial_weights.rank()

Jan 11, 2016

Hi Giuseppe,

It works for me, any chance you're running in daily mode? It needs to be minutely as far as I know.

Thanks,
Delaney

Disclaimer

Yachi Wang

Jan 12, 2016

I tried to increase the leverage to 1, which I supposed the exposure should still be zero (someone please correct me if I was wrong), But at some point of backtesting, the exposure changed to -1 over night, what could be the case?

David Edwards

Jan 12, 2016

@ Giuseppe, is it possible that you are limiting the percentile of selected stocks so that only one stock is in the long or short baskets? If so a length 1 series will coerce to a float and you would get the error you're describing.

@ Yachi, your assumption that the exposure should still be 0 is correct. I am not sure what happened in your situation (I'll take a closer look when I find a minute). There are some issues with stocks being delisted around the 08 debacle that cause similar problems. You will have to hone in on the trades the algo made that caused it to go nutty. I find it interesting that the allocations never went back to normal after another rebalance.

Giuseppe Moriconi

Jan 12, 2016

Thanks David and Delaney, i actually was running the algo in daily mode... this was the issue

Yachi Wang

Jan 13, 2016

@David, Thank you for your explanation, I take into consideration of delisting in my code this time by adding a "if get_open_orders(sid = stock): /n continue "statement, and it did alleviates the frenzy of leverage. However, it did not explain the abruptly decreasing of leverage. I also tried out the code in different period, interestingly, I found out the leverage changing is not consistent in different timeframe, I tried to figured out if it is caused by the algo... For example, if took timeframe from 2008-05-01 to 2008-12-31, the deleveraging seemed to occurred periodically

Yachi Wang

Jan 13, 2016

What confused me is if I extend the timeframe from 2008-02-01 to 2009-04-30, the leveraging only occurred once, on Jun,1 2008,
what could be the cause of this situation?

Blue Seahawk

Jan 14, 2016

At the leverage drop from 1.86 to 1.08, you can examine the trades at that time by adding track_orders(). You might want to set a start date in its options for convenience and to avoid overwhelming the logging window with a boatload of output, no stop date required, just let 'er rip and see what happens.

Edit: I had a bug in PvR code now fixed where it wasn't always accounting for all shorting as risk and have had to drop some of what I said here previously about this code. Note this code only spends ~1/4 of initial capital however it shorts beyond initial capital ($2.3M) and that's extra risk. The beta at way below zero is often (or always, I'm not sure) a sign of copious shorting. If you adjust to go long more, to use more of that starting capital, and short less, the mountain during the '08 crash will be less and I think the long-term separation above SPY will be correspondingly higher. Consider/try increasing longleverage and decreasing shortleverage and see what the differences would be thru 2016. That'll take hours. :/ :) You can run two or more at the same time.

I was seeing a flatline for about a year, all in cash, to look into, a bit beyond the range of the test above.

Shawn Emhe II

Jan 15, 2016

I have a question about the use of log prices in this algo. Because the prices are converted to log on line 50, shouldn't they be subtracted rather than divided on line 52?
Replacing:

    prices = np.log(history(context.lookback, '1d', 'price').dropna(axis=1))  
    R = (prices / prices.shift(context.return_window)).dropna()

with:

    prices = np.log(history(context.lookback, '1d', 'price').dropna(axis=1))  
    R = prices.diff(context.return_window).dropna()

The pct_change calculation on line 63 would also need a similar change.

Shawn Emhe II

Jan 20, 2016

I played around in the research environment and found the answer to my own question. By dividing the log prices the results are normalized. Subtracting them gives a ranking that's weighted toward higher priced stocks, which may have a higher log return but lower percent return. But that brings me to the question, why use log prices? What advantage does it offer over just dividing the normal prices?

Apr 19, 2016

The algorithms in this thread may cease working briefly as we're doing maintenance during the switch to Q2. Please let me know if this causes any issues.

Disclaimer

kay star

May 30, 2018

Do you by chance have a fresh link to the paper?