Have a look at here

Thanks - I've had a look & still can't get it working based on how mine is set up. My intuition is the following process:
1. Calculate beta of the short portfolio (high beta)
2. Calculate beta of the long portfolios (low beta)
3. Find the hedge ratio ( -beta_long/beta_short ). Should give a value like 1.5 or something.
4. Adjust "context.short_leverage" to the short_leverage*hedge ratio for each iteration to make beta=0. Maybe my thinking is wrong & alternatively you could be leveraging the weights of the short position by the hedge ratio.

Below is the code I added in, it is causing some whacky things with the leverage which I cannot understand & cant get beta to normalize. Can't tell who to interpret the output from pipeline, copying the code into a notebook isn't working for me, maybe the math is wrong.

def before_trading_start(context, data):  
    # Call pipeline_output to get the output  
    # Note this is a dataframe where the index is the SIDs for all  
    # securities to pass my screen and the columns are the factors which  
    output = pipeline_output('ranking_example')  
    ranks = output['factor']  
    long_ranks = ranks[output['longs']].rank()  
    short_ranks = ranks[output['shorts']].rank()  
#    context.beta_long = context.output.loc[context.long_ranks.index]['beta'].mean()  
    beta_long = output.groupby('longs')['beta'].mean()[True]  
    beta_short = output.groupby('shorts')['beta'].mean()[True]  
    hedge_ratio = beta_long/-beta_short  
    context.short_leverage = context.short_leverage*hedge_ratio

    context.long_weights = (long_ranks / long_ranks.sum())  
    log.info("Long Weights:")  
    log.info(context.long_weights)  
    context.short_weights = (short_ranks / short_ranks.sum())  
    log.info("Short Weights:")  
    log.info(context.short_weights)  
    context.active_portfolio = context.long_weights.index.union(context.short_weights.index)

Heres the latest backtest

Lex,

Two comments.

I believe your hedge_ratio calculation should be

hedge_ratio  =  beta_long / (beta_short + beta_long)  
short_leverage  = -hedge_ratio     # negative because we want to go short these securities  
long_leverage   = 1-hedge_ratio    # will be positive and therefore long if both betas are positive

As long as the betas are both greater than 0, the hedge_ratio will be between 0 and 1. This ensures the net portfolio leverage is 1. Your code may already include this but make sure the long_leverage is also modified by the hedge ratio.

Second, remember that beta may be negative. "Beta_short" and/or "beta_long" could potentially be negative. I assume the intention of the algorithm is to short the "short" securities. However, if it turns out the beta for those securities is negative, then in order to achieve a net zero beta, you would actually need to go long on those "shorts". Consider the case where beta_long = .5 and beta_short = -.5. To get a net 0 beta one would need to go LONG an equal amount of each.

The other problem with negative betas is the absolute value of the hedge_ratio could go above 1. This in turn causes the portfolio leverage to increase above 1. Consider the case where beta_long = .5 and beta_short = -.4. The hedge_ratio then equals -.4 / (-.4 + .5) or -4. The short_leverage would then be -4 and the long_leverage would be 5. The portfolio leverage would then be around 9.

The above issues aren't insurmountable but need to be considered in the algorithm.

Good luck.

Hi sorry I figured out what the noob error was, just need some help trying to fix it. My beta_long (should be 0.87ish) & beta_short(meant to be 1.28ish) calculations are wrong. If I hard code the value for deleverage as 0.68 - (0.87 / 1.28) then it works perfectly. Can anyone help with figure out the beta calc?

    long_ranks = ranks[output['longs']].rank()  
    short_ranks = ranks[output['shorts']].rank()  
    beta_long = output.groupby(long_ranks)['beta'].mean()[True]  
    beta_short = output.groupby(short_ranks)['beta'].mean()[True]  
    deleverage = beta_long/beta_short  

I have also tried these from Luca to calculate beta but is not working because I do not understand the index method.

    beta_long = context.output.loc[context.long_ranks.index]['beta'].mean()  
    beta_short = context.output.loc[context.short_ranks.index]['beta'].mean()  

Here's a function I use, that mixes in SPY, to bring beta close to zero. The assumption is that the stocks in the universe are, on average, highly correlated to SPY, which for a relatively large portfolio of equities, in practice, will be the case. Apparently, using such a "hedging instrument" as SPY is not copacetic for the Q fund, but I haven't heard a cogent argument from them yet as to why it would be problematic, and an all-equity solution would be preferred.

denom = np.sum(np.absolute(a))  
    if denom > 0:  
        a = a/denom  
    allocate(context, data, a)  

def allocate(context, data, desired_port):  
    stocks = context.stocks + [sid(8554)]  
    for stock in context.portfolio.positions.keys():  
        if stock not in stocks:  
            if data.can_trade(stock):  
                order_target_percent(stock,0)  
    pct_ls = np.sum(desired_port)  
    record(pct_ls = pct_ls)  
    scale = 1.0-0.5*abs(pct_ls)  
    m = len(context.stocks) + 1

    weight = np.zeros(m)  
    for i, stock in enumerate(context.stocks):  
        weight[i] = scale*context.leverage*desired_port[i]  
    weight[-1] = -0.5*context.leverage*pct_ls  
    for i, stock in enumerate(stocks):  
            if data.can_trade(stock):  
                order_target_percent(stock, weight[i])  

There are a couple of things wrong with your leveraging, but first want to clarify what you are trying to accomplish. Typically, one tries to create a portfolio which has a net beta of zero. However, you could strive for a net beta of any number. If the market is going up maybe try for a positive beta. If the market is going down then maybe a negative one. The Quantopian contest is looking for betas less than .3. Maybe that would be a good number?

So, starting with some basic math, how to calculate the net beta?

Consider two securities, ABC and XYZ, with betas of .5 and 1.5 respectively. One holds $1000 of each so equal dollar values of each. The net beta, beta_net, equals (.5 + 1.5) / 2 = 1.

In general, for an arbitrary number of securities, with d1, d2 ... d_n representing the amount (in dollars) held, and beta1, beta2 ... beta_n the betas, then beta_net can be calculated as below. It's really just a weighted average of betas. The weights (the d's) are the dollar amounts and not the quantity of shares.

beta_net  =  ((d1 * beta1) + (d2 * beta2) ... + (d_n * beta_n) ) / (d1 + d2 ... + d_n)

# in Pythonic pseudo-code this looks like

beta_net = d * beta / d.sum

# a convenient Numpy method is "average" which has a "weights" parameter to give a weighted average

beta_net  =  np.average(beta, weights=d)

Now, IF there are equal dollar amounts of each security then d1 = d2 = d_n , so beta_n can be simplified to

beta_n  =   ((d * beta1) + (d * beta2) ... + (d * beta_n)) / (d*n)  
beta_n  =   ( beta1  +  beta2 ... +  beta_n) / n            # which is just the mean or average beta

The formulas that both you and Luca used incorporate the "mean" method. That is fine as long as one later goes on to order equal amounts of each security. However, that is not the way your order logic is written. Your order logic seems to weight each security by it's rank. Therefore you also need to weight each beta by this same amount.

See if that fixes your logic.

Also, I'm not really familiar with the term "deleverage". Ultimately, you don't want to calculate the net beta (as above) but rather calculate how much to weight the long basket of stocks and how much to weight the short basket. This gets back to the question of what you want for the net beta. So, let's assume one has a dollar amount of long stocks and call that "dollars_long". What one wants to know is how much of the short stocks to purchase so the net beta of the portfolio is some value (call it beta_net). It's really just the equation above but instead of solving for beta_net one solves for the dollars_short.

# starting with the equation for beta_net and two positions (long and short)  
beta_net  =  ((dollars_long * beta_long) + (dollars_short * beta_short)) / (dollars_long + dollars_short)  
(dollars_long + dollars_short) * beta_net = (dollars_long * beta_long) + (dollars_short * beta_short)
dollars_short * beta_net  =   (dollars_long * beta_long) + (dollars_short * beta_short) - (dollars_long * beta_net)  
dollars_short * beta_net  =   ((dollars_long * (beta_long - beta_net)) + (dollars_short * beta_short)  
(dollars_short * beta_net)  -  (dollars_short * beta_short) =   dollars_long * (beta_long - beta_net) 
((dollars_short * (beta_net - beta_short))  =  dollars_long * (beta_long - beta_net)  
dollars_short  =  (dollars_long * (beta_long - beta_net)) /  (beta_net - beta_short)

# so finally.... move the negative sign out front and we solve the amount of dollars short  
# notice the negative sign implies "sell" or "short" this amount

dollars_short  =  -(dollars_long * (beta_net - beta_long)) /  (beta_net - beta_short)   

# we can check this with $1000 of longs with a beta of .5 and shorts with a beta of 1.5  
# How much of the shorts to purchase to net a zero beta?

dollars_short  =  -1000 * (0 - .5)/(0 - 1.5)  = -(1000*-.5)/ (-1.5)  = -333.3   

# Put this value into the original equation for beta_net to double check. We do indeed get 0 beta if we short $333.3 of the shorts.

beta_net  =  ((1000 * .5)  +  (-333.3 * 1.5)) / (1000 -333.3)  
beta_net  =  (500 - 500) / -666.6 = 0

I won't go into the math here, but if you are trying to get a zero beta, a more useful number may be the "hedge_ratio" or the percentage of the portfolio which should be allocated to the "hedge". In this case the "shorts" are our hedge. This only works when you are trying for a zero beta. A non-zero beta has a little different equation.

hedge_ratio  =  beta_long / (beta_short + beta_long)  
short_percentage  = -hedge_ratio     # negative because we want to go short these securities  
long_percentage   =  1-hedge_ratio  # will be positive and therefore long if both betas are positive

Make sure you define what net beta you are trying to achieve and then calculate the correct values for the amounts of longs and shorts. Also, as I mentioned in a previous post, maybe ensure that both the long and short betas are positive.

Maybe something like this

# output is a pandas dataframe returned from the pipeline. It has columns named "beta", "longs", "shorts", and "rank"  
# I assume the "longs" and "shorts" columns are boolean.  
# There are many ways to index and slice a dataframe, but one convenient one is the query method.  
# Both of the below statements are equivalent since "longs" is bolean.  
long  = output.query('longs')  
long  = output.query('longs == True')

# long is a dataframe with the same columns as output but only the rows where the column "longs" is True  
# The betas can be accessed as follows. The result is a Pandas series type.  
long['beta']

# Pandas series can be multiplied as long as the lengths are the same and then returns another Pandas series.  
# Pandas series also have a convenient "sum" method.  
# Putting this together we can get the average beta weighted by rank  
beta_long  =  (long['beta'] * long['rank']).sum()  /  long['rank'].sum

# Another (perhaps faster) method is to use the numpy "average" method (remember to import numpy)  
beta_long  =  numpy.average(long['beta'], weights = long['rank'])

I'd finally make a separate pandas series which has all the weights. Assuming one wants to weight based upon rank.

long_weights = output.query('longs')['rank'] / output.query('longs')['rank'].sum()

The order logic could then be something like this

short_percentage  =  beta_long / (beta_short + beta_long)    #  This should be positive. Remember to order a negative amount.  
long_percentage  = 1 - short_percentage

for stock in long_weights:  
        if data.can_trade(stock):  
            order_target_percent(stock, context.leverage * long_percentage * long_weight[stock])

for stock in short_weights:  
        if data.can_trade(stock):  
            order_target_percent(stock, context.leverage * short_percentage * -short_weight[stock])

Hope this helps? I didn't actually run any of the code or double check my math, so please excuse any mistakes and let me know.

Hi Dan,

The idea is that this tells me how long/short the portfolio will be:

pct_ls = np.sum(desired_port)  

For example, for equal sums of weights, long and short, pct_ls = 0, then no SPY is needed. If pct_ls = 1.0, then an SPY weight of -1.0 is needed (and vice versa), times a factor of 0.5 to normalize the leverage.

You are correct, though, that it is a hack. For the algo I'm working on, it keeps beta well within the +/- 0.3 range, and I don't have to mess around a more complicated approach. Thanks for the input. It might not be that difficult to incorporate the actual betas, so that at each portfolio update, beta ~ 0. I'll think about it.

Grant

Good input.Simplest is often best. I've tried an algorithm with weighted average betas like above but it leads to a lot of complexity. The devil is in the details. The case of daily trading that doesn't also just use a simple "rebalance everything at one time" approach is problematic. Unless I rebalanced everything, I ran into issues with how to really size a single new position. Additionally, negative betas, and how they impacted the average, created problems.

Moreover, as they say, "past performance is no indication of the future". I had some portfolios which had net zero betas (based upon past price history) which nonetheless moved up and down with the market anyway. The 250 day betas were much different than the 5 or 10 day betas.

Thanks for your in-depth reply Dan. I am trying to implement this at the most basic level possible. It is correct that I am equal-weighting the securities & hence simply using 'mean()' should be fine. I am trying to neutralise beta, beta=0. & am using a decent chunk of correlated S&P500 securities so both long & short betas will be positive. Since I am short high-beta securities & long low-beta securities, in this case long portfolio X, has beta = 0.87, and short portfolio Y, has beta 1.28, I am 'deleveraging' or allocating less to the short portfolio (beta 1.28) so that it equals the beta of the long portfolio (beta 0.87). In this particular case, I have cheated & know the beta_long & beta_short through the backtesting, so know that the leverage for the short portfolio should = 0.68 & the leverage for the long portfolio = 1. This makes the beta of the combined portfolio = 0; (long weights * long beta) -> 1 * 0.87 = 0.68 * -1.28 <- (short weights adjusted for deleverage * short portfolio beta). This makes combined portfolio beta = 0.

Very simple & basic logic here, I have tested the logic by hard-coding the short leverage as 0.68 (as I know what the beta_long & beta_short are meant to be from backtesting) & it indeed makes beta close to 0. My problems lie with the simple python calculation of the portfolio betas.

    beta_long = output.groupby('longs')['beta'].mean()[True]  
    beta_short = output.groupby('shorts')['beta'].mean()[True]  

The above does not seem to return the correct portfolio betas & I don't know where its wrong. I cant copy paste this & check in a notebook as the code is different & too difficult for me to decipher how to replicate it. I think the dataframe coming from below isn't quite how I expect it:

output = pipeline_output('ranking_example')  
ranks = output['factor']  
long_ranks = ranks[output['longs']].rank()  
short_ranks = ranks[output['shorts']].rank()

I have attached the latest backtest where you can see I have hard-coded the leverage in the line below:

    context.short_weights = (short_ranks / short_ranks.sum())*0.68  

I would like to replace 0.68 with 'deleverage' & be able to achieve the same output after beta is properly calculated under beta_long & beta_short.

Thanks

Forgot to attach backtest

Ok so only just found out you can actually print stuff in the algorithm which is a great help to me.... Here are the results of my print:

beta_long: -0.0538497417778
beta_short: 5.43614342964
deleverage: -0.00990587214535

So for now back to the drawing board & Dan's first comment.

Lex

I saw those beta numbers last night and glad you noticed them too. Basically it calls into question the ability to construct a beta zero portfolio based upon past betas.

Attached is a backtest that I changed 4 things:

1. line 109 removed the .68 "fudge factor"
   
2. lines 114-118 added logic to calculate weighted average for betas
   
3. lines 121-123 added logging for beta variables
   
4. line 165 commented out the short ordering (so only long orders)

Look at the log and it shows betas similar to what you posted. Primarily note that beta long is about zero and the weighted beta long is a negative .3.

With the short ordering commented out, the algorithm only buys the long picks. The premise is that the resulting "long only" portfolio will have a beta similar to the historical beta of the long stocks (about zero or about -.3 depending upon the calculation). However, the portfolio beta turns out to be .87. I didn't try it, but commenting out the long buys and going only short will probably yield similar disagreement with historical betas. One can also play with the beta regression length but I've found this doesn't improve things much.

My thoughts are that past betas don't ensure similar future betas. One may get a little better correlation using randomly picked stocks, but the effect seems exacerbated using "good picks". What I mean is our algorithms start by picking stocks that we feel will over or under perform the market. They are stocks that are changing. They are outliers. We expect the future prices of these stocks to be different from the past prices. It shouldn't be a surprise then that their future betas are different from their past betas?