Building a Better Beta¶

Today we're announcing the release of SimpleBeta, a new Pipeline API factor that makes it easy to efficiently calculate market beta for all stocks in a trading universe.

In this notebook, we introduce the idea of market beta and show how SimpleBeta improves upon previously-available tools for calculating beta in Pipeline.

Outline¶

• Introduce market beta and explain why we might care about it in the context of a trading algorithm.
• Show how we could previously use RollingLinearRegressionOfReturns to compute market beta for a large universe of assets.
• Show how we can build a faster beta implementation from first principles as a CustomFactor.
• Introduce the new SimpleBeta built-in factor, which is over 130x faster than RollingLinearRegressionOfReturns and can handle missing input data.

What is Beta and Why Do We Care?¶

When stock prices move, they tend to move in sync with the movements of the overall market. Not all stocks exhibit this tendency equally, however: some stocks have historically been more or less sensitive to the movements of the market.

One simple way to measure how much a stock tends to move with the rest of the market is to run a linear regression between the returns of the market and the returns of the stock in question. The slope of such a regression gives us a measure of how strongly the returns of an asset tend to be affected by the returns of the market as a whole.

Since linear regression equations are often written as:

$$ Y \sim \alpha + \beta X$$

(read: Y is distributed as alpha plus beta X) we often refer to the slope of the regression line between an asset's returns and the market's returns as the asset's "market beta".

Let's see how we would calculate beta for AAPL. One simple method is to use scipy.stats.linregress to run a linear regression on daily returns. In practice we often use the price of a broad-market ETF like SPY as a proxy for the return of "the market".

linregress returns a namedtuple containing the slope, intercept, and several other useful statistics about the regression.

This regression tells us that AAPL's beta to SPY was a little over 1.0 for the year of 2010. This shouldn't be terribly surprising: Apple is such a large company that its returns account for a substantial portion of the overall market's return all by itself.

We can visualize the results of a regression using matplotlib and seaborn.

Most stocks display at least some historical tendency to move along with the overall market, but different stocks can display different sensitivities at different times.

United Steel (X), for example, has historically been more sensitive to the market than most stocks.

McDonald's (MCD), on the other hand, has historically been less sensitive to changes in the market as a whole.

Calculating Beta For All Stocks With Pipeline¶

In an algorithm that trades a large number of assets, it's often useful to know each stock's historical beta to the market. One way to try to control an algorithm's market beta, for example, is to keep the beta-weighted exposure of the algorithm's positions close to zero.

For a long time, the Pipeline API has provided a built-in factor named RollingLinearRegressionOfReturns that could be used to calculate simple linear regressions for all assets in the Quantopian database.

The implementation of RollingLinearRegressionOfReturns essentially boils down to something like this:

(The full source for RollingLinearRegressionOfReturns as of the time of this writing can be found in Zipline here.)

This class uses the multiple outputs feature of CustomFactor to simultaneously output all the values computed by scipy.stats.linregress every day.

We can use this to calculate a rolling beta for every asset in Quantopian's database like this:

The computation graph for this pipeline looks like this:

Unfortunately, for RollingLinearRegressionOfReturns, looping over every column of input every day and calling scipy.stats.linregress each time isn't very efficient, especially if we only care about the slope of the regression.

Looping over numpy arrays in pure python is slow, and the implementation of linregress isn't designed with the assumption that it will be called a large number of times, so it does some expensive things like defining a new namedtuple subclass on every invocation.

linregress also calculates an intercept, a p-value, an R-squared value, and a standard error of the estimate (all relatively expensive computations), but those calculations are wasted if all we care about is the slope of the regression.

We can measure roughly how long it takes to calculate betas with RollingLinearRegressionOfReturns using time.time and a bit of cleverness:

The 6 seconds per day translates to about 25 minutes per backtest-year, which means that a two year backtest using RollingLinearRegressionOfReturns to calculate beta for all assets might spend over an hour just calculating betas!

Fortunately for us, we can speed up our beta calculations by writing them ourselves as vectorized numpy operations.

Speeding Up Beta Calculations¶

Let's review the mathematics of linear regression and see if we can find a faster way to implement our beta calculation.

The simplest and most common way to calculate the slope of a simple linear regression for an independent variable $X$ and dependent variable $Y$ is to use the formula:

$$\beta = \frac{Cov*{sample}(X, Y)}{Cov*{sample}(X, X)}$$

where $Cov_{sample}(X, Y)$ is the sample covariance of $X$ and $Y$.

The $_{sample}$ subscript I'm using here is a bit of non-standard notation Covariance is a statistical measure of how much a pair of random variables "move together". Sample covariance is a particular way of estimating the covariance of a distribution from a set of observations.

There are many subtleties to robustly estimating covariance (you can read about some of those subtleties in the Quantopian Lecture Series article on covariance estimation). For now, however, we're just going to focus on re-creating what scipy.stats.linregress does, but faster.

Given observations $x_1, x_2, \ldots, x_N$ drawn from $X$, and observations $y_1, y_2, \ldots, y_N$ drawn from $Y$, the formula for sample covariance is:

$$Cov*{sample}(X, Y) = \frac{1}{N - 1}\sum*{i=1}^{N}{(x_i - \bar{X}))(y_i - \bar{Y}))}$$

where $\bar{X}$ is the sample mean of $x_1, x_2, \ldots x_N$, and $\bar{Y}$ is the sample mean of $y_1, y_2, \ldots y_N$.

Here's one way of thinking about what this formula "says":

For each pair of $(x_i, y_i)$ observations, look at how much $x_i$ is above or below the average value of all the $x$'s and multiply that value by how much $y_i$ is above or below the average value of all the $y$'s.

• If $x_i$ and $y_i$ are both above or both below their average, we'll get a positive value for $(x_i - \bar{X})(y_i - \bar{Y})$.
• If $x_i$ and $y_i$ are on opposited sides of their averages, we'll get a negative value for $(x_i - \bar{X})(y_i - \bar{Y})$.

To calculate sample covariance, we take the average of all of these multiplied mean-deviation samples, except we divide by $N - 1$ instead of $N$ when calculating the mean to correct for a small bias introduced by the fact that we're estimating the population mean and covariance at the same time.

• If above-average $x$ samples tend to correspond to above-average $y$ samples in our data, then we'll end up averaging mostly positive values, so we'll estimate a positive covariance.
• If below-average $x$ samples tend to correspond to above-average $y$ samples, we'll end up averaging mostly negative values, so we'll estimate a negative covariance.
• If there isn't any relationship between $x$'s being above their mean and $y$'s being above their mean, then we'll end up averaging a mix of positive and negative values, so we'll end up with a covariance near zero.

Since we're interested in measuring how the returns of each asset depend on the returns of the market, our independent variable $X$ will be the returns of SPY, and our dependent variable $Y$ will be the returns of the asset whose beta we're calculating.

To get started, lets see if we can speed up our single-stock beta calculation by copying the formula above as closely as possible.

So far so good. We can reproduce scipy's result exactly, and we've gotten a significant speedup by avoiding extra calculations we didn't need.

We can speed up our implementation a bit more by noticing that we're dividing both covariance and spy_variance by $N - 1$ even though the next thing we do is divide covariance by spy_variance. Since the divisions by $N - 1$ are going to end up canceling out, we don't need to bother with them in the first place.

Once we have an efficient single-asset implementation of beta, we can adapt it to efficiently perform a vectorized regression against multiple assets using multi-dimensional arrays and numpy broadcasting.

Let's see how our vectorized beta handles calculating beta for every asset in the QTradableStocksUS.

Now we're starting to get somewhere! If we spend 0.002 seconds calculating betas every trading day, then that means we're only spending about half a second per backtest-year on calculating beta.

Let's see how we would plug our vectorized_beta function into a CustomFactor:

We've gone from spending 6 seconds to calculate betas for a day to spending just over half a second!

We're actually doing even better than it might seem from those numbers: most of that half second was spent loading pricing data, not calculating betas, so our scaling will be much better if we run for a longer period:

Our updated MyBeta factor can calculate betas for a year in less than the time it used to take to calculate one day!

That's more than a 250x speedup!

The SimpleBeta Built-In Factor¶

There are still a few ways that we can improve our MyBeta factor:

• We can add support for regressing against assets other than SPY (for example, against sector ETFs).
• We can add better support for dealing with missing data (for example, if we're calculating a 1 year beta, we might still want to compute a "best-effort" beta for stocks that IPO'ed six months ago using the data we have.)
• We can speed up performance a bit more by playing a few tricks to avoid making unnecessary copies when possible.

Fortunately, we don't need to implement all of those improvements here, because we can just use quantopian.pipeline.factors.SimpleBeta, which implements them for us!

SimpleBeta has two required parameters:

• target, the asset whose returns should be used as the independent regression variable.
• regression_length, an integer indicating the number of days of data that should be used to calculate regressions.

SimpleBeta also accepts an parameter, allowed_missing_percentage, which controls how missing data is handled. By default, SimpleBeta will produce a beta for an asset as long as we have valid returns observations on at least 75% of days. You can tune this behavior by passing a value between 0 and 1 as allowed_missing_percentage. For example, passing 0.1 would mean that only 10% of values are allowed to be missing, while passing 0.9 would mean that 90% of values can be missing.

Here's an example of using SimpleBeta to compute the same 252-day betas we've been calculating, but with robust handling of missing values:

Because SimpleBeta robustly handles missing data, it produces many fewer NaN outputs than our MyBeta factor:

Review and Conclusions¶

In this notebook, we:

• Introduced market beta and explain why we might care about it in the context of a trading algorithm.
• Looked at the RollingLinearRegressionOfReturns Pipeline factor and showed how we could build a much faster vectorized implementation of beta ourselves using numpy.
• Learned about the new SimpleBeta built-in factor.

If you want to learn more about how these improvements were implemented, you can see the source code as of the time of this writing here in Zipline. You can also see the Zipline pull request here.

Future Work¶

There are many different ways to estimate beta. In this notebook, we focused on a the simple case of calculating beta by running an OLS regression on daily returns observations, but there are many more sophisticated estimation methods we might consider in the future.

It's common, for example, to use a Shrinkage Estimator for beta, which allows us to combine a "raw" beta estimate with a prior belief about how we expect stocks to behave. It's also common to use longer time intervals for returns (e.g., weekly rather than daily returns), to try to smooth out daily fluctuations.