Very useful, thanks for sharing!

Personally, I use the moving/trailing window for my dynamic updating of the OLS beta coefficient, as you mention yourself as an alternative solution. Have you been able to determine whether the Kalman Filter approach in general performs better?

I've found that moving or rolling linear regression windows can give large oscillating values of the regression coefficients which is not very desirable. You can see this clearly in the attached notebook even with a 100 day window length. I've also found that it is more likely to break down out of sample than the Kalman filter method.

Thank you, Aidan.

On a different note, does anyone know when the slides of all presentations at QuantCon Singapore will be emailed to the participants ? Thank you.

Hi Colin,

Not sure about this one, I already got an email with the slides. I suggest contacting Kelly at Quantopian as she organised the conference.

@Aidan: This is a great notebook with a very clear example that demonstrates the value of this approach. Have you thought about using returns instead of prices? That would reduce a lot of the non-stationarity the Kalman filter is trying to capture.

On the flip side, I think prices are usually used for pair trading as the economic hypothesis is that the value of the two companies will converge to some stationary ratio. I'm not as clear as how this would be reflected in the returns, as they would be one level removed from that hypothesis. Basically care would have to be taken to ensure that you were still testing the same hypothesis and not a subtly different one.

Thanks for the article. Great code as well. Quick question/comment. How does Kalman updating of alpha and beta differ from updating via the plain old recursive OLS equations ? Does not Kalman approach need a guesstimates of observation noise properties, etc.?

@Thomas: As Delaney pointed out the economic hypothesis is that a linear relationship between the prices or log prices is stationary and this is used to exploit the relative mispricing of the assets. However, I think it might be possible to do this with returns using arbitrage pricing theory.

@aby: I've never implemented a recursive OLS but if I remember right it's a very similar implementation to the Kalman filter. The Kalman filter requires the user to input the transition and observation variance / covariance matrices.

to calculate spread beta, i got the error:

what's wrong and how to solve it?

KeyError Traceback (most recent call last)
in ()
----> 1 spread_kf = data_all[secs[1]] - data_all[secs[0]] * beta_kf['beta'] - beta_kf['alpha']

/usr/local/lib/python2.7/dist-packages/pandas/core/frame.pyc in getitem(self, key) 1995 return self.getitem_multilevel(key)
1996 else:
-> 1997 return self.getitem_column(key)
1998
1999 def _getitem_column(self, key):

/usr/local/lib/python2.7/dist-packages/pandas/core/frame.pyc in getitem_column(self, key) 2002 # get column
2003 if self.columns.is_unique:
-> 2004 return self.get_item_cache(key)
2005
2006 # duplicate columns & possible reduce dimensionality

/usr/local/lib/python2.7/dist-packages/pandas/core/generic.pyc in get_item_cache(self, item) 1348 res = cache.get(item)
1349 if res is None:
-> 1350 values = self.data.get(item)
1351 res = self._box_item_values(item, values)
1352 cache[item] = res

/usr/local/lib/python2.7/dist-packages/pandas/core/internals.pyc in get(self, item, fastpath) 3288
3289 if not isnull(item):
-> 3290 loc = self.items.get_loc(item)
3291 else:
3292 indexer = np.arange(len(self.items))[isnull(self.items)]

/usr/local/lib/python2.7/dist-packages/pandas/indexes/base.pyc in get_loc(self, key, method, tolerance) 1945 return self.engine.get_loc(key)
1946 except KeyError:
-> 1947 return self.engine.get_loc(self.maybecast_indexer(key))
1948
1949 indexer = self.get_indexer([key], method=method, tolerance=tolerance)

pandas/index.pyx in pandas.index.IndexEngine.get_loc (pandas/index.c:4154)()

pandas/index.pyx in pandas.index.IndexEngine.get_loc (pandas/index.c:4018)()

pandas/hashtable.pyx in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:12368)()

pandas/hashtable.pyx in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:12322)()

KeyError: 'beta'

It seems the beta key is missing, try Run -> Run All in the menu.

hi Aidan thx for replying but I tried to Run All, still doesn't work

I see the problem now, you are right the keys are incorrect. To fix it change beta to Slope and alpha to Intercept.

Hi Aidan, thanks for correction, got it!

Applied Aidan's fix to the "PEP - KO Spread with dynamic beta" section.

This is absolutely fabulous, thank you for sharing.
I am wondering regarding cointegration test, is there any thing convenient that I can use to test cointegration for 2+ securities? I know in theory I can use the Johansen test but it doesn't look like that there is anything well developed for that.

The ADF test uses the spread time series as input so you can still use it for 2+ securities. An example would be to construct the spread for n securities using the residual of a linear regression and pass that spread to the ADF test.

The Johansen test is not currently available in a Python library as far as I know. However, there is an open issue to include it in statsmodels package here:
https://github.com/statsmodels/statsmodels/pull/453
https://github.com/statsmodels/statsmodels/issues/448

Yeah in general finding a cointegrated set of securities just means finding a set for which a linear combination of the time series is stationary. I think of it as the time series mutually explain all non-stationarity in each other. We go into this in this lecture.

Finding the right linear combination may be more difficult, but testing a specific linear combination isn't too tricky.

Very cool, I never thought one could show correlation over time in such an elegant manner.

Hi Aidan,

Thanks for your great posts on Kalman filter on Quantopian and your blog. I am trying to replace my multivariate regression with Kalman filter. Is there an equivalent multivariate Kalman filtering library available in Python? Could you please explain the difference between running several Kalman filters on each stock vs. a multivariate Kalman filter? Hope you find time to reply.

Best regards,
Pravin

I don't know if anyone noticed but the Kalman filter approach to regression sets the intercept and beta to same value in the first iteration after fitting. This seems to be an anomaly. Why are the two coefficients same?

Why are the two coefficients same?

This is the initial state of the Kalman filter, which is set by initial_state_mean argument. You can set it to a different value but it shouldn't change the end result very much.

Aidan, Thanks for the great article. I am looking at the trading aspect of this strategy i.e. when the strategy signals the buy or sell, how many shares of the two underlying stocks are bought or sold, how to calculate the P&L for each trade?
Since the spread = Y - BX - Intercept, when a buy is signaled, does that mean that we are buying 1 share of Y and sell B shares of X? When a sell is signaled, does that mean Buy B shares of X and sell 1 share of Y?
Also, is the P&L of each trade = [BX(t+n) - Y(t+n)] - [Y(t) - BX(t)], i.e Sell P&L - Buy P&L ?

What's the best way to trade when the spread (Y-BX) is inversely proportional to (Y-X)? How do you recognize that they are both moving in opposite directions ? Here B is the dynamic beta we calculated using Kalman filter. I'm opening the trade looking at Y-BX spread, but since Y-X is moving in opposite direction, all these trades are losing trades.

Aidan,

I ran this pair through Kalman filter and it is giving some not very good slope estimates.
have I done something wrong?

is it because we are using only 1 Y observation, maybe we should use 5 consecutive Y's as observation

thanks
Gizat

meanwhile will experiment with exponentially weighted least squares regression to estimate slope & intercept