cool, thanks for this example.

The Kalman filter is very powerful once you grasp how it works. If you're doing offline work, check out the pykalman module. The order weighting idea is nice, I hadn't thought of that.

Is there a reason you only rebalance weekly? You can reduce the trading frequency by reducing the transition covariance (delta) and increasing the observation covariance (Ve). This has the effect of slowing down the rate at which the regression coefficients change.

Anyone know if the kalman filter can be extended to more than two securities where the dependent and independent variables are no so clear?

@Simon, it's easy to extend to n-dimensional linear regression by increasing the dimensions of the observation matrix for the Kalman filter (variable x in the algo). This backtest does a regression between SPY and the sector ETF's XLE, XLF and XLI.

Hi Aidan,

First, thank you for your great work. I was going to explore this Kalman Filter algorithm before you posted. You helped me and many people here.

Second, for most pair trading algos, we need to test cointegration of the pair by using mostly either ADF or Johansen test. In this example, EWA-EWC pair is tested with Johansen test and it showed that the pair has good eigenvector. Is there any way that a pair trading algo includes one of these tests and gives us a confirmation if the pair is good or not before it runs?

Third, going a little bit further, is there any way for an algo to search the best pair by using those tests, pick right pair(s) and run with Kalman Filter algo?

Lastly, I'm wondering why EWA-EWC pair model orders same shares for long and short. Does it already include risk adjusted factors? What if I set a pair that has very best eigenvectors but the price ratio and beta is too high? In this case, ordering same shares for long/short does not make any sense. How does Kalman Filter pair model work regarding risk adjustment?

Thank you.

Is there any way that a pair trading algo includes one of these tests and gives us a confirmation if the pair is good or not before it runs?

You could use the adfuller function in the statsmodels package to test for cointegration. In my experience, a good Dickey-Fuller test statistic does not necessarily imply a profitable pair for trading. AFAIK, the Johansen test is not available in any public python library but there is an open issue to include it in statsmodels.

is there any way for an algo to search the best pair by using those tests, pick right pair(s) and run with Kalman Filter algo?

Finding good pairs for trading is a difficult and computationally expensive problem. Have a look at Simon's thread, Grid-searching for profitable cointegrating portfolios, for a good discussion on this.

I'm wondering why EWA-EWC pair model orders same shares for long and short

It doesn't order the same number of shares for long and short. If we are going "long" on the EWA/EWC spread, we buy X amount of shares in EWC and we sell beta * X of shares in EWA. This is a hedged position and we are never long and short on the spread at the same time.

Aidan,

Oooops...Right, it was not same shares.

Thank you. I will look up adfuller function and Simon's thread.

Yes, finding right pairs is very difficult. Even this Kalman filter algo does not work with some other pairs. I guess we need to work more and harder.

Again,thank you for the great work.

Aidan - my point was that even with a Kalman filter, one must decide a priori which variable is the dependent variable and which are independent regressors. In a basket situation, this seems nonsensical?

The Kalman filter works under the assumptions that you have a prediction function good at approximating the behavior of the phenomenon you're trying to track and that the error in the output of that function is normally distributed. In simulations of physical phenomena this works out well, because you know that a missile you're trying to shoot with a laser, for example, will not suddenly turn left 90 degrees. Physical laws limit the possible next states of that missile to a pretty small range of values.

Markets on the other hand, can turn 180 degrees without notice and there is no physical momentum to make tracking them tractable. Maybe I'm missing something, but I don't see why a Kalman filter should do well.

A particle filter would at least improve the situation by removing the assumption about linear behavior. Thoughts?

i read through the description on wikipedia: http://en.wikipedia.org/wiki/Kalman_filter

While I don't consider myself a data scientist, I have done the modeling (programming) for some. Regarding it's apparent success for this example, to me this looks to just be a "black box" who's end result is to average the trends, and give you a predictor of the next timestep. I'm not saying that's all it is, but for those (like me) who don't fully understand the math behind the algorithm, that's what it is.

oh also, you don't need a separate "dependent" variable. just one input stream, that's predicting the next value of itself.

edit: at least that's what the kalman filter as described on wikipedia is about. I didn't look at the implementation that Aidan gave.

@Simon, you're right, this isn't a scientific experiment and it's not obvious which asset to use as the dependent variable. The important question is does it really matter which asset we choose to be dependent? In my own tests, I've found it doesn't make a lot of difference to the end result. As a rule of thumb, I choose the dependent variable that gives the largest value of the cointegration coefficient, which implies the asset with the lower volatility is independent.

@Eric and @Jason, there is reasoning behind using a Kalman filter for this application!

First, let's look at what we are trying to achieve. We want to construct a stationary time-series (spread) from two individually non-stationary time-series (EWA and EWC). Let's assume there exists a parameter (beta or cointegration coefficient) such that the residual from the linear combination of the non-stationary time-series is stationary and we can use the resulting stationary series to generate trading signals. In a perfect world, beta would be constant and our job would be done.

Our problem now is that beta is not constant and changes gradually (assuming the series stay cointegrated) over time. To use a Kalman filter, we need to formulate an observation and transition equation. The observation equation is the linear combination of the non-stationary time-series plus Gaussian white noise. This is really just a linear regression of EWA and EWC. The transition equation assumes that beta is equal the previous beta plus Gaussian white noise. This is a random walk.

Using more complex non-linear models probably won't improve things, as Ernie Chan said on non-linear models "experiences showed me that nonlinear models have mostly been unmitigated disasters in terms of trading profits" and "One is almost certain to overfit a nonlinear model to non-recurring noise".

haha, thanks mate :)

i just got a copy of Ernie Chan's book, so looking forward to reading about this in detail :)

Thanks for posting this. I haven't made it through Ernie's second book yet, but I'm curious why anyone would use this strategy given that it does horribly as soon as you factor in slippage and commissions. Even with 10 times as much capital it still loses money.

I'm new to trading, so it's likely I'm misunderstanding something here. Can someone explain when this strategy (or similar pair trading strategies) would make sense?

Using the current setup of the algo, it trades frequently, almost every day, and the profit from each trade is not sufficient to cover transaction costs. It trades frequently because the Kalman filter responds quickly to changes in the EWA/EWC spread. Reducing the sensitivity of the Kalman filter will decrease trading frequency and only trade larger deviations in the spread, which will increase the expected payoff. To answer your question, it only makes sense to use this strategy, or any other strategy, when the expected payoff of a trade is greater than transaction costs.

Hi Paul,

Here is a complete stab in the dark:

ewashares=context.portfolio.positions[context.ewa].amount

Hello,
I wanted to ask what is the meaning of this line:

if e < 5:  
            record(spread=float(e), Q_upper=float(sqrt_Q), Q_lower=float(-sqrt_Q))  

Why don't you want to plot values where the spread is >=5?

Hi Karel,

I can't remember exactly why I did that. It was something to do with keeping the y-scale on the record graph output small so it's easier to see. It won't have any effect on the results if you remove it.

Yes, I have cloned the algorithm and found out already. It is only there to filter out initial extremes from the graph output. Thanks.

I can see one issue when trying to use Kalman filter for trading - the infinite memory. I just tried to include more samples preceding the backtest start date - it still affects the prediction of beta/alpha even when I use thousands of samples (no convergence). I think for trading application, the actual Kalman filter implementation should be more biased on recent data and be able to "forget" about data from ancient past. Of course, you can set the "pre-train" phase length to some arbitrary value, but to which one?

There are some implementations like this discussed (for instance, search for Adaptive Fading Kalman Filter). Does anyone have experiences with these modifications?

How would I change the algorithm for daily trading?

Is there a reason that orders only take place in the last 5 minutes of the trading day?

Also, when you run a full backtest and look at the transaction details, there are sometimes multiple orders on the same day. For example, on 5-03-06, there were the following order: -1137 EWA, -1125 EWA, +1000 EWC, and +1000 EWC. Why are there multiple orders on the same day?

On 4-26-06, there was only one order: -1000 EWC. Why was there only one order and no corresponding pairs trade, i.e. +EWA?

Thanks for sharing the algorithm and answering questions.

Is there a reason that orders only take place in the last 5 minutes of the trading day?

Ernie Chan's original algo runs on daily close data so to replicate that on Quantopian I used the close price at 3.55PM to update the Kalman filter and allow 5 minutes before market close for any orders to fill.

Thank you very much for your response.

Changing the time that orders take place by 15-30 minutes can significantly change the results of a backtest (for example, changing from 3:55 pm to 3:30 pm). Why does this happen? Is there an ideal time to execute trades? Thanks.

Why does this happen? Is there an ideal time to execute trades?

I'm not certain of the reason but if I had to guess I would say that arbitrage algo's perform better with increasing volatility and, in general, volatility is higher around market open and market close. I'm not sure if there is an "ideal time" to execute trades, it really depends on what you're trying to exploit.

How is the entry and exit points chosen? A short position is opened when e>sqrt(Q). Then we should wait for e to decrease. But the we just exit when e revert to sqrt(Q) again. Why is it?

How come the transaction always start with one leg instead of two legs?

Why does this happen? Is there an ideal time to execute trades?

There is no specific ideal time to execute trades, however your thesis/analysis is based on daily close price data. Therefore, you should aim to minimise slippage against the forthcoming close price on each given day. Executing trades 15 mins before market close still subjects you to the probability of slippage against the close price on that day, but how do you decide the optimal time...maybe 5 mins before the close?..or 2 mins? Setting the execution/order as close as possible towards the close increases the probability that your order may not be filled so you will need to assess the trade off between close price-slippage/probability of execution.

Thanks for sharing Aidan. This has been so helpful along with your Kalman Filter notebook.

I have a question though about the initialization setup you have here. In the notebook, you assumed the measurement error to be 10, or Ve to be 100, but here you set Ve to something dramatically smaller, 0.001. Is there a reason behind such huge discrepancy? This number can obviously change the results dramatically.

When I rerun the numbers, I can get negative value for Q which result in errors when I calculate sqrt(Q). How do you manage negative Qs?

problem solved for only one leg in the beginning.

However, mean reversion quality for bond product is very bad.

Better try on other equity or FX spreads.

If we are using a Kalman filter as in the the example above, would we still need to reblanace any open positions periodically to reflect the position in the latest spread as it is getting updated by the Kalman Filter?

(sorry but i dont seem to be able to see any rebalancing in the algorithm?)

hi thx aidan for the nice post. It certainly helps to enlightened more about Kalman filter application. Just wonder what will be the best practice in using this Kalman filter to find the cointegrated pairs. Will following two approaches both reasonable?

1. Testing cointegration based on simple OLS residual (spread) analysis
   
2. Testing cointegration based on new residual derived from adaptive beta state from Kalman process

I've found option 2 wasn't really appropriate since it makes an assumption on evolving state of beta which might impact the regression residual of the actual price measurement. Let me know what others think as well. Thanks

HI Ridwan,

Agree with your analysis. Approach 1 is reasonable but you'll find most of the time the conitegration relationship will break down and will never revert to the mean. Approach 2 doesn't make a lot of sense as the relationship is constantly evolving.

thx aidan. Guess i will try to combine both analysis (i.e. in sample cointegration analysis based on simple OLS to find the pairs and out sample forecast based on Kalman filter for trading signal). Do you think this approach makes sense?

Are there any leverage used in this algo?

Hi, I'm seeking help here. While I'm understanding Kalman Filter and trying to apply this algorithm in real trades, there are something I don't quite understand.

In the sample algorithm, the kalman filter takes one price for each symbol, and get hedge ratio, mean, standard deviation etc. So that it further decides if we can enter long/short at a moment. However, in real trades, for every tick/bar, there are bid price and short price for every symbol, so they are two values of each symbol, four values for pair trading. Then how should we use the four values to calculate trading signal?

Actually this question is not only about Kalman Filter, but general to any backtest program. Back test data has only one value for each time (which might be OPEN or CLOSE), but in real trades, there are lowest ask, highest bid and other order prices. Which values should we use to decide trading signal?

Appreciate any help!

@Matt Sun
"but in real trades, there are lowest ask, highest bid and other order prices. Which values should we use to decide trading signal?"

You can use the last traded price