Since this week's topic is more about what not to do I decided to create an algorithm in response to a question during the last lecture on Kalman filtering.

I was asked, "where would you go next with this algorithm?" My response was that I would factor the entire thing into a class, that way I can create multiple instances of it and trade several pairs at once. That's what I have done here, this algorithm trades 6 pairs at once using a class nearly the same as Kalman filtering algo from last time. The pairs are not special, I just selected two companies from the same industry for 6 industries.

Please "clone and tweak" this algorithm, this is only meant to be a scaffolding for you to build upon, you still need to add your own twist.

love these series Guys! Always enjoy looking at your code David, very clean and well structured.

Thanks Ted, I appreciate the positive feedback!

The recordings from this lecture are ready. You can view the full lecture or check out the 3 separate topics below:

• Violation of Regression Model Assumptions
  
• Instability of Regression Coefficients
  
• Dangers of Model Misspecifiation

Does anyone understand the relationship between autocorrelation and non-stationary?

Is it necessary to use both sm.tsa.stattools.acf and sm.tsa.stattools.adfuller?

Answer

Great link, Jonathan. I found it myself while researching the topic. Wikipedia, as usual, also has a good explanation of non-stationarity. The way I think about it is that non-stationarity is describing whether or not a data generating process changes over time. Autocorrelation is describing one aspect of how that change occurs. This is a very simplified and hand-wavy definition of course, so I hope I haven't confused anyone further.

One thing to note is that many many series in finance are auto-correlated. Prices and returns, especially, are influenced by the recent prices and returns. It's important to at least check to see how auto-correlated a series is before trying to model future values.

David's multiple-pairs Kalman-filter-based algo deserves more attention, this is good stuff.

I agree Simon, I liked making this one, there's a few gems in it from an implementation perspective. The pattern of factoring a strategy into a class and creating instances of it is really useful. I find it easier to track several things when each instance is doing its own bookkeeping.

Kalman filtering is pretty cool, I hadn't really played with the idea much before the lecture series, but I've been looking for excuses to use them since.

Thanks for sharing that algo.

Looking over the source code, isn't it overkill to feed in filtered average prices into the Kalman filter?

Can't you replicate the same thing by just adjusting the observation_covariance?

Hey Jonathan, just linking to your other Kalman filters post here so that others reading through the thread can reference both discussions.

Anybody looking for a discussion purely about Kalman filters and their usage should see here. Also, the original Kalman filters lecture thread, available as part of the Quantopian Lecture Series.

www.quantopian.com/lectures

Hi Johnathan,
It could very well be overkill, I'm not sure really. This algo was a demo of the concepts in the Kalman filtering lecture, which discussed using it for moving averages and regression. I wanted to make sure there was code for both of those concepts in there so I filtered the prices also.

David

As an example for Kalman filters, that makes sense. Thanks!

Hello, I've just cloned the algorithm of David above, but it's not working anymore. I guess because of the new API. I've tried to changed it, so that it works again:
I changed sid in asset in line 82. So now the algorirthm is working again, but the results are different ( I got a max drawdown of 8.8%. Any idea how to fix it?

I think that some strategies will have slightly different results under the new API, and it's actually more accurate. We fixed a bunch of bugs and infidelities with the upgrade, so the new results should be treated as the correct ones now. We pulled the Kalman algorithm from the lectures page for now, as we're in the middle of building a ton of other lecture content. At some point we may upload a new version, but yours seems to be working for now.

...we're in the middle of building a ton of other lecture content...

Good news! They are excellent, I enjoyed all of them. Thanks.

I really like the economic thesis behind trading pairs, so I cloned this algorithm and have been experimenting with finding additional pairs, tweaking the algorithm, etc. In doing so, I uncovered something that doesn't seem quite right to me, and I would like to get the forum's thoughts on it. I was running the algorithm with just one single pair to see results, and I realized that if I switched the order of the securities (put security A in the _x position and security B in the _y position, then run again with A in the _y position and B in the _x position), the results are different.

From my understanding of the "idea" behind this algorithm, I believe the results should be the same regardless of how the 2 securities in the pair are ordered. I'm not quite at the level yet where I am able to debug why something like this would be happening. Would anyone care to take a look and see if they can explain to me why this should be happening or help change the algorithm so this no longer happens? Also, I have had trouble getting it updated so that the 3 instances of deprecated code are removed despite several attempts.

I will attempt to share 2 backtests that are identical except for the ordering of the pairs, and you can see that the outcomes are different.

Thanks for anyone who might be able to give this a shot,

Joseph

Follow-up: I give up for now, but will try to attach the backtests again later. Take my word for it - try cloning the earlier backtest, picking one pair of securities, run it, reverse the order of the securities and run it again... Sorry I can't get the backtests to attach right now!

Joseph

Here is one with F and GM...

And here is another one with F and GM, just in the reverse order, which I would have thought wouldn't have changed the overall results.

Hey Jason,

We're incredibly busy with QuantCon SG right now, but will take a look at this ASAP.

Thanks,
Delaney

Thank you Delaney! I really appreciate it!

Joseph

Hi Joseph,

I suspect that the issue is due to the calculation of the hedge_ratio function. Since it calculates linear regression parameters and the intercept term is going to be different, you end up with different (and non-symmetric) values for the beta, or "hedge ratio". This will result in different calculations depending on which equity in the pair is the independent variable and which is the dependent. Unfortunately, I haven't had a lot of time to explore it to figure out the root cause, so this is just my suspicion.

An alternative to doing the linear regression might be to try computing the ratio of the stocks instead of the linear combination and using that as the trading signal. So your hedge ratio would instead be calculating Y/X or X/Y (where X and Y are the elements of your pair) and you would be checking the z-score of a rolling calculation of that ratio as your trading signal. Your ordering logic will probably be more annoying than the linear combination case because you have to calculate the correct number of stocks to approximate that ratio, but I suspect that it will be more likely to be symmetric.

Cheers,
Max

Hi Joseph and Max,
The hedge ratio found by a linear regression will vary when the X/Y variables are switched because of the error calculation. Ordinary least squares is the most common method, which assumes that the error is in the output variable and optimizes only to minimize the residuals between Y and the predicted value. If total least squares is chosen for the error function the residual will be calculated assuming that both the X and Y contain errors. When using total least squares the hedge ratio should not change (other than by inverting) when the order is swapped.
I haven't dug into pyman filters deep enough to determine if this is why this is happening in this particular case. Intuitively I wouldn't think so, since the kalman filter was designed to make predictions with the assumption that the inputs contain error. Maybe someone with more knowledge on kalman filters can comment.

In order to implement a Kalman filter you need to supply a model of the system in question. In this case, the system is modelled as a simple linear regression between two securities. It is assumed in the Kalman filter that the system is noisy and the hidden state (regression coefficients or hedge ratio here) of the system can't be observed directly. In addition to this the observations, which is the dependent variable in the linear regression, is assumed to contain noise.

Since we are assuming that the model of our system is a simple linear regression the results will change depending on which security you choose to be the dependent variable. It may be possible to model the system in the Kalamn filter as a total least squares regression which would remove this choice but I have never looked into it.

Thanks, Aidan. For those of you who don't know, Aidan recently spoke at QuantCon SG and presented on precisely this. It was a great talk and whereas we didn't record it, he did share his materials here.

https://www.quantopian.com/posts/quantcon-2016-using-the-kalman-filter-in-algorithmic-trading

Nice