Quantopian Lecture Series: Kalman Filters

http://www.thealgoengineer.com/

Jul 21, 2015

A big thanks to one of our own users, Dr Aidan O'Mahony, for allowing us to use his Kalman Filter example. You can find more on his blog.

Disclaimer

Matthieu Lestel

Nice intro on Kalman filters. One little point on the financial side though, you can use prices to estimate the hedge ratio in a cointegration relationship but you should use returns to estimate alpha and beta to avoid spurious relationships.

You're completely correct, Matthieu. We have a note near the end explaining that you'd generally want to use returns, but perhaps we should make that clearer. We used prices because the pictures are clearer due to the increased range.

Disclaimer

Jenny Nitishinskaya

I found these two sites incredibly helpful for figuring out which variables in the Kalman filter do what.

Disclaimer

David Edwards

Here is the companion algorithm for the Kalman filter talk. It's an implementation of the pair trade from this post that uses Kalman filters to smooth the prices and calculate the spread between the two stocks.

Deleted User

Jul 30, 2015

thank you very much for the lecture. It was very instructive ;). From France :)

Jul 30, 2015

Very glad you enjoyed it, Tao. Please don't hesitate to share any feedback.

Disclaimer

JOHN CHAN

Aug 1, 2015

How do read the heat map does 2012 to 2013.. is a good year.. and 2014 is a bad year..?? from blue to red?? which one.. indicates overheating..??

Aug 5, 2015

The heat map doesn't say anything about the quality of the year. Rather it just indicates how the relationship between two securities, and therefore regression coefficients, changes over time. There is a distinct movement through the time space as indicated by color. The point here is that you have to be careful when taking a regression, as at any given time the actual underlying conditions may be changing.

Disclaimer

JOHN CHAN

Aug 5, 2015

Thanks Delany for clarifying things..

Bharath Rao

Aug 7, 2015

What do the valleys in the equity curve represent? Times when Kalman got the underlying state completely wrong?

Aug 7, 2015

Hi Bharath, are you referring to the algorithm returns or something in the notebook?

Disclaimer

Dash Boyer-Olson

Sep 1, 2015

Hi, did you put out the link to the video? Sorry if I'm missing it.

James Christopher

Sep 1, 2015

@Dash

Check out the Quantopian Lectures page it is a living collection of all of the Quantopian Lectures Series material including notebooks, backtests, and videos.

Disclaimer

Sep 21, 2015

http://nbviewer.ipython.org/format/slides/url/alphamaximus.com/assets/notebooks/kalman_filters.ipynb#/ -- looks interesting
http://arxiv.org/abs/1509.04072 -- read this on the train this morning, looks good, but I haven't worked through the maths for the actual feature transformation yet.

https://www.quantopian.com/posts/quantopian-lecture-series-this-time-youre-more-wrong?c=1

Sep 21, 2015

Also, for future readers of this thread, David shared an algo which generalizes the Kalman-based pairs trading to multiple pairs, in this thread:

I was working through the Kalman Filter notebook and I noticed in the Linear Regression portion if you regress SPY over SPY you dont get a Beta of 1 throughout and it shows that there is alpha to be gained.

It should approach 1 though, over time? It's just an estimator.

Shouldnt Alpha approach zero though? It hangs around 0.96 - 0.99

I am not sure how you got it to do that, for me, alpha approaches 0. I'm attaching my clone.

I am by no means an expert at python, but I am assuming that the +9.916e-1 at the top of the second graph gets added to each y interval in the second plot making it around 0.900.

Your notebook is the same exact as mine. I have attached my notebook. I changed my start and end date and it goes from 2010-2016

Hmm you might be right. I tried to fix this with the advice here: http://stackoverflow.com/questions/3677368/matplotlib-format-axis-offset-values-to-whole-numbers-or-specific-number but matplotlib.ticker is blocked in Quantopian.

I am no expert on kalman filters so I will have to leave that one to the crowd.

Will go ahead and run some simple regression checks tonight, but I imagine it has do due something with the calculation and not the input prices.

how kalman filter works in pictures

Feb 20, 2016

Attempted other pairs of the same securities, and none of them seem to approach a beta of 1 or alpha of 0 when regressed over eachother (for example, tried SPY & SPY, TSLA & TSLA, and IWM & IWM).

Kyle Hegeman

Feb 28, 2016

Miles, the issue is related to the choice of the prior mean and covariance. I'm not sure what the justification is for choosing np.ones((2,2)) as the prior covariance matrix. I found this link very helpful to understand how the filter works. In the attached notebook I ran some experiments to better understand how choice of prior affects convergence.

Feb 28, 2016

Thanks Kyle,

I will definitely look into this as I want to move away from an EMA to something that has inputs that are more appropriate

Mark Segal

Mar 30, 2016

I’m curious if anyone has an opinion as to when you would exactly use a Kalman filter to calculate a moving average. For example in momentum and trend following strategies moving averages are used to smooth a time series to give an indication of a trend or change in momentum. It seems like using a Kalman filter by virtue of giving a closer fit to the actual time series reduces the smoothing effect. In the moving average example in Delaney's original notebook the 90 day MA looks smoother than the 60 day MA which is smoother than the 30 day MA which is smoother than the Kalman estimate of the MA. So applying a Kalman estimate of a moving average to momentum and trend following strategies would in general result in extra unwanted trading. What am I missing?

Mar 31, 2016

A Kalman filter is used to smooth out noise, and the parameters to a Kalman filter can be adjusted to make the smoothing effect more or less severe, just like in a moving average. The trade-off is that a smoother signal will lag the true state of the world, whereas a signal that follows the measurements of the world can be quite noisy. Why some people use Kalman filters is that they are effectively more mathematically generalized moving averages. You can put in other models, and the parameters to control the smoothness actually mean something (measurement error, etc.). In a moving average what does 29 days mean vs 47 days? In practice Kalman filters may be able to get you closer to a non-parametric model. A non-parametic model is one in which you don't have to tweak parameters, which is highly desirable -- parameter tweaking leads to a lot of overfitting. Again, because the parameters to a Kalman filter actually mean something, you may be able to estimate them by say measuring variance on a dataset. This would allow you to not have to tweak and pick a smoothness, like you would have to with a moving average.

Disclaimer

Adithya MR

Apr 14, 2016

One continuing question on using Kalman as a replacement for MAs. I'm not sure what the "correct" values of transition covariance and observation covariance should be. According to me, we must take

observation cov = variance of underlying data over some ballpark lookback window (how long?)
transition cov = variance of MA of underlying data over some ballpark lookback window (how long?)

Without estimating these (and setting both to some default value like 1) will likely defeat the non-parametric nature of the Kalman filter?
Is my understanding correct? And if yes, is there a better way to estimate the observation / transition covariances?

Apr 14, 2016

You're correct, there's no easy answer, and in the end a Kalman filter may end up being just as parametric as a MA. However, with an MA there is no hope of being able to derive the proper parameter, as the window length really means nothing. With a Kalman filter you can at least have some hope of occasionally doing so. There may be a process (less frequent in finance, more frequent in other fields) in which you can estimate observation and transition covariances through another variable rather than the data itself, or maybe even derive theoretical values. The idea is that worst case you're just as parametric as a MA, and have to optimize covariance values, best case you find another way to estimate/derive covariances in a non-parametric fashion. The way in which you would derive the covariances is probably pretty specific to each system, so I'm not sure there's a general way of doing so.

Disclaimer

Peter Ho

May 3, 2016

Can you please help calculate Q(t), the measurement variance prediction, in Python? I'm assuming that Q(t) can be calculated from the state_covs array in the notebook. The square root of Q(t) can then be used to determine entry/exit signals (as described in Ernest Chan's book Algorithmic Trading). Thanks.