Hello James (and Thomas),

It might be helpful to get a better feel for the technical details here. There is a reference given on http://scikit-learn.org/stable/modules/generated/sklearn.cluster.AffinityPropagation.html#sklearn.cluster.AffinityPropagation:

Brendan J. Frey and Delbert Dueck, “Clustering by Passing Messages Between Data Points”, Science Feb. 2007

A Google search on the title yields numerous hits with PDF links to the paper (e.g. www.psi.toronto.edu/affinitypropagation/FreyDueckScience07.pdf).

One thought is to get a feel for how clustering changes versus time. For example, if the list of securities in a cluster is stable versus time and then one of the securities (e.g. XYZ) drops out of the cluster, perhaps it would be a signal that XYZ should be bought/sold?

Another thought is that perhaps the algorithm could be useful in constructing and periodically re-balancing a diversified portfolio?

@James, @Grant, what about something simple, like a linear regression between pairs in a cluster?

Fawce & James,

The graph on http://scikit-learn.org/stable/auto_examples/applications/plot_stock_market.html shows a distinct cluster of big defense contractor stocks (Boeing, Raytheon, General Dynamics, Lockheed Martin, and Northrop Grumman). When I plot them in Google finance here along with the S&P 500, they definitely appear to be marching lock-step to one another (but also to the S&P 500). Might be a good test case to understand what's going on.

Regarding picking pairs, I think in this case, it might be more appropriate to do some form of multiple regression on the whole cluster.

We renamed open and close to open_price and close_price. Attached is the new backtest (yep, you can respond with attached backtests now!).

Hello James/Thomas/Fawce,

Was anyone able to sort out an application of this clustering technique to trading?

Grant

Hi Grant,

Not yet. I think this will be really useful once universe is available (currently heavy underway) to select stocks with the potential of cointegration. On a related note, there is a really cool paper on checking for cointegration in a Bayesian framework which, as they show, as a lot of benefits:
http://web4.cs.ucl.ac.uk/staff/C.Bracegirdle/BayesianConditionalCointegration.php

Thanks Thomas,

Nothing like some light bedtime reading.

Grant

Hello Thomas,

I'll work my way through the paper to see what I can learn (managed to get it on my Kindle with K2pdfopt). Some questions:

1. Why would one expect the clustering technique highlighted above to be useful in identifying pairs?
   
2. More generally, I've been wondering what's so special about pairs? It seems the same approach could be applied to N securities (where N > 2). Have you seen more general formulations of "pairs" trading?
   
3. Is Quantopian restricted from making specific trading recommendations (e.g. providing a list of potential pairs trading candidates)? It seems that you guys should have free-wheeling access to the data and could be running screening algorithms and then provide guidance to users, right? There must be standard algorithms out there (pairs trading has been around since the 80s [ref.]). Or would this be akin to offering specific investment advice (albeit free)?
   
4. Given that pairs trading is well-known, wouldn't the big guys be constantly screening for opportunities and cause them to evaporate relatively quickly? Intuitively, it seems like pairs trading might be a dead end?

Thomas,

I did a first quick read of the paper http://web4.cs.ucl.ac.uk/staff/C.Bracegirdle/BayesianConditionalCointegration.php. Lots of math, but I figure that the algorithm can't be all that complicated. If you have an outline of what the code would look like, please let me know.