Market Regime detetection using Hidden Markov Model or Anomaly Detection (OneClassSVM)¶
For many algorythmic trading strategies, fixed asset allocation weights and model paramters are static and hence, do not adapt to abrupt market regime changes. I believe many algorithms could improve their overall performance by adjusting certain of their paramters based on the current underlying market regime. For example, an algo could bias its weights toward more short positions during market contractions.
I have searched the web for such techniques and came across one using Hidden Markov State Machine. It is shown here below. The training is done on the SPY as we are trying to determine the broad market regime. It could be applied to specific sectors as well.
The second technique is my own creation. It uses sklearn's OneClassSVM as an unsupervised anomaly detector. The idea is to feed the svm with normalized short time series of the SPY as input features. 20 trading days are used. The OCSVM should hopefully determine which short time series are "abnormal" or "unclustered". We should treat those abnormal time series as a different market regime, one which signals high volatilty and contraction. I think a 1D convnet would work better then a OneClassSVM, but Keras/Tensorflow is not supported. On the otherhand, I would also not be surprised that a convnet overfits.
HMM¶
from sklearn import hmm
import pandas as pd
from matplotlib import pyplot as plt
import numpy as np
from quantopian.research.experimental import history
from matplotlib import cm
df = history(
symbols('SPY'),
fields=['close_price', 'volume'],
frequency='daily',
start='2004-11-19',
end='2018-11-19'
)
rets = df['close_price'].pct_change()[1:]
volume = df['volume'][1:]
dates = df.index[1:]
close_v = volume
model = hmm.GaussianHMM(n_components=2, covariance_type="full", n_iter=1000)
X = np.column_stack([rets, np.log(volume)])
model.fit([X])
hidden_states = model.predict(X)
def plot_in_sample_hidden_states(hmm_model, df):
"""
Plot the adjusted closing prices masked by
the in-sample hidden states as a mechanism
to understand the market regimes.
"""
# Predict the hidden states array
hidden_states = hmm_model.predict(X)
# Create the correctly formatted plot
fig, axs = plt.subplots(
hmm_model.n_components,
sharex=True, sharey=True
)
colours = cm.rainbow(
np.linspace(0, 1, hmm_model.n_components)
)
for i, (ax, colour) in enumerate(zip(axs, colours)):
mask = hidden_states == i
ax.plot_date(
df.index[mask],
df["close_price"][mask],
".", linestyle='none',
c=colour
)
ax.set_title("Hidden State #%s" % i)
ax.grid(True)
plt.show()
HMM results¶
We see that the HMM hidden state #0 seems to capture volatile and downturning market regimes whilst hs #1 represents normal, more profitable regimes.
plot_in_sample_hidden_states(model, df)
One Class SVM anomaly detection¶
data = df['close_price'].tolist()
# build the mini 20 day long time series normalized.
out = []
for i in range(len(data)):
chunk = np.array(data[i-20:i])
try:
out.append(chunk/chunk[0])
except:
out.append(np.ones(20))
X = np.array(out)
from sklearn.svm import OneClassSVM
from sklearn.preprocessing import PolynomialFeatures
# nu represent the expected ratio of outliers. 30% in this case. Gamma is a fitting param.
model = OneClassSVM(kernel='rbf', nu=0.30, gamma=1.1)
# we use poly feat to increase the number of features and try to suss out meaningful relationships.
# Another approach could be to manually engineer features from the data set. example, diff, rolling slopes, etc.
model.fit(PolynomialFeatures(degree=3).fit_transform(X))
y = model.predict(PolynomialFeatures(degree=3).fit_transform(X))
def plot_in_sample_clusters(model, df):
"""
Plot the adjusted closing prices masked by
the in-sample hidden states as a mechanism
to understand the market regimes.
"""
# Predict the hidden states array
hidden_states = model.predict(PolynomialFeatures(degree=3).fit_transform(X))
hs = pd.Series(hidden_states).rolling(window=5).mean().fillna(1)
hs[hs >= 0] = 1
hs[hs < 0] = 0
hidden_states = hs
hidden_states.index = df.index
# Create the correctly formatted plot
fig, axs = plt.subplots(
2,
sharex=True, sharey=True
)
colours = cm.rainbow(
np.linspace(0, 1, 2)
)
for i, (ax, colour) in enumerate(zip(axs, colours)):
mask = hidden_states == i
ax.plot_date(
df.index[mask],
df["close_price"][mask],
".", linestyle='none',
c=colour
)
ax.set_title("Cluster #%s" % i)
ax.grid(True)
plt.show()
Results¶
Similarly to the HMM, it appears that the OCSVM was able to identify market regime changes. Of course, this is all insample.
plot_in_sample_clusters(model, df)
