Hi @Leo,

in backtest periods for short durations the beta is much higher than the bounds I set in the order_optimal_portfolio

Which "beta" are you looking at in backtest? Can you post a pyfolio tearsheet?

Hi @jonathan,

I'm getting error pyfolio tearsheet "TearSheetNotAvailable: Algorithm with id '...' lacks sufficient data to create a tear sheet. Please make sure it has run for at least 6 months before creating tear sheets". I am looking at the regular "Run full backtest" from algorithm page, just for different time periods.
From a beta of -1.06 in the last two weeks, I went down to beta of -0.25 in last six weeks and further down to -0.04 over last two years. I am wondering if the wide changes are a result of the algorithm entering positions on day 1 that get averaged out over longer time periods.

From 2017-07-20 to 2017-08-03 with $10,000,000 initial capital

Beta
-1.06

Same algo (no code change)

From 2017-06-20 to 2017-08-03 with $10,000,000 initial capital

Beta
-0.25

Seems to get really low over a 2 year backtest and in general all multi year backtests.
From 2015-06-01 to 2017-07-19 with $10,000,000 initial capital

Beta
-0.04

My constraint is as follows:
MAX_BETA_EXPOSURE = 0.25

beta_default = 0.66*RollingLinearRegressionOfReturns(
target=sid(8554),
returns_length=5,
regression_length=260,
mask=long_short_screen_default
).beta + 0.33*1.0

pipe = Pipeline(columns = {  

....
'market_beta_default':beta_default,
....
}

risk_factor_exposures = pd.DataFrame({
'market_beta_default':pipeline_data.market_beta_default.fillna(1.0)
})
neutralize_risk_factors = opt.WeightedExposure(
loadings=risk_factor_exposures,
min_exposures={'market_beta_default':-MAX_BETA_EXPOSURE},
max_exposures={'market_beta_default':MAX_BETA_EXPOSURE}
)

@Leo, a stock's beta is an estimated value. In the pipeline expression, you are calculating a regression beta per stock. This is the historical beta of the stock. What we really care about, of course, is the future actual beta of the stock. The weighting of 2/3 and 1/3 is called Blume's Adjustment from Betas and Their Regression Tendencies. It basically posits that the historical regression is noisy and tends to overestimate or underestimate beta; over time betas should revert to 1.0. A more modern interpretation is that we have a prior belief that betas are all "1.0" and we shrink the regression estimate toward this value.

In your optimization constraints, you are constraining the portfolio to a weighted sum of this predicted beta. The reality could be something different.

The beta in the backtester is a regression of your algo's return stream versus the SPY. You can think of this as what actually happened. Your portfolio is constrained in the optimizer to what you are predicting the betas will be, and what actually happened in the future is what is reported as the beta in the backtester (and pyfolio). However, this backtester regression beta is also an estimate. Like any regression coefficient, you need a sufficiently long time series to get a good estimate. This reported beta is an estimated value and has a standard error. In your examples, you can see that with a very short period of time you get a very noisy estimate, but that the estimates converge to a reasonable value as you add more time.

Thanks a lot @jonathan for the detailed explanation.