The first implementation takes gains out of the portfolio for income, better results may be obtained from reinvesting the gains of each re-calibration period, as seen below.

Someone pointed out to me that I had a serious head start on the benchmark witch is absolutely true. Once the historical data management system is running well this will not be a problem, for now I fixed it by buying into the benchmark while the data is gathered so that the benchmark and my returns start from the same ground.

Hi Wayne,

Here's your most recently posted algorithm, starting in 2011. It trades pretty infrequently and I'm wondering if the return relative to the benchmark might be more due to your mix of securities rather than trading?

Grant

The strategy trades every 126 days (half of a trading year) unless you change the parameters. Outperformance is not necessarily the goal of this portfolio. The minimum variance portfolio is a commonly used tool in asset allocation. The securities that I chose were chosen because they span the assets within the S&P 500 while allowing for diversification across the industry sectors as they are determined by State Street (or possibly by MSCI (GICS) or someone else). This means that when estimating the covariance matrix we have only 9*8/2+9 = 45 parameters to estimate as opposed to doing all 500 constituents of the S&P 500 which would be 500*499/2+500 = 125,250 parameters. With 252 observations estimating 45 parameters gives 5.6 observations per parameter which is statistically rather shaky however estimating 125,250 parameters with 252 observations is... Another approach would be to select the assets which compose some high percentage of the S&P 500 like the top N securities for example and run the same algorithm on those securities. I am not sure how this would be accomplished in quantopian but it could prove quite powerful with a long enough observation window.

The short answer to your question is that this algorithm performs the way it does precisely because of the securities that I have chosen. The point is that this algorithm is all about selecting the weights which maximize diversification benefits across industrial sectors. The algorithm is re-calibrated every six months to account for the fact that we do not have stationary multivariate return distributions; correlations shift and covariance matrices are time varying.

Why the strategy backtest performance will difference that when you modify the ticker/sid order sequence ?

example like the sequence compare to original sequence:

    context.tickers = ['xli','xlk', 'xlb', 'xlp', 'xly', 'xlv','xlf', 'xle', 'xlu']  
    context.sids = [   sid(19657),sid(19658),  
                       sid(19654), sid(19659),  
                       sid(19662), sid(19661),  
                       sid(19656),sid(19655), sid(19660) ]  

I found the context.data.cov() will arrange the ticker by alphabet sequence,
so in following code ,the problem will occur in pimv to match the context.tickers dictionary,
How should we could to avoid the situation ?

pimv = { e:pimv[i,0] for i,e in enumerate(context.tickers) }  

the log for print context.data.cov() :

2005-07-05handle_data:39INFOrecalibrating...
2005-07-05PRINT('context.data.cov()=', '\n', xlb xle xlf xli xlk xlp xlu xlv xly
xlb 0.000129 0.000096 0.000055 0.000074 0.000058 0.000041 0.000052 0.000039 0.000064
xle 0.000096 0.000212 0.000038 0.000056 0.000042 0.000026 0.000057 0.000023 0.000045
xlf 0.000055 0.000038 0.000059 0.000048 0.000045 0.000031 0.000041 0.000032 0.000047
xli 0.000074 0.000056 0.000048 0.000065 0.000053 0.000036 0.000042 0.000036 0.000055
xlk 0.000058 0.000042 0.000045 0.000053 0.000073 0.000035 0.000037 0.000034 0.000054
xlp 0.000041 0.000026 0.000031 0.000036 0.000035 0.000034 0.000026 0.000029 0.000037
xlu 0.000052 0.000057 0.000041 0.000042 0.000037 0.000026 0.000065 0.000023 0.000036
xlv 0.000039 0.000023 0.000032 0.000036 0.000034 0.000029 0.000023 0.000046 0.000034
xly 0.000064 0.000045 0.000047 0.000055 0.000054 0.000037 0.000036 0.000034 0.000062)

UPDATE: I had try add sorted function, not sure this is right or not!
pimv = { e:pimv[i,0] for i,e in enumerate(sorted(context.tickers)) }

my BT :

A couple suggestions
- If you use minute mode you will be able to use the history() function, that way your algo will not have a warm up period while it collects data.
- Get rid of context.tickers all together to make the indexing simpler. If you want the symbols you can get them from the sids in context.sids via sid.symbol.

I did not go through and spot check this, but I believe the indexing should be good again. I got rid of context.tickers to simplify indexing. I also log the symbol when orders are placed so you can see how to to that too.
Hope this is helpful. You should probably try to move over to minute mode at some point too.

EDIT:
The indexing is off still in the example backtest. The simplest way to get it to work is to define context.sids as sorted([list of sids]), then they will have the same ordering. I also included a function for the minimum variance calculation with correct indexing.

def minimum_variance_weights(R, nobs=252, leverage=1.0):  
    '''  
    Minimum Variance Portfolio weights.

    params  
      R: DataFrame of historical returns.  
      nobs: Number of observations.  
      leverage: scalar multiplier for target weights

    returns  
      Series of target weights indexed by columns of R  
    '''  
    R = R.tail(nobs)  
    C_inv = np.linalg.inv(R.cov())  
    ones = np.ones(len(C_inv))  
    v = C_inv.dot(ones)  
    w = v / ones.T.dot(v)  
    return pd.Series(w, index=R.columns) * leverage  

I took the changes made by all above and also added the TLT (bond ETF) to the basket. It really seems to smooth the back test and gives impressive returns with very low draw down over the time period (2002 - 2014). Now I do notice that it appears that the algo is going short, which I was surprised at. Should this be happening? I was under assumption that this was long only.

Going short is normal for a minimum variance portfolio. The equation returns the weights that should minimize the volatility, but it does not distinguish between upside and downside volatility, so it will short in order to reduce the total volatility.

Could this be modified to be constrained to be long only?

It can, but there is no correct way, you could use an optimization routine, or just shift the results. In this backtest, I use the same weights returned from the minimum variance formula, but if any of them are negative, it will add the absolute value of the smallest weight to all of them and normalizes the result. To me, this seems like a simple and intuitive way to turn this into a long only strategy.

Looking at the initial algorithm and the one where you made changes and removed the ticker list - what accounts for the major discrepancy in returns and trades? Is it the issue that was brought up about the ticker/sid order sequence ?

Yes, it was actually not fixed in the last example. When using the dict comprehensions to make a series, the sids get sorted into ascending order, so I made the context.sids a sorted list to avoid indexing error later. I think its good here, but I have not picked through it.

Here's an attempt at a minimum variance long-only algorithm, using scipy.optimize.minimize. Seems to work nicely. --Grant

Over a longer time period. --Grant

Hey guys, I took a shot at minimum variance portfolio myself, although the results are only available in ipython notebook as for now. Perhaps you will find it interesting http://nbviewer.ipython.org/github/Marigold/universal-portfolios/blob/master/modern-portfolio-theory.ipynb.

Here is how it looks if rebalanced monthly after adding TLT to the list.

Mojmir Vinkler, very interesting work book,
Can you or somebody else make it shareable in research platform.