Thanks for the well-written algorithm with comprehensive comments!

Hi Naoki,

Thank you very much for sharing this, it looks really nice.
I was wondering though: How is it possible that the script chooses to keep the cash and not buy any ETF's?
The bins only consider the money to be put in at least one of the ETF's, not none, am I right?

Also, just as a remark, there is a numpy function np.nan_to_num(arr), doing exactly what it implies.

At least one ETF get the money, and in all such C(10,8) portfolios all of the funds are invested. But when taking the average of portfolio holdings weighted by the return of each portfolio, return happened to be negative across most portfolios. The script decides not to buy any ETFs if all holdings turns negative in this way.

Thanks for the tip on nan_to_num!

That makes sense..

Something else I was wondering; The bins you are generating are basically just linear combinations of the identity matrix (all money in one of the securities)
Why does the outcome differ (it does) when you consider all these bins separately, and not just using the identity matrix as 'bins'?

Even turning off the 'clipping' operations such as zeroing negative weights and disregarding trades under 1000, there still is a difference between the two..

The result from portfolios with all money in each security, and result of all possible portfolio is different, because we rebalance it daily. Return of average of 100% stock and 100% bond is different from the return of 50% bond/50% stock if you rebalance it (if you don't rebalance at all, it should be the same).

In terms of algebra, we are taking S = X B where matrix X (252 days of return * m securities) dotted by matrix B (m securities * all possible portfolios), which will produce matrix S (252 days of return * all possible portfolio). So using B = all portfolios or B = I (identity matrix) will produce completely different S.

Thank you for sharing. It's very interesting this script holds cash during downturn. The high beta and return are mainly due to leverage. Can you elaborate why you choose leverage of 2?

@Joe Black

I dig this strategy a little bit. There is an explanation here[1]. In general, this strategy does not perform well when the volatility of the underlying securities is low. Hence we need to improve the performance of this strategy by amplifying the volatility with a leverage. You could try a leverage of 10, that has a lot a fun :)

[1]http://epchan.blogspot.sg/2007/01/universal-portfolios.html

I did some light reading on the algorithm and it seems to be a bit slow in reaching the optimal return with hindsight. Very cool nonetheless. Doubly nice that it holds cash on a downturn.

Hi Joe, I chose leverage of 2 because of more attractive return, with still reasonable drawdown.

I reviewed the paper again and realized I misinterpreted one thing.

In line 94, after calculating returns for all portfolio, the script subtract 1 (e.g. 1% loss is -0.01). Then it takes the weighted average of all portfolio. However, what the paper meant was not subtracting 1 (e.g. 1% loss is 0.99) and taking the weighted average.

Now I did removed the -1 and backtested, but 1 year is too short and 'universal portfolio' ends up almost just the equal weight. So as in the original paper, this version below takes the the power 20 of the 1 year return to emulate 20-year return, and takes the weighted average.

Because there is no negative returns (just the returns less than 1) you never get negative weights in universal portfolio. So no need to skip shorting ETFs in trading according to the theory. I was checking the paper for how to treat negative weight and realized the above discrepancy.

But interestingly, the original -1 version has better return and drawdown. Would this -1 universal portfolio still approach the optimal, even if you subtract 1 before integration?

"But interestingly, the original -1 version has better return and drawdown"

As expected both versions have about the same risk-adjusted Return in "normal" situations. I made a backtest post-crisis (starting 1 January 2010) and the new/corrected version has higher beta (hence the bigger DD) and higher return, but about the same Shape-ratio.

The diference comes from the 1st version going to cash on the Great Recession correction. You should probably focus on that and to try to understand why/diferences.

Hi Naoki,
Yes the paper doesn't subtract 1 from the return. Your first script also used sum(abs()) as denominator, which leads to negative weights. The idea is to weight performance for all possible constant rebal portfolio. Subtract 1 or not doesn't change the idea of weighting based on performance, however that sum(abs()) did make a difference.
Also I think the paper doesn't suggest use a certain look back period. You could start on any day with equal weight, and keep re-balancing since that day based on actual performance.

A couple of things.
I tried this algorithm from 2004 and it didn't seem to do well, any idea why?
Also, I haven't tried all of this, but can we extend this algorithm with the following etfs?
1. And spx short etf for the bear market. This didn't work for me.
2. Some international etfs, like Asia pac, Europe, emerging markets, etc etf

Any thoughts?

hi this is my first comment on quantopian . nice to meet you here

Hi Saravanan, it does not work well for 2004 because some ETFs were't there back then (the algo needs 1 additional year of back data as well). SPX does not work well because it's an inverse of SPY. The Universal Portfolio considers every possible combination of stocks in the universe. But putting large cap (close to SPY) and SPX together distorts calculation.

By the way, this is not the kind of algo Quantopian is looking for at the moment. I would say there are 3 type of algo trading: portfolio optimization (like this one), momentum (go with trend) and arbitrage (buy and sell something at the same time). Quantopian is looking for arbitrage type algo (i.e. low-beta) because you can create value regardless of the stock market going up or down, and that's what the institutional investors want in hedge funds, to diversify their portfolio.

Nevertheless, Universal Portfolio is a cool theory with mathematical backing, and it is an interesting problem to code. I hope techniques and tricks used in the script would be useful in creating up a great algo.

Starting with 1 million, the first algo exchanges 4.5 million dollars to profit 2 million for a return (taking leverage into account) of only 45%. The second is 41%. Aside from that, the flatline in '08 looks great. Sure would be good to even make money during that downturn, more difficult than expected when I restrict myself to only what I think can truly do well going forward.

Naoki,

I like this code a lot and wanted to experiment . It worked for me the other day, but now I get following error

Something went wrong. Sorry for the inconvenience. Try using the built-in debugger to analyze your code. If you would like help, send us an email.
ValueError: cannot convert float NaN to integer
There was a runtime error on line 171.

Line 171 points to your execute function. It appears an NaN is creeping into the data, probably on this line

tradingvolume = history(3, frequency='1d', field='volume', ffill=True).mean()

but doesn't mean adjust for NaN's be using a 0 value?

I then inserted following

 if  pd.isnull(tradingvolume[sec]):  
      continue

but this did not eliminate error.

Before I go crazy with this I figure somebody else has already caught an fixed this bug. btw, comes up in run for dates 01/01/2003 - 01/01/2005

I know this is really old but how can I port over the logic that determines the market is in a downturn and chooses to hold cash?

I would like to use it for other Pipeline Algos I have and especially the one attached. I know the attached uses XIV but we could change that to prove the logic works during 2008. Thanks!

Here is another Algo I would like to use that logic in. Would it be possible to implement it with the attached?