I'm a big fan of the minimum variance strategies/portfolios/ETFs. Marcos Lopez de Prado recently gave a talk describing his paper and algorithm ( http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2197616 ) for doing numerically robust constrained portfolio optimization. Might be helpful here if you were to want to add inequality constraints to the portfolio weights...

Jorge,

Nice result. Perhaps you can fill me in...can we now use more than 10 sids in an algorithm? If so, what is the limit?

Grant

With set_universe, we did a ton of performance improvements on the backtester. You can now test up to 100 individually-named sids. The set_universe feature permits you to test even more - more than 700 when in daily mode. Over time we plan to loosen the limits even further, as we invest in performance fixes.

Dan

I have tried doing this same algorithm with the set_universe, however I've run into some problems. (I did only try it when the set_universe feature just came out so there may have been some bugs fixed by now).

These are the issues that I run into when trying this on the set_universe feature:
1. As far as I know you cannot set the exact number of stocks to return when using set_universe, it just returns all the stocks that fit within a certain range, which you can tweak but cannot set the exact number that are returned to you.
2. When I use set_universe, it sometimes misses price data and returns 0's for the prices for an entire security. When this happens, my algorithm will calculate 0's for the daily_returns for a specific stock. Later, when I use the daily_returns to calculate the covariance_matrix it will give me 0's for the row and column of the covariance_matrix that contains that specific security. When this happens, I get an error "Singular Matrix" when trying to take the inverse of that matrix, and this prevents me from calculating the GMV portfolio unless I loop through beforehand and remove any securities that have 0's in the returns.
3. Also, when I use set_universe, it returns to me a much larger set of stocks, which I know it is supposed to do. However, for this specific algorithm, and a few others, I would like to use a set number of securities to make up my portfolio. To do this, I iterate through each of the possible combinations of say 10 securities out of the say 40 securities that set_universe returns. The number of possible combinations for this set is very high and when I loop through all of them to find the optimal set of 10 securities with the least variance, the backtester will time-out and say that it has spent too much time in handle_data. Is there a way to possibly increase the time that it will allow you in handle_data before timing out?
4. For this specific example I am reallocating my portfolio every 10 days on the past 40 days of data. I have a circuit breaker to skip the entire handle_data function if the current day % 10 is not = 0, allowing me to only run handle data every 10 days when all the data refreshes. However, I have debugged and printed out both data.keys() and the keys in my batch transform and it looks like they overlap sometimes. So the set of stocks I am getting back from my batch transform are no longer in data.keys(). This causes an issue when I am in the process of reallocating because I am unable to loop through data.keys() and hit every security that I would like to deal with.

Not sure if anyone has come across these issues when dealing with set_universe, or whether or not some of these things have been fixed. But if so, please let me know. It would be much appreciated.

Jorge, that's great feedback. We greatly appreciate the time you're giving to this.

1) You're right, the function doesn't deliver an exact number of stocks. That's an interesting feature request. I'll mull that one over.

Looking at the next three questions, I started writing a response, and then I realized for each one I want to look at your latest version. Can you share the code?
2) I'd like to look at this one. Can you share the code on that one? There shouldn't be zeros in our price data from set_universe, in general.
3) We can adjust the handle_data timeout, but we also can improve the performance so that it doesn't time out. Can you share the code that is causing your timeout so we can look at it?
4) ditto ;)

Hey Dan,

1. Also, while were talking about set_universe, is there any plan for a way to filter by other metrics as well? Industry? Sub-industry? Fundamentals?...etc?
   
2. Let me include this one on the next reply, I need to go back and look for it and believe I can only include one backtest in a reply.
   
3. I've attached here a modification of the GMV Portfolio using the set_universe. I have it loop through the unique combinations of a set of 10 stocks in the set of stocks that set_universe returns (should be ~40) and search for the optimal one based on criteria I made up. It times out going through the loop the first time. Understandably, there are many unique combinations of a set of 10 in a set of 40, but I would just like to know the limits available.
   

Hey Dan,

Here's the backtest for number 4. I've printed out data.keys() and the batch transform as well as their lengths.

@Jorge, looks really interesting. A couple of suggestions:

1. You might want to place the portfolio selection code into the batch_transform. Because you set a refresh_period get_past_prices() will return the same, cached, array for 10 days. But you are recomputing the same portfolio (for 10 days) every day.

2. Compute one large covariance matrix over all stocks in the portfolio instead of all unique combinations. I think there is a lot of unnecessary recomputing going on which you might be able to put outside the loop.

If that doesn't help with the time-out problem we could either increase the time-out for you.

Hi Thomas,

Which backtest are you referring to with your suggestions?

Jorge

The 2nd to last.

For your first suggestion, does the block of code on lines 34-36 not function as a circuit breaker if it is not on a day that is an increment of 10, which should be each time the batch transform refreshes?

For the second, that's a good suggestion, I will look into that.

Re 1: You are correct, I missed that. I think it'd be more readable if you place it into the batch_transform but that's up to you.

Newbie question: I cloned the algo and then modified the universe to context.secs = [sid(8554),sid(22972),sid(25485),sid(26669),sid(28054)].

Everything else was kept constant. I receive the following error when backtesting:

KeyError: securities
File testalgorithm_sycheck.py:41, in handle_data
File /zipline/utils/protocol_utils.py:85, in __getattr_

Scott,

Looks like you named the universe context.secs, but you are trying to access context.securities?

That would do it.

Another newbie question: I cloned the algo and get the error below when building it. Any idea what is wrong? Thanks!

AttributeError: 'BarData' object has no attribute 'has_key'
USER ALGORITHM:46, in handle_data
if data.has_key(security):

Quant Trader, it looks like a recent change we made to the object that backs data, for performance reasons, removed the has_key method.
We will look at restoring has_key functionality.

In the meantime, you can change the has_key line to:

if security in data:  

and that should work in your algo.

That did the trick. Thank you!

Is there a way to eliminate cash from this algo so it's just calculating allocation based on securities listed within the algo?

@Jorge: I noticed that for the solution of the global variance minimum you use the inv(Cov) * [1 .. 1]', which gives the sum of each elements in the rows. However, in Carol Alexanders "Market Models", pg 188, she states that it should be the sum of each element in the columns, thus [1 .. 1] * inv(Cov). Is this relevant?
I'm not too familiar with dynamic asset allocation yet and Ernie Chan states here that it is not necessarily effective due to the high costs of re-balancing (at least for retail traders) and I would be interested to hear what your thoughts are on this.

@Jorge: Just ran some calcs and both ways seem to give the same results. This makes sense, of course, since the covariance matrix is symmetrical (please excuse my poor linear algebra skills).
Thanks for putting it up. I learn a lot here.

This algorithm would be a good candidate for kelly position/portfolio sizing for dynamic leverage.

Has anyone gotten this, or an algorithm like this, to work with set_universe? After the first universe change, I am getting a singular matrix on the inversion, which I am guessing is because the batch transform to get the prices is failing to get enough data...

@Tom - Yes, you are correct. You can use either calculation to get the weights for the min variance portfolio as long as the order of the calculation matches with the dimensions of the matrices

@Simon - I have been struggling trying to get this algorithm to work with set_universe. I have encountered the same issue - getting a singular matrix on the inversion. The problem is that the batch transform is returning either "NAN" or 0 for one of the daily returns. I have also had this occur when I hand pick securities that are missing certain data. The way I found around this issue is to loop through the matrix of daily_returns and eliminate any rows that contain "NAN" or 0. The problem is when you do this you are reducing the amount of securities you trade (you could think you are trading say 1% of the universe which is ~40 securities but when you run that loop you could cut down on that significantly if a lot of securities have tainted data). Another issue I found when trying to implement set_universe, which other people have also mentioned, is that there is no way to tell when a stock "falls out" of your universe. For example, if you only want to trade stocks that are say 80-81 percentile, every quarter it will readjust what securities fall in or out of that range. The problem with that is when the universe changes, it keeps every stock you currently have a position in in the universe. In an algorithm like this, that is every single stock. So after one quarter, you now have more stocks in data_keys than you really want and you are no longer only dealing with stocks that fall in your specified universe range. As far as I know, there is no way to tell which stocks should have been eliminated from the universe so that you can exit those positions. If anyone knows a solution to this issue please share.

@Jorge,

This is a fascinating api problem. It never occurred to me that people would want securities in the current portfolio to fall out of the universe, but your case makes it perfectly clear. I think we need a new builtin method, get_universe, which returns a set containing all the securities in your universe. Then you could mask the keys in handle_data's data parameter, and in batch transform's datapanel parameter.

Sound reasonable?

thanks,
fawce

@fawce - Thanks, that'd be great. In the API docs it says it reapplies the criteria periodically to create the universe, is that every quarter or is that becoming more frequent? Is there any way to set the frequency that the universe changes?

@Jorge,
Universe updates on the calendar quarter, which is a fixed period. Could you tell me a bit more about the flexibility you'd want to have for the period? I'd really like to understand how higher frequency changes would affect your algo strategy.

thanks,
fawce

@fawce - Ah okay, thanks for clearing that up. I dont really have a basis for requesting that as of yet. I really just wanted to see what effect changing the universe change frequency had on the performance of my algorithm...

Hi,

Can someone help me use this strategy using the the lowest 15 stocks by price amongst the S&P500. The results should be interesting. Regards, D