Here's the same algo as above, but with:

    # equal weighting  
    rebalance_portfolio(context, context.b_t)  
    return  

The portfolio is equal weight and re-balanced daily.

Very interesting Grant. Thanks for sharing. The sharpe @4 is amazing but the beta is high.

Thanks Satyapravin,

I'm still working on it. I'm not confident that the optimizer doesn't sometimes end up off in the weeds, resulting in spurious values. This code is a Band-Aid:

    allocation = res.x  
    allocation[allocation<0] = 0  
    allocation = allocation/np.sum(allocation)  

It eliminates negative values in the allocation, which shouldn't result based on the bounds.

Grant

Grant: Very cool. Just to make sure, here you're trying to replicate the OLMAR algorithm? I suppose that's a good approach as once that is working it's easier to change to solve a slightly different optimization problem (e.g. a market-neutral one).

Hello Thomas,

Yes, I am trying to replicate the OLMAR algorithm. I haven't attempted an side-by-side comparisons, yet. And yes, the idea is to have more flexibility in the optimization problem. Just out of curiosity, what are the various optimizers available on Quantopian? Did CVXOPT ever get up and working?

Grant

@Grant, yeah, that's a tight little group you've got there...

def initialize(context):  
    set_symbol_lookup_date('2015-01-01')  
    context.stocks = symbols("VRTX","AKAM","BIIB","MNST","AMGN","MSFT","ROST","INTC","NFLX","WYNN",  
                             "TSLA","MU","CELG","FB","GILD","YHOO","WDC","ILMN","PCLN","AAPL")  
def handle_data(context, data):  
    if bool(get_open_orders()): return  
    for stock in context.stocks: order_target_percent(stock, 1.0 / len(context.stocks))

Did I miss something? Were you using OLMAR to pick them? Is that what your code ended up doing? If so, wicked! What universe did you use to pick them from?

Hello Market Tech,

Nothing fancy to pick the securities. As I said above:

Please note that the backtest is biased by the selection of some high-performance securities (i.e. intentional look-ahead bias).

As I recall, I did a Google search for the top Nasdaq 100 stocks of 2013 & 2014. I found that a couple of them did not trade all the way into 2015, so I just replaced them with a couple big tech names off the top of my head.

At this point, the algo only handles long positions, so I figured why not work with a portfolio that has some umph.

Besides the look-ahead bias, there may be a risk of over-fitting here, too. There are enough knobs to turn that with high-performing stocks, a little tweak here and there could end up making a big difference in returns. So, it would be reasonable to wonder if the algo is working at all. I've played around with a very different set of securities, and the performance is ugly:

    context.stocks =  [sid(19662),  # XLY Consumer Discrectionary SPDR Fund  
                       sid(19656),  # XLF Financial SPDR Fund  
                       sid(19658),  # XLK Technology SPDR Fund  
                       sid(19655),  # XLE Energy SPDR Fund  
                       sid(19661),  # XLV Health Care SPRD Fund  
                       sid(19657),  # XLI Industrial SPDR Fund  
                       sid(19659),  # XLP Consumer Staples SPDR Fund  
                       sid(19654),  # XLB Materials SPDR Fund  
                       sid(19660)]  # XLU Utilities SPRD Fund  

Grant

Here's one that seems to work with:

   context.stocks =  [sid(19662),  # XLY Consumer Discrectionary SPDR Fund  
                       sid(19656),  # XLF Financial SPDR Fund  
                       sid(19658),  # XLK Technology SPDR Fund  
                       sid(19655),  # XLE Energy SPDR Fund  
                       sid(19661),  # XLV Health Care SPRD Fund  
                       sid(19657),  # XLI Industrial SPDR Fund  
                       sid(19659),  # XLP Consumer Staples SPDR Fund  
                       sid(19654),  # XLB Materials SPDR Fund  
                       sid(19660)]  # XLU Utilities SPRD Fund  

There's a drawdown at the end (due to oil?), but otherwise it does o.k.

Grant

I suppose what I was wondering was that since your results in the second test above are precisely the results that that spot of code I posted produced, and rightly so, rebalance everyday for 3 years, what was the optimize code bringing to the table? If the straight even-weight beats the optimized, then what am I missing? Even with the X ETFs above, a straight rebalance, even weight beats the test.

Are you searching for the efficient frontier in continuously rebalanced weights? Are you sub-selecting from your group? Your statement "To solve the portfolio optimization problem" would seem to want to re-weight every instrument, on the fly, and then use those weights to rebalance. Is that the intent? Hey, if this is an ongoing WIP then just ignore me (my wife does (and my kids)).

Market Tech,

As you said, it's a work in progress. If you have a peek at Section 4.2 of the paper I posted above, there is an optimization problem described there. The basic idea is to minimize the difference between a new portfolio allocation vector and an existing one, subject to an inequality constraint. In this case, the constraint should result in distributing the capital such that under-performing securities relative to their mean prices are weighted more heavily than ones at or above their means. So, every security could get tweaked when there is an update. It may be that the strategy fundamentally doesn't work, or I have made mistakes in the code or my application of it. Since there's no great secret, I thought I'd post it for the masses to review.

Regarding the S&P 500 sector ETFs above, and an equal-weighted portfolio re-balanced every day, I've attached a backtest. It's what I'd expect---basically tracks the SPY benchmark. The optimization algo sputters at the end, but seems to be doing something right otherwise. Not too sexy, but interesting.

Grant

Here's another installment. I think I'll try to replicate the result Thomas W. published on https://www.quantopian.com/posts/olmar-3-dot-0-using-new-order-and-history-features. Note that this algo based on scipy.optimize.minimize does not yet have the averaging over the trailing window that Thomas has in his algo. --Grant

Here's another tweak, and I added more high-flying stocks and SH, to reduce the volatility. Not sure if it helps, but note that I added the derivative of the objective function. --Grant

Hey, grant. I had a quick question, what do you mean when you say your algorithm has equal weight and that you rebalance it daily. What sort of benefit does this give you?

Hello Jason,

I posted it as a crude way of showing that the algorithm might be working. If the return of the algo is greater than a simplistic equal weight portfolio, then maybe the algo is actually taking advantage of mean reversion.

Grant

Why can't you start a backtest on, say 1 Jan 2012? It gets an error from before that. The error stops between 1 Jan 2012 and 1 Jan 2013.

This is the error:
"There was a runtime error. AttributeError: 'SIDData' object has no attribute 'price'
... USER ALGORITHM:51, in tradeGo to IDE
context.b_t[i] = context.portfolio.positions[stock].amount*data[stock].price"

Thanks,
Nick

This error usually means you try to trade a symbol that does not have price date available. Usually because it's just trading, stopped trading or has no data for other reasons. Use the debugger to trace the error

That's exactly right - the algo tried to access data on a minute bar the stock didn't trade. So there wasn't a price available in "data[stock].price".

So fix that you can add the guard,

for i, stock in enumerate(context.stocks):  
  if stock in data:  
    context.b_t[i] = context.portfolio.positions[stock].amount*data[stock].price  

@Grant Great Algorithm, Could you elaborate what are you doing in this 2 lines?

 cons = ({'type': 'eq', 'fun': lambda x:  np.sum(x)-1.0},  
            {'type': 'ineq', 'fun': lambda x:  np.dot(x,x_tilde) - context.eps})  
 res= optimize.minimize(norm_squared, context.b_0, args=context.b_t,jac=norm_squared_deriv,method='SLSQP',constraints=cons,bounds=bnds)

Hello Erick,

The first line provides the constraints for the optimization. The first equality constraint corresponds to the normalization of the portfolio vector, x; it sums to 1. The second inequality constraint comes from the paper referenced in the algo (see section 4.2 of http://icml.cc/2012/papers/168.pdf). It is meant to maximize the expected return of the portfolio, based on the assumption of mean reversion.

The second line is the call to the optimization routine (see http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html). The arguments are the objective function, initial value, additional arguments for the objective function, derivatives of the objective function, the optimization method, contraints, and bounds.

There's another example of optimization in https://www.quantopian.com/posts/long-only-minimum-variance-portfolio-using-scipy-dot-optimize-dot-minimize, if you're interested.

Grant

Thanks Grant, for the explanation. By the way congrats, you truly deserved to be the first winner. Changing topic I like the approach of this strategy however it looks like as you mention needs to have good set of securities that behave well in mean reversion.

EG

Thanks Erick,

Yes, it may be the case that there needs to be an overall upward trend for the portfolio. Over the long-haul, one would hope that the market trends upward, so that if the portfolio is sufficiently diversified, there will be an edge.

Grant

Here's another update, using get_fundamentals. Sorry, the code is a bit of a wreck, but I'm still struggling to understand how get_fundamentals works. In any case, it seems to be working, and the performance looks decent. --Grant

Hi Grant,

Today I cloned your algo(last one with fundamentals) and tried to run it without changin anything, but got following error :

ValueError: shapes (1,20) and (1,20) not aligned: 20 (dim 1) != 1 (dim 0)  
USER ALGORITHM:119, in norm_squared  
return 0.5*np.dot(delta_b,delta_b)  

Is it possible that this error is related to this Quantopian upgrade?

Yes, that's certainly related to the update (numpy in this case). I fixed the dot() call and it should run with this version.

Thanks Thomas,

What would happen, if I had put this algo to live trading?

Hasan

We would have emailed you before the upgrade and helped you to fix your algorithm. This is what we did for all live and contest algorithms.

Hi Thomas,

In one spot in the updated code, I see:

np.dot(delta_b, delta_b.T)  

But in other spots, I see:

np.dot(x,x_tilde)  
np.dot(allocation,x_tilde)  

Is the transpose of one of the vectors needed in some cases, but not in others?

Grant

Hi Grant,
Why the results change when backtesting in daily mode? Is this because of different price value when trade method is executed?
Is there any solution to make them the same?

Hello Hasan,

Well, I wouldn't recommend expending energy running on daily bars, since Quantopian doesn't support live trading in this mode. Is there a reason you've gone this route?

Grant

Thanks Grant,
There is no special reason, I just backtested in daily mode, and when decided to go for paper testing, confused by different results.

I guess I'd circle back and do some backtesting on minute bars. --Grant

Grant, Sorry for the delay. This has been really helpful. Thanks :)