I reduced the initial capital to $25K, and added:

    open_orders = []  
    for stock in context.stocks:  
        open_orders.append(bool(get_open_orders(stock)))  
    if True in open_orders:  
        return  

I use the built-in slippage and commission models, and plot the cash balance.

Grant

Hello Grant,

I wish I knew what it's doing! But you might as well 'return' as soon as you find the first pending order:

    #open_orders = []  
    for stock in context.stocks:  
        if bool(get_open_orders(stock)):  
            return  
    #if True in open_orders:  
    #    return  

P.

Thanks Peter,

The link to the paper describing the original algorithm is http://icml.cc/2012/papers/168.pdf. It uses prices rather than dollar-volumes.

Grant

Hi Grant,

Nice idea to use the dollar-volume data! How does the performance compare to using OLMAR with price data over the same period?

Aidan

Hello Aidan,

Here you go. It's a simple code change:

@batch_transform(refresh_period=R_P, window_length=W_L) # set globals R_P & W_L above  
def get_data(datapanel,sids):  
    p = datapanel['price'].as_matrix(sids)  
    # v = datapanel['volume'].as_matrix(sids)  
    # return np.multiply(p,v)  
    return p  

Grant

Here's an attempt at minute-level trading. Seems to improve the return, but then it completely tanks at the end...not sure why. --Grant

Hi Grant,

I think it would be really cool if we could add a penalization term to the optimization problem when we readjust the portfolio. So that you don't have to buy or sell too many stocks. The transaction costs probably wipes out profit if we try to match b to b* exactly every period.

Note that I don't think that the dollar-volume computation in this algorithm is quite right, since the batch transform forward-fills both price and volume for missing bars (not yet confirmed). If a bar is missing, the volume should be zero in the dollar-volume code. I'm considering ways of fixing the problem; if you have a solution, please post it. --Grant

@Grant - studying this one... I like the potential of OLMAR... is anyone running this in live trading yet?

when u say " If a bar is missing, the volume should be zero in the dollar-volume code." so what happens now? it's using a previous dollar-volume from a previous get_data or the as_matrix call "datapanel['volume'].as_matrix(sids) " is not returning a 0?

Hello Miguel,

Grant will expand but Grant and myself have just realised that Quantopian data forward fills empty bars with price and volume information. This surprised me and perhaps I should have considered this before. Some discussion from the last few days in these threads:

https://www.quantopian.com/posts/thinly-traded-stocks-why-no-gaps
https://www.quantopian.com/posts/why-does-the-data-in-handle-data-have-different-dates-for-some-securities

and some examples here:

https://www.quantopian.com/posts/feature-request-display-hh-mm-in-log-entries-in-a-minutely-backtest

I think, although he can confirm or deny it, that Grant would like either no forward fill of price and volume or a forward fill of price with volume set to '0'.

Related, Simon asked some questions here about the forthcoming 'history' feature:

https://www.quantopian.com/posts/draft-proposal-of-new-data-history-api

i.e.

Also re forward filling, are you forward filling a nil tick to produce a degenerate bar with 0 volume? Or are you actually forward filling whatever the last bar was, causing any code that has exponential decay, volume accumulation or really any calculation at all to change...?

EDIT: I just re-read the reply from Eddie to Simon re: 'history':

'ffill=True' - fill the pricing information forward with the last known value (with the exception of inserting a "0" for volume)

P.

Hello Miguel and Peter,

I don't know if anyone is running the OLMAR algorithm in paper trading. I tried and there was an error. I haven't gotten back to it.

Regarding the forward filling, there are a couple issues at play here. I need to check if the forward filling used by the batch transform sets the volume to zero. If not, I think that I can turn off the forward filling and then just ignore the missing data (and with the upcoming history method, there will be a way to deal with missing data). The issue described on https://www.quantopian.com/posts/thinly-traded-stocks-why-no-gaps is different in that I think that it impacts order submission/fulfillment/slippage, since I think the filling is applied before the backtest is started...we'll have to see what the Quantopian response is to my questions.

Grant

Hi Grant,

Glad to see our OLMAR is having a revival, the results are also very promising.

Have you played around with eps and window_length at all? In my optimization experiments I found that those have a huge effect, especially epsilon. This also leads to @Ben's question: you can clone the algo and play around with eps which essentially is a parameter of how aggressively the portfolio should be rebalanced; the lower the more conservative.

Thomas

Hello Thomas,

I've played around a bit with both eps and window_length...I may do a more systematic study once I sort out a few things. I got the algo to run on minute data, with slippage by adding:

for stock in context.stocks:  
        if bool(get_open_orders(stock)):  
            return  

This way, the portfolio can't be reallocated until all orders are filled.

Particularly in minute mode, I'm concerned how missing bars (thinly traded securities) impact the algorithm. I can fix up the batch transform, but I'm not sure how to interpret my finding described on https://www.quantopian.com/posts/thinly-traded-stocks-why-no-gaps. My work-around is:

    for stock in data:  
        if data[stock].datetime < get_datetime():  
            return  

The portfolio won't get reallocated if any of the securities did not trade (historically). However, I think that orders still get filled, right? Or not?

Also, how does the backtester handle orders for fractional shares? I suspect as the algo is written now, there can be a difference between the actual portfolio state and the one that I record with:

# update portfolio  
    context.b_t = b_norm  

Grant

Hi Grant,

Yeah, those work-arounds make sense. Orders should still get filled if you just return, yes.

I also share you concern regarding the internal portfolio allocation vector b_t to get out of sync with the actual portfolio. Instead we should probably just recalculate b_t at the beginning of each handle_data().

Thomas

good

Another minute-level run. Still crashes at the end, though...hmm? --Grant

Here's an update with the sid fix (https://www.quantopian.com/posts/list-of-nasdaq-100-sids-to-use-in-your-algo). Note that this algorithm isn't as great as it might appear, since it seems that most of the return can be obtained by just buying the entire Nasdaq 100, with equal dollar weights (set context.eps = 0).

--Grant

Here's an update (switched to using VWAP for the moving average, per Ben V. on https://www.quantopian.com/posts/olmar-implementation-fixed-bug). I also only allow shares to be transacted in even lots (multiples of 100). There's likely some bias in this approach, since I ran the algorithm a number of times to find 5 random stocks out of the Nasdaq 100 that juiced up the return. --Grant

Here's basically the same algo as immediately above, except run over a longer time period. --Grant

@Grant:

I tried to paper trade the last version you posted but get:

IndexError: index -1 is out of bounds for axis 0 with size 0
USER ALGORITHM:168, in simplex_projection
rho = np.where(u > (sv - b) / np.arange(1, p+1))[0][-1]

Seems like some issue with missing sids? Anyway, just wondering if you perhaps encountered this as well and know more.

Thomas

Hello Thomas,

The problem might be due to missing sids, but I'm not sure. I'll tinker around with it when I get the chance.

Grant

Thomas,

Here's basically the same algo. I'll try live trading with it to see if I get an error, too.

Grant

Hello Thomas,

I got through my first day of paper trading with no error. Here's the code I'm running:

import numpy as np  
import math

# globals for get_data batch transform decorator  
R_P = 0 #refresh period in days  
W_L = 5 #window length in days

def initialize(context):  
    context.stocks = [sid(1419),sid(12652),sid(6546),sid(5061),sid(6683)]  
    for stock in context.stocks:  
        print stock.symbol  
    context.m = len(context.stocks)  
    context.b_t = np.ones(context.m) / context.m  
    context.eps = 1 # change epsilon here  
    context.init = False  
def handle_data(context, data):  
    cash = context.portfolio.cash  
    record(cash=cash)  
    if not context.init:  
        rebalance_portfolio(context, data, context.b_t)  
        context.init = True  
    # get data  
    d = get_data(data,context.stocks)  
    if d == None:  
       return  
    # skip tic if any orders are open or any stocks did not trade  
    for stock in context.stocks:  
        if bool(get_open_orders(stock)) or data[stock].datetime < get_datetime():  
            return  
    # update portfolio  
    for i, stock in enumerate(context.stocks):  
        context.b_t[i] = context.portfolio.positions[stock].amount*data[stock].price  
    context.b_t = np.divide(context.b_t,np.sum(context.b_t))  
    prices = d[0]  
    volumes = d[1]    

    m = context.m

    x_tilde = np.zeros(m)

    b = np.zeros(m)  
    # find relative moving volume weighted average price for each secuirty  
    for i, stock in enumerate(context.stocks):  
        vwa_price = np.dot(prices[:,i],volumes[:,i])/np.sum(volumes[:,i])  
        x_tilde[i] = vwa_price/prices[-1,i]  
    ###########################  
    # Inside of OLMAR (algo 2)

    x_bar = x_tilde.mean()

    # Calculate terms for lambda (lam)  
    dot_prod = np.dot(context.b_t, x_tilde)  
    num = context.eps - dot_prod  
    denom = (np.linalg.norm((x_tilde-x_bar)))**2

    # test for divide-by-zero case  
    if denom == 0.0:  
        lam = 0 # no portolio update  
    else:  
        lam = max(0, num/denom)  
    b = context.b_t + lam*(x_tilde-x_bar)

    b_norm = simplex_projection(b)

    rebalance_portfolio(context, data, b_norm)

@batch_transform(refresh_period=R_P, window_length=W_L) # set globals R_P & W_L above  
def get_data(datapanel,sids,clean_nans=False):  
    p = np.nan_to_num(datapanel['price'].as_matrix(sids))  
    v = np.nan_to_num(datapanel['volume'].as_matrix(sids))  
    return [p,v]

def rebalance_portfolio(context, data, desired_port):  
    #rebalance portfolio  
    current_amount = np.zeros_like(desired_port)  
    desired_amount = np.zeros_like(desired_port)  
    if not context.init:  
        positions_value = context.portfolio.starting_cash  
    else:  
        positions_value = context.portfolio.positions_value + context.portfolio.cash  
    for i, stock in enumerate(context.stocks):  
        current_amount[i] = context.portfolio.positions[stock].amount  
        desired_amount[i] = desired_port[i]*positions_value/data[stock].price

    diff_amount = desired_amount - current_amount

    for i, stock in enumerate(context.stocks):  
        # order(stock, int(math.floor(diff_amount[i]/100.0))*100) #order_stock  
        order(stock, int(math.floor(diff_amount[i])))

def simplex_projection(v, b=1):  
    """Projection vectors to the simplex domain

Implemented according to the paper: Efficient projections onto the  
l1-ball for learning in high dimensions, John Duchi, et al. ICML 2008.  
Implementation Time: 2011 June 17 by [email protected] AT pmail.ntu.edu.sg  
Optimization Problem: min_{w}\| w - v \|_{2}^{2}  
s.t. sum_{i=1}^{m}=z, w_{i}\geq 0

Input: A vector v \in R^{m}, and a scalar z > 0 (default=1)  
Output: Projection vector w

:Example:
>>> proj = simplex_projection([.4 ,.3, -.4, .5])  
>>> print proj  
array([ 0.33333333, 0.23333333, 0. , 0.43333333])  
>>> print proj.sum()  
1.0

Original matlab implementation: John Duchi ([email protected])  
Python-port: Copyright 2012 by Thomas Wiecki ([email protected]).  
"""

    v = np.asarray(v)  
    p = len(v)

    # Sort v into u in descending order  
    v = (v > 0) * v  
    u = np.sort(v)[::-1]  
    sv = np.cumsum(u)

    rho = np.where(u > (sv - b) / np.arange(1, p+1))[0][-1]  
    theta = np.max([0, (sv[rho] - b) / (rho+1)])  
    w = (v - theta)  
    w[w<0] = 0  
    return w  

I'm confused what happened, though, since I should have approximately the same dollar amount in each security. Instead, I have:

CERN
$32,915.62

SBUX
$28,902.06

DLTR
$24,547.25

MSFT
$8,029.78

ROST
$4,988.75

For some reason, the algo both bought and sold shares. Due to the warm-up? I'd expected, after the first trading day to have about the same dollar amount in each security, with only buys.

Grant

Hi Grant,

Thanks, that code runs fine now. I did notice the same behavior as you did though. Do we have a bug in the portfolio filling?

Btw. we have some new order methods in zipline that allow better control over how many shares to order that should make it to quantopian soon.

Thomas

Thomas,

I think that the start-up behavior under paper trading is due to the warm up. I had coded to do an initial buy, with equal dollar amounts in each security, before the batch transform window was full. Apparently, paper trading went back in time, applied the initial buy, then the warm-up period ensued, followed by the first day of trading, when the re-balancing was reported. So, I don't think that there is a bug in portfolio filling.

Grant

Grant,

I used the flags in many of my own algos and shares, and it always causes problems. Using a boolean flag for the init is brittle in simulated trading, precisely because of the warmup sequence for the algorithms you mention. However, we've found that simple switches also tend to break in real trading, because the switch assumes a clean slate start in the account. This assumption breaks down when you have pre-existing positions in your account, or you make transfers/transactions from outside the algorithm, or your broker updates your account, etc.

Instead of the boolean flag, we've been writing snippets to look at the current portfolio and determine if we want our algorithm to return the positions to some baseline. Sometimes that means liquidating, sometimes it means rebalancing to equal weights, etc. In this case, it might be as simple as checking if any of your stocks are missing from the portfolio. In other algorithms, we've looked at the total percent invested as a trigger to return to baseline.

thanks,
fawce

Thanks Fawce,

In this case, the algo is initialized by buying equal dollar amounts across all of the securities in the portfolio. Then, the portfolio is re-balanced. I could have just as well initialized after the warm-up period, so that on the first day of paper trading, I would have seen equal dollar amounts in each security. There is no provision to return to a baseline (e.g. equal dollar amounts), although it would be straightforward.

Grant

Grant, I'm very pleased to see this implementation on Quantopian. I have run many experiments in the past using many of the predecessor algorithms cited in the paper (Cover's alg, EG, and the Hungarians' KNN).

One thing I can't tell by looking at the code: Is this a long only portfolio? No negative allocations?

Hello Ken,

The algorithm is long only (no negative allocations). The question has come up before, but to my knowledge, no one has posted a solution.

Grant

As Fawce informed me here, "All live trading, paper or real money, is minutely." So, I added some code to execute the algorithm at market open under minutely trading. Here's the result, which basically replicates the one above.

In the end, it might be better to make a trading decision every minute, but based on my testing, it seems that there may need to be some sort of smoothing/filtering of the data to replicate the results I obtain with daily re-balancing.

--Grant

To demonstrate the usefulness of the new features (history() and new order methods; see here for more description), I updated the OLMAR algorithm. As you can see the code is much simpler now, especially order_target_percent() allows us to get rid of the rebalance_portfolio() logic.

so what happens here ? :S

The returns are fantastic. But seem correlated with the benchmark. Is there evidence of alpha (say bear market performance)? Also, a big question is how do we select the universe of stocks? I would think limiting to bluechips may help ensure reversion to the moving average in the long run.