Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Optimise _API is converting my shorts to longs after just 1 day

Hi all!

Below is the code from my Algo, minus the signals which are generated in Pipeline.

The majority of the non-signal code has been "borrowed" from other Algos and questions answered by @dan whitnable on these message boards. (Thanks Dan!).

I've been through the code many times but cannot identify the problem.

Longs are entered and then exited correctly (longs should be held for "days_held" days).

Shorts are entered correctly, but then on the 2nd day they are flicked from Shorts to longs. They are then held for the correct amount of time ("days_held" days).

This is of course playing havoc with the Algo's turnover in addition to obscuring the genuine signals.

Can any eagle eyed coders spot what I've done wrong?

Or any suggestions how i could track down the bug?

Thanks :-)

   return Pipeline(  
        columns={  
            'latest_close' : latest_close,  
             'shorts': shorts,  
             'longs': longs

        },  
        screen=(securities_to_trade),  
    )



def before_trading_start(context, data):  
    # Gets our pipeline output every day.  
    context.output = pipeline_output('my_pipeline')  


    context.portpos_count = len(context.portfolio.positions)  
    print "port pos count:"+str(context.portpos_count)    

    context.shorts = context.output[  
        context.output['shorts']].index.tolist()

    context.longs = context.output[  
        context.output['longs']].index.tolist()

    context.YestPrice = context.output['latest_close']

    context.shorts_sig_count = len(context.shorts)  
    print "shorts sig count:"+str(context.shorts_sig_count)

    context.longs_sig_count = len(context.longs)  
    print "longs sig count:"+str(context.longs_sig_count)  


    context.MyShortsCandidate = cycle(context.shorts)  
    context.MyLongsCandidate = cycle(context.longs)

    for stock in context.portfolio.positions:  
        CurrPrice = float(data.current([stock], 'price'))  
        if stock in context.age:  
            context.age[stock] += 1  
        else:  
            context.age[stock] = 1  
    for stock in context.age:  
        if stock not in context.portfolio.positions:  
            context.age[stock] = 0  
        message = 'Stock.symbol: {symbol}  :  age: {age}'  
        log.info(message.format(symbol=stock.symbol, age=context.age[stock]))

    pass


def rebalance(context, data):  
    cancel_open_orders(context, data)


    # Create position size constraint  
    constrain_pos_size = opt.PositionConcentration.with_equal_bounds(  
        -context.max_pos_size,  
        context.max_pos_size  
    )

    # Ensure long and short books  
    # are roughly the same size  
    dollar_neutral = opt.DollarNeutral()

    # Constrain target portfolio's leverage  
    max_leverage = opt.MaxGrossExposure(context.max_leverage)

    # Constrain portfolio turnover  
    max_turnover = opt.MaxTurnover(context.max_turnover)


    #checking each stock in the protfolio to see if is ready to exit  
    for stock in context.portfolio.positions:  
        if not get_open_orders(stock):  
            if (stock in context.age and context.age[stock] <days_held ):  
                pass  
            elif (  
                stock in context.age and context.age[stock] >=days_held  
            ):

                if context.portfolio.positions[stock].amount >0:  
    ###EXIT LONGS###  
                    weight = 0  
                    weights = pd.Series(index = context.longs, data = weight)  
    # Next create a TargetWeights object using our weights series  
                    target_weights_longs = opt.TargetWeights(weights)  
                    orders = order_optimal_portfolio(objective = target_weights_longs, constraints = [max_leverage,dollar_neutral,constrain_pos_size,max_turnover,])

    ###EXIT SHORTS###  
                if context.portfolio.positions[stock].amount <0:  
     # set the weight for the stock to zero  
                    weight = 0  
                    weights = pd.Series(index = context.shorts, data= weight)  
                    target_weights_shorts = opt.TargetWeights(weights)  
                    orders = order_optimal_portfolio(objective = target_weights_shorts, constraints = [max_leverage,dollar_neutral,constrain_pos_size,max_turnover,])


            else:  
                context.age[stock] = 1


    ####Now process new longs#####  
    if context.longs_sig_count == 0:  
        WeightThisBuyOrder = 0  
    else:

        weight = min(0.04,float((1.00 /days_held)/ context.longs_sig_count))  
        weights = pd.Series(index = context.longs, data = weight)  
    # Next create a TargetWeights object using our weights series  
        target_weights = opt.TargetWeights(weights)  
        orders = order_optimal_portfolio(objective = target_weights, constraints = [max_leverage,dollar_neutral,constrain_pos_size,max_turnover,])  


    ####Now process new shorts#####  
    if context.shorts_sig_count == 0:  
        WeightThisSellOrder = 0  
    else:

        weight = min(0.04,float((1.00 /days_held)/ context.shorts_sig_count))  
        weights = pd.Series(index = context.shorts, data = weight)  
        target_weights = opt.TargetWeights(-weights)  
        orders = order_optimal_portfolio(objective = target_weights, constraints = [max_leverage,dollar_neutral,constrain_pos_size,max_turnover,])             


def my_record_vars(context, data):  
    """  
    Record variables at the end of each day.  
    """

    # Record our variables.  
    record(leverage=context.account.leverage)  
    record(positions=len(context.portfolio.positions))  
    if 0 < len(context.age):  
        MaxAge = context.age[max(  
            context.age.keys(), key=(lambda k: context.age[k]))]  
        print ("MaxAge: ",MaxAge)  
        record(MaxAge=MaxAge)


def log_open_order(StockToLog):  
    oo = get_open_orders()  
    if len(oo) == 0:  
        return  
    for stock, orders in oo.iteritems():  
        if stock == StockToLog:  
            for o in orders:  
                message = 'Found open order for {amount} shares in {stock}'  
                log.info(message.format(amount=o.amount, stock=stock))


def log_open_orders():  
    oo = get_open_orders()  
    if len(oo) == 0:  
        return  
    for stock, orders in oo.iteritems():  
        for o in orders:  
            message = 'Found open order for {amount} shares in {stock}'  
            log.info(message.format(amount=o.amount, stock=stock))


def cancel_open_orders(context, data):  
    oo = get_open_orders()  
    if len(oo) == 0:  
        return  
    for stock, orders in oo.iteritems():  
        for o in orders:  
            cancel_order(o)

# This is the every minute stuff  
def handle_data(context, data):  
    pass  
6 responses

Found it.My problem is that I cal the order_optimal_portfolio statement twice (one for longs and one for shorts). Each call is constrained by "dollar_neutral" and hence distorts the signals.

When i removed "dollar_neutral" from the order_optimal_portfolio statement the problem goes away, However, my algo does not now meet the dollar_neutrality risk criteria.

I wonder if there is a way to combine the longs and shorts into a single call of the optimise_API (am sure that is what I should be doing, but don't know how). I could then constrain as required.

Any hints tips appreciated! :-)

Actually, it's generally best to call optimize just once rather than separate calls for the longs and the shorts. As noted, there won't be any way for optimize to ensure dollar_neutral, or really any other constraints for that matter, if called separately.

So, once one has the weights for longs and shorts in two separate series (assume long_weights and short_weights), then simply combine the two series together using the pandas 'concat' method. First ensure that all the values in longs_weights are positive and short_weights are negative.

all_weights = pd.concat([long_weights, short_weights])

Now use 'all_weights' as the target weight objective in a single 'order_optimal_portfolio' statement.

See if that works for you. Good luck.

BTW. thank you for the callout in your post.

Thanks Dan. That is now working! Onwards!

Hi Dan

Quick follow up. I implemented the approach you suggested as best I could (code below). There is now just a single call to order_optimal_portfolio that handles all the entry and exits orders.

When I backtest, it works for long periods of time (2003 to 2009 are fine), but then throws exceptions.

The error message
*
ValueError:cannot reindex from a duplicate axis
Line: 370 inrebalance
orders = order_optimal_portfolio(objective = target_weights, constraints = [max_leverage,dollar_neutral,constrain_pos_size,max_turnover,])
Go to IDE
*

The new code (based on your idea) as follows:

    #checking each stock in the protfolio to see if is ready to exit  
    for stock in context.portfolio.positions:  
        if not get_open_orders(stock):  
            if (stock in context.age and context.age[stock] <days_held ):  
                pass  
            elif (  
                stock in context.age and context.age[stock] >=days_held  
            ):

    ####First EXIT positions that have reached days_held#####  
                weight_e = 0.0 #min(0.04,float((1.00 /days_held)/ context.longs_sig_count))  
                long_weights = pd.Series(index = context.longs, data = weight_e)  
                short_weights = pd.Series(index = context.shorts, data = weight_e)  

    ####Now process new longs#####  
    if context.longs_sig_count == 0:  
        WeightThisBuyOrder = 0  
        weight_l= 0  
        long_weights = pd.Series(index = context.longs, data = weight_l)  
    else:

        weight = 0.05 #min(0.04,float((1.00 /days_held)/ context.longs_sig_count))  
        long_weights = pd.Series(index = context.longs, data = weight)  
    ####Now process new shorts#####  
    if context.shorts_sig_count == 0:  
        WeightThisSellOrder = 0  
        weight_s= 0  
        short_weights = pd.Series(index = context.shorts, data = weight_s)  
    else:

        weight = -0.05 #min(0.04,float((1.00 /days_held)/ context.shorts_sig_count))  
        short_weights = pd.Series(index = context.shorts, data = weight)  


##Now run the optimise portfolio API ###  
    all_weights = pd.concat([long_weights, short_weights])  
    target_weights = opt.TargetWeights(all_weights)  
    orders = order_optimal_portfolio(objective = target_weights, constraints = [max_leverage,dollar_neutral,constrain_pos_size,max_turnover,])                     

Hmmm.

Could it be that sometimes you have a security in both the long AND short list? Maybe use the following to drop any duplicates in the all_weights series.

    all_weights = pd.concat([long_weights, short_weights])  
    all_weights = all_weights[~all_weights.index.duplicated(keep=False)]

See if that works. If thats the issue, maybe ensure there are no duplicates before the 'concat' method and then don't bother with this.

Also, could you attach a backtest. It's much easier to help troubleshoot.

Tks Dan - Good point. I think u are right. I'll try that. (edit IT WORKED!)

Really keen to keep the code that generates the signals private - been tinkering with it on it off and on literally for 25 years! :-)

When you say "attach a backtest" - is that possible without revealing the code?