While I am asking questions, has anyone tried to do a backtest with realistic short locate fees? Is it possible to programmatically withdraw cash from the account to simulate the various fees?

In production, IB holds the order until it's able to be submitted. For stocks which don't have available borrow, that can take a long time, since it requires more borrow to appear or for someone who's selling short to buy the stock back. In TWS, you can identify that the state of the order is 'held' but I'm not sure if this info is available through Quantopian. In production, it's actually quite annoying. . .

I haven't tried backtesting with realistic borrow fees. Do the stocks you're planning to short have a high borrow cost? If so, it's likely they will also have a highly variable borrow cost -- 5% in some days, 10% a month later so figuring out which number to use is hard and I don't think that IB offers a history of cost-of-borrow. For me, I just view borrow as a fixed expense and subtract off a couple of % of return a year to compensate for it. I'm sure you can do it more accurately (generally, borrow is only charged for overnight positions so it's easy enough to keep track off if you know what rate to charge) but am rather dubious on the value.

To remove cash at the time of a trade, perhaps you could use a custom commission model, as in this post about taxes.

IB has a Short Loan Borrow rates tool in their Mosaic, with historical rates: https://institutions.interactivebrokers.com/en/index.php?f=1572

Pretty awesome. I will probably just start to avoid the stocks with high rates, and try them out in a more nimble algo.

Thanks Simon, that's actually pretty amazing. I've tried checking the borrow rates through IB before but through a much clumsier approach.

If you're planning a long-short portfolio, keep in mind that IB doesn't net proceeds of short sales against charges for long purchases and charges different interest rates for both. IE, if your account has a net value of 100k and in it, you're long 200k of stock and short 100k, from IB's perspective, you are borrowing 100k (at 1.63 %) and lending 100k (at 0%). When interest rates are higher, this spread gets larger. This spread is likely to be more than the actual cost to borrow the stock unless you're short-selling some hard-to-borrow products.

Hmm, I see. It's possible to recoup 50% of their stock lending revenue by joining their Stock Yield Enhancement program, but one probably cannot avoid the margin loan spread...

In production, IB holds the order until it's able to be submitted. For stocks which don't have available borrow, that can take a long time, since it requires more borrow to appear or for someone who's selling short to buy the stock back. In TWS, you can identify that the state of the order is 'held' but I'm not sure if this info is available through Quantopian. In production, it's actually quite annoying. . .

Okay so I just saw this problem today in IB paper trading, and now I'm worried. Quantopian folks - is there any way to probe whether an order is in this state? I am trying to leg into baskets, and if one leg gets held, my P&L is at risk since the hedges are not in place. If it is still held after a couple of minutes, I'd like to be able to figure that out and back out of the other legs and try again tomorrow or something...

https://www.quantopian.com/help#api-orderobj

There's nothing like "status" in here, but in the IB live trading, the web app knows the order is held. That's frustrating!

Hopefully Quantopian can implement the correct solution in their API.

As a quick hack, you can put in logic to check for the condition where if you have a sell order out, the last market price is above your limit price and you haven't gotten a fill, the borrow fail must be happening.

Yeah I'm going to have to do something, or else this strategy is kaput on Quantopian!

This data is definitely available in their API too, so there's no reason in theory we couldn't find out the shortability of a stock without having to attempt to short it, time out on the fill, then go flat everything else that we bought/sold, which is going to cost money every day that we fail.

https://www.interactivebrokers.com/en/software/api/apiguide/tables/using_the_shortable_tick.htm

Is it possible to programmatically withdraw cash from the account to simulate the various fees?

Have a look at https://www.quantopian.com/posts/removing-cash-from-portfolio-to-pay-taxes to see if anything posted or referenced there might work.

Yeah, something like that might be possible. More concerned about the short fail problem.

Simon,
You can certainly check to see what's in your portfolio to determine if the stock you thought you were going to buy was actually purchased. Also should show up as unexecuted in the order status right? I've been working around this too, and I've been trying to do the short first, then checking on the next minute bar to see if it executed before I do the long side of the hedge. Gives me a minute slippage but better than the short side not filling. The real show stopper for me is what happens when I'm forced to cover the short, since IB just liquidates it without telling Q? Since this could happen at any time, I really have to check my portfolio every minute to make sure it hasn't unexpectedly changed from the previous minute, and I haven't figured out how to pass state between minutes in order to do that.

Hmm I wasn't going to worry about THAT, since I believe IB will call you during the day before they force cover at the close, no? At that point, I figure you have to stop the algo, close all the legs by hand and restart it without them, or something? I am not sure how reliable it would be to write algo code that attempts to handle that situation automatically. Without knowing if it's available for short again the next day, would it attempt to re-open that basket?

If you want to pass state between minutes, you can throw stuff in the context, FYI. I am just not sure if it's a good idea for this situation, although I am certainly thinking about it now. I wonder if both situations could be handled the same way - inside context, store your expected portfolio. During a rebalance minute, update that, and order_target_percent it. During a non-rebalance minute/scheduled, if they differ in share count, shut down that basket and close out all the legs. That gives it a minute to execute the rebalance, which should be plenty.

This brings me to another question - is there any way for an algo to send an email? I would certainly like to be alerted if the algo had to shut down a basket due to a discrepancy between expected and actual portfolios...

Thanks, I'll give that a shot and shoot you the code if I figure it out first.

In my experience IB will not only not call you when they liquidate you, but when you call them to ask why your position was mysteriously closed out they'll give you three mutually exclusive answers and then tell you that they don't have any control over what their risk algorithm does. I recently had them auto-liquidate a position because their algorithm felt a 5 point option spread could lose 8 points, and there was no convincing them otherwise. If you haven't yet had the opportunity to work with their autoliquidation routines and "customer service" you're in for a treat!

The trouble of course is that although things will almost certainly fill in a minute in IB, Quantopian won't fill them if there wasn't a trade that minute, and so we can't use the same code for backtesting Quantopian vs IB.

Here is what I came up with (with the baskets removed, just add in your baskets at the top). I hate this sort of code, completely not what I want to be doing.

import datetime  
import pytz  
import pandas as pd  
from zipline.utils.tradingcalendar import get_early_closes

def initialize(context):  
    set_slippage(slippage.FixedSlippage(spread=0.00))  
    set_commission(commission.PerShare(cost=0.0035, min_trade_cost=0.35))  
    A = sid(???)  
    B = sid(???)  
    C = sid(???)  
    D = sid(???)  
    E = sid(???)  
    F = sid(???)  
    context.baskets = [  
        {  
            A: +0.2,  
            B: -0.2,  
        },  
        {  
            C: +0.15,  
            D: -0.15,  
        },  
        {  
            E: +0.15,  
            F: -0.15,  
        }  
    ]  
    # these will be calculated during our ordering_logic  
    context.desired_positions = []  
    context.spy           = sid(8554)  
    start_date = context.spy.security_start_date  
    end_date   = context.spy.security_end_date  
    context.early_closes = get_early_closes(start_date, end_date).date  
    # this is the grace period that we give orders to work.  Should be short for IB  
    # since we don't want to be unhedged for long, but unfortunately, might need to  
    # be very long for Quantopian backtesting, since they do not fill during no-trade  
    # bars  
    context.order_cancel_working_time = datetime.timedelta(0,30*60,0)  
    schedule_function(ordering_logic,  
                      date_rule=date_rules.every_day(),  
                      time_rule=time_rules.market_close(minutes=40),  
                      half_days=True)

    # TEST FUNCTION DO NOT RELEASE!!!  
    schedule_function(simulate_call_in,  
                      date_rules.month_start(days_offset=10),  
                      time_rules.market_open(minutes=30),  
                      half_days=True)  
# TEST FUNCTION  
def simulate_call_in(context, data):  
    sid_to_call_in = list(context.baskets[0].keys())[0]  
    log.warn(str(get_now()) + ": CALLING IN " + str(sid_to_call_in.symbol))  
    order_target(sid_to_call_in, 0.0)  
def get_now():  
    return pd.Timestamp(get_datetime()).tz_convert('US/Eastern')

def percent_to_shares(context, data, sid, percentage_of_port):  
    port_val = context.portfolio.portfolio_value  
    cash_target = port_val * percentage_of_port  
    last_price = data[sid]['price']  
    shares_target = cash_target / last_price  
    return int(shares_target)  
def calculate_desired_basket(context, data, basket):  
    basket_all_traded = True  
    for sid in basket:  
        if sid not in data:  
            basket_all_traded = False  
    desired_basket = {sid: (percent_to_shares(context,data,sid,basket[sid]) if basket_all_traded else 0.0)  for sid in basket}  
    return desired_basket  
def calculate_desired_positions(context, data, baskets):  
    desired_positions =  [ calculate_desired_basket(context, data, basket) for basket in baskets ]  
    return desired_positions

def ordering_logic(context, data):  
    record( leverage=context.account.leverage )  
    context.desired_positions = calculate_desired_positions(context, data, context.baskets)  
    rebalance(context, data) 

def rebalance(context, data):  
    now = get_now()  
    for basket in context.desired_positions:  
        # this is silly, but apparently necessary for Quantopian  
        basket_all_traded = True  
        for sid in basket:  
            if sid not in data:  
                basket_all_traded = False  
        if basket_all_traded:  
            for sid in basket:  
                log.info(str(now) + ": Targeting " + str(basket[sid]) + " for " + str(sid.symbol))  
                order_target(sid, basket[sid])  

def cancel_all_stale(context, data):  
    now = get_now()  
    sids_cancelled = set()  
    fresh_orders = False  
    open_orders = get_open_orders()  
    for security, orders in open_orders.iteritems():  
        for oo in orders:  
            if ((get_datetime() - oo.dt) > context.order_cancel_working_time):  
                log.warn(str(now) + ": Cancelling order placed at " + str(oo.dt) +  " for " + str(oo.amount) + " shares of " + str(oo.sid.symbol) + "!")  
                sids_cancelled.add(oo.sid)  
                cancel_order(oo)  
            else:  
                fresh_orders = True  
                #log.info(str(now) + ": NOT CANCELLING order for " + str(oo.amount) + " shares of " + str(oo.sid.symbol) + " because it's fresh!")  
    return (sids_cancelled, fresh_orders)

def tolerable(context, data, sid, a, b):  
    last_price = data[sid]['price']  
    a_cash = a*last_price  
    b_cash = b*last_price  
    cash_diff = abs(a_cash - b_cash)  
    port_value = context.portfolio.portfolio_value  
    diff = cash_diff / port_value  
    tol = 0.01 # 1% of port value is ok  
    return diff < tol

def verify_basket(context, data, basket_desired_positions):  
    basket_okay = True  
    for sid in basket_desired_positions:  
        desired_position = basket_desired_positions[sid]  
        if not tolerable(context, data, sid, desired_position, context.portfolio.positions[sid].amount):  
            basket_okay = False  
    new_basket = {sid: (basket_desired_positions[sid] if basket_okay else 0.0) for sid in basket_desired_positions}  
    return (new_basket, basket_okay)  
def verify_positions(context, data, desired_shares):  
    all_baskets_okay = True  
    new_desired_positions = []  
    for basket in context.desired_positions:  
        (new_desired_basket, basket_okay) = verify_basket(context, data, basket)  
        if (not basket_okay):  
            all_baskets_okay = False  
        new_desired_positions.append(new_desired_basket)  
    return (new_desired_positions, all_baskets_okay)  
def handle_data(context, data):  
    now = get_now()  
    (sids_cancelled, fresh_orders) = cancel_all_stale(context, data)  
    # only bother checking out portfolio if we actually have one  
    if (len(context.portfolio.positions) > 0):  
        # if this is the same minute as our ordering_logic, we won't cancel those orders, they have  
        # one minute to work.  
        if (fresh_orders == False):  
            # if we cancelled some orders, presumably held because of no shorts available, give them a minute  
            # to cancel  
            if (len(sids_cancelled) == 0):  
                # if there was nothing to cancel and nothing fresh, double check that our positions haven't been  
                # changed from underneath us  
                (new_positions, all_positions_okay) = verify_positions(context, data, context.desired_positions)  
                if (not all_positions_okay):  
                    log.error(str(now) + ": Portfolios not converging, backing out of unbalanced baskets!")  
                    context.desired_positions = new_positions  
                    rebalance(context, data)

Note that without all this crappy error handling code that we are trying to do every minute, this algorithm would otherwise be nothing but a list and dictionary comprehension calling order_target_percent! Literally 10 lines of code! This is insane!

Anyway, basically, we try to proactively rebalance once per day, and give those orders N minutes to work. After those minutes are up, we cancel the orders, and give those orders a minute to cancel. This is how we catch short fails.

If we have no working orders and nothing to cancel, we check to make sure our portfolio is all what we intended. If it's not, due to a short fail not filling, or a short getting called in from underneath us, we log an error, set that entire hedged basket to 0 and rebalance towards it. Those orders will clearly have to abide by the same logic, but that's okay, we'll be 0 on that basket for the rest of the day. The next morning, we try again. Or that's the theory anyway! Comments welcome.

is there any way for an algo to send an email?

To the best of my knowledge, no. However, you could "roll your own" by writing a script to monitor the dashboard 'Logs' output, and send you an e-mail under pre-defined circumstances (“Danger, Will Robinson!”)

The trouble of course is that although things will almost certainly fill in a minute in IB, Quantopian won't fill them if there wasn't a trade that minute, and so we can't use the same code for backtesting Quantopian vs IB.

In the research platform, you could apply a model to fill in the missing bars, so that the backtest trades every minute for security XYZ, as it does for the frequently traded SPY, for example. You'd probably also want to fiddle with the slippage model. And commissions. And don't forget to automatically cancel all orders at the end of the day. In theory, at least in my feeble mind, you could bring a backtest into alignment with real-world trading at IB, particularly if you add some non-deterministic, Monte Carlo jazz.

An order can be detected as 'held' using the order status. See this thread: https://www.quantopian.com/posts/how-to-detect-if-an-order-has-been-canceled.

The corresponding order status is 4: https://github.com/quantopian/zipline/blob/master/zipline/finance/blotter.py#L42. It looks like this is missing from the API documentation and we will add that in.

I'd love to see automatic alerts sent from your IB algo! Currently this is not possible, you can log when an event happens in the algo. In the future, I'd like to see email and/or SMS messages to be triggered within the code.

http://ibkb.interactivebrokers.com/article/845

More details on IB's short fail handling.

Simon,
Thanks for that. This part really scares me:
"Given the volume of formal recalls which we receive but are not later acted upon, IB does not provide customers with advance warning of these recall notices. In the event the recall does result in buy-in, the lender executes the buy-in transaction and notifies IB of the execution prices. IB, in turn, allocates the buy-in to customers based upon their settled short stock position and unsettled trades are not considered when determining liability. Recall buy-ins are viewable within the TWS trades window once posted to the account with notifications sent, on a best efforts basis, by approximately 17:30 EST."

In other words. "We'll know 3 days in advance if your short position is going to be closed against your will, but we aren't going to bother to tell you. If it does get closed, we'll make our best effort to tell you by 5:30 that night, but won't guarantee that we will." Seriously IB?

Yeah, that's a bit iffy. Regrettably, it does seem like nightmarish logic is required to confirm that what's actually in the portfolio is congruent with what is desired and what was expected. I just hate adding so much python to such simple algorithms, since every line of code is another potential bug, especially in dynamically typed non-automatically-regression-tested code.

it does seem like nightmarish logic is required to confirm that what's actually in the portfolio is congruent with what is desired and what was expected

This type of control problem is not new under the sun, so if it can be abstracted, then standard approaches can be applied. It sounds like you are basically trying to transition your system (positions at IB) into a specified state within a certain amount of time, and then monitor the system, taking action when indicated.

The "box" you are trying to control (IB account) has numerous states, some desirable, some not-so-desirable. You might start by elaborating all of the states. I gather that it is something like this:

1. Fully hedged basket.
   
2. Partially hedged basket.
   
3. Empty basket.

I'm guessing that you want to be either in state 1 or state 3; the risk is ending up in state 2. I'm gathering that the problem comes in when you try to make the transition 3 --> 1 or 1 --> 3 (or you are sitting in 1 or 3), you end up in state 2, which is a failure. When your system is in a state of failure, the question is what action do you want to take? Naively, if a failure of any degree is detected, and state 3 is safe, then you'd want to transition immediately to state 3. Or you could put in a fixed wait time, to see if the system self-corrects (e.g. when a failure is detected, wait 15 minutes, and then pull the plug). Failures could be handled differently, depending on the situation. If you are sitting fat, dumb, and happy in state 1 and out-of-the-blue, you end up in state 2, you could immediately transition to the safe state 3 (and similarly, if somehow you flop into state 2 from state 3, you'd immediately force the system back to state 3). You could also consider adding a fourth state, which would effectively bring everything to a halt, requiring you to stop the algorithm. It would be a hack, but you could actually call a function that would cause handle_data() to time-out, and induce a crash, which hopefully would trigger Q support to contact you via e-mail and/or phone (instead of a time-out, a memory overload error would work, too). Or you could write a script to monitor the log output of your Q algo dashboard, and send you an e-mail when certain flags are encountered.

I don't think that there is any way around the tedious engineering work to control the remote system (IB) properly. It's kinda like when I put our oven into self-clean mode, and the door must be closed, and is locked automatically, so I don't end up setting the house on fire, or end up with a family member in the emergency room.

Yes, I suppose you are right. It is just such a pain, and compounded by the fact that it has to be done in python at one minute polls and without types, provers and tests. At this point I am just complaining, I'll stop.

FWIW I am testing a variant of the above algorithm, which rather than aborting a basket to flat when some leg executions fail, it just calculates the "implied portfolio value" from the realized sizes of all the legs (ie: the reverse of the order_target_percent calculation), takes the min of that, then resizes the unbalanced basket to that size.

So, if 5% of one leg couldn't get shorted, it'll just resize the other legs to 95% too, resulting in a balanced but smaller-than-ideal basket.

Simon,

Sounds good. I'm assuming that you'd need to cancel the order that did not go through completely? And assuming that the other leg's order got filled completely, then you'd need to submit another order to balance that leg against the one that couldn't be established fully (presumably due to the "short fail" problem)? Are you making adjustments the next minute bar, or are you allowing some time for things to percolate at IB?

Also, what if you intentionally put in a larger order than necessary on the short side? Maybe you'd land closer to the mark, and if you end up with too many shares short, then you could trim back until you are perfectly hedged.

By the way, does Interactive Brokers paper trading simulate this kind of thing accurately?

Grant

Grant: yes to everything. I give ib orders two minutes to fill, but that has to be increased to 6 hours or whatnot when testing in Quantopian.

I did get a short fail reject order and a short fail hold order in IB paper trading. I simulated the buy-in case myself by adding a test schedule_function which kicks out part of the leg. That's in the code above the too.

As for ordering extra, I don't think they are allocated pro rata, so I don't think it would help. Either you get them, or you don't, AFAIK.