Hello Jess,

An important consideration for live trading is that the algorithm gets re-launched every day (at market open, 9:30 am typically?). Presumably, for the algo you shared above, this won't be a problem. However, I can imagine that there could be some pitfalls. Is there any way to emulate the daily re-launching in backtesting?

Grant

Hi Grant,

Actually I think the algo re-launch process is going to be transient, so while there are two important items in the list of current limitations that are related, I don't think there is good reason to go to too much effort building around this structure for the long term.

Basically you want your algorithm to be deterministic, if you get the same results when you click 'backtest' and then click it again you are probably good. The two subtle cases that can violate determinism are (1) using a random seed variable in your algo, just don't do it. (2) relying on historical fetcher data that is overwritten - make sure that your fetcher data source is fixed, with only updates via appending new data.

Algos (and their host servers) are launched each morning several hours before market open to make sure there is time to get things initialized and to catch any potential problems prior to open. A good rule of thumb is to make sure new fetcher data is updated by midnight to be on the safe side.

Best, Jess

Is the 'context' saved between restarts?

Thanks Jess,

I also have Simon's question regarding 'context' persistence between re-starts, and a more general one. I gather that there is no way to permanently store data generated by the algorithm, for later access (e.g. from a hard drive). Is this on the roadmap?

Also, I'm not sure I understand how generating random numbers "violates determinism" and how it would affect execution of the algorithm. Say one wanted to pick, randomly, a different stock to trade every day. How could this be done?

Grant

I'm looking for an opinion or thoughts on this.

Rebalancing, as is in done in this algo, can mean leaving a percentage of your cash idle. In this particular example that isn't the case but if say one of the traded items was google at over $1000 per share and a 10% stake (30k account value) would leave hundreds of dollars not used.

This code can help a bit but I suspect that there is a better way.

    targets = [40,30,15,10,5]  
    constraint = ({'type':'eq', 'fun': lambda targets: sum(targets)-100 })  
    optimized = scipy.optimize.minimize(excessCash,  
                                        targets,  
                                        (context.portfolio.cash+context.portfolio.positions_value,data),  
                                        constraints=constraint,  
                                        method='SLSQP')  

I was playing with a rebalancing algorithm I cloned that does essentially this same thing, and I started to wonder about how exactly this can work without margin trades. It seems like you'd want to first determine which positions you're going to sell, and make those trades--ensuring they actually execute--before you buy the others. Otherwise, you could hypothetically end up buying on margin for several days while your sell orders fail to execute, right?

def rebalance(context,data,exchange_time):  
    #Rebalance the portfolio to the predetermined target weights:  
    #  
    #  40%   - US Stock Market  
    #  30%   - Foreign Stock Market  
    #  15%   - Real Estate  
    #  10%   - Natural Resources  
    #  5.0%  - Bonds  
    order_target_percent(context.USStockMkt,0.40)  
    order_target_percent(context.IntlStockMkt,0.30)  
    order_target_percent(context.RealEstate,0.15)  
    order_target_percent(context.NatResources,0.10)  
    order_target_percent(context.Bonds,0.05)

Hello Eric,

You may be right. I've been trying to get my head around generalized order management, and the series of order_target_percent lines just submit a bunch of market orders simultaneously (unless there is something more sophisticated under the hood). Note that there is a 5% cash buffer, so for small changes, perhaps it'll be o.k. But it is not clear what'll happen for larger changes, with thinly traded securities or if an order gets rejected/cancelled or partially filled.

Grant

Thanks Grant,

I hadn't heard about the cash buffer. It might help the situation, but I would rather not make the assumption that my orders will fit within that. I don't like ambiguity in the behavior of my code, and I certainly wouldn't feel comfortable live trading without knowing exactly how my orders are going to be executed. For that reason, I think I'll start manually placing the orders and monitoring their execution status myself. By "manually" I mean using order() instead of order_target_percent(). ;)

Hello Grant,

The 5% cash buffer is mentioned here https://www.quantopian.com/posts/paper-trading-with-interactive-brokers-open-beta-launch but isn't it just for that algo? (Where it seems to be 10%, anyway.)

P.

Thanks Peter (and sorry Eric),

Yes, for the algorithm above there is no cash buffer (my ability to do math drops exponentially after ~ 8 pm). For the one posted on https://www.quantopian.com/posts/paper-trading-with-interactive-brokers-open-beta-launch, it is 1%, right?

context.weights = 0.99/len(context.secs) or context.weights = 0.11
9*0.11 = 0.99, which leaves 0.01 or 1% in cash

I've attached a backtest as a check, which results in ~$10K in cash (out of $1M total).

Simon Thornington touches on the order management problem on https://www.quantopian.com/posts/grid-searching-for-profitable-cointegrating-portfolios:

Well, stop orders on mean reversion systems are particularly tricky -- they're much more straight-forward on momentum-based systems.

For the time being, I am not going to target Quantopian, but try going it alone. I'll be keeping a close eye of course though! The challenges will be the same, structuring order placement to minimize transaction costs, handling partial and delayed fills correctly to avoid unhedged exposure, aggregating and disaggregating orders from multiple concurrent systems, per-system and portfolio level risk management and user overrides, etc etc. Lots to do!

For the algorithm posted by Jess above, it may be a matter of defining a dependent series of orders (1-->2-->3...) so that capital flows versus time out of securities that have appreciated and into securities that have dropped in price, without triggering any borrowing.

Grant

Hello Grant,

I think you're looking at the one from Dan and I'm looking at the one from Jess!

P.

Well...no matter...I think that Eric's point above applies. --Grant

If the market orders are big enough that they might take a while to execute then I suspect you are going to have bigger problems with slippage.

As I see it, the issue is this:

Initial state: 100% in A, 0% in B
Change state to: 0% in A, 100% in B

order_target_percent(context.A,0)  
order_target_percent(context.B,1)  

Should work, right? But the proceeds from the sale of A are needed for the purchase of B, so if the sale of A does not go through, then what happens at IB?

There should be a way to code the algorithm so that it is more robust, right? Basically, as the sell order for A is filled, a series of buy orders for B can be submitted, until the transaction is complete.

Grant

Grant,

If you are rebalancing your portfolio once a month, I'm not sure the performance of your algo is going to be impacted in any significant way if you have to submit the order to sell A, wait a few minutes for it to clear, and then submit a followup order to purchase B. The gap in the time scale between one minute and one month is just too large for a few minutes to matter in an algorithm that does something monthly.

However, as noted previously, if you keep a cash buffer in your account, this is moot since the cash buffer will be used to purchase B even if the sell order for A goes through after the buy order for B.

Furthermore, if your account is large enough to be allowed to trade on margin, the buy order for B will go through even if you don't have a cash buffer.

It is also worth noting that unless you're trading a pretty illiquid stock, the odds are that if you place the sell and buy orders sequentially, they'll be processed sequentially, which means even with a 0% cash balance you can probably safely place both orders in the same minute. Worst-case scenario, the order for B get cancelled and you retry it a minute later.

I believe that among the 60-odd order types that IB offers are some that group multiple orders involving multiple securities into a single basket order that constrains how and when all the orders in the basket get filled. See, e.g., Scale Orders. We may consider adding support for such order types in the future if there is sufficient interest in them. However, we have to keep in mind that we will eventually be supporting multiple brokers, and I don't think we want our functionality to diverge greatly from broker to broker, so we may avoid supporting order types that are unique to specific brokers.

@Eric - as you point out my algo is not currently robust to trading illiquid stocks - it simply blindly calculates the order sizes need to reach the target allocation and places all those orders at once. Perhaps too simplistic for some use cases.

If you want to do something different then you can drop back to the basic order() type. Also keep in mind that our backtester is open-soure software (https://github.com/quantopian/zipline)

The order_target_percent method was contributed by a member of our open source community, Jeremian Lowin (see code below). If you come up with a way to extend this and make it more robust please share it back so we can pull in any improvements. I've also thought that a nice addition would be to add a parameter to define a threshold for rebalance, so for example if you don't want to place trades for less than 100 shares, or 10 shares or whatever you could specify that. Haven't gotten to it yet though.

def order_target_percent(self, sid, target, limit_price=None,  
                         stop_price=None):  
    """  
    Place an order to adjust a position to a target percent of the  
    current portfolio value. If the position doesn't already exist, this is  
    equivalent to placing a new order. If the position does exist, this is  
    equivalent to placing an order for the difference between the target  
    percent and the current percent.

    Note that target must expressed as a decimal (0.50 means 50\%).  
    """  
    if sid in self.portfolio.positions:  
        current_position = self.portfolio.positions[sid].amount  
        current_price = self.portfolio.positions[sid].last_sale_price  
        current_value = current_position * current_price  
    else:  
        current_value = 0  
    target_value = self.portfolio.portfolio_value * target

    req_value = target_value - current_value  
    return self.order_value(sid, req_value, limit_price, stop_price)

which then uses:

def order_value(self, sid, value, limit_price=None, stop_price=None):  
    """  
    Place an order by desired value rather than desired number of shares.  
    If the requested sid is found in the universe, the requested value is  
    divided by its price to imply the number of shares to transact.

    value > 0 :: Buy/Cover  
    value < 0 :: Sell/Short  
    Market order:    order(sid, value)  
    Limit order:     order(sid, value, limit_price)  
    Stop order:      order(sid, value, None, stop_price)  
    StopLimit order: order(sid, value, limit_price, stop_price)  
    """  
    last_price = self.trading_client.current_data[sid].price  
    if np.allclose(last_price, 0):  
        zero_message = "Price of 0 for {psid}; can't infer value".format(  
            psid=sid  
        )  
        self.logger.debug(zero_message)  
        # Don't place any order  
        return  
    else:  
        amount = value / last_price  
        return self.order(sid, amount, limit_price, stop_price)  

Thanks Jonathan (& Jess),

This caught my interest, since I started working on an algorithm that seems to work pretty well for low dollar-volume stocks (via set_universe), and I'd like to apply stop loss orders as the order is filled (assuming that the stocks are thinly traded). And as the position is exited (again assuming thin trading), reduce the loss coverage accordingly. Perhaps the lingo "wind up" and "wind down" applies here. The problem Eric pointed out is analogous in that basically one has to write a Quantopian routine that will do the wind up and then update the allocation in a dynamic fashion, based on the IB state.

I agree that with liquid ETFs, a cash buffer, and the intent of the algorithm above, this may be no problemo.

Grant

Jessica,

Thanks for pointing out that zipline source code. That will certainly help. I would like to extend it in some ways, and the threshold feature you describe sounds great to me as well. :)

I noticed that this algo seems to rebalance each day at exactly the same time as the time that it was initially deployed even though there seems to be a much larger window for rebalancing. I am curious why this is. Can anyone explain?

Hi Aaron,

The algo is set up to rebalance only once in a given day within the trading window defined, the window is left larger than a single minute to make the code robust to unforseen difficulty in completing a rebalance in live market conditions (possible market data delay, system goes down, I forget to have the algoirthm logged into my account at market open, what have you), but normally should indeed rebalance at the first minute of the trading window. Once the rebalance logic has been triggered the context variables storing the last and next rebalance date are updated and the algo will do nothing until it reaches the next time when the trading window AND rebalance date are both reached.

Hopefully that's clear - happy to step through the code in more detail if its not, or if I missed your Q entirely let me know.

Best regards,
Jess

That is what I thought. The window is 10am-3pm. I would expect the algo to rebalance sometime after 10am. I deployed this algo at 13:16 on Monday to take advantage of Quantopian's generous free-trading-for-life offer! I was surprised to see it rebalance at 13:17 on Tuesday. I am waiting to see if it repeats this behavior today. I was trying to step through the code to see if there is something I do not understand.

Aaron - sorry, yes you're right. As coded right now, if you deploy this algo in between 10am and 3pm the timestamp (date + time) is what's used to key subsequent rebalances. In backtest the algo is always initially deployed at market open, so I hadn't actually noticed this subtlety.

If you'd like the algo to rebalance at 10am you can either a) relaunch the algo outside of the trading window (post 3pm through 10am following day) or b) modify the rebalance logic to key off of the date and not the timestamp (I'll try to get that second option coded up and post back here, but wanted to get you an answer that you're correct with what you've observed in the short term).

Thanks for pointing this out! Best wishes,
Jess

I was just trying to understand how the code worked. I think I understand now. I can fix it if I want to. Thanks!

exchange_time > context.rebalance_date + datetime.timedelta(days=context.Rebalance_Days)

Yup, so this should do it, truncate the rebalance date to be just a date, not a datetime (on line 79/80 of original algo). Make sense?

# Update the current and next rebalance dates
context.rebalance_date = exchange_time.date()

Hi Jessica,

In Quantopian, how are daily returns calculated from monthly rebalanced strategies? Obviously weights between rebalancings are going to differ than the original because of differing performance between stocks. Just wanted to see how that was done! Or if there is also any reference in Zipline.

Thank you.