Hello AM,

Thanks. I'm paper trading my example above at the moment as follows:

I've also attached your example - there are two transactions:

2013-04-18 01:00:00 AAPL    BUY      100    $391.93 $39,193.00  
2013-04-19 01:00:00 AAPL    SELL    -100    $390.64 ($39,064.00)  

P

Hi Peter,

Let's say that you have 100 shares of a stock and you want to take a long position. Then you will need to add them to your portfolio by saying:

order(stock,  100)  

Now, you want to protect yourself and close the entire position if the stock price falls below the 5% threshold. Then you will add this bit of code below.
order(stock, -100, stop_price = price * 0.95)

But keep in mind that elsewhere in your algo, you would have needed to track the price of the stock when you entered the position. You can do it with this statement:

if context.enter == True:  
  context.entered_price = data[context.stock].price  
  log.info("Price of stock bought is:  " + str(context.entered_price))  
  context.enter = False  

So your initial notation from your first post (pasted here) is correct, assuming you're going for a long buy strategy and not shorting. First you need to open the position, and THEN create a stop order. There indeed will be 2 transactions.

price = data[stock].close_price  
order(stock,  100)  
order(stock, -100, stop_price = price * 0.95)  

Let me know if that helps!

Hello Alisa,

Can you elaborate on the mechanics of a stop order as an example, please? Assume I go long with a market order that gets filled. I then place a sell stop at a price 5% below the purchase price. The stop order show as a pending order and is cancelled overnight. Is that correct? Although in paper trading with Quantopian (not IB) or in the backtester the order is not cancelled.

I see two possible scenarios:

(i) the stop order resides on the Quantopian server. When the stop is reached a market order is sent to IB.

or

(ii) the stop order is sent to IB but is still recorded on Quantopian as pending. If the stop is reached a market order is executed. Some communication would be required from IB to alert Quantopian.

In (i) the stop would be triggered by a Nanex price and in (ii) by an IB price. This makes me think (i) is more likely.

P.

Hi Alisa,

A related question is what if the orders go through per your example above:

price = data[stock].close_price  
order(stock,  100)  
order(stock, -100, stop_price = price * 0.95)  

Then, at some later point, the shares are sold:

order(stock,  -100)  

Is the limit order automatically cancelled? Or will it still be hanging around? It seems like it would need to be cancelled at the same time that the long position is exited, right?

[EDIT] Well, I'm confused again. It seems that there would need to be a linkage at IB between the separate orders, so that the limit order is cancelled just after the sell order is confirmed filled. If the limit order gets cancelled before the fill, then it is risky.

Grant

More confusion. Anony's logic above would seem to make sense for entering the two orders. First, the market order is submitted. Once it is confirmed to be filled, then immediately the limit order is submitted to cover the risk. Why? Because, presumably the limit order will be in place at IB immediately, while the market order will only be filled when the market supports the transaction. So, if both the market and limit orders are submitted simultaneously, then one could end up with a limit order covering a yet-to-be filled market order.

An obvious question is can all of this be done within a single call to handle_data, or will the algorithm need to wait until the next call to handle_data to determine if the market order has been filled, to then submit the limit order?

I'm curious...wouldn't Interactive Brokers have devised an single order type for this sort of thing?

Grant

Hello Grant,

I'm sure we will get some clarification soon. My best guess? Quantopian only send market orders to IB and manage stops and limits on their own server because they use Nanex prices.

P.

Anony,

I rejiggered your code a bit, and it runs now. Appears to exhibit the correct behavior, although I haven't checked it down to the nth detail.

Grant

Hi Peter et al,

I'd be happy to further elaborate on my response.

Can you elaborate on the mechanics of a stop order as an example, please? Assume I go long with a market order that gets filled. I then place a sell stop at a price 5% below the purchase price. The stop order show as a pending order and is cancelled overnight. Is that correct? Although in paper trading with Quantopian (not IB) or in the backtester the order is not cancelled.

When the market price hits your 5% loss threshold, this will convert the stop-order into a market-order. The order will then be executed as a regular order and attempt to sell your designated shares. You mention "stop order.. cancelled overnight"... that's almost correct. Let's imagine this edge-case scenario:

4:00PM - stop-order price is hit and converts order into a market-order. However the market is closed and no more trades can be placed. All open orders are expired at the end of the day, as explained in this thread.

9:30AM: The algo is running the next day and looks at the market price. We don't know what happened overnight. The price could have wildly risen, in this case, the stop-order is no longer valid and it will not attempt to exit the position. Alternatively the price stayed below the threshold, in which case we execute the stop-order and convert it to a market order. We place the order to sell, which can be filled within a few seconds depending on the liquidity.

Either way, your algo is checking your stop-order price against the market price. When the threshold is met, it will place an order to exit the position.

I see two possible scenarios:
(i) the stop order resides on the Quantopian server. When the stop is reached a market order is sent to IB.
or
ii) the stop order is sent to IB but is still recorded on Quantopian as pending. If the stop is reached a market order is executed. Some communication would be required from IB to alert Quantopian.

The first scenario is accurate. Our market feed comes from Nanex, which provides the pricing for all positions. The stop order is triggered by the Nanex price, alerts us, and sends the market order to IB for execution.

@Grant:

In my example to Peter I was using a stop-order, not a stop-limit order and I think Anony gave a great description.

if both the market and limit orders are submitted simultaneously, then one could end up with a limit order covering a yet-to-be filled market order.

The onus is on the code writer to prevent that scenario from happening - and this would likely get caught during backtesting before deploying to paper or live trading. Your example shows a great way of checking if we've filled our position and then setting the stop-order.

I too am a fan of "rejiggered", I'll see if I can throw it in sometime :)

Thanks Alisa,

Sorry for not quite following. Which order types are sent to Interactive Brokers immediately, and which are held at Quantopian, and triggered by Nanex prices? Also, which order types stay in place overnight? And which are cancelled EOD?

Grant

When you are paper/live trading with IB the behavior is as follows:

• Deploy algo to live market
  
• algo receives aggregated minute bars using the Nanex price feed that runs handle_data once per minute
  
• When an order trigger is hit, the algo sends the order to IB -- depending on the liquidity, it can be filled within seconds
  
• Handle_data() runs again in the next minute and updates any positions and checks for any order triggers

Based on this, all order types are triggered by Nanex prices and immediately sent to IB. We do not hold any orders, we pass them right to the broker.

Thanks Alisa,

So, if I understand correctly, only market orders are sent asynchronously to IB and filled ASAP by them. The other order types are recorded by Quantopian asynchronously:

limit order: Use order(sid, amount, limit_price=price) to place a limit order. A limit order executes at the specified price or better, if the price is reached.
stop order: Call order(sid, amount, stop_price=price) to place a stop order (also known as a stop-loss order). When the specified price is hit, the order converts into a market order.
stop-limit order: Call order(sid, amount, limit_price=price1, stop_price=price2) to place a stop-limit order. When the specified stop price is hit, the order converts into a limit order.

However, they are not sent to IB, and are potentially converted to market orders at a minutely frequency, if a trigger condition is met based on Nanex aggregated minutely bars (presumably all OHLC values are used, not just the close to determine the trigger condition).

Also, all market orders sent by Quantopian to IB are good until EOD/market close, when they are cancelled automatically. Similarly, any orders still at Quantopian (not yet converted to market orders) are cancelled at EOD, as well.

I've gathered that get_order, get_open_orders, and cancel_orders are all asynchronous, including providing order status at IB, within the minute. Or no?

Correct?

Also, what information is returned by get_order and get_open_orders?

[EDIT] If my thinking is correct, a market order would be filled by IB almost immediately, but it would take ~58 more seconds to have an associated stop loss order effectively in place at Quantopian (since it could not be triggered until the next minutely bar). Seems like one would want the stop loss to go in right after the IB order is filled, right?

Grant

I'm still trying to sort out how all of this works. I've gathered that the only IB order type that Quantopian supports is Market (see https://www.interactivebrokers.com/en/software/webtraderguide/webtrader/orders/supported_order_types.htm). And any market order sent to IB from Quantopian has a Time in Force of Day (it will be automatically cancelled at the end of the trading day).

Regarding orders other than market orders, they are not really orders in the sense that they are not sent to the broker, IB, but rather are managed by Quantopian as an intermediary. So, in a Quantopian algorithm, this is an order:

order(sid, amount)  

The other supported "order" types (limit, stop, & stop-limit) are effectively helper functions that use the feed of aggregated Nanex minute bars to decide if a market order should be submitted to IB. In other words, one could just as well write custom limit, stop, and stop-limit order functions (which would utilize the basic market order, order(sid, amount)).

I've tried to sort out the difference between a limit order and a marketable limit order...perhaps someone could clarify.

Here's a book that may be of interest:

Trading and Exchanges: Market Microstructure for Practitioners by Larry Harris
http://amzn.com/0195144708

http://raudys.com/kursas/2002%20Trading%20and%20exchanges.pdf (draft)

And another apparently relevant reference:

Order Submission Strategy and the CuriousCase of Marketable Limit Orders
http://faculty.babson.edu/sirri/research/limit%20orders%20JFQA%20june%202002.pdf

Hello Grant,

I would really like to know if your understanding is correct.

It seems that if manually trading through IB's TWS that Stop and Stop Limit orders are simulated by IB as they are not native to the exchange i.e. for NYSE see here: https://www.interactivebrokers.co.in/en/?f=%2Fen%2Ftrading%2Fexchanges.php%3Fexch%3Dnyse%26amp%3Bshowcategories%3DIND%26amp%3Bib_entity%3Din

But if you are right then Stop, Stop Limit and also Limit orders are additionally simulated by Quantopian. By contrast, and I'm guessing again, it seems that something like IB-Matlab has access to the full range of order types at IB i.e. anything you can manually do in TWS can be done in Matlab.

P.

Yep...I've been getting some offline guidance from Alisa, and gradually, it is sinking in how the Quantopian system works with respect to IB. One concern (perhaps not valid) is that if one submits a market order, it goes immediately to IB, however, if there is an applicable stop-loss order, it would not be evaluated for another ~ 60 seconds. I ain't no trader (obviously), but it would seem better to apply the stop-loss as soon as the market order is filled (and only put a stop-loss in for the fill amount). There's another timing issue I just considered, which is that Quantopian does not clarify exactly when the limits are evaluated. If the evaluation is just before the call to handle_data, and it takes up to 2 seconds for a triggered market order to be filled at IB, then it wouldn't be until the next call to handle_data that the order status and portfolio are updated (unless they've built in some latency in their system to account for the delay).

--Grant

Upon quick scanning, the order management code is here:

https://github.com/quantopian/zipline/blob/master/zipline/finance/blotter.py

https://github.com/quantopian/zipline/blob/master/zipline/finance/slippage.py

Presumably, under live trading w/ IB, this code is used.

Hello Grant,

(Funnily enough I wrote check_order_triggers at https://github.com/quantopian/zipline/blob/master/zipline/finance/slippage.py although Eddie improved it! It's my only contribution to zipline.)

This topic is interesting. I don't know if it will happen but I'm hoping to get an IB account by the end of March, for obvious reasons. Once I can run an algo against the paper trading account I will be happier.

This discussion seems relevant:

https://groups.yahoo.com/neo/groups/TWSAPI/conversations/topics/31158

This stands out concerning an algo accessing the IB API or (worse?) an algo running in Quantopian accessing the IB API:

Where the exchange doesn't offer a native stop order, IB usually has a
simulated stop order which automatically does rapid order amendments with
stop-limit orders in the way you described - but much more effectively than
you can do it yourself, because IB are very close to the exchange.

P.

Yes, I think effectively the same thing can be accomplished in Quantopian, with a series of stop orders applied over a series of calls to handle_data, as the applicable market order is filled at IB. I've started to tinker around with code that would accomplish this. --Grant

Thanks Anony,

Yes...the idea of a diagram has crossed my mind.

Could you elaborate on the difference between a limit order and a marketable limit order, you mention above?

Grant

Thanks Anony,

I'll have to think about if any of this is relevant in the context of Quantopian. As I understand, Quantopian will only send market orders to IB, so there will be no "sweeping the book" correct?

Grant

Well, I count 1.107 M shares on the Ask side, so the order would be filled completely, but at an average price somewhat above 29.63 (since some shares at the higher price would need to be included). Correct?

Thanks...so, back to my question. I gather that there is no such thing as a marketable limit order in the context of Quantopian. In other words, there is no way to, say, buy 100 shares of XYZ at or below $10, since the Quantopian limit order isn't working with the IB book (a Quantopian limit order is, if I understand correctly, more of an algorithm helper function, since I could just write one myself). I could end up buying at higher than $10, since only a market order is executed at IB. Correct?

--Grant

Good point regarding the effective "Quantopian order book." They have not detailed the order of precedence of orders from Quantopian to IB, but maybe it is first in, first out? I think the argument will be that they are not offering HFT, so it just doesn't matter.

Another thought that occurred to me is that since the Quantopian limit orders are triggered on aggregated Nanex minute bars, wouldn't it make sense to use the high or low price, rather than the close? I gather that the close is used. For example, if I am trying to avoid a loss, I think I'd prefer to use the low price, rather than wait another minute for the close to finally get below my trigger price to sell.

Hey all,

Just to clarify, Quantopian does not hold any orders. We send all orders (even stop/limit orders) immediately to the broker (IB).

-Rich

Hi Rich,

Maybe you can clarify. My understanding (perhaps incorrect) is that although IB supports a variety of order types (e.g. https://www.interactivebrokers.com/en/software/webtraderguide/webtrader/orders/supported_order_types.htm), the only order type that is sent to IB from Quantopian is a simple market order. Triggered orders are managed ("held") by Quantopian until the trigger is satisfied based on the aggregated Nanex minute bar feed, at which point a market order would be sent to IB from Quantopian.

But again, I'm probably totally confused...

Grant

Hi Grant, no problem.

As you know, Quantopian supports 4 order types, based on the combination of arguments with which your algorithm calls the order method. Quantopian actually handles all of these order types exactly the same: when an algorithm places an order, it's sent immediately to IB (with IB's corresponding order type). In this way, IB manages all the triggers you mentioned.

So if your algorithm places a limit order, IB will work that limit order for you.

-Rich

Thank Rich,

But what about Alisa's post above:

When you are paper/live trading with IB the behavior is as follows:
Deploy algo to live market
algo receives aggregated minute bars using the Nanex price feed that runs handle_data once per minute
When an order trigger is hit, the algo sends the order to IB -- depending on the liquidity, it can be filled within seconds
Handle_data() runs again in the next minute and updates any positions and checks for any order triggers
Based on this, all order types are triggered by Nanex prices and immediately sent to IB. We do not hold any orders, we pass them right to the broker.

Was her description simply incorrect?

Also, from what you are saying, the trigger levels are determined by IB, not Quantopian, correct?

Just trying to figure out what to flush out of my mind, and what should remain.

Grant

Great questions Grant, and sorry for the confusion. Yes, that description does have some inaccuracies, or maybe requires a liberal interpretation of the phrase "order trigger." And that's right, Quantopian doesn't manage any triggers for converting orders, etc - that's all IB.

To revise Alisa's description:
Each minute, handle_data will run if there were any trades in your universe (according to the Nanex feed). If so, and your algorithm decides to place an order via the order method, it goes right to IB. The next minute, handle_data may run again, and your algo code will be able to see the latest fills and positions from IB when it decides whether to place more orders.

-Rich

Thanks Rich,

And I think that you answered one of my other questions:

https://www.quantopian.com/posts/calls-to-handle-data-every-minute-under-live-trading

If I hear you correctly, handle_data will remain event-driven under live trading (and presumably there will be no way to force it to run every minute, other than a work-around such as adding SPY to the universe).

Grant

Glad to help.

That's right - live trading and backtesting behave the same in terms of when handle_data is called: on those minutes when any securities in the algo's universe were traded.

-Rich

Thanks Rich,

You state above "The next minute, handle_data may run again, and your algo code will be able to see the latest fills and positions from IB when it decides whether to place more orders." My interpretation is that Quantopian only polls the current IB state just before each call to handle_data, and then makes the state available to the algorithm, correct? The algorithm cannot access the current (real-time) IB state within a call to handle_data, correct?

Also, in addition to asynchronous (immediate) ordering, order cancellation instructions are sent asynchronously to IB, as well, correct?

Grant

Hi Grant,

Yes, that's exactly the behavior you'll see - the state of the live algorithm's orders and portfolio will not change for the duration of a call to handle_data, even if the IB state is changing in real-time.

Right, order cancels are sent immediately to IB, just like requests to place new orders.

-Rich

I tried a simple test on my IB paper account:

def initialize(context):  
    context.SPY = symbol('SPY')  
    context.flag = False


# Will be called on every trade event for the securities you specify.  
def handle_data(context, data):  
    if context.SPY not in data:  
        return  
    log.info('{}'.format(data[context.SPY].price))  
    if context.flag == False :  
        order(context.SPY, 100)  
        order(context.SPY, -100, stop_price = data[context.SPY].price - 0.03)  
        context.flag = True  

The original order was filled, as expected, and the 'stop' order was recorded as 'open' - also as expected.

However, as can be seen from the log, although the stop price was penetrated, no order was executed at IB at that price.
Shouldn't the 'stop' have been executed?

Here is the dashboard and log output:

dashboard

LOG

2014-04-08 21:48handle_data:11INFO185.145
2014-04-08 21:49handle_data:11INFO185.185
2014-04-08 21:50handle_data:11INFO185.18
2014-04-08 21:51handle_data:11INFO185.101
2014-04-08 21:52handle_data:11INFO185.11
2014-04-08 21:53handle_data:11INFO185.02
2014-04-08 21:54handle_data:11INFO185.04
2014-04-08 21:55handle_data:11INFO185.058
2014-04-08 21:56handle_data:11INFO185.09
2014-04-08 21:57handle_data:11INFO185.15
2014-04-08 21:58handle_data:11INFO185.189
2014-04-08 21:59handle_data:11INFO185.17
2014-04-08 22:00handle_data:11INFO185.105

isn't that at 4:45pm EST? (after market) so you'd have to wait until the next day? also looking at your log, I don't see why the stop price shown is 185.12. since the sale was at 185.10, am I right to assume that your stop should trigger at 185.07?

Hi Jason,

the algo was run while the market was open - I'm in London at the moment, hence the time difference.

The stop price was set 3c below the purchase price deliberately, so that I could be reasonably sure that the stop-price would be penetrated. The dashboard shows the stop-price as 185.12, the buy-price at 185.10. I assume the small discrepancy is due to commission/slippage? In any event, the log shows the price going as low as 185.02, so the stop order should have executed at IB, right?

Hi Dave, iirc all quantopian log times are in UTC. I thought there was a 5 hour difference from EST to UTC so I thought it was 4:48 EDT (maybe I'm off by 1 hour due to daylight savings time?)

actually i just checked, and isn't 21:48 UTC actually 17:48 EDT? seems after market to me..... EDIT: sorry eastern usa is EDT, not EST

re the price trigger, yeah if it's in market it should have triggered. I"m just wondering why the stop price is 185.12....

Hi Jason, difference is 6 hrs due to daylight saving time in the UK, so the market was definitely open. The order was filled at price 185.16 at 15:49., stop price 185.12 (difference probably due to commission and slippage, not relevant to this anyway). At 15:51 price dropped below stop price and stayed there 'til 15:57. The stop should have triggered....

UPDATE:
I stopped the algo at 16:41 (10:41 exchange time) on the 9th April. The dashboard was showing NO POSITIONS(!), and the stop-order was showing CANCELLED.
Logging in to my paper account, I was surprised to see that 100 SPY share were sold at 23:15 (17:15 exchange time) at price 185.02!

SLD 100 SPY Stock 185.02 ARCA APR 8 23:15:23 1.41 null

Either something is wrong, or I'm misunderstanding how order(security, stop_price=...) is supposed to work.

Please can someone help as I'd really like to add a working stop-loss to my real-money live algo.

Hi Dave/Jason,

I copied your code and ran it against an IB paper account today at 2:00PM EDT. The code worked as written, but perhaps not as you intended.

In your code, the "order" and "stop_order" are submitted in the same bar. The price used for the stop_order is the close price of the previous bar. I imagine you only want to enter the "stop_order" once your order is filled. In that case, you first need to check if there are open orders, log the fill price, and then reference the fill price in your "stop_order" function.

Additionally, you are only logging the close_price. If the stop_order threshold is crossed during the minute bar (for example, as the low_price), the stop order is converted to a market order. but you won't see the details in the logs! To see the full price range in the minute, I would recommend logging the high, low, and close to understand the behavior.

One final note, the live algorithm dashboard shows the fill price, which is determined by IB. The logs are driven by a real-time pricing data feed from Nanex that is aggregated into one-minute bars. So there may be a couple small discrepancies between the data feed prices, but you will see the IB fill prices for your securities in the dashboard. Hope that helps!

Cheers,
Alisa

Alisa, I understand what u r saying, but my question is why the stop did not trigger when the stop price was clearly touched while the exchange was open and then triggered at 23:15 (17:15 eexchange time). I don't understand why you say the code is behaving 'as written'. am I missing something?

There are a couple possibilities here. First, what are you calling “exchange_time”? We use NYSE market hours which are from 9:30AM - 4:00PM EDT.

The log output in your original post says “2014-04-08 21:48". The 21:48 timestamp is in UTC, which is equivalent to 4:48PM EDT and is after market close. Similarly, 23:15 (or 17:15) is after market hours. In the dashboard screenshot, under the orders and fills, you can also see the same timestamp of the bar when the order was submitted.

Your algorithm placed the order as directed, but it is up to the broker (IB) and their routing system to fill the order. If you have questions about IB's order routing, I would suggest to contact them directly (unfortunately I don't have those details!). We are using IB's Smart Order Router algorithm for order execution. Feedback is welcome if another execution algo is preferred!

21:48 is 15:48 EDT (as the UK is now on summer time) - see my discussion above with Jason. Quantopian stops outputting prices at 16:00 EDT, which you can see clearly from the log.

Digging through timezones can make one's head spin (including mine!), so here is a summary of the algo details:

• Your log output is in UTC
  
• Your dashboard screenshot shows your laptop time, which is the Johannesburg timezone
  
• The algorithm was deployed at 3:48PM EDT with order price of $185.145, meaning the stop_order price was $185.115
  
• Your IB dashboard reported the stop order was filled at 5:15 EDT
  

Now that we have the correct timezones and sequence of events, it indeed looks strange that the order was not filled. What it looks like is that the order was sent to IB as we'd expect, but IB didn't fill it. What's even more odd is that every order we send to IB is sent with an expiration time of market close. We never expect an order to be filled outside of market hours.

Once we've sent an order to IB, it is up to IB to actually fill the order. We do not have control over the order fulfillment. We've noticed strange things before in IB's paper environment. In other IB paper accounts, both ours and other member's, they occasionally experience odd behavior. You may have ran into an IB blip. You might call them to explain the situation, but they like to spend their time on real money calls, and don't always worry about paper trading.

Another possible explanation is that the order was filled as expected, but IB was slow to notify/update the dashboard. In either case, IB is the right place to ask about the order fill details.

Alisa,

Can you help me to enter OCA -- one cancels all stop/stop limit orders?

-Russell

Ugh -- does the above discussion really mean that STOP orders have to be re-submitted everyday because in IB (unlike the backtester, zipline) orders are canceled at the end of the day?

Can someone at Quantopian please implement GTC orders, if only for STOPs? That would be super swell guys!! B)

Just pass on the the GTC flag to the order in IB and let IB take care of it. I'm guessing, maybe, this causes some incompatibility with ZipLine.

Hi Russell,

In live trading, all market orders are submitted to IB with a same-day expiration time of 4PM. This means that any open orders (stop, limit, regular orders) will get cancelled at the end of the trading day. Your algo will enter a new STOP order each day if it wasn't previously executed.

I saw your private message about OCA orders, which I believe we answered. Let me know if you need anything else!

Alisa

Is there a "call back" so we can know whether a stop order triggered, or do we have to cycle through open orders ourselves and check the last list of orders vs. the the current?

There isn't an explicit "call back" per se but you can look at the get_order object and see the stop price status. When the stop_price is triggered it will return as True.

Here's an example of how to check if your stop_price was reached: https://www.quantopian.com/posts/how-to-set-a-stop-loss-with-a-set-universe