@Bill L. Yeah, I'm not sure what's going on with these limit orders. They should get cancelled, yet don't. And they seem to stay in the market (because they eventually all fill...???)

Add some logging to your strat or try this one:

def initialize(context):  
    context.stock = symbol('TLT')  
    set_benchmark(symbol('TLT'))  
    schedule_function(func = StartOfDayEntry,  
                      date_rule = date_rules.every_day(),  
                      time_rule = time_rules.market_open(minutes=15))  
    schedule_function(func = UpdateState,  
                      date_rule = date_rules.every_day(),  
                      time_rule = time_rules.market_close(minutes=15))  
    schedule_function(func = EndOfDayCancel,  
                      date_rule = date_rules.every_day(),  
                      time_rule = time_rules.market_close(minutes=15))

def handle_data(context, data):  
    record(Close     = data[context.stock].close_price)  
    record(PnL       = context.portfolio.pnl)  
    record(Outlay    = context.portfolio.positions_value)  
    record(Leverage  = context.account.leverage)  
    record(CostBasis = context.portfolio.positions[context.stock].cost_basis)

# limit order to buy at specific level  
def StartOfDayEntry(context, data):  
    for stock in data:  
        order(stock, 100, style=LimitOrder(limit_price=85.5))  
        print("   Open order >>>")

# cancel at EOD  
def EndOfDayCancel(context, data):  
    for stock in data:  
        print("  Open position {0:>5} Net: {1:<6.0f}".format(stock.symbol, data[stock].NetQuantity))  
        if (data[stock].OpenLimit):  
            cancel_order(data[stock].OpenLimit)  
            print("Limit cancelled {0:>5} @ {1:>7.2f}".format(stock.symbol, data[stock].OpenLimit.limit))

#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  
def UpdateState(context, data):  
    for stock in data:  
        data[stock].NetQuantity   = context.portfolio.positions[stock].amount  
        data[stock].CostBasis     = context.portfolio.positions[stock].cost_basis  
        data[stock].HasOpenOrders = False  
        data[stock].OpenLimit     = None  
        data[stock].OpenStop      = None  
        if (get_open_orders(stock)):  
            data[stock].HasOpenOrders = True  
            for order in get_open_orders(stock):  
                if order.limit:  
                    data[stock].OpenLimit   = order  
                    data[stock].LimitLeaves = order.amount - order.filled  
                elif order.stop:  
                    data[stock].OpenStop    = order  
                    data[stock].StopLeaves  = order.amount - order.filled  

I get logs like this:

2002-10-01 Open order >>>
2002-10-01 Open position TLT Net: 0
2002-10-01 Limit cancelled TLT @ 85.50
2002-10-02 Open order >>>
2002-10-02 Open position TLT Net: 0
2002-10-02 Limit cancelled TLT @ 85.50
2002-10-03 Open order >>>
2002-10-03 Open position TLT Net: 0
2002-10-03 Limit cancelled TLT @ 85.50
2002-10-04 Open order >>>
2002-10-04 Open position TLT Net: 0
2002-10-04 Limit cancelled TLT @ 85.50
2002-10-07 Open order >>>
2002-10-07 Open position TLT Net: 0
2002-10-07 Limit cancelled TLT @ 85.50
2002-10-08 Open order >>>
2002-10-09 Open order >>>
2002-10-09 Open position TLT Net: 0
2002-10-09 Limit cancelled TLT @ 85.50
2002-10-10 Open order >>>
2002-10-10 Open position TLT Net: 0
2002-10-10 Limit cancelled TLT @ 85.50
2002-10-11 Open order >>>
2002-10-14 Open order >>>
2002-10-14 Open position TLT Net: 0
2002-10-14 Limit cancelled TLT @ 85.50
2002-10-15 Open order >>>
2002-10-15 Open position TLT Net: 0
2002-10-15 Limit cancelled TLT @ 85.50
2002-10-16 Open order >>>
2002-10-16 Open position TLT Net: 0
2002-10-16 Limit cancelled TLT @ 85.50
2002-10-17 Open order >>>
2002-10-17 Open position TLT Net: 300
2002-10-18 Open order >>>
2002-10-18 Open position TLT Net: 400

@Market Tech,

Thanks for having a look.

I have replicated the issue using your much neater code, and I see many of the same strange fills. The log output agrees with what you have above (we never trade more than 100 shares), but the prices of the fills still seem wrong.

Running a full backtest with your code and parameters: From 2002-10-30 to 2002-11-30 with $1,000,000 initial capital (minute data).

If I look at the "transactions" tab I see:

2002-11-01 buy 100 @ 86.03 (in 2 clips of 50 shares each)
2002-11-04 buy 100 @ 85.45
2001-11-05 buy 100 @ 85.76
2001-11-06 buy 100 @ 85.49
2001-11-27 buy 50 @ 85.55 and 50 @ 85.86
2001-11-29 buy 100 @ 85.25

Which is what I really don't understand. On days where we buy lower than 85.50, it makes sense (the market is below the limit when we place our order, and we get filled at the market level).
But on the 1st, 5th and 27th, what is happening? How can I buy 20 or 30 cents above my limit?

@Bill L. No doubt this is part of the Q's market model. I mean, it definitely IS built into Quantopian's market model (what they call a slippage model). I'd be more concerned with the missed cancels that the prices you get for limits. In the real market, if you are filled, you are filled at your limit or better. The fact that you're getting variations in fill price (above your limit) is Q's means of introducing volatility and volume restrictions risk.

I think a simple logging test with the enter limit / cancel limit, on the same day, needs to be done...

[Update]

It is absolutely Quantopian's market model at work here. How it interprets adequate volume; when it determines that the market is open for execution, what prices it should calculate for fills, all that goes into a market model affects these minutely back tests. Having built market models myself, I'm not entirely sure their representation of the market is as accurate as it could be. But, what can you do? If you want to play in the Open game, then you have to use their market model. If you're here trying to build a system for your own use then you may want to build your own slippage model. (I'm not sure how full featured a model could be given the surface of the environment quant's get exposed to.)

Evidence of slippage variations: change the minutes in the schedule_function from 1 to 60 to 180 and see the differences in order cancels and P&L.

def initialize(context):  
    context.stock = symbol('TLT')  
    set_benchmark(symbol('TLT'))  
    schedule_function(func = StartOfDayEntry,  
                      date_rule = date_rules.every_day(),  
                      time_rule = time_rules.market_open(minutes=180))  
    schedule_function(func = EndOfDayCancel,  
                      date_rule = date_rules.every_day(),  
                      time_rule = time_rules.market_close(minutes=180))

def handle_data(context, data):  
    costBasis   = context.portfolio.positions[context.stock].cost_basis  
    costBasis   = costBasis if costBasis > 0 else None  
    netQuantity = context.portfolio.positions[context.stock].amount  
    netQuantity = netQuantity if netQuantity > 0 else None  
    record(CostBasis   = costBasis)  
    record(NetQuantity = netQuantity)

def StartOfDayEntry(context, data):  
    order(context.stock, 100, style=LimitOrder(limit_price=85.5))  
    print("   Limit order entry >>>")

def EndOfDayCancel(context, data):  
    print("   Open position {0:>5} Net: {1:<6.0f}".  
          format(context.stock.symbol, context.portfolio.positions[context.stock].amount))  
    for order in get_open_orders(context.stock):  
        if order.limit:  
            cancel_order(order)  
            print("<< Limit cancelled {0:>5} @ {1:>7.2f}".format(context.stock.symbol, order.limit))  

@Market Tech, thank you again for looking at this with me. I was really unsure if this basic test was showing such odd results because I had misunderstood the framework, or there really was an error.

@ Quantopian staff, in term of using Q as backtesting platform, this really does seem to be outside of the realms of valid choices for a market model. If a trader places a limit order for 100 shares at 85.50, the only possible results are
1. completely filled at 85.50 or better
2. partially filled at 85.50 or better
3. zero fill

No matter what the assumptions are with regard to liquidity or volume restrictions, these are the only three possible outcomes (commissions aside). I don't see how any results of more complex trading algos can be judged if a simple test like this gives results that would be impossible in a trading environment, such as getting filled at 86.03 on a limit buy at 85.50.

I'm sure everyone at Quantopian HQ is very busy with the results of the first round of the live trading Open, but I'd be very interested to get some feedback on this issue. Apart from anything else it seems to invalidate historical backtesting results, so seems relevant to the Open as well!

Can Q reproduce these results? Do you agree that they are incorrect from a market simulation point of view, or is there an element at play here that we have missed?

Hi Bill,

Thanks for opening up this issue. It's quite a confusing topic and there have been a good number of questions asked on it. I think Market Tech summed it up well when he attributed to the difference in cost you're seeing your limit order versus the transactions tab in the full backtest (which adds on costs from the default slippage model).

I just quickly printed out 'context.portfolio.positions[context.stock]' in order to check out what the logs would produce and this is what I got

2002-11-01PRINTEvent({'status': 0, 'limit_reached': True, 'created': Timestamp('2002-10-30 14:32:00+0000', tz='UTC'), 'stop': None, 'reason': None, 'stop_reached': False, 'commission': 0.0, 'amount': 100, 'limit': 85.5, 'sid': Security(23921, symbol='TLT', security_name='ISHARES 20+ YEAR TREASURY BOND', exchange='NYSE ARCA EXCHANGE', start_date=datetime.datetime(2002, 7, 24, 0, 0, tzinfo=<UTC>), end_date=datetime.datetime(2015, 2, 3, 0, 0, tzinfo=<UTC>), first_traded=None), 'id': '5f855a5f74954434bca6fdd187d00376', 'dt': Timestamp('2002-11-01 15:52:00+0000', tz='UTC'), 'filled': 50})

2002-11-01PRINTPosition({'amount': 50, 'last_sale_price': 85.47, 'cost_basis': 86.0041875, 'sid': Security(23921, symbol='TLT', security_name='ISHARES 20+ YEAR TREASURY BOND', exchange='NYSE ARCA EXCHANGE', start_date=datetime.datetime(2002, 7, 24, 0, 0, tzinfo=<UTC>), end_date=datetime.datetime(2015, 2, 3, 0, 0, tzinfo=<UTC>), first_traded=None)})  

You can see that the last_sale_price is 85.47 which is below your limit price, but on a cost_basis the total ended up being more than your limit.

I wonder if that can clear up the confusion a little?

Hi Seong,

Thanks for taking a look, I appreciate it. I am very impressed with he potential of Quantopian and hope to use it extensively, which is why I'm concerned with simple sanity tests like this. Overall, I'm amazed at how much the team has achieved so quickly.

If the explanation that Market Tech gave is correct, it seems to mean that the backtesting results are not correct, in the sense that they are significantly different from what would be observed in practice. If the difference in cost between my limit order versus the transactions tab in the full backtest is due to the default slippage model, then the slippage model has added 50 cents slippage to a limit order, which is incorrect. Even if it was a market order, it would be pretty extreme (for 50 or 100 shares, of an instrument which traded about 100,000 shares daily at that time). But I don't want to get sidetracked into the details of the slippage model.

The key issue, I think, is that for a limit order, there cannot be slippage on price. Your cost basis can only be the limit price. The choices on the simulation side are how much of your order gets filled: from none, to all. But there is no possibility of trading above your limit. Does that correspond to your understanding of trading using limit orders?

Bill,

I see what you're saying.

The key issue, I think, is that for a limit order, there cannot be slippage on price. Your cost basis can only be the limit price.

I may be wrong on this, but I personally haven't encountered a situation where a limit order doesn't encounter commission costs. Thoughts on this?

Hi Seong, thanks for continuing to give this some attention, I think there is a relatively fundamental problem here, but I seem to be having some issues clearly communicating it.

I may be wrong on this, but I personally haven't encountered a situation where a limit order doesn't encounter commission costs.

Although it is true, I'm not sure that this is really relevant to the example we are trying to understand.

A limit order to buy 100 shares @ 85.50 receives a fill at 86.03. Are you implying that the 53 cent difference is due to the default 3 cent commissions?
If it was somehow due to commissions (for example, the commission model giving results 20 times too large), all the fills in this simple test would be at (85.5 + commission), but instead we see fills at 86.03, 85.76, 85.55 and 85.86.
So, with the clarification:

The key issue is that for a limit order, there cannot be slippage on price. Your cost basis can only be the limit price (plus commission)

I don't think we are any closer to understanding why this backtest fails to give fills that are compatible with actual trading results. Is it possible to explain these prices?

Hi Bill,

I understand what you're saying, although I don't have a clear grasp on the problem either. To me, because the last_sale_price is under your limit price, it seems that commissions per share + trade has to account for the difference between last_sale_price and cost_basis. Still investigating a bit further

Bill,

Here's a better explanation of what's going on. So there are two factors at work:

1) Commission costs are factored into the LIMIT order so if you were to set slippage and commissions to ZERO you would get limit orders that match your criteria.

    set_commission(commission.PerShare(cost=0, min_trade_cost=None))  
    set_slippage(slippage.FixedSlippage(spread=0))  

2) Also what's going on is that it's also a problem on our end that once you submit a limit order, we incorrectly go through the following steps:

• If the bar price triggers the order, the order goes into the execution engine.
  
• The execution engine then applies the slippage model.
  
• The order is filled using the slippage-adjusted price.

We need to take the slippage model out of it for limit orders - that will be a better model.

We're also thinking about modifying the slippage model instead of removing it in these cases. What do you think about looking at the high and low prices of the bar, and then filling the order depending on availability? For instance, imagine you have a limit order to buy 100 shares if the price dips to 10.00. In a one-minute bar, the high is 10.05, the low is 9.95, and the volume is 100. That order shouldn't completely fill - it should only fill for the parts of the bar that are under 10. Do you think we should head in that direction?

Seong

The sequence should be:

1) Apply price market model
2) Apply fill quantity market model
3) Apply commission model

For #1 price is a function order type and bar range. There is no way to speculate where price spent most of the bar and where most of the volume for that bar was transacted.
For #2 the % fill for the bar should be only volume based, again no way to know how much volume was executed above or below some limit or stoplimit price.

So one would determine the price to execute the order. Limit order prices are obvious. Market order prices are of course the close of the bar. Stop orders, however are trickier. Touching the stop price does not mean your order was necessarily transacted at that price. So, in this case only would you want to provide some reproducible variation in execution price. Whatever would be reasonable I suppose. What does the Q use now? (Perhaps x% of high/low range added to the stop -- if close > open, else subtracted from the stop if close < open)

Then determine the volume available to fill the order. A null volume model may consume all the volume available for that bar. Not realistic but hey, it's the null model. A realistic fill quantity model would take some % as you currently allow.

Add any commissions as specified by the model.

Hi Seong,

Thanks, that explanation makes sense. At least that applying slippage to limit orders can only make system results worse in a backtest, not better.

buy 100 shares if the price dips to 10.00. In a one-minute bar, the high is 10.05, the low is 9.95, and the volume is 100. That order shouldn't completely fill

Agreed, some kind of heuristic for fill volume is needed. But the details will not affect the system results too much: if the market trades lower for a few bars, the limit order will get filled anyway, just a few bars later. The only difference will be if someone trades at the exact low or high of the life of the limit order, which is generally not the case (or a good basis for a system).

Filling a number of shares linearly proportional to where the limit was on the bar seems as good as anything. So, 50 shares would fill that minute in your example, and 100% of the volume traded would fill if your limit was fully above the minute bar. (But as an aside, if the limit price is above the market at the time the limit order is submitted, it will be marketable and the slippage rule will apply up to the limit price: if high/low is 10.05/9.95 and I submit a buy at limit 10.5 in small enough volume that I should be totally filled, the price still needs to be determined)

Edit: clarified the last parental comment

Hi,

So far this example has been useful in pointing out a possible improvement to the backtesting engine. If it is possible, I'd like to continue looking at the same example, as I still think there may be additional steps needed to fully reconcile these fills.

First, we remove any effect of commissions and slippage, as you recommend with

set_commission(commission.PerShare(cost=0, min_trade_cost=None))  
    set_slippage(slippage.FixedSlippage(spread=0))  

Sticking with the same script and dates (trying to buy at limit 85.5 every day in November 2002)

Expected results (with no slippage or commission)
On Nov 1, 5 and 27, we should buy 100 TLT at 85.50
On Nov 4, 6, 29 we should get filled at market, below 85.50 when we place the limit order
On the other days we get no fill

Observed results
Nov 1 buy 100 @ 85.47
Nov 5 buy 100 @ 85.39
Nov 27 buy 100 @ 85.41

I suppose this is because these are the first last_sale_price which is less than the order's limit. But again, this is not the right level: the fills will still be at the limit exactly (in this case without commission). The way this is currently implemented unrealistically improves the trader's fills to the low of the bar they would have traded in. This has much less of an impact than the other point, but for algos which trade frequently it will nonetheless continually improve their backtested results.

Price market model
• On a bar by bar basis
• • For limit order
• • • Bar range contains limit order price
• • • • Fill @ limit price
• • • Bar high < limit price
• • • • Fill @ bar high
• • • Bar low > limit price
• • • • NO Fill

If Price market model == Filled
   Apply Fill quantity market model

If Fill quantity market model filled (size found to fill order)
   Apply Commission market model