I believe this:

if exchange_time.hour != 15 and exchange_time.minute != 55:  

should be this:

if exchange_time.hour != 15 or exchange_time.minute != 55:  

or perhaps tidier:

if not (exchange_time.hour == 15 and exchange_time.minute == 55):  

otherwise that conditional is going to return false and not return at any time during the 3pm hour, and any hour in the 55th minute.

Thanks Tom,

That was a typo on my part while modifying the algorithm to demonstrate the issue. Even when corrected the problem still exists. Here are some of the times being reported (See updated code below).

8.12pm
3.44pm
2.45pm
6.44pm
8.56pm
5.02pm

I'm in the UK so I would imagine that US Eastern time conversation would be in place. But the multiple times is odd. It also seems to change depending on the pair I have. Some pairs always trade around 8.55pm (which is UK time when NYSE is at 3.55pm). Am I missing something here? Does the backtest data contain exchanges in other time zones and is that factoring into the results somehow? i.e. Chicago?

Cheers,

A

How about if (exchange_time.hour == 15 and exchange_time.minute == 55):
then execute a trade.
Much cleaner and you do not have to guess what is going on.

This was a tricky one for me to unravel, but I think I figured it out. You're trading with thinly traded stocks, and that leads to two unexpected things.

1. When there is no data in a given call to handle_data, when there are no trades, we just skip that bar. This is one of those things that seemed like a great idea two+ years ago, but looks less good tonight. Your code only trades at 3:55. And if neither stock has trades at 3:55 - we just skip that bar, and no orders are placed.
   
2. Even when orders are placed, they don't always fill at 3:56. If there is no data. . . we skip that bar.

So here's how to make this example more clear. First, add a sid that trades essentially every minute.

context.spy = symbol('SPY')

Second, log your open orders. You'll get more insight into what is failing.

    open = get_open_orders()  
    if len(open) > 0:  
        log.debug('at ' + str(exchange_time) + ' open order count is ' + str(len(open)))  

Hi Dan,

I'm trying to sort out if this issue of skipping handle_data under backtesting when there is no data becomes more serious under live trading. Without the addition of a clock-like security such as 'SPY' would it be impossible to guarantee minutely feedback from the broker to the algorithm (since handle_data would not be called every minute for thinly traded securities)? For example, the algo could submit an order at minute 0, but not be able to query order status at minute 1 (due to a complete absence of trades for all securities listed in the algo between minutes 0 & 1). Or do live algos also trigger handle_data based on the availability of broker data?

By the way, there is a missing part of the story here, since if I recall correctly, the addition of 'SPY' in the algo above will also result in filling of gaps in data for thinly traded securities, correct?

Grant

Yes - After adding SPY or any other reliable minutely updated security I can see that the algo executes at 8.56pm UK Time (3.56pm US/Eastern). For the timebeing I will have to go back to the drawing board to find higher volume/frequency securities for this algo. However, I agree with Grant's point and would like to know how this would work in a live trading scenario with thinly traded securities.

What would happen in live trading is essentially the same, but with the added change that when your order is filled, that would result in a trade bar, and that would trigger the handle_data() run.

This is one of those design decisions that looked neat two years ago, but may not be the best choice going forward. We may change it.

For now, if you throw "context.spy = symbol('SPY')" into initialize() when you're worried about thinly traded stocks, that will keep any odd behavior from cropping up.

Thanks Dan,

Good point. For an order filled by IB, presumably your data feed from Nanex NxCore would pick up the trade immediately and you (or you vendor, perhaps) would generate a bar the next minute. But for the algorithm to see other events at the broker (e.g. rejected orders), SPY would need to be added, correct?

In the long run, it seems like the work-around of adding SPY as a clock is not gonna be the best approach. Has there been any consideration of an alternative?

Regarding gaps in the data and filling, here's a relevant discussion:

https://www.quantopian.com/posts/thinly-traded-stocks-why-no-gaps

To detect bar filling, code like this can be applied (see attached backtest):

for stock in data:  
        if data[stock].datetime < get_datetime():  
            log.info("no trades in {s}".format(s=stock.symbol))  
            print data[stock].datetime  
            print data[stock].volume  

Alternatively, history can be used with the forward filling turned off.

Grant