Live trading and backtesting behave the same in terms of when handle_data is called: on those minutes when any securities in the algo's universe were traded.

-Rich

Thanks Rich,

Looking over the help docs, I see "With Fetcher, the CSV file's date/time information can change the event frequency." So, is it possible, under live trading, to call handle_data more frequently than every minute? Also, could handle_data be called at 9:30 am (market open), if the right data are fed in with Fetcher? And if so, could an order be submitted right at the open?

Grant

As Rich mentioned, live trading and backtesting call handle_data the same way - it will get called once per minute. If you're using Fetcher in live trading and your file contains sub-minute information, it will only return the last datapoint during the minute run. It will not act upon second or tick data. For more information, here is the Zipline code.

Just as in backtesting, you cannot live trade at the market open (although we're working on it!). Any market orders from Fetcher during the minute of 9:30AM will be rolled into the first call of handle_data at 9:31AM.

Thanks Alisa,

So, I still don't understand what is meant by "With Fetcher, the CSV file's date/time information can change the event frequency." From what you are saying, the date/time information in a CSV file cannot change the event frequency--it will always be one call to handle_data per minute. Aren't these contradictory?

Grant

Yes - it looks like our help doc in that spot is out-of-date. We'll get that updated shortly. I just checked with other people here, and that section was written when a specific problem with Fetcher came up. That underlying problem has been fixed, and that section needs to be obsolete. Regardless, it's confusing as written.

The problem was that if you mixed daily fetcher data with minutely simulations, then you could get an "extra run" of handle-data to handle the "daily" data point. That problem doesn't happen anymore. So, you see, the "change event frequency" language in the help was pretty tortured and I see how it is confusing.

Bottom line: handle_data() will run every minute that there is data present, no more, no less.

Thanks Jess,

I started to read the fetcher docs, and plan to give it a try sometime. I'll offset the timestamps and see what happens. Presumably, the data with offset timestamps will be ignored? Or no?

Grant

Huh...a switcheroo Jess --> Alisa? Kind of an odd practice...what if users started swapping the identities of their posts...would get confusing for y'all.

Jess/Alisa/somebody, using fetcher with offset timestamps (fractional minutes), what happens?

Hello Grant,

Have a look at the logs for the algo at https://www.quantopian.com/posts/forex-data-2 which uses sub-minute date i.e.

02/01/2013 09:31:00 87.032  87.045  87.039  87.0391  
02/01/2013 09:31:01 87.021  87.034  87.029  87.0291  
02/01/2013 09:31:02 86.99   87.04   87.02   87.0208  
02/01/2013 09:31:03 87.021  87.022  87.023  87.0234  
02/01/2013 09:31:04 86.99   87.03   87.013  87.0131  
02/01/2013 09:31:06 87.009  87.021  87.016  87.016  
02/01/2013 09:31:09 87.009  87.021  87.015  87.0154  
02/01/2013 09:31:10 87      87.017  87.009  87.0093  

This data is then available in the algo as minutely:

2013-01-02 PRINT 14:31:00 87.162  
2013-01-02 PRINT 14:32:00 87.13  
2013-01-02 PRINT 14:33:00 87.168  
2013-01-02 PRINT 14:34:00 87.169  
2013-01-02 PRINT 14:35:00 87.15  
2013-01-02 PRINT 14:36:00 87.166  
2013-01-02 PRINT 14:37:00 87.145  
2013-01-02 PRINT 14:38:00 87.12  
2013-01-02 PRINT 14:39:00 87.162  

The minutely price seems to be the close from the previous bar i.e.

02/01/2013 14:30:59 87.162  87.165  87.165  87.1651  
02/01/2013 14:31:00 87.162  87.163  87.163  87.1633  
02/01/2013 14:31:01 87.162  87.165  87.163  87.1631  

which is ambiguous and

02/01/2013 14:31:58 87.13   87.21   87.169  87.1696  
02/01/2013 14:32:02 87.157  87.169  87.164  87.1647  

which is not. In

02/01/2013 14:38:59 87.156  87.168  87.164  87.1647  
02/01/2013 14:39:00 87.162  87.173  87.166  87.1665  
02/01/2013 14:39:01 87.16   87.18   87.168  87.1683  

'14:39:00' seems to be included in the previous bar

P.

Mysterious...Quantopian folks, perhaps you could shed some light here? How should fetcher work with misaligned timeseries data? --Grant

Hello Grant,

What's a "misaligned time series"?

As Alisa says above "If you're using Fetcher in live trading and your file contains sub-minute information, it will only return the last datapoint during the minute run." That is what we see in the logs. And we now know that the the bars comprising sub-minute data are closed on the right at 'hh:mm:00'.

P.

Hi Peter,

By "misaligned" I just mean that the data loaded via fetcher sometimes fall at fractional minutes. All of the data are not aligned with the security data stream provided by Quantopian.

I think I'll have to try fetcher myself sometime, and play around with it. I gather that sub-minute data can be used in an algorithm, but with some limitations? In any case, once I try fetcher, I'll sort it out.

Grant