Possible bug in data transform

@batch_transform(refresh_period=1, window_length=1) def get_day_prices(datapanel,sid): closePriceSeries= datapanel['close_price'][sid]; log.debug('This is a tail of the time series'); log.debug(closePriceSeries.tail()); log.debug('The last entry close price is '+ str(closePriceSeries[-1])); log.error("Observe the %f is on %s and not on %s" % (closePriceSeries[-1] , closePriceSeries.index[-1].date(), closePriceSeries.index[0].date())); return closePriceSeries[-1];

2013-03-27get_day_prices:17DEBUGThis is a tail of the time series 2013-03-27get_day_prices:18DEBUG2013-03-26 20:57:00+00:00 460.97 2013-03-26 20:58:00+00:00 460.81 2013-03-26 20:59:00+00:00 460.98 2013-03-26 21:00:00+00:00 461.13 2013-03-27 14:31:00+00:00 454.08 Name: 24 2013-03-27get_day_prices:19DEBUGThe last entry close price is 454.08 2013-03-27get_day_prices:20ERRORObserve the 454.080000 is on 2013-03-27 and not on 2013-03-26

Hi Guillermo,

Thanks for sharing your findings, and thank you for probing into the batch transform's behavior.

The reason you are seeing two different dates is that the window_length is measured in days. Since you are testing with minutely data, the batch_transform holds a day of data, but does a rolling update. So, when the first trade of 3/27 arrives, the window stretches back to 3/26. We explain the behavior a bit more in our help docs: https://www.quantopian.com/help#ide-batch-transforms

We chose to have the minute data "roll" with partial days in order to keep the number of data points in a full window as consistent as possible. We cope with weekends and holidays so that the window is really measured in trading days.

I updated your algo a bit to log the head and tail of the window, as well as the full timestamp for the observations. Here's the output:

2013-03-27get_day_prices:17 DEBUG This is a tail of the time series  
2013-03-27get_day_prices:18 DEBUG  
2013-03-26 14:32:00+00:00    463.9425  <<------ first row at 9:32 EST (stamp is UTC)  
2013-03-26 14:33:00+00:00    463.4523  
2013-03-26 14:34:00+00:00    463.1300  
2013-03-26 14:35:00+00:00    463.5000  
2013-03-26 14:36:00+00:00    463.9801  
Name: 24  
2013-03-27get_day_prices:19 DEBUG  
2013-03-26 20:57:00+00:00    460.97  
2013-03-26 20:58:00+00:00    460.81  
2013-03-26 20:59:00+00:00    460.98  
2013-03-26 21:00:00+00:00    461.13  
2013-03-27 14:31:00+00:00    454.08 <<---- last row at 9:31 EST (stamp is UTC)  
Name: 24  
2013-03-27get_day_prices:20 DEBUG The last entry close price is 454.08  
2013-03-27get_day_prices:21 ERROR Observe the 454.080000 is on 2013-03-27 14:31:00+00:00 and not on 2013-03-26 14:32:00+00:00  
End of logs.

thanks,
fawce