Question about using data.history() and data.current()

Back to Community

edited

The following sample code runs over the period from 2016-08-31 to 2016-10-30 and uses AAPL US as an example.
The schedule_function() runs every [email protected] minute before market close and it prints out the last 4 daily close prices using data.history() function.

Please refer to the log message @2016-08-31 03:59

First, why does the closing price of 2016-08-30 (data.history()) and 2016-08-31 (data.current()) show the same value (I have verified from other data source they should not be)?

Second, the closing price of 2016-08-30 should have known when the schedule_function() run at 2016-08-31 03:59, but it does not show the correct value until 2016-09-31 03:59. It does not seem to be a single data point issue. It consistently shows the same issue/feature throughout the back-test period.

Any suggestion? Thank you.

Anthony

def initialize(context):  
    context.sma_lookback = 4  
    # AAPL US  
    context.security = sid(24)  
    schedule_function(test_price_1, date_rules.every_day(), time_rules.market_close(minutes=1)) 

def test_price_1(context,data):  
    price_history = data.history( context.security, fields='price', bar_count=context.sma_lookback, frequency='1d')  
    print(price_history)  
    current_price = data.current(context.security, 'price')  
    log.info("{}, current_price:{}".format(get_datetime('US/Eastern').time(),current_price))

############# Log Message ######################

2016-08-31 03:59 PRINT
2016-08-25 00:00:00+00:00 107.580
2016-08-26 00:00:00+00:00 106.940
2016-08-29 00:00:00+00:00 106.820
2016-08-30 00:00:00+00:00 105.895
Freq: C, Name: Equity(24 [AAPL]), dtype: float64
2016-08-31 03:59 test_price_1:46 INFO 15:59:00, current_price:105.895

2016-09-01 03:59 PRINT
2016-08-26 00:00:00+00:00 106.94
2016-08-29 00:00:00+00:00 106.82
2016-08-30 00:00:00+00:00 105.99 <- only at 2016-09-01 it shows the correct close price of 2016-08-30
2016-08-31 00:00:00+00:00 106.20
Freq: C, Name: Equity(24 [AAPL]), dtype: float64
2016-09-01 03:59 test_price_1:46 INFO 15:59:00, current_price: 106.2

2016-09-02 03:59 PRINT
2016-08-29 00:00:00+00:00 106.82
2016-08-30 00:00:00+00:00 105.99
2016-08-31 00:00:00+00:00 106.11 <- only at 2016-09-02 it shows the correct close price of 2016-08-31
2016-09-01 00:00:00+00:00 106.70
Freq: C, Name: Equity(24 [AAPL]), dtype: float64
2016-09-02 03:59 test_price_1:46 INFO 15:59:00, current_price:106.7

2016-09-03 03:59 PRINT
2016-08-30 00:00:00+00:00 105.990
2016-08-31 00:00:00+00:00 106.110
2016-09-01 00:00:00+00:00 106.725
2016-09-02 00:00:00+00:00 107.750
Freq: C, Name: Equity(24 [AAPL]), dtype: float64
2016-09-03 03:59 test_price_1:46 INFO 15:59:00, current_price:107.75

2016-09-07 03:59 PRINT
2016-08-31 00:00:00+00:00 106.110
2016-09-01 00:00:00+00:00 106.725
2016-09-02 00:00:00+00:00 107.730
2016-09-06 00:00:00+00:00 107.690
Freq: C, Name: Equity(24 [AAPL]), dtype: float64
2016-09-07 03:59 test_price_1:46 INFO 15:59:00, current_price:107.69

2016-09-08 03:59 PRINT
2016-09-01 00:00:00+00:00 106.725
2016-09-02 00:00:00+00:00 107.730
2016-09-06 00:00:00+00:00 107.700
2016-09-07 00:00:00+00:00 108.330
Freq: C, Name: Equity(24 [AAPL]), dtype: float64
2016-09-08 03:59 test_price_1:46 INFO 15:59:00, current_price:108.33

2016-09-09 03:59 PRINT
2016-09-02 00:00:00+00:00 107.73
2016-09-06 00:00:00+00:00 107.70
2016-09-07 00:00:00+00:00 108.38
2016-09-08 00:00:00+00:00 105.46
Freq: C, Name: Equity(24 [AAPL]), dtype: float64
2016-09-09 03:59 test_price_1:46 INFO 15:59:00, current_price:105.46

2016-09-10 03:59 PRINT
2016-09-06 00:00:00+00:00 107.700
2016-09-07 00:00:00+00:00 108.380
2016-09-08 00:00:00+00:00 105.510
2016-09-09 00:00:00+00:00 103.185
Freq: C, Name: Equity(24 [AAPL]), dtype: float64
2016-09-10 03:59 test_price_1:46 INFO 15:59:00, current_price:103.185

2016-09-13 03:59 PRINT
2016-09-07 00:00:00+00:00 108.38
2016-09-08 00:00:00+00:00 105.51
2016-09-09 00:00:00+00:00 103.14
2016-09-12 00:00:00+00:00 105.64
Freq: C, Name: Equity(24 [AAPL]), dtype: float64
2016-09-13 03:59 test_price_1:46 INFO 15:59:00, current_price:105.64

6 responses

Dan Whitnable

The backtester is designed to ensure there is no lookahead bias. What that means is at the last minute of the day one only knows the price as of the PREVIOUS minute. One cannot look into the future and see what the close of the current minute will be.

What you are seeing in the log is actually the last known price as of the last minute of trading (ie the price at the end of 15:58:00). The next day, and only the next day, can one get the close of the previous day (ie the price at the end of 15:59:00). Make sense?

I changed your code a bit to look at minute data to be able to see this. Here is the log output (edited a bit for clarity)


Name: Equity(24 [AAPL]), dtype: float64  
2016-08-30 15:59 price_market_close:15 INFO 15:59:00, current_price:105.895  
# the above is the latest known price during the last minute of trading or really the close of 15:58:00 NOT market close

# if the next day we get 4 minutes of data at market open (so 1 minute of todays data and 3 minutes of yesterday), we get  
2016-08-31 09:31  PRINT  
2016-08-30 15:58:00+00:00    105.888  
2016-08-30 15:59:00+00:00    105.895   # this is the close at time 15:58 reported at 15:59 which matches what was printed previously  
2016-08-30 16:00:00+00:00    105.990   # this is the close of the previous day  
2016-08-31 09:31:00+00:00    105.881   # this is the the close at time 09:30 (really the open) reported at minute 09:31

One can't look ahead at the close of the current time period. One needs to wait until the next time period to get the previous close.

Dan Whitnable

Here is the backtest

Anthony Ng

Dan, thank you for your reply. I understand your explanation that "The backtester is designed to ensure there is no lookahead bias".

In this particular example, I am only interested in obtaining daily close price.
price_history = data.history( context.security, fields='price', bar_count=context.sma_lookback, frequency='1d')

Let's focus on what's returned from data.history().

2016-08-31 03:59 PRINT
2016-08-25 00:00:00+00:00 107.580
2016-08-26 00:00:00+00:00 106.940
2016-08-29 00:00:00+00:00 106.820
2016-08-30 00:00:00+00:00 105.895

"price_history" is printed @2016-08-31 15:59:00, at this time we should have already known the close price of 2016-08-30 (the previous day), am I correct? (Otherwise the above statment is still at 2016-08-30, then it make sense that 2016-08-30's close is still not yet available.)

The correct close price of 2016-08-30 is 105.99 (see below) , which is only avaiable when when "price_history" is printed @2016-09-01 15:59:00.

2016-09-01 03:59 PRINT
2016-08-26 00:00:00+00:00 106.94
2016-08-29 00:00:00+00:00 106.82
2016-08-30 00:00:00+00:00 105.99
2016-08-31 00:00:00+00:00 106.20

Perhaps I can tell you what I would like to achieve.
Say today I am @2016-08-31, I want to send a buy/sell order based on the signal generated from the last 4 days (26,28,30,31 Aug 2016) of closing prices (including today). I want this buy/sell order to be executed the next day ie. [email protected] or Open.

After the removal of Day Mode in Quantopian 2, I find it harder to simulate daily type of strategy. Any suggestion?

Anthony Ng

Dan,

I have read through the API documentation of data.history() in details and confirmed that the last entry is the "current time" (despite I am using "d").

Prior to that I was not aware of the fact that this DataFrame contains mix of intraday data (current time in price_history[-1]) and daily data (in price_history[-2], price_history[-3] etc)

The second confusion I had (see the following PRINT message running your code with the price_market_open() and price_market_open() function on my Q platform).

It seems the timestamp is messed up when the code is running on my Q platform (Is that because I am located in Asian time zone?).

price_market_open() and price_market_open() should be showing the same date. As you can see below while price_market_open() is printed on 2016-08-30 but price_market_close() is printed on 2016-08-31. Hence I was wondering why 2016-08-30's Close is not yet available when I am already on 2016-08-31. Now I understand because I am still on 2016-08-30 (before close) when that message was printed.

Do you know how to configure the Q system to show the right timestamp? For example:

2016-08-31 03:59 price_market_close:15 INFO 15:59:00, current_price:105.895 (mine)  
2016-08-30 15:59 price_market_close:15 INFO 15:59:00, current_price:105.895 (yours)

############## Log

2016-08-30 21:31 PRINT #output printed from price_market_open()
2016-08-29 19:58:00+00:00 106.84
2016-08-29 19:59:00+00:00 106.83
2016-08-29 20:00:00+00:00 106.82
2016-08-30 13:31:00+00:00 106.09
Name: Equity(24 [AAPL]), dtype: float64
2016-08-31 03:59 price_market_close:15 INFO 15:59:00, current_price:105.895 # output printed from price_market_close()

2016-08-31 21:31 PRINT #output printed from price_market_open()
2016-08-30 19:58:00+00:00 105.888
2016-08-30 19:59:00+00:00 105.895
2016-08-30 20:00:00+00:00 105.990
2016-08-31 13:31:00+00:00 105.881
Name: Equity(24 [AAPL]), dtype: float64
2016-09-01 03:59 price_market_close:15 INFO 15:59:00, current_price:106.2 # output printed from price_market_close()

Thank you

Anthony

Dan Whitnable

The brute force method to changing the log timezone is simply to change the timezone on your computer. On a Mac and it's as simple as going to the clock and clicking on NY City on the timezone map (and un-checking 'set timezone automatically'). PCs have something similar. Just remember to set it back. There may be something more elegant but if so it's not documented.

Anthony, In the previous post you asked...

Perhaps I can tell you what I would like to achieve. Say today I am
@2016-08-31, I want to send a buy/sell order based on the signal
generated from the last 4 days (26,28,30,31 Aug 2016) of closing
prices (including today). I want this buy/sell order to be executed
the next day ie. [email protected] or Open.

What you are asking is EXACTLY what pipeline is designed to do. Use pipeline. Don't use the data.current or data.history functions. Generally only use those functions for intraday prices (like fine tuning an order price) or if you have a day trading strategy.

Create a pipeline with factors for all the data you need for your signal. Create filter(s) based upon that signal. Define a pipeline with those factors and filters. Put the pipeline_output method in the 'before_trading_start' method. Do your order execution in a scheduled function based upon the pipeline output data.

This will do exactly what you want. This will ensure:

your signal will be generated by the pipeline using the latest data
from 26,28,30,31 Aug 2016. This is because it will be run
'before_market_open' of 09-01 (and AFTER market close of 08-31).
your orders can be placed anytime during the following day (ie
09-31) by scheduling them using the 'schedule_function'

Look at the the pipeline tutorial https://www.quantopian.com/tutorials/pipeline.

Anthony Ng

Dan, thank you for your suggestion!

You've successfully submitted a support ticket.

Our support team will be in touch soon.