Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Mispricing Data
Mike Massari
edited Feb 22, 2019
Seeking Help Research Case Studies Lectures

Hello!

I ran into this issue and would really like to understand where the problem lies.

It occurs to me that the data "recorded" at run-time is correct, while the same data, retrieved "historically", is way mispriced.

Am I missing something?

Thanks for any clarification.

P.S. The info is in the log of the Backtest

2018-02-20 14:45 before_trading_start:26 INFO 2018-02-16 00:00:00+00:00 75.06
Freq: C, Name: Equity(21652 [IYR]), dtype: float64 2018-02-21 14:45 before_trading_start:26 INFO 2018-02-20 00:00:00+00:00 74.13
Freq: C, Name: Equity(21652 [IYR]), dtype: float64 2018-02-22 14:45 before_trading_start:26 INFO 2018-02-21 00:00:00+00:00 72.9
Freq: C, Name: Equity(21652 [IYR]), dtype: float64 2018-02-23 14:45 before_trading_start:26 INFO 2018-02-22 00:00:00+00:00 73.57
Freq: C, Name: Equity(21652 [IYR]), dtype: float64 2018-02-26 14:45 before_trading_start:26 INFO 2018-02-23 00:00:00+00:00 74.83
Freq: C, Name: Equity(21652 [IYR]), dtype: float64 2019-01-02 14:45 before_trading_start:17 INFO 2018-02-16 00:00:00+00:00 72.579
2018-02-20 00:00:00+00:00 71.680
2018-02-21 00:00:00+00:00 70.491
2018-02-22 00:00:00+00:00 71.138
2018-02-23 00:00:00+00:00 72.357
Freq: C, Name: Equity(21652 [IYR]), dtype: float64 End of logs.

Here's a better comparison:

Run-time:
2018-02-16 00:00:00+00:00 75.06
2018-02-20 00:00:00+00:00 74.13
2018-02-21 00:00:00+00:00 72.9
2018-02-22 00:00:00+00:00 73.57
2018-02-23 00:00:00+00:00 74.83

Historical:
2018-02-16 00:00:00+00:00 72.579
2018-02-20 00:00:00+00:00 71.680
2018-02-21 00:00:00+00:00 70.491
2018-02-22 00:00:00+00:00 71.138
2018-02-23 00:00:00+00:00 72.357

9 responses
Vladimir
Feb 22, 2019

Try this way:

def initialize(context):  
    context.sec = symbol('IYR')  
    context.stop = 5  

def before_trading_start(context, data):  
    price = data.current(context.sec, 'close')  
    log.info(price)  
    context.stop = context.stop - 1 

    if context.stop == 0:  
        prices = data.history(context.sec, 'close', 250, '1d')  
        log.info(prices.iloc[-5:])  
'''
START  
02/11/2019  
END  
02/18/2019  
'''

2019-02-11 05:45 before_trading_start:7 INFO 84.14
2019-02-12 05:45 before_trading_start:7 INFO 84.35
2019-02-13 05:45 before_trading_start:7 INFO 83.84
2019-02-14 05:45 before_trading_start:7 INFO 84.25
2019-02-15 05:45 before_trading_start:7 INFO 84.52

2019-02-15 05:45 before_trading_start:12 INFO
2019-02-08 00:00:00+00:00 84.14
2019-02-11 00:00:00+00:00 84.35
2019-02-12 00:00:00+00:00 83.84
2019-02-13 00:00:00+00:00 84.25
2019-02-14 00:00:00+00:00 84.52

Mike Massari
Feb 22, 2019

Yes it works, but why is the price different when you retrieve data after a long time, and not after few days?

Vladimir
Feb 22, 2019

I have tried it on different dates:

START
02/11/2017
END
02/18/2017

2017-02-13 05:45 before_trading_start:7 INFO 78.51
2017-02-14 05:45 before_trading_start:7 INFO 78.77
2017-02-15 05:45 before_trading_start:7 INFO 78.38
2017-02-16 05:45 before_trading_start:7 INFO 78.229
2017-02-17 05:45 before_trading_start:7 INFO 78.635

2017-02-17 05:45 before_trading_start:12 INFO
2017-02-10 00:00:00+00:00 78.510
2017-02-13 00:00:00+00:00 78.770
2017-02-14 00:00:00+00:00 78.380
2017-02-15 00:00:00+00:00 78.229
2017-02-16 00:00:00+00:00 78.635

START
02/11/2015
END
02/18/2015

2015-02-11 05:45 before_trading_start:7 INFO 80.06
2015-02-12 05:45 before_trading_start:7 INFO 79.9
2015-02-13 05:45 before_trading_start:7 INFO 80.76
2015-02-17 05:45 before_trading_start:7 INFO 80.32
2015-02-18 05:45 before_trading_start:7 INFO 80.1

2015-02-18 05:45 before_trading_start:12 INFO
2015-02-10 00:00:00+00:00 80.06
2015-02-11 00:00:00+00:00 79.90
2015-02-12 00:00:00+00:00 80.76
2015-02-13 00:00:00+00:00 80.32
2015-02-17 00:00:00+00:00 80.10

Mike Massari
Feb 22, 2019

Maybe I did not explain myself correctly.

Yes, your algo shows exact data, but that is because you are retrieving data right after you retireve your data run-time.
Try this and see if you understand what I mean.. This one retrieves data after 1 year.. so the algo requests the data after one year.
I have trouble writing it out.

See the version below, please.

def initialize(context):
context.sec = symbol('IYR')
context.stop1 = 5
context.stop2 = 200

def before_trading_start(context, data):
if context.stop1 != 0:
price = data.current(context.sec, 'close')
log.info(price)
context.stop1 = context.stop1 - 1 

context.stop2 = context.stop2 - 1  
if context.stop2 == 0:  
    prices = data.history(context.sec, 'close', 250, '1d')  
    log.info(prices.iloc[51:55])
Mike Massari
Feb 22, 2019

This is as simple as it gets.
You can check online wherever you want (investing.com, bloomberg ...) that the data returned is NOT correct.

Thoughts?

def initialize(context): 

context.iyr = symbol('IYR')

def before_trading_start(context, data):
h = data.history(context.iyr, 'close', 250, '1d')
log.info(h.head(20))

Vladimir
Feb 22, 2019

Longer period:

def initialize(context):  
    context.sec = symbol('IYR')  
    context.stop = 20  

def before_trading_start(context, data):  
    price = data.current(context.sec, 'close')  
    log.info(price)  
    context.stop = context.stop - 1 

    if context.stop == 0:  
        prices  = data.history(context.sec, 'close', 250, '1d')  
        log.info(prices.iloc[-20:])

START
01/21/2015
END
02/18/2015

2015-01-21 05:45 before_trading_start:7 INFO 81.02
2015-01-22 05:45 before_trading_start:7 INFO 80.95
2015-01-23 05:45 before_trading_start:7 INFO 82.49
2015-01-26 05:45 before_trading_start:7 INFO 82.37
2015-01-27 05:45 before_trading_start:7 INFO 83.13
2015-01-28 05:45 before_trading_start:7 INFO 83.01
2015-01-29 05:45 before_trading_start:7 INFO 82.43
2015-01-30 05:45 before_trading_start:7 INFO 82.58
2015-02-02 05:45 before_trading_start:7 INFO 81.18
2015-02-03 05:45 before_trading_start:7 INFO 81.0
2015-02-04 05:45 before_trading_start:7 INFO 81.77
2015-02-05 05:45 before_trading_start:7 INFO 81.42
2015-02-06 05:45 before_trading_start:7 INFO 82.35
2015-02-09 05:45 before_trading_start:7 INFO 80.18
2015-02-10 05:45 before_trading_start:7 INFO 79.79
2015-02-11 05:45 before_trading_start:7 INFO 80.06
2015-02-12 05:45 before_trading_start:7 INFO 79.9
2015-02-13 05:45 before_trading_start:7 INFO 80.76
2015-02-17 05:45 before_trading_start:7 INFO 80.32
2015-02-18 05:45 before_trading_start:7 INFO 80.1

2015-02-18 05:45 before_trading_start:12 INFO
2015-01-20 00:00:00+00:00 81.02
2015-01-21 00:00:00+00:00 80.95
2015-01-22 00:00:00+00:00 82.49
2015-01-23 00:00:00+00:00 82.37
2015-01-26 00:00:00+00:00 83.13
2015-01-27 00:00:00+00:00 83.01
2015-01-28 00:00:00+00:00 82.43
2015-01-29 00:00:00+00:00 82.58
2015-01-30 00:00:00+00:00 81.18
2015-02-02 00:00:00+00:00 81.00
2015-02-03 00:00:00+00:00 81.77
2015-02-04 00:00:00+00:00 81.42
2015-02-05 00:00:00+00:00 82.35
2015-02-06 00:00:00+00:00 80.18
2015-02-09 00:00:00+00:00 79.79
2015-02-10 00:00:00+00:00 80.06
2015-02-11 00:00:00+00:00 79.90
2015-02-12 00:00:00+00:00 80.76
2015-02-13 00:00:00+00:00 80.32
2015-02-17 00:00:00+00:00 80.10

Mike Massari
Feb 22, 2019

But again, you are retrieving the data after 20 days even if the backtest lasts 4 years.

Try retrieving the data after one year and the data will be wrong.

Vladimir
Feb 23, 2019

Now I see what you called "mispricing".

def initialize(context):  
    context.sec = symbol('IYR')  
    context.stop = 20  

def before_trading_start(context, data):  
    price = data.current(context.sec, 'close')  
    log.info(price)  
    context.stop = context.stop - 1 

    if context.stop == 0:  
        prices  = data.history(context.sec, 'close', 250, '1d')  
        log.info(prices.iloc[-20:])

START
12/03/2015
END
01/01/2016

2015-12-03 05:45 before_trading_start:7 INFO 74.72
2015-12-04 05:45 before_trading_start:7 INFO 73.63
2015-12-07 05:45 before_trading_start:7 INFO 74.84
2015-12-08 05:45 before_trading_start:7 INFO 74.64
2015-12-09 05:45 before_trading_start:7 INFO 74.47
2015-12-10 05:45 before_trading_start:7 INFO 74.03
2015-12-11 05:45 before_trading_start:7 INFO 73.56
2015-12-14 05:45 before_trading_start:7 INFO 73.25
2015-12-15 05:45 before_trading_start:7 INFO 73.43
2015-12-16 05:45 before_trading_start:7 INFO 74.24
2015-12-17 05:45 before_trading_start:7 INFO 75.69
2015-12-18 05:45 before_trading_start:7 INFO 75.32
2015-12-21 05:45 before_trading_start:7 INFO 74.37
2015-12-22 05:45 before_trading_start:7 INFO 74.76
2015-12-23 05:45 before_trading_start:7 INFO 75.12
2015-12-24 05:45 before_trading_start:7 INFO 74.969872
2015-12-28 05:45 before_trading_start:7 INFO 74.97
2015-12-29 05:45 before_trading_start:7 INFO 75.27
2015-12-30 05:45 before_trading_start:7 INFO 75.995
2015-12-31 05:45 before_trading_start:7 INFO 75.62

2015-12-31 05:45 before_trading_start:12 INFO
2015-12-02 00:00:00+00:00 73.775
2015-12-03 00:00:00+00:00 72.699
2015-12-04 00:00:00+00:00 73.894
2015-12-07 00:00:00+00:00 73.696
2015-12-08 00:00:00+00:00 73.528
2015-12-09 00:00:00+00:00 73.094
2015-12-10 00:00:00+00:00 72.630
2015-12-11 00:00:00+00:00 72.324
2015-12-14 00:00:00+00:00 72.501
2015-12-15 00:00:00+00:00 73.301
2015-12-16 00:00:00+00:00 74.733
2015-12-17 00:00:00+00:00 74.368
2015-12-18 00:00:00+00:00 73.430
2015-12-21 00:00:00+00:00 73.815
2015-12-22 00:00:00+00:00 74.170
2015-12-23 00:00:00+00:00 74.970
2015-12-24 00:00:00+00:00 74.970
2015-12-28 00:00:00+00:00 75.270
2015-12-29 00:00:00+00:00 75.995
2015-12-30 00:00:00+00:00 75.620

Are you aware of Quantopian split and dividend price adjustment?
The data returned for equities will be adjusted for splits, mergers, and dividends as of the current simulation time.
This may help.

Mike Massari
Feb 24, 2019

Thanks a lot!!!!

About Quantopian
Careers
Community
Events
Documentation
Academia
Lectures
Status
Facebook
Twitter
LinkedIn
YouTube
Terms of Use
Privacy Policy

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.