It seems that Futures data retrieved by History is returning data from the future instead of historical data.
I have reproduced the problem in the attached algo.
It seems that if you ask for 10 minutes of data at 06:30 EST you will receive data for the ES futures as of 11:20 CST. I suspect the platform (falsely) treats the time of the Futures data as UTC, while the data is in fact in CST. The result is that we falsely get a lookahead of several hours of data that does not exist in reality of the simulation.
I have cross referenced this with sample data from AlgoSeek.
Here is a link to the cross reference data. You can see that the output matches 11:20 CST and not 05:20 EST as it should:
https://www.algoseek.com/static/files/sample_data/futures_and_future_options/ESH5.Futures.1MinBars.TradesOnlly.20150128.csv
Hi Oren,
The timestamp at the start of a log message is in your browser's local timezone. However, the datetimes that you see in the index of the data.history result are in UTC. The 11:20 UTC time corresponds to 6:20am ET. Is that data source you are comparing to adjusting the prices at all? Contract prices are adjusted as of the simulation date, whereas many sources will adjust all prices as of today.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.
Hi Jamie,
I am aware of the way Quantopian works with timezones. However, in this case I am comparing the data I am receiving to that from Algseek (see link in original posting) and under the assumption that the CSV file is in CST (according to Algoseek documentation), the data I am getting from data.history for this future is 5 hours ahead (meaning it did not exist when calling the data.history). I suspect that while loading the Algoseek data, Quantopian platform may be assuming the data is in UTC, while it is in fact in CST.
I have attached a sample algo + a link to the data from Algoseek for your reference.
Hi Oren,
I'm not too familiar with AlgoSeek, but the documentation I found on their Trade+Quote minute bar data suggested that their data is timestamped in UTC: https://www.algoseek.com/static/files/documentation/futures_and_future_options/AlgoSeek.US.Futures.TradeAndQuotes.OHLC.Bars.1.2.pdf. Not sure if this helps. It might be worth looking for the 5pm-6pm maintenance window (for most US contracts) to see where that lines up in the file.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.
Hi Jamie,
After several more hours of investigation, it seems you are correct. The AlgoSeek data is UTC (you're tip with the maintenance window was very helpful). I was cross-referencing my algorithm to QuantConnect and I now suspect that the bug above actually exists in QuantConnect and not Quantopian (https://github.com/QuantConnect/Lean/issues/1534).
Thanks!
No problem, Oren. Thanks for bringing it up in the first place.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.
In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.