I put together a graphic that marks the start and end dates for each future currently existing in the Quantopian database. You'll note some interesting things like the fact that some futures stopped trading, and we didn't start getting data for certain futures until much later than 2002.
Hopefully you find this helpful when trying to assess what data Quantopian has to offer.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.
I should probably add a word of caution about lookahead bias. Entirely removing futures that stopped trading from an algorithm introduces lookahead bias as you wouldn't have known ahead of time that the future would stop trading. Instead, you should consider adding logic to handle such an event. I'd suggest something along the lines of detecting that the current chain of futures is only 1 contract and you are near the auto_close_date or something like that.
If anyone has ideas on handling the end of trading for a future, feel free to post them here.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.
Thanks for posting this Jamie - it is very useful.
Are there any plans to further backdate any of the data? I'm mainly interested in VX which only starts from mid-2012.
No plans right now to extend the futures data back any further since that would probably require getting a new vendor. However, I'm going to reach out to our current vendor and see if they can extend the data back any further.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.
This post is helpful Jamie.
I am thinking of comparing
data.current_session.date() with future.start_date, future.end_date to see if a contract is currently traded, perhaps with some buffer for my holding period. Does anyone see any problems with this approach?
Here is some simple code to test if a futures contract can be traded.
Let me know if you see any problems with this approach.
Peter, that looks reasonable to me. Of course, there's a degree of lookahead bias in there, so just be conscious of it. Futures are a little bit different from equities on Quantopian in that they are not actively listed/delisted. The list of futures will likely remain relatively static while the contracts themselves will be dynamic so I think it's a reasonable approach.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.
I am using the platform to test CMEGROUP future such as continuous_future('BP',roll='calendar') and encounter two questions below.
CMEGROUP British Pound / USD forex (and also other futures such as GOLD) is trading from Sunday - Friday 6:00 p.m. - 5:00 p.m. with a 60-minute break each day beginning at 5:00 p.m. However, if i select US Future Calendar at quantopian platform, it works at 6:30AM and closes at 5:00PM. This is a difference here. How can i "tailor-make" a calendar for handling CMEGROUP future product trading ?
When running full backtest and if i click tansaction detail, i see the unit price as 2 decimal digits only (e.g. 1.23 instead of 1.2345). In future FOREX trading, we talk about PIP which means..say.. 1.2301 and 1.2310 are 9 PIP in difference. If we round this to 1.23 only it does not work . For BP POUND/USD future the profit or loss here we are talking about (1.2310-1.2301) x 62500 = USD 56.25. Can we make the platform supporting 4 decmial digits somehow ? Any trick ?
Hi Tony,
Currently, the US futures calendar running from 6:30am-5:00pm is the only one available for trading futures on Quantopian. You have access to the 24 hour data for these futures, you're just limited to placing trades in the 6:30am-5:00pm window. At some point in the future, we'd like to support a full 24 hour calendar.
That looks like a display bug. We'll have to get it fixed. Under the hood, the correct precision is used. We'll have to fix the display to reflect the correct precision.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.
Hi Jamie McCorriston,
Thank for your speedy response. I fully appreciate the support on CMEGROUP trading hour (chicago time 6PM - 5PM) in the near future. Cheers.
HI Jamie,
When I try to get data for Feeder Cattle (FC) I get the following error:
NoDataAfterDate: No data on or after day=2017-02-23 00:00:00+00:00 for sid=1036201701
However the continuous future FC tells me FC 91834509703577600 starts at: 2002-01-02 00:00:00+00:00 and ends at: 2017-07-13 00:00:00+00:00.
If you want to reproduce this you could add the following line after the
tradeable_contacts list comprehension.
prices = data.history(tradeable_contacts, 'price', 60, '1d')
Any idea on why this is happening?
Hi Peter,
In your example snippet, is tradeable_contracts a list of ContinuousFutures? We were recently made aware of a bug with continuous futures and data.history. We're looking into it now. As a temporary workaround, you might be able to use data.current to get the current contract of your continuous futures, and pass that as input to data.history. If you were already passing Future objects (contracts), I'd be curious to know so I can tell our engineers that the problem isn't just with continuous futures.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.
If I run this query in research for the list of 61 continuous contracts which are still traded (72-11 not traded anymore) I can only get data up to '2017-07-01
but according to the graphs it should be possible to go back at least till 2010. In any case how can the error be avoided? I use dropna to clean the data after the data is returned but at the moment the error prevents to do anything. Using dateI=2017-06-01 i get the error below:
dateI='2017-07-01'
dateF='2018-03-05'
dataFC= history(
symbol_list[0:61],
fields='close_price',
frequency='daily',
start=dateI,
end=dateF,
handle_missing ='ignore'
)
NoDataForSidTraceback (most recent call last)
in ()
22 frequency='daily',
23 start=dateI,
---> 24 end=dateF
25 )
26
/build/src/qexec_repo/qexec/research/api.py in history(symbols, fields, start, end, frequency, symbol_reference_date, handle_missing, start_offset)
537 findata_dir=findata_dir,
538 user_id=user_id,
--> 539 start_offset=start_offset,
540 )
541
/build/src/qexec_repo/qexec/research/_api.pyc in inner_history(symbols, fields, frequency, start, end, symbol_reference_date, handle_missing, asset_data_dir, findata_dir, user_id, start_offset)
691 findata_dir=findata_dir,
692 user_id=user_id,
--> 693 start_offset=start_offset,
694 )
695
/build/src/qexec_repo/qexec/research/_api.pyc in inner_get_pricing(symbols, start_date, end_date, symbol_reference_date, frequency, fields, handle_missing, asset_data_dir, findata_dir, user_id, start_offset)
555
556 asset_specific_data = _get_pricing_internal(
--> 557 data_portal, assets, pinched_start, pinched_end, frequency, fields,
558 )
559
/build/src/qexec_repo/qexec/research/trades.pyc in _get_pricing_internal(data_portal, assets, start_date, end_date, frequency, fields)
109 start_date=ensure_timestamp(start_date),
110 end_date=ensure_timestamp(end_date),
--> 111 data_frequency=frequency,
112 )
113
/build/src/qexec_repo/qexec/research/trades.pyc in ohlcv_panel_from_source(data_portal, sids, start_date, end_date, data_frequency, fields)
183 freq,
184 _field,
--> 185 data_frequency,
186 )
187 # For date_indexes which extend past the last available dt, reindex
/build/src/qexec_repo/zipline_repo/zipline/data/data_portal.pyc in get_history_window(self, assets, end_dt, bar_count, frequency, field, data_frequency, ffill)
958 else:
959 df = self._get_history_daily_window(assets, end_dt, bar_count,
--> 960 field, data_frequency)
961 elif frequency == "1m":
962 if field == "price":
/build/src/qexec_repo/zipline_repo/zipline/data/data_portal.pyc in _get_history_daily_window(self, assets, end_dt, bar_count, field_to_use, data_frequency)
802
803 data = self._get_history_daily_window_data(
--> 804 assets, days_for_window, end_dt, field_to_use, data_frequency
805 )
806 return pd.DataFrame(
/build/src/qexec_repo/zipline_repo/zipline/data/data_portal.pyc in _get_history_daily_window_data(self, assets, days_for_window, end_dt, field_to_use, data_frequency)
825 field_to_use,
826 days_for_window,
--> 827 extra_slot=False
828 )
829 else:
/build/src/qexec_repo/zipline_repo/zipline/data/data_portal.pyc in _get_daily_window_data(self, assets, field, days_in_window, extra_slot)
1108 days_in_window,
1109 field,
-> 1110 extra_slot)
1111 if extra_slot:
1112 return_array[:len(return_array) - 1, :] = data
/build/src/qexec_repo/zipline_repo/zipline/data/history_loader.pyc in history(self, assets, dts, field, is_perspective_after)
548 dts,
549 field,
--> 550 is_perspective_after)
551 end_ix = self._calendar.searchsorted(dts[-1])
552
/build/src/qexec_repo/zipline_repo/zipline/data/history_loader.pyc in _ensure_sliding_windows(self, assets, dts, field, is_perspective_after)
430 adj_dts = prefetch_dts
431 prefetch_len = len(prefetch_dts)
--> 432 array = self._array(prefetch_dts, needed_assets, field)
433
434 if field == 'sid':
/build/src/qexec_repo/zipline_repo/zipline/data/history_loader.pyc in _array(self, dts, assets, field)
572 dts[0],
573 dts[-1],
--> 574 assets,
575 )[0]
576
/build/src/qexec_repo/zipline_repo/zipline/data/dispatch_bar_reader.pyc in load_raw_arrays(self, fields, start_dt, end_dt, sids)
118 end_dt,
119 sid_groups[t])
--> 120 for t in asset_types if sid_groups[t]}
121
122 results = []
/build/src/qexec_repo/zipline_repo/zipline/data/dispatch_bar_reader.pyc in ((t,))
118 end_dt,
119 sid_groups[t])
--> 120 for t in asset_types if sid_groups[t]}
121
122 results = []
/build/src/qexec_repo/zipline_repo/zipline/data/continuous_future_reader.pyc in load_raw_arrays(self, columns, start_date, end_date, assets)
37 start_date,
38 end_date,
---> 39 asset.offset
40 )
41
/build/src/qexec_repo/zipline_repo/zipline/assets/roll_finder.pyc in get_rolls(self, root_symbol, start, end, offset)
124 if prev < prev_c.contract.auto_close_date:
125 break
--> 126 if back != self._active_contract(oc, front, back, prev):
127 # TODO: Instead of listing each contract with its roll date
128 # as tuples, create a series which maps every day to the
/build/src/qexec_repo/zipline_repo/zipline/assets/roll_finder.pyc in _active_contract(self, oc, front, back, dt)
214 return front
215
--> 216 front_vol = get_value(front, prev, 'volume')
217 back_vol = get_value(back, prev, 'volume')
218 if back_vol > front_vol:
/build/src/qexec_repo/zipline_repo/zipline/data/resample.pyc in get_value(self, sid, session, colname)
626 # This was developed to complete interface, but has not been tuned
627 # for real world use.
--> 628 return self._get_resampled([colname], session, session, [sid])[0][0][0]
629
630 @lazyval
/build/src/qexec_repo/zipline_repo/zipline/data/resample.pyc in _get_resampled(self, columns, start_session, end_session, assets)
577 range_open,
578 range_close,
--> 579 assets,
580 )
581
/build/src/qexec_repo/zipline_repo/zipline/data/minute_bars.pyc in load_raw_arrays(self, fields, start_dt, end_dt, sids)
1272
1273 for i, sid in enumerate(sids):
-> 1274 carray = self._open_minute_file(field, sid)
1275 values = carray[start_idx:end_idx + 1]
1276 if indices_to_exclude is not None:
/build/src/qexec_repo/zipline_repo/zipline/data/minute_bars.pyc in _open_minute_file(self, field, sid)
1080 )
1081 except IOError:
-> 1082 raise NoDataForSid('No minute data for sid {}.'.format(sid))
1083
1084 return carray
NoDataForSid: No minute data for sid 1072201706.
for dateI='2017-07-01' the query works, just to confirm the list of continuous futures is correct, but for for dateI='2017-06-01' I get the error above
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.
In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.
