Thanks for implementing this. It will definitely bring in a lot of new insight and improve a lot of existing algorithms.

Just one question.... If minutely data will be available for backtesting, does this mean that live (deployed) algorithms will be able to look at live data (rather than 10-20 minute delayed, as futures such as $VX are)? Or, will all minute-resolution backtests need to be run with timeshifted data?

That's a great news. When you will implement European Futures as well?

The Bean Report - a quick look at bean futures on Q:

p.s. most code borrowed from Q tutorials.

Are there any sample codes based on reading futures data?

@John, Are you asking about Quantopian paper trading? Currently, algorithms that are broker backed (paper or real $) all receive live updating data each minute. I'm not yet sure what paper trading will look like in terms of delayed data for futures. At the very least, a delayed algorithm will not run with timeshifted data. If it's delayed, it simply runs 15 minutes delayed. For example, the first bar (9:31) is simulated 15 minutes later (9:46), but runs as though it's the first bar of the day, and uses data from the first bar (9:31).

@Vincenzo, No timeline yet, but we definitely want to expand on the current list of available futures.

@Frank, That looks like an awesome study! Have you considered making a standalone post for it? You might have an easier time getting eyes/feedback on it. Also, have you considered generalizing the study by using ContinuousFutures?

@Macro Investor, If you're asking about backtesting, futures are currently only available in research. We will be launching futures in backtesting, as well as tutorials, lectures, and documentation at QuantCon.

@ Jaime

It was just a Saturday afternoon project because I was excited to see that futures were available in research. I will create a stand alone post next time I go back to the study and enhance it. Will also consider generalizing with Continuous Futures. Thanks again for Futures. Awesome to have this available !!!!

Frank, this is a really cool notebook. I only got through a quarter of it and even that was a ton of cool content.

@ Delaney, thanks, will make a stand alone post after fixing some issues and adding some features to the model. I have not checked the Q Data availability, but am wondering if any agricultural or weather data is available?

Sounds like a plan, Max may even be interested in borrowing snippets to use as exercises in our upcoming futures lectures. He'll reach out to you if so. I don't believe we currently have that data, but it is certainly in the realm of things we would like to integrate at some point. Seong would know better about this.

Sounds good!

We're always on the lookout for future partners. We've spoken to a few agricultural providers as well - and if you know any potential data firms that provide that data, you can send them my way. SLEE @ quantopian.com

For continuous_future, it will be useful to have a bit more flexibility in specifying the roll, e.g. number of days offset vs the auto_close_date in the case of calendar, and/or rolling a deferred future on the same day as the nearby future. This is because if I want to look at strategy of futures spreads, then having them all roll on their respective expiry severely restricts what I can do.

Hi Han,

Thanks for the feedback. Continuous futures are not assets that can be ordered, so there won't be any automatic rolling from contract to contract. To roll from one contract to the next, you will be able to get the current contract of a continuous future, as well as the next contract (from a function that will be available in backtesting). So you can decide when you would like to roll from one contract to the next. The roll logic in continuous futures is simply for looking at price histories.

That being said, I recognize that being able to set a custom roll logic (both for historical time series' of data, and for rolling from one held contract to the next), so that's something we may add going forward.

Are there any sample codes available to clone?

Futures are not yet available in backtesting. The launch of futures in backtesting will happen at QuantCon at the end of April at which point there will be sample algorithms to clone.

Very happy to see VIX Futures included. Excited for full launch

If I live trade via Robinhood (which does NOT support futures), will my algorithm even be able to look at live futures data? I will need to use Interactive Brokers to facilitate this, right?

Is it normal that VIX prices are available from July 2012 only? Will earlier data be added at some point?

@everyone: Futures have launched in backtesting, check out the official announcement.

@John: Futures are available in research and backtesting right now. I'm not yet sure how futures data would interact with Robinhood accounts.

@Samuel: I was a little surprised by the start of data in Jul. 2012 for our VIX data as well. Our provider only started collecting pricing and volume data for VIX futures in 2012 it seems. I actually called the exchange and spoke with a few people trying to figure out why this may have been the case. From what I understand, VIX futures techincally started trading in 2004, but they were quite illiquid for a long time. My best guess now is that there just wasn't enough activity for our vendor to get data prior to 2012. We don't have immediate plans to add more data for VIX futures, but in the long run, we would like to be able to have histories going back to 2002 for all the instruments we added. I'm working on a notebook to show the 'lifespans' of the futures we have in our database which should paint the picture nicely.

@Jamie Thanks for the update. It would be definitely interesting to have VIX data around 2008 for the stress-testing of volatility strategies.

@Jamie I'd like to bump up what @Han Zheng wrote above.
There very much ought to be a way to build a continuous contract based on calendar but able to roll +/- N days before the auto_close_date. I imagine most interest will be in rolling a few days earlier. There are many reasons to do this, but among the most important are that for physically delivered commodities, those dates can be particularly volatile in ways that many traders prefer to avoid.
Is there a plan to do this?

@knuckledragger: I definitely agree with you and Han. I actually posted an example algorithm that rolls a continuously held position in CL 2 days before the auto close. The key to note with continuous futures is that they don't actually determine any sort of rolling of your portfolio. The auto_close_date of a contract is the only attribute that actually takes any sort of automated action on your positions. Even if you use a continuous future that roll before the auto close, you'd need to include some logic in your algo to close out your existing position.

I'm hoping the example I shared in the other thread demonstrates how you can customize the roll behavior in your algo. Let me know if you find it helpful.

Thanks, Jamie. I'm more interested in the behavior of the continuous contract for right now. I am using the research / ipython notebook rather than the backtester.

Ah, I see. A limitation for continuous futures right now is that they can't really be used to simulate a return stream if your algo roll logic doesn't line up with either of the default rolls on continuous futures. At some point, we'd like to build out that sort of customizability. In the meantime, the backtester is your best bet at getting a return stream since it computes returns based on the contracts you hold.

Note that you could then pull the return stream back into research via get_backtest().

This is my first time on Quantopian and this was a great place to start. this was awesome!

I modified the original notebook a bit to include a section that looks at the spread between Crude Oil and Gasoline. There's also a tearsheet for the final algorithm in the Futures Tutorial which runs a pair trade between these two futures.

Here is the backtest that was used to generate the tearsheet in the above notebook. It was copied from the Futures Tutorial.

Hi @Jamie, how to backtest if the data is outside of the trading calander?

I tried to test grians in IDE, but seems it can just be traded from 6:30-5pm(future). Since the soybean's trading time is 19:00-7:45 & 8:30-13:20.....So every day it miss 10-hour data (from 19:00 to 05:30 UTC) when backtest in IDE..... I want to know how can I add those missing data by setting trading time frame? otherwise the data set is not continuous, and is hard to do algo...

Hi Manlin,

Right now, trades can only be scheduled between 6:30am-5pm Eastern Time. While this doesn't capture the entire trading window for corn, it captures the vast majority of it (see attached notebook). You can place trades any time during that window.

Additionally, you have access to 24 hour data. If you want data from 3:30am-6:30am ET, you can make a data.history call with 'm' frequency for 180 minutes at 'market open' of the US Futures calendar. The 6:30am-5pm restriction is only on schedule function so returns and historical lookups will be continuous.

There are some strategies that depend on executing trades around the clock. Unfortunately, those can't be implemented with the current US futures calendar. We would like to extend the calendar at some point in the future to support these types of strategies. In the meantime, we're hoping that there are still a large number that can be written that place trades in the 6:30am-5pm window.

Does this help?

@Jamie Thanks for your quick reply . It's helpful to me!

I'm getting nans in the NG futures when using data.current for price vs data.history. Does anyone know why?

            print(data.current(sec,'close'))  
            print(data.history(sec, 'price', 1, '1m')[0])  

2011-11-28 06:31  PRINT 3.714  
2011-11-28 06:31  PRINT 3.714  
2011-11-29 06:31  PRINT nan  
2011-11-29 06:31  PRINT 3.543  
2011-12-27 06:31  PRINT 3.128  
2011-12-27 06:31  PRINT 3.128  
2011-12-28 06:31  PRINT 3.137  
2011-12-28 06:31  PRINT 3.137  
2012-01-26 06:31  PRINT 2.826  
2012-01-26 06:31  PRINT 2.826  
2012-01-27 06:31  PRINT 2.667  
2012-01-27 06:31  PRINT 2.667  
2012-02-24 06:31  PRINT nan  
2012-02-24 06:31  PRINT 2.776  

Hi Honver,

The close is the close price from the previous minute bar, but is not forward filled which is why you see the nan. The price is also the close price from the previous minute bar but it is forward filled.

I hope this helps.

@Jamie after I clone the algorithm "Futures Trend Follow " available in section "Sample Algorithms" at https://www.quantopian.com/help and run a backtest starting with 1/1/2017 I get following Error Message:

NoDataAfterDate: No data on or after day=2017-02-23 00:00:00+00:00 for sid=1034201612
There was a runtime error on line 63.

Could you please help?

^ This error has been there for a month or so, Q said they are working on this, however, there seems to be no progress...

Yes, I've been waiting for several weeks for this issue to be fixed. But its still been unresolved.

I'm thinking to switch over to Zipline, the open source backend of Quantopian, but I found that its lacking 1) ingestable Futures data, and 2) examples to get started. It seems these requests have been in GitHub dating back to 2015 with no resolution.

Its frustrating there's no straightforward way to make progress with Futures backtesting and trading. At this point, I'm getting the feeling Quantopian isn't taking Futures trading seriously.

I would like to see Futures on Zipline.
@Jeremy Cohn, Quandl does have free Futures data. You could download that and load it into a Zipline bundle.

@Peter Harrington,
Thanks, yes I am aware of those. Unfortunately Quandl only offers daily bars, but I'm an intraday trader and require minute bars. In any case, yes, I'll have to access minute bars elsewhere and figure out how to ingest into Zipline.

When querying historical data on continuous futures with data.history, the closing price for the same date can be different on different backtesting dates. Does anyone know if historical pricing on continuous futures is somehow dynamically calculated? This unexpected behavior of unreproducible pricing can make debugging code more challenging. If this is not a bug, then it would be very helpful to have an option that ensures pricing reproducibility over time. Attached is an example backtest showing this effect in the log output, also reproduced here:

2016-03-24 12:00 fetch_prices:33 INFO query on 2016-02-17 has a price of 1885.25 on 2016-02-15 for ES
2016-03-24 12:00 fetch_prices:35 INFO query on 2016-03-21 has a price of 1876.60 on 2016-02-15 for ES

Is this bug fixed alread? any update?

I don't think this is a bug. See my attached notebook for a full explanation.

Quantopian's backtester is built to avoid lookahead bias. You do not know what the future roll adjustment is going to be until that future roll date.

Therefore, in the example that "@David Silverstein" posted, the 2016-02-15 price query for the 'ES' continuous future on the query date '2016-02-17' would just be the 'ESH16' (the front month at the time) price of 1885.25.

When the backtester queries the price on the '2016-03-21' query date, the backtester will return that 1885.25 times the multiplier of 0.99541 calculated from the 2016-03-16 roll. This comes out to be 1876.60.

At the risk of a ridiculously basic and newbie question, when I execute the first cell:

clf16 = symbols('CLF16')
clf16

It returns: Future(1058201601 [CLF16]), much less than the detailed properties of the object that are returned in the example notebook. Anything I can do to get the entire properties?

Hi Wei,
Please try :

clf16.to_dict()  

Thanks taro shinsei, worked a charm. Much appreciated!

I directly cloned this notebook yet I am receiving an error. I haven't modified anything.

TypeErrorTraceback (most recent call last)
in ()
4 frequency='daily',
5 start_date='2015-10-21',
----> 6 end_date='2015-12-21'
7 )
8

TypeError: history() got an unexpected keyword argument 'start_date'

Hello Adam,

The signature for the history method went through some changes. The error you are getting can be fixed by changing start_date to start, and end_date to end.

I hope this helps, and apologies for the confusion.

Futures Data available but no more Live Trading? & Paper Trading is delayed? Why?

Hello Guys,

I do not understand the following:
"Note that there are some points where this looks a little off, most notably in the transition from CLK16 to CLM16 between April and May. This is because the rolls are currently computed daily, using only the previous day's volume to avoid lookahead bias."

The volume of the continuous contract should always equal to the volume of either CLK16 or CLM16, depending which contract you are trading? Why it is neither of them?

Thanks!

Hi @Jamie McCorriston,

Nice way to aggregate the values by 30mins. By the way the time for the data it is EST ?

@steven: Sorry if that note was confusing. The comment was supposed to convey that the plot might make it look like the volume is off, but it is only because it is hopping from the volume of one contract at the end of day N to the volume of the next contract on day N+1.

I hope this helps clear things up.

@Erick: Thanks. The notebook I shared which shows the volume per 30 minute bar is displaying the data in UTC.

Quick question friends,

Was exploring my IB account, and noticed that the root symbols for futures on Quantopian don't match with those listed on IB... I know this may be a complete noob question, but is this resolved somehow by the API...? I looked for multiple assets and didn't see any of them reflected in the actual IB platform.

For example, say I wanted to create a strategy that trades VIX Futures. Well, according to the documentation, I would use the symbol 'VX' to denote that. But when I search for 'VX' in the IB platform, I don't see any such listing. Could this maybe be that I'm using a demo account? Not likely... I would think they would provide the same assets on a paper trading account as a live one.

Again, sorry for such a "noob" question. This is my first forum post, and I am relatively new to the space. I would really appreciate any clarification on this issue!

So I think I figured it out!!! The symbols listed on the IB charts are those for the E-CBOT, not the CBOT itself.
Still, I am wondering how the Quantopian API resolves this in the backend.

Hi this might be rather unrelated to the topic, but anyhow: is there any plan to include futures into the competitions? I think this would help the community to churn out more interesting futures algos. Is this given a priority?

I understand that even though you say "your market hours" for futures are 6:30 AM - 5:00 PM ET, you trade futures only between 9:30 AM - 4:00 PM ET.
But, you call the function before_trading_start() at 8:45 AM ET.

1) So, what do you expect an algorithm to do between the time 6:30AM - 8:45AM ?

2) what is the best place (i.e. function) to get last 20 days of 1-minute interval data for futures ? It didn't allow data.history() to be called in before_trading_start(). Nor can I use pipeline. The best way (I know) is in some function I have scheduled. Is that the recommended way ?

3) Also, is there any recommended way to select 30 minute interval rows from the 1-minute interval data obtained using data.history() ? I am finding it time consuming to convert each row's index into datetime and then selecting.

@Ryan Newby: For futures, there are multiple symbols used for same instrument , e.g. street symbol, exchange symbol, globex symbol etc. It is upto the software which one to use and there is one-to-one mapping using to target. As another example, I think Bloomberg have their own symbols.
Don't worry about the symbols. Use whatever the company is giving and find the appropriate mapping.

I got answer to my question# 3 i.e. how to group data by time interval in above notebook. The groupBy() method !

@Jamie
Looks like Futures is not usable for either, research nor backtesting.
Data is not available since early 2017...
I was wondering if you guys at Q have any word on this?
Thanks in advance!

Futures data are not available anymore according to me from the end of 2018.

* future_contract = symbols('NGF20') futures_position_value = get_pricing(future_contract, start_date = '2017-01-01', end_date = '2020-01-01', fields = 'price') futures_position_value.name = futures_position_value.name.symbol futures_position_value.plot() plt.title('NG Futures Price') plt.xlabel('Date') plt.ylabel('Price'); *

Any info on futures data availability? what happened?