most futures listed on U.S. exchanges, including fixed income, stock index, and commodity futures  

Will currency futures be supported?

13 years of minute-bar history

Would be nice to know specifications on:
• how you constructed a continuous contract. Or, from where you acquired your data?
• trades reported in fabricated continuous contract or in monthly rolling contracts?
• contract roll timing, procedure?
• Initial and maintenance margin data, both historic and ongoing?
• margin call handling during backtests?
• equity / futures mixing of positions in single algo? (for hedging, stat-arb, etc.)
• applicability to your Fund?

Seconded for Market Tech's questions, very curious to know the implementation details of the futures rolling, whether rolls using spreads are supported and so forth.

Hey folks! I'm one of the engineers working on the futures trading project. Many of our plans are still in flux, but I'll answer what we know so far. We are looking for user feedback and ideas to finalize some design choices.

@Michael
We plan to support currency futures. We do not yet have an exact list of which contracts we will support. We will publish that list as soon as we are certain.

@MT
1. The Zipline additions are flexible to support constructed continuous contracts and monthly/quarterly/annual rolling contracts, though we are working first toward the rolling contracts. We are building tools to present the synthetic continuous data for use in your algos. How would you like to see continuous contract data represented in an algo?
2. Our data supplier is providing us with minute-level data for the rolling contracts.
3. We do not have a fixed plan for our rolling API, but we've talked about providing a roll method that would close the given future and open a position in the subsequent contract, similar to a two-sided order() call. This also plays in to the previous conversations about removing expired assets from portfolios, as all futures would need to be out of the portfolio by their notice dates, but could be rolled prior to that date. What would make the most sense to you to perform rolling in an algo?
4. We do not have a source for historic margin requirements, and they also vary from brokerage-to-brokerage. The standing idea is to provide a margin requirement model, similar to the commission and slippage models, that could be modified to suit the user's needs. These models would be to adjust on price variance or other factors.
5. We plan to implement margin call simulation for both futures and equities.
6. We have been building toward mixed equities/futures portfolios, and see no barriers to providing it. Zipline now supports these mixed-asset portfolios.
7. Futures algos will absolutely be applicable to our fund.

@Simon
We definitely want to support rolling with spreads. How do you see a spread roll looking in your algo code?

Please keep the questions coming! -James

Oh I don't know, something like:

def roll(sid):  
  contractSeries = sid.contractSeries  
  expiry = sid.contractExpiry # contractExpiry will need to know the contractSeries too, for the below to work properly  
  nextExpiry = expiry + futures.ExpiryOffset(quarters=1)  
  nextSID = futuressymbols(contractSeries, nextExpiry)  
  contractsHeld = __context.portfolio.positions[sid].amount  
  spread = [ {sid: sid, amount: -contractsHeld}, {sid: nextSid, amount: contractsHeld} ]  
  order(spread, 1, style=MarketOrder)  

Note that is completely made up, I haven't looked at the zipline PR at all. The trick is making sure that whenever possible, that gets executed as a single order of the spread contract on the exchange, rather than just two legs and two tickets. As a bonus, now you support spread trading. :)

Captain Kirk,

Done right, the possibilities boggle the mind.

Let's see:

1. "continuous contract data represented"... I'm thinking in/out sample end dates which would represent different assemblies of rollups, and therefore different point-in-time prices, but that's probably not necessary. The problem comes with representing trades, true contract trades that must have been made in the expiries at the time of the trades, as opposed to seemingly disconnected continuous contract prices. This is easily one of the hardest parts of futures back testing. Sure you can gloss it over and just assume the continuous has been calculated correctly. But all the other factors dealing with true trades in real time trading just don't jive. It's almost as if you'd want to provide both price streams for a single futures, one to watch and one to trade.

2. So trades will show up in the actuals and not in the continuous? You'd have to do this for all the rolling anyway so I assume this is true. What a hassle no?

3. Some traders like to roll out early or opportunistically. Probably a fixed n days prior to expiry would work. A nice to have would be a volume traded roll at a specified ratio, e.g. when the next expiry's volume exceeds the current contract by 2:1 or 4:1 then roll. That way you keep the volume up and liquidity high in your position.

4. Most systems I've been involved with always assume the trader was solvent enough to always answer margin calls. I guess you guys can't do this with retail. I suppose a switch to either assume a bottomless account, or as I think you do in your actual trading by just aggressively closing down positions until the condition is ameliorated.

5. Margin calls, as events? That would be wise. Shift the onus onto the algo to determine how to best come out of hock.

6. Margin changes over time per contracts. You mentioned that you don't have historic feed for this but there have been some pretty severe shocks to commodities over the years regarding drastic margin requirement boosts (have there ever been relaxations?). I guess they're seldom enough to just have real time algos deal with them when they happen. But unlike the various errors in dividends and splits in the equities you've been running there would be no weird jumps/drops in the margin requirements. You might be trading (backtest) money you don't really have. Another tough nut.

7. Re: the Fund, are you sure? Are the risks not considerably different? And the clientele? Maybe a managed futures fund vs a stodgy, lumbering equities fund?

8. Getting calendar and cross market spreads rolling, right and accurate would be a huge boon to analysis. Now, 13 years is "OK" but there is some deep data out there regarding futures going back into the late 1900's. I mean, there's Mar 1960 Corn available! I guess 13 years of minutely data would be good enough though. Do you have a contract list yet? Are you going to team up with CQG or TT or NewEdge or other brokerages now? Have you guys implemented a proper FIX 5.n yet?

Nice addition! I am looking forward to bust my simulated account with some margin calls :D

This also plays in to the previous conversations about removing expired assets from portfolios, as all futures would need to be out of the portfolio by their notice dates, but could be rolled prior to that date.

@ James,

Are you planning to move to a more generic asset object (see https://github.com/quantopian/zipline/blob/master/zipline/assets/assets.py):

# Expected fields for an Asset's metadata  
ASSET_FIELDS = [  
    'sid',  
    'asset_type',  
    'symbol',  
    'asset_name',  
    'start_date',  
    'end_date',  
    'first_traded',  
    'exchange',  
    'notice_date',  
    'expiration_date',  
    'contract_multiplier',  
    # The following fields are for compatibility with other systems  
    'file_name',  # Used as symbol  
    'company_name',  # Used as asset_name  
    'start_date_nano',  # Used as start_date  
    'end_date_nano',  # Used as end_date  
]

What would be the distinction between end_date, notice_date, and expiration_date? And end_date_nano? For equities? For futures?

For equities, would notice_date correspond to the date when a delisting was announced, and expiration_date would be the scheduled delisting date (presumably, a trading suspension would be handled differently, since the security would remain active, i.e. the end_date would keep pushing out into the future..."Time keeps on slippin', slippin', slippin'...")? I figure it would be worth some thought for future compatibility, assuming Q eventually acquires a data feed that provides the delisting info.

Also, if you are moving to a generic asset object, what will become of the security object, https://www.quantopian.com/help#api-sidinfo? I assume that you wouldn't "break" algos, so how do you plan to manage this?

Regarding your sampling of market data (for equities, you create OHLCV minute bars), why don't you open-source the code, along with details on how you are maintaining timing/synchronization?

Our data supplier is providing us with minute-level data for the rolling contracts

You are talking about your backtest data provider, right? For live trading, you'll be computing minute level data on the fly, as you do for equities, correct? Will the backtest provider and live trade data provider be different? Will you reconcile the backtest and live trade data, to make sure that they are exactly the same?

@Simon
I like that pattern - thank you. I will dig in to see if we can provide the spread through the exchange.

@MT
1. This all makes sense. The challenge in providing a continuous pricing stream is that there are simply so many schools of thought on how to calculate it. My standing plan is to provide a pricing history that is an un-adjusted stitching of the individual contracts on their roll dates for use in finding trading signal. The individual streams would still be available for anyone who wants to perform a continuous pricing calculation by their own methodology.
2. Trades will show up in the actuals. This means that rolling must be handled manually, though we aim to make that as painless as possible with API rolling methods.
3. That makes sense, and would be supported in our plans.
4. Parallel to the futures project, we are working out how we want our margin models to function. One possibility for the first versions would present the user with warnings (possibly exceptions) that they had passed their margin capabilities, but leave the execution of margin call transactions in backtest optional.
5. Yep, but in my mind I go back-and-forth on this point: if a margin call is a situation to be avoided whenever possible by the algorithm, then why simulate margin calls when a margin call practically represents a failure case? My thinking is that the back-tester should simulate a margin call, but be very verbose to the user that something bad has happened and should be avoided. What do you think?
6. A tough nut, indeed! But if these problems were easy, it wouldn't be fun to solve them. For equities, modeling the RegT margin requirements would cover most cases, as the RegT margins would be the most conservative requirements. For futures, possibly using a variance-based approximator over time would model the brokerages' margin models closely enough for testing.
7. I'll defer to Jess Stauth re: the fund. I've asked her to drop in.
8. We're finalizing the contract list now. We're securing >13 years of data for Futures and we're retooling our infrastructure to support trading and analysis over these longer historical periods. For initial release, we are looking at the same tradable range as equities (13 years). We will be supporting futures trading initially through Interactive Brokers, as with equities.

@Grant
We are planning to move to a more generic Asset object that will be extended to new classes that are needed as new tradable Assets are added. The generic Asset, and the specific Equity, are almost exactly the same as Security in the current system, and will be interacted with identically as Security is deprecated so as to not 'break' algos.

You can see the definitions of these Asset objects in _assets.pyx (a Cython file, though it is more-or-less readable as Python).
You can see a PR here for compatibility (adding security_*_date properties).

Not all of the fields you pasted there apply to every Asset sub-class. An Equity object will not have a notice_date, expiration_date, or contract_multiplier. The *_nano fields are for compatibility with other systems and overwrite their corresponding date fields, if provided.

start_date: First day of trade data for an Asset
end_date: Last day of trade data for an Asset (today, if still trading)
notice_date: First notice for a Future
expiration_date: The day at which a Future is no longer trading (the day after the final end_date)

We are considering open-sourcing our data-loading infrastructure, as you've suggested, but it would be of limited use to someone who doesn't have contracts with our same data suppliers. It's definitely still on the table, though.

Hi guys,

It is awesome to see so much interest and so many great questions about this work!

On the fund front, we're absolutely going to be looking to invest in strategies that use futures. Opening up the playing field of asset classes to allow for broader diversification of algorithms was a big part of why we decided to tackle this project.

That said we've got a big head start with the equities part of the platform, so it remains to be seen whether futures algos will be included in the first fund product we launch. I think it is too soon to speculate on the details of how we'd combine and market a cross-asset product at this point.

Please feel free to keep sharing ideas and questions, either on this thread or privately with us at [email protected].

Best, Jess

Two requests:

1. Generic naming for the first most liquid contracts and subsequent contracts (Eurodollars will get complicated) plus database of roll and expiry dates.

2. Eurex products please?

(not futures related but Asia Equities would be nice as from my experience these products are much less efficient than US markets)

My request is Europe most important indices (DAX, EUROSTOCK, IBEX...)

I am coming to Quantopian for the first time. I am also working with Zipline - again for the first time. I have traded futures for many years. One thing I have not gathered yet:
Support for non USD denominated instruments. I trade anything from Euro denominated contracts through to Yen, HKD and even such delights as ZAR and KRW.

In backtesting, does the P&L account for the varying FX rates? If so, there is of course no reason not to use Zipline for non US shares also.

Karen, any idea when this will roll out? And will it integrate cleanly with IB Live Trading?
Also, functionality to calculate margin maintenance requirements and any margin expenses (both for futures + other) would be great.

any updates?

I'm interested to find out if the VIX futures (symbol /VX) data feed is part of the futures rollout.

@Max
We've entered early alpha stages of the project. We are finding and fixing bugs and working with users to ensure that our new API for working with futures is intuitive and complete. If you (or anyone reading this) are interested in participating in our futures alpha, please e-mail our team at: futures (at) quantopian.com .

@Daniel
VIX (VX-CBOE) futures are in the initial batch that we plan to offer.

any updates?

Bump

@Miles & gffs
Development is ongoing, and we are currently running a closed alpha test for our new futures-trading features. We do not have a timeline for an open beta test yet, but we will update this thread when we do.

Live long and prosper.

My request is FTSE China A50, a high volume and being traded actively
investing.com/indices/china-a50

Hi James -- we're waiting for it!!!

@Lake Austin -- daily volume only 1,400...
http://www.barchart.com/commodityfutures/SGX_FTSE_China_A50_Futures/HN

@Russell I got data from the follow link:
http://www.investing.com/indices/china-a50-historical-data
the daily volumne is between 500k - 1M, between june-july.
recently china market is closed because of lunar new year.
have no idea which one reflect the reality.... wish the comment field allow us to attach screenshots.

Not that you would want to copy but how does Tradestation handle the margin calls and rollovers

@Dennis -- not sure. Be careful, sell/buy before the rollover.

From where are you guys sourcing the data? Will this be tick-data post processed into minute bars? Will VX have the same resolution as say, ES or ZN?

I worked with the Ned Davis platform for many years and it used a perpetual contract calculation which was superb. It closely mimic's real life as one contract nears expiration the next contract out gains strength. Our models would never see the switch it was so gradual and smooth. I have the logic for this if it is of interest at all, it's very unique. Mark

Will the Futures data have the volume BID, volume ASK, so that delta in bar can be calculated. Look up delta divergence on you tube by market delta

@Kevin
We source our backtesting data from the same provider as our Equities data. They have asked us to keep their name private. For live trading, futures algorithms will receive their trade data from Nanex's NxCore product, who also provide our Equities live data feed.

As it stands, the tick data is processed in to minute and daily bars and presented to your algorithm similarly to how Equities trade data is presented in our system. All products will have the same resolution in this regard.

@David
Initially, the futures data will not have volume bid or volume ask. That would certainly be valuable, though, and we are looking at ways that we may add it.

Will you have continuous contracts ? How the gap adjustment will be done ? WiIl this data only be available for front month or for all months?

One problem with continuous contracts is that the adjusted prices can become negative for commodities with long periods of backwardation. I believe it would be preferable to use actual prices and then implement a roll logic similar to rebalancing. There would be 'standard quantopian' parameters for each contract so that results can be compared across strategies, but an algo developer should be able to implement his or her own roll strategy if desired. In addition, such a framework allows trading across time, e.g long Dec16 short Jun16.

@James - have you all looked into the free data program available directly from CME? The data would be more reliable than other providers, in my humble opinion...http://fintechsandbox.org/

Are futures released in 2.0?

any update on this feature release?

Yeah, then the lunatic fringe can really do some damage. If you want to trade a low leverage system on reasonably wide stops, you guys are going to need a huge capital base.

But I thought that the massive amount of leverage was the whole point of futures. If I'm going to blow up my account, I want to do it really fast.

Very well put Tar Xvzf. You will be following a long favoured path.

Long surviving CTAs tend to have a pretty modest margin to equity Ratio. I think mine avereraged around 10 to 15%. This still lead to significant leverage on low vol instruments such as the 3 month Eurodollar in isolation but in general I used only very modest leverage. People can check out this statement by looking at margin requirements.

I have seen so many hopefuls over the years using high leverage relying on back tested results. The results have usually not prooved very satisfactory!😠😂😞😃😉😭

Hi Everyone,
We continue to work away on adding futures support to Quantopian. Last week we merged a big change to Zipline that enables support for multiple trading calendars. This is critical to allow for futures trading outside of NYSE market hours. We've also made great progress at ingesting futures data.

The next steps for the project are to work on building out the calendars for the CBOE, CME and ICE exchanges. Part of this will be solving problems around allowing algorithms to trade both equities and futures at the same time, and dealing with time management throughout the system.

We will be launching an alpha after this next phase is complete. You can sign up for it here, and also learn a little bit more (such as which contracts we plan to support).

Can you give some timeline? Are you expecting to make futures trading available sometime this year (2016)?

Hi Karen,

This seems like a pretty big change ("trade both equities and futures at the same time" presumably integrated in the same platform). Do you expect another Q2 (i.e. Q3)? Will testdrive be revived? Maybe publish a schedule well in advance? I guess my concern would be that if you roll things out as you did for Q2, there won't be sufficient heads up and time to incorporate and test changes.

Grant

Certainly don't change the engine under running contest algorithms.

Hi,
I noticed this announcement happened over a year ago, I'm wondering if we could have access to the futures API and futures data on Research? that would serve as a way to beta test it before making it available on the platform.

Bump. How is it going with the futures integration?
Cheers

Any updates ?

If you go to the zipline GitHub you can see lots of commits of code relating to futures and handling dates. I think the guys are working on it, but avoiding any promises as these kind of projects have a nasty habit of missing deadlines and disappointing people.

https://github.com/quantopian/zipline/pull/1456

Do you have an approximate timeline for the futures ? Is it Q1 2017 or later?

Can't wait, thanks to all the team!

Margin calls sure are a very good idea. Do commodities futures include softs by any chance?

Thanks again

can't wait either. Thanks to the team guys

Awesome! I'm looking forward to it. Are there going to be news report feeds for it as well? For example, in CL there are weekly oil API and EIA reports.

Im ecstatic!!!

Thank you guys for the awesome work!!!

Hey guys,

The futures project is currently in a private alpha stage. If you would like to participate in the alpha, please email in to [email protected]. It's a bit tough to give a timeline right now, there's still some work to do, but we are definitely making progress! If you're interested, a lot of the work can be tracked in Zipline.

@Patrick: When we publicly launch futures in backtesting, we won't have data feeds for them right away, but we definitely plan to add them going forward.

Finally this is good news!

great news!

Are futures going to be accessible only via the Pipeline mechanism, or in other ways as well?

Thanks,

TIm

To start, futures data will only be available through the data object (i.e. data.history and data.current). The goal will be to add futures to pipeline going forward, but they won't be available in pipeline at the start.

Thank you, Jamie!

What's the current thinking on data sources for futures backtesting? Where's the data coming from? How far back will the data go?

I would like to sign up for futures alpha testing, if it is still possible. Many thanks.

The data for Futures is now available on Quantopian Research. Jamie outlines the API and other concepts in the forum post making the announcement.

Futures implementation is Great. Eurex futures would be Super Great. Great Job guys

Futures are now available in research and backtesting. Check out the announcement post for more details!

Any progress with Eurex futures?

Any idea when futures will be included in live trading please?

What about Dax and Bund futures?