Notebook

Futures API Introduction

Futures data is now available in Quantopian research for 72 US futures going back to the start of 2002 (where applicable). Data includes daily and minutely open, high, low, close, and volume (OHLCV) data for 24 hours x 5 days a week and is collected from electronic trade data.

The list of US futures currently available on Quantopian can be found at the end of this notebook.

Let's start by looking at the data for a particular contract for the Crude Oil future. Similar to equities, a reference to a futures contract is obtained via the symbols function. Futures contracts are denoted by a base symbol + a code for month/year of delivery. CLF16 is a contract for crude oil (CL) with delivery in January (F) 2016 (16). -- Here is a reference for a mapping of month codes.

In [1]:
clf16 = symbols('CLF16')
clf16
Out[1]:
Future(1058201601, symbol=u'CLF16', root_symbol=u'CL', asset_name=u'Light Sweet Crude Oil', exchange=u'NYMEX', start_date=Timestamp('2012-01-19 00:00:00+0000', tz='UTC'), end_date=Timestamp('2015-12-21 00:00:00+0000', tz='UTC'), first_traded=None, notice_date=Timestamp('2015-12-23 00:00:00+0000', tz='UTC'), expiration_date=Timestamp('2015-12-21 00:00:00+0000', tz='UTC'), auto_close_date=Timestamp('2015-12-17 00:00:00+0000', tz='UTC'), tick_size=0.01, multiplier=1000.0, exchange_full=u'NYMEX')

Here is a brief explanation of some of the properies of the Future object:

  • root_symbol: The root symbol of the underlying. For example, 'CL' for crude oil.
  • start_date: The first date that the contract exists on Quantopian. Note that the price of a contract might be a NaN near the start_date as it may not be traded actively until closer to delivery.
  • auto_close_date: The auto_close_date is 2 days prior to the earlier of the end_date (last traded date) and the notice_date. See this explanation of last traded date and notice date. In backtesting, positions in contracts will be automatically be closed out on their auto_close_date.
  • tick_size: The price of a future can only change in increments of its tick_size. For example, CL changes in increments of \$0.01.
  • multiplier: The number of units per contract. A contract for CL is for 1000 barrels of oil.

history

Right now, futures OHLCV data is available in research via a new history function in the experimental library. It's possible that this API might change in the near future depending on the feedback that we get from you, the community. The history function is very similar to get_pricing.

To start interacting with pricing and volume data for futures, let's import history:

In [2]:
from quantopian.research.experimental import history

In short, history can be called like this:

history(assets, fields, frequency, start_date, end_date)

To see the full doc string for history, uncomment and run the next cell.

In [3]:
#print history.__doc__

Let's get the close price and volume of our crude oil contract for the 2 months leading up to the delivery date (2015-12-21). Note that all pricing data for futures are unit prices. For a CL contract, the price is the price per barrel, and a contract is for 1000 barrels (the multiplier). In backtesting, the value of a portfolio will change by an amount equal to the price change * multiplier.

In [4]:
clf16_data = history(
    clf16, 
    fields=['price', 'volume'], 
    frequency='daily', 
    start_date='2015-10-21', 
    end_date='2015-12-21'
)

clf16_data.head()
Out[4]:
price volume
2015-10-21 00:00:00+00:00 45.97 21833.0
2015-10-22 00:00:00+00:00 46.27 17198.0
2015-10-23 00:00:00+00:00 45.65 27287.0
2015-10-26 00:00:00+00:00 44.63 18940.0
2015-10-27 00:00:00+00:00 44.30 22071.0

Now let's plot each field.

In [5]:
clf16_data.price.plot();
In [6]:
clf16_data.volume.plot();

The rise and subsequent drop in trading volume prior to the delivery date of a contract is typical behavior for futures. Let's see what the volume looks like for a series of consecutive contracts.

In [7]:
cl_contracts = symbols(['CLF16', 'CLG16', 'CLH16', 'CLJ16', 'CLK16', 'CLM16'])
In [8]:
cl_consecutive_contract_volume = history(
    cl_contracts, 
    fields='volume', 
    frequency='daily', 
    start_date='2015-10-21', 
    end_date='2016-06-01'
)
In [9]:
cl_consecutive_contract_volume.plot();

Trading activity jumps from one contract to the next. Transitions happen just prior to the delivery date of each contract.

This phenomenon can make it difficult to work with futures. Having to explicitly reference a series of transient contracts when trading or simulating futures can be a hassle.

In order to trade consecutive contracts for the same underlying future, we can use what's called a "Continuous Future".

ContinuousFutures

Continuous futures are abstractions over the 'underlying' commodities/assets/indexes of futures. For example, if we wanted to trade crude oil, we could create a reference to CL, instead of a series of CL contracts. Continuous futures essentially maintain a reference to a 'current' contract deemed to be the active contract for the particular underlying.

N.B. Continuous futures are not tradable assets. They maintain a reference to the current active contract related to a given underlying.

To create a continuous future on Quantopian, we must import the continuous_future function from the experimental library. Similar to history, the continuous_future function is in the quantopian.research.experimental.

Let's import it and take a look at the documentation.

In [10]:
from quantopian.research.experimental import continuous_future
In [11]:
print continuous_future.__doc__

    Create a specifier for a continuous contract.

    Parameters
    ----------
    root_symbol : str
        The root symbol for the continuous future.

    offset : int, optional
        The distance from the primary contract. Default is 0.

    roll : str, optional
        How rolls are determined. Options are 'volume' and 'calendar'. Default
        is 'volume'.

    adjustment : str
        Method for adjusting lookback prices between rolls. Options are
        'mul', 'add', and None. Default is 'mul'.

    Returns
    -------
    continuous_future : ContinuousFuture
        The continuous future specifier.

So there are 4 arguments that we need to consider.

  • root_symbol: The root symbol of the underlying. For example, 'CL' for crude oil.
  • offset: The distance from the primary contract. 0 = primary, 1 = secondary, etc. We'll get into this more later.
  • roll: How to determine the 'current' contract of the continuous future. Current options are 'volume' and 'calendar'. The 'volume' approach chooses the current active contract based on trading volume. The 'calendar' approach chooses the current active contract based simply on the auto_close_dates of each contract.**
  • adjustment: How to adjust historical prices from earlier contracts. We'll get into this more later. Options are 'mul', 'add', or 'None'.
In [12]:
cl = continuous_future('CL', offset=0, roll='volume', adjustment='mul')
cl
Out[12]:
ContinuousFuture(90999980378095616, root_symbol='CL', offset=0, roll_style='volume', adjustment='mul')

To determine the active contract of a continuous future, we can ask for the 'contract' field of a ContinuousFuture object from history.

In [13]:
cl_active = history(
    cl, 
    fields='contract', 
    frequency='daily', 
    start_date='2015-11-10', 
    end_date='2015-11-25'
)
In [14]:
cl_active.tail(10)
Out[14]:
2015-11-12 00:00:00+00:00    Future(1058201512 [CLZ15])
2015-11-13 00:00:00+00:00    Future(1058201512 [CLZ15])
2015-11-16 00:00:00+00:00    Future(1058201512 [CLZ15])
2015-11-17 00:00:00+00:00    Future(1058201512 [CLZ15])
2015-11-18 00:00:00+00:00    Future(1058201601 [CLF16])
2015-11-19 00:00:00+00:00    Future(1058201601 [CLF16])
2015-11-20 00:00:00+00:00    Future(1058201601 [CLF16])
2015-11-23 00:00:00+00:00    Future(1058201601 [CLF16])
2015-11-24 00:00:00+00:00    Future(1058201601 [CLF16])
2015-11-25 00:00:00+00:00    Future(1058201601 [CLF16])
Freq: C, Name: ContinuousFuture(90999980378095616 [CL, 0, volume, mul]), dtype: object

To get a better sense of what this is doing, let's get the volume history of our CL continuous future and plot it with the individual contract volumes that we plotted earlier.

In [15]:
cl_continuous_volume = history(
    cl, 
    fields='volume', 
    frequency='daily', 
    start_date='2015-10-21', 
    end_date='2016-06-01'
)
In [16]:
import pandas as pd

cl_volume_history = pd.concat([cl_consecutive_contract_volume, cl_continuous_volume], axis=1)
In [17]:
cl_volume_history.plot(style={cl: 'k--'});

From this plot, we see that the volume for the CL ContinuousFuture is essentially the skyline of the individual contract volumes. Note that there are some points where this looks a little off, most notably in the transition from CLK16 to CLM16 between April and May. This is because the rolls are currently computed daily, using only the previous day's volume to avoid lookahead bias.

Now let's take a look at pricing.

In [18]:
# Pricing data for our consecutive contracts from earlier.
cl_consecutive_contract_pricing = history(
    cl_contracts, 
    fields='price', 
    frequency='daily', 
    start_date='2015-10-21', 
    end_date='2016-06-01'
)

# Pricing data for our `ContinuousFuture`.
cl_continuous_future_pricing = history(
    cl, 
    fields='price', 
    frequency='daily', 
    start_date='2015-10-21', 
    end_date='2016-06-01'
)

Plotting the pricing data for consecutive contracts over a 7 month span is a little bit tricky because at the start, we have prices for all 7 contracts, which makes it a bit hard to visualize:

In [19]:
cl_consecutive_contract_pricing.plot();

To better visualize this, let's look at the historical price of the active contract of our CL ContinuousFuture.

In [20]:
cl_continuous_future_pricing.plot();

This represents the price of the underlying commodity, CL, on the most actively traded contract. Much easier to look at.

You might notice that the price at the start of this plot exceeds \$60, but when we plotted the individual contracts, it barely made it above \$50. This is because the historical price is getting adjusted for jumps between contracts.

The best way to explain this is to plot the prices history of the unadjusted continuous future.

In [21]:
cl_unadjusted = continuous_future('CL', offset=0, roll='volume', adjustment=None)

# Pricing data for our `ContinuousFuture`.
cl_cf_pricing_unadjusted = history(
    cl_unadjusted, 
    fields='price', 
    frequency='daily', 
    start_date='2015-10-21', 
    end_date='2016-06-01'
)
In [22]:
cl_cf_pricing_unadjusted.plot();

It's still a bit tricky to tell what's going on. Let's see if we can make it a little more obvious by plotting the active contracts separately.

In [23]:
cl_unadjusted_history = history(
    cl_unadjusted, 
    fields=['contract', 'price'], 
    frequency='daily', 
    start_date='2015-10-21', 
    end_date='2016-06-01'
)
In [24]:
cl_unadjusted_history = cl_unadjusted_history.pivot(index=cl_unadjusted_history.index, columns='contract')
cl_unadjusted_history.plot();

The increase in price when jumping from one contract is not considered to be an increase in value in the future. Instead, it is associated with the carrying cost of the underlying commodity. This will be covered more in the Lecture Series (futures lectures in progress).

Let's see what the unadjusted ContinuousFuture is doing.

In [25]:
pd.concat(
    [cl_unadjusted_history, cl_cf_pricing_unadjusted], 
    axis=1
).plot(
    style={cl_unadjusted: 'k--'}
);

And now the adjusted ContinuousFuture.

In [26]:
pd.concat(
    [cl_unadjusted_history, cl_continuous_future_pricing], 
    axis=1
).plot(
    style={cl: 'k--'}
);

Adjustment Types

In this example, our ContinuousFuture is using the 'mul' adjustment technique. This essentially computes the adjustment as the ratio of new contract price / old contract price whenever the active contract rolls to a new contract.

The 'add' technique computes the adjustment as the difference new contract price - old contract price.

Offset

The offset of a continuous future determines whether the active contract is the 'primary', the 'secondary', 'tertiary', etc. contract. So far in this notebook, we've been looking at the primary contracts, which are the contracts trading with the earliest delivery. The secondary contract is the one with the subsequent delivery date, tertiary is the next, etc.

For example, if we want to look at the term structure of a particular future, we can get a chain of contracts like this:

In [27]:
cl_cfs = []
for i in range(5):
    cl_cfs.append(continuous_future('CL', offset=i, roll='volume', adjustment='mul'))
    
pricing = history(
    cl_cfs, 
    fields=['price', 'contract'],
    frequency='daily',
    start_date='2015-05-05',
    end_date='2015-05-05'
)

Let's convert this Panel into a DataFrame and change our index to the auto_close_date of the active contracts.

In [28]:
term_structure = pricing[:, '2015-05-05', :]
term_structure.set_index(pd.Index([c.auto_close_date for c in term_structure.contract]), inplace=True)
In [29]:
term_structure.plot(marker='o');

This is a normal futures curve.

Feedback Welcome

This is a pre-release of futures in research. We're looking for feedback on the API presented in this notebook. If you have suggestions, difficulty using certain parts of the API, or any questions, please let us know.

Backtesting

We are working on adding futures to the backtester. The API for futures in backtesting will look very similar to the API presented in this notebook. We're planning to officially launch futures in backtesting at QuantCon.

List of Available Futures

These are the 72 US futures that are currently available on Quantopian.

Symbol Future
BD Big Dow
BO Soybean Oil
CM Corn E-Mini
CN Corn
DJ DJIA Futures
ET Ethanol
FF 30-Day Federal Funds
FI 5-Year Deliverable Interest Rate Swap Futures
FS 5-Year Interest Rate Swap Futures
FV 5-Year T-Note
MB Municipal Bonds
MS Soybeans E-Mini
MW Wheat E-Mini
OA Oats
RR Rough Rice
SM Soybean Meal
SY Soybeans
TN 10-Year Deliverable Interest Rate Swap Futures
TS 10-Year Interest Rate Swap Futures
TU 2-Year T-Note
TY 10-Year T-Note
UB Ultra Tbond
US 30-Year T-Bond
WC Wheat
YM Dow Jones E-mini
VX VIX Futures
AD Australian Dollar
AI Bloomberg Commodity Index Futures
BP British Pound
CD Canadian Dollar
EC Euro FX
ED Eurodollar
EE Euro FX E-mini
ES S&P 500 E-Mini
EU E-micro EUR/USD Futures
FC Feeder Cattle
JE Japanese Yen E-mini
JY Japanese Yen
LB Lumber
LC Live Cattle
LH Lean Hogs
MD S&P 400 MidCap Futures
ME Mexican Peso
MI S&P 400 MidCap E-Mini
ND NASDAQ 100 Futures
NK Nikkei 225 Futures
NQ NASDAQ 100 E-Mini
NZ New Zealand Dollar
SF Swiss Franc
SP S&P 500 Futures
TB TBills
GC Gold
HG Copper High Grade
SV Silver
CL Light Sweet Crude Oil
HO NY Harbor ULSD Futures
HU Unleaded Gasoline
NG Natural Gas
PA Palladium
PL Platinum
PB Pork Bellies
QG Natural Gas E-mini
QM Crude Oil E-Mini
XB RBOB Gasoline Futures
EI MSCI Emerging Markets Mini
EL Eurodollar NYSE LIFFE
MG MSCI EAFE Mini
XG Gold mini-sized
YS Silver mini-sized
RM Russell 1000 Mini
SB Sugar #11
ER Russell 2000 Mini
In [30]:
# All of the available futures categorized in a dictionary for convenience.
available_futures = {
    # Agricultural Commodities
    'ag_commodities': {
        'soybean': continuous_future('SY', offset=0, roll='volume', adjustment='mul'),
        'soybean_emini': continuous_future('MS', offset=0, roll='volume', adjustment='mul'),
        'soybean_oil': continuous_future('BO', offset=0, roll='volume', adjustment='mul'),
        'soybean_meal': continuous_future('SM', offset=0, roll='volume', adjustment='mul'),
        'wheat': continuous_future('WC', offset=0, roll='volume', adjustment='mul'),
        'wheat_emini': continuous_future('MW', offset=0, roll='volume', adjustment='mul'),
        'corn': continuous_future('CN', offset=0, roll='volume', adjustment='mul'),
        'corn_emini': continuous_future('CM', offset=0, roll='volume', adjustment='mul'),
        'oats': continuous_future('OA', offset=0, roll='volume', adjustment='mul'),
        'rough_rice': continuous_future('RR', offset=0, roll='volume', adjustment='mul'),
        'sugar': continuous_future('SB', offset=0, roll='volume', adjustment='mul'),
        'ethanol': continuous_future('ET', offset=0, roll='volume', adjustment='mul'),
        'feeder_cattle': continuous_future('FC', offset=0, roll='volume', adjustment='mul'),
        'live_cattle': continuous_future('LC', offset=0, roll='volume', adjustment='mul'),
        'lean_hogs': continuous_future('LH', offset=0, roll='volume', adjustment='mul'),
        'pork_bellies': continuous_future('PB', offset=0, roll='volume', adjustment='mul'),
        'lumber': continuous_future('LB', offset=0, roll='volume', adjustment='mul'),
    },

    # Non-Agricultural Commodities
    'non_ag_commodities': {
        'crude_oil': continuous_future('CL', offset=0, roll='volume', adjustment='mul'),
        'crude_oil_emini': continuous_future('QM', offset=0, roll='volume', adjustment='mul'),
        'natural_gas': continuous_future('NG', offset=0, roll='volume', adjustment='mul'),
        'natural_gas_emini': continuous_future('QG', offset=0, roll='volume', adjustment='mul'),
        'unleaded_gasoline': continuous_future('HU', offset=0, roll='volume', adjustment='mul'),
        'rbob_gasoline': continuous_future('XB', offset=0, roll='volume', adjustment='mul'),
        'heating_oil': continuous_future('HO', offset=0, roll='volume', adjustment='mul'),
        'gold': continuous_future('GC', offset=0, roll='volume', adjustment='mul'),
        'gold_mini': continuous_future('XG', offset=0, roll='volume', adjustment='mul'),
        'silver': continuous_future('SV', offset=0, roll='volume', adjustment='mul'),
        'silver_mini': continuous_future('YS', offset=0, roll='volume', adjustment='mul'),
        'copper': continuous_future('HG', offset=0, roll='volume', adjustment='mul'),
        'palladium': continuous_future('PA', offset=0, roll='volume', adjustment='mul'),
        'platinum': continuous_future('PL', offset=0, roll='volume', adjustment='mul'),
    },

    # Currencies
    'currencies': {
        'jpy': continuous_future('JY', offset=0, roll='volume', adjustment='mul'),
        'jpy_emini': continuous_future('JE', offset=0, roll='volume', adjustment='mul'),
        'cad': continuous_future('CD', offset=0, roll='volume', adjustment='mul'),
        'mxn': continuous_future('ME', offset=0, roll='volume', adjustment='mul'),
        'aud': continuous_future('AD', offset=0, roll='volume', adjustment='mul'),
        'nzd': continuous_future('NZ', offset=0, roll='volume', adjustment='mul'),
        'gbp': continuous_future('BP', offset=0, roll='volume', adjustment='mul'),
        'chf': continuous_future('SF', offset=0, roll='volume', adjustment='mul'),
        'eur_emicro': continuous_future('EU', offset=0, roll='volume', adjustment='mul'),
        'euro_fx': continuous_future('EC', offset=0, roll='volume', adjustment='mul'),
        'euro_fx_emini': continuous_future('EE', offset=0, roll='volume', adjustment='mul'),
        'eurodollar_nyse_liffe': continuous_future('EL', offset=0, roll='volume', adjustment='mul'),
    },

    # Equities
    'equities': {
        'sp500': continuous_future('SP', offset=0, roll='volume', adjustment='mul'),
        'sp500_emini': continuous_future('ES', offset=0, roll='volume', adjustment='mul'),
        'sp400_midcap': continuous_future('MD', offset=0, roll='volume', adjustment='mul'),
        'sp400_midcap_emini': continuous_future('MI', offset=0, roll='volume', adjustment='mul'),
        'nasdaq100': continuous_future('ND', offset=0, roll='volume', adjustment='mul'),
        'nasdaq100_emini': continuous_future('NQ', offset=0, roll='volume', adjustment='mul'),
        'russell1000_emini': continuous_future('RM', offset=0, roll='volume', adjustment='mul'),
        'russell2000_emini': continuous_future('ER', offset=0, roll='volume', adjustment='mul'),
        'nikkei225': continuous_future('NK', offset=0, roll='volume', adjustment='mul'),
        'MSCI_emerging_markets_mini': continuous_future('EI', offset=0, roll='volume', adjustment='mul'),
        'MSCI_EAFE_mini': continuous_future('MG', offset=0, roll='volume', adjustment='mul'),
        'dow_jones': continuous_future('DJ', offset=0, roll='volume', adjustment='mul'),
        'dow_jones_emini': continuous_future('YM', offset=0, roll='volume', adjustment='mul'),
        'big_dow': continuous_future('BD', offset=0, roll='volume', adjustment='mul'),
        'vix': continuous_future('VX', offset=0, roll='volume', adjustment='mul'),
        'bloomberg_commodity_index': continuous_future('AI', offset=0, roll='volume', adjustment='mul'),
    },

    # Rates
    'rates': {
        'us_2y': continuous_future('TU', offset=0, roll='volume', adjustment='mul'),
        'us_5y': continuous_future('FV', offset=0, roll='volume', adjustment='mul'),
        'us_10y': continuous_future('TY', offset=0, roll='volume', adjustment='mul'),
        'us_30y': continuous_future('US', offset=0, roll='volume', adjustment='mul'),
        'deliverable_interest_rate_swap_5y': continuous_future('FI', offset=0, roll='volume', adjustment='mul'),
        'deliverable_interest_rate_swap_10y': continuous_future('TN', offset=0, roll='volume', adjustment='mul'),
        'interest_rate_swap_5y': continuous_future('FS', offset=0, roll='volume', adjustment='mul'),
        'interest_rate_swap_10y': continuous_future('TS', offset=0, roll='volume', adjustment='mul'),
        'municipal_bonds': continuous_future('MB', offset=0, roll='volume', adjustment='mul'),
        'utra_tbond': continuous_future('UB', offset=0, roll='volume', adjustment='mul'),
        'tbills': continuous_future('TB', offset=0, roll='volume', adjustment='mul'),
        'fed30day': continuous_future('FF', offset=0, roll='volume', adjustment='mul'),
        'eur': continuous_future('ED', offset=0, roll='volume', adjustment='mul'),
    },
}