Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Trading VIX - Quandl Data Now In Pipeline For Backtesting And Live Trading

Now that Quandl data is available natively through Quantopian Data I thought I'd whip up a little volatility trading algo. I chose doing a volatility algo because it is consistently one of the more popular trading strategies discussed on the site, and folks are always asking how to get VIX data in.

I took the strategy outlined here and adapted it for Quantopian. It estimates what the author calls the 'term structure impact' and uses the value to determine entry and exit points for trading the volatility ETFs VXX (S&P 500 short term futures) and XIV (inverse S&P 500 short term futures). Honestly, its pretty raw but should give you an idea on how to get started with Quandl data in Pipeline and the backtester. The author from the above link suggested doing a simple moving average cross in combination with the 'term structure impact' to determine entry and exits points, but I'd like to see what else we can come up with.

Try out this algo and explore some of the other free datasets available natively on the platform. For more information on the integration check out Josh's post.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

38 responses

If I understand correctly, this algo would be basically ineligible for both the contest and the fund, correct?

Correct. There are a number of reasons like it doesn't short, high beta, etc. But It looked like a simple implementation to try out first.

Will this work for me if I sign up on Quandl?

Error: Runtime exception: UsageNotAllowed: You do not have access to the following dataset(s): quandl.yahoo_index_vix (quantopian.com/data/quandl/yahoo_index_vix)

Hi Gary,
Similar to the Accern thread, I'll answer here for other viewers :)

For any data set from our partners, listed at quantopian.com/data, you need to enable access to the set for yourself. This means (in the case of Quandl) that you need to hit the "Get" button on the page for the data set you want to use. So head over to https://quantopian.com/data/quandl/yahoo_index_vix and hit "Get" and then try to run the algo again.

In the case of the premium sets, you can "Get" a free sample the same way. The full dataset requires a monthly subscription.

Hope that makes sense

Josh

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Hi all,

A newb question here, but how do I use this yahoo_index_vix object with the data object in the handle_data function? So when I do backtesting, how can I access the close price of the previous day for the VIX?

Thanks

As far as I understand, the only place you can access historical values of these new datasets is within a custom factor class of the Pipeline framework. To get the N previous days of VIX in handle_data you'd need to create N custom factors which returned the [-N] row of the pipeline array. Hopefully you can do all the calculations you need from within the custom factor.

Sounds good, I'll have a try, thank you!

@James, Thanks for the code - can we get Vxv data in aswell ?

I'll add your +1 for VXV

Please add +1 for VXMT and VXST as well!

@James and Simon,
I understoof marketsci’s #1 strategy a bit dfferently, i took it as using the point between contango/backwardation as a trigger for switching between Vxx and Xiv. Heres the line from marketsci:

Keeping it really simple, let’s assume we bought VXX at the close when the values in the chart above were
positive, and bought XIV when the values in the chart were negative.

By "values in the chart above" he is refferring to the custom impact calulation that he has, but he sais in the post that one can also use the basic Vix term structure (first month vs second month) with similar results, thats what i think i have done in the attached backtest

you could get in the ballpark simply comparing the first/second month futures contracts

I commented the code as much as possible so you can perhaps take a look and let me know if i misunderstood marketsci’s post
Thanks for your time

I am not really sure of the intent of the original strategy here; the one that I trade uses the transformed slope(s) of the VIX spot complex and the VIX implied vs realized vol VRP(s) as an analog signal for VXX/XIV/SVXY/VIXY, combined with an inverse-volatility position sizing.

ps - you are logging ContangoOrBackwardation = 1

I think the algo as it is suffers from intra-day look-ahead bias.

The code does its rebalancing 30 minutes after market open. It does so by using the output of compute(), which in turn uses today's yahoo_index_vix.close. That value won't be available in real-life until the end of the day.

You might want to drop the last element of vix_close and of use_close (and modify your window_length accordingly).

Hi Gillaume,

Good catch. We think there's a problem in the lag we've modeled for the historic load of data for this Quandl data set (this is on Quantopian, not Quandl, to be clear). To explain deeper, we track two key dates for each data set from our partners: asof_date and timestamp. asof_date represents the time to which the data record applies. The timestamp represents the time at which that record made it into our database.

For our daily processing of new data points, timestamp is generated based on our actual processing of the data. For the data that we bootstrapped the system with from the past, we need to model what the lag would be between the asof_date and the timestamp. In this case, it looks like we mistakenly set this lag to 0 and thus, it's being made available 1 day early. We're in the process of calculating a realistic lag to use and reprocessing the Quandl data.

Thanks for the help,
Josh

Tammer posted an article on Quandl' site:

https://www.quandl.com/blog/interview-with-a-quant-part-one

@James, Despite activating the data, i still get a runtime error that i should activate the data,
Tried logging out, back in, making new clean algo etc - has not worked so far

Thanks

Sorry about that. You should have access now.

We're in the process of calculating a realistic lag to use and reprocessing the Quandl data.

Has the fix been implemented? I thought I'd give it a try, but if it has a bug, I can wait.

By the way, you gotta consider a user-facing bug/issue tracker, ideally with notifications to users. Also, in this case, I get the impression that you just left the bad data out there. As soon as the problem was found, shouldn't you have blocked access? A one day look-ahead bias would seem to be a critical bug that could affect user's decisions and potentially result in financial losses.

Hi Grant,
Interesting idea on the bug database.

Not yet on the bug fix -- we're currently testing the solution in our test environment. You can check in on the status of fix by running the following in a research notebook:

from quantopian.interactive.data.quandl import yahoo_index_vix as vix  
vix.sort('asof_date')  

When the asof_date and the timestamp are different, you'll know that the fix has been made.

In the meantime, this bug doesn't affect data which we have processed 'live'. So you can look for when the timestamps for the data are no long simulated and reflect the actual date of processing. This date is December 11, 2015:

In [4]:  
vix[vix.asof_date != vix.timestamp].sort('asof_date')  
Out[4]:  
asof_date   open_   high    low close   volume  adjusted_close  timestamp  
0   2015-12-11  21.360001   25.270000   20.879999   24.389999   0   24.389999   2015-12-12 19:01:34.530229  
1   2015-12-14  24.700001   26.809999   21.469999   22.730000   0   22.730000   2015-12-15 06:36:07.893760  
2   2015-12-15  20.760000   21.620001   20.020000   20.950001   0   20.950001   2015-12-16 21:35:08.360000  
3   2015-12-16  19.250000   20.240000   17.120001   17.860001   0   17.860001   2015-12-18 03:07:28.729848  
4   2015-12-17  16.180000   19.049999   16.129999   18.940001   0   18.940001   2015-12-18 06:04:15.290164  
5   2015-12-18  19.340000   23.299999   18.750000   20.700001   0   20.700001   2015-12-19 05:02:12.136719  
6   2015-12-21  19.639999   20.209999   18.700001   18.700001   0   18.700001   2015-12-22 07:05:26.978916  
7   2015-12-22  17.610001   18.219999   16.600000   16.600000   0   16.600000   2015-12-23 06:03:41.145885  
8   2015-12-23  15.860000   16.250000   15.330000   15.570000   0   15.570000   2015-12-24 09:04:05.117921  
9   2015-12-24  15.440000   15.880000   14.450000   15.740000   0   15.740000   2015-12-25 06:03:28.041797  
10  2015-12-28  17.650000   18.129999   16.879999   16.910000   0   16.910000   2015-12-29 03:04:59.535674  

This provides us with daily VIX data, what if I needed the current VIX level in my algorithm?

Thanks
AR

This provides us with daily VIX data, what if I needed the current VIX level in my algorithm?

Thanks
AR

Hi Albert, today we don't have a facility for delivering intra-day, external data. This request (especially for VIX) is a common one and I'll put a +1 next to it for you in our backlog.

Go ahead and add another +1, sir.

Thank you Josh, your work is appreciated!

@Josh - Not only VIX, but other indexes should be made available as minute bars. We should be able to see the S&P500, not have to rely on SPY.

Hi all,

I wanted to provide notification that the bug causing look-ahead bias with the Quandl data has been fixed. The timestamp column for data we loaded initially now reflects a lag. Previously timestamp matched the asof_date field for data loaded up in our initial load. Now, each timestamp is lagged properly based on data observed from our daily loading. See the attached notebook for a simple illustration of this. Note, timestamps for data that is newly loaded each day receives a timestamp value the reflects the datetime at which the data was saved in the database.

The impact of this fix is that it removes a source of lookahead bias in the backtesting since the close price was available to the backtester before trading occurred.

This fix covers all data sets sourced from Quandl.

Thanks
Josh

Hi James,

The link in your original post and the links in the code to the strategy such as :

https://marketsci.wordpress.com/2012/04/11/strategy-1-for-trading-volatility-etps-term-structure-following/

are password protected. Is there somewhere else we can see the ideas behind this strategy?

Tony

Tony, I couldn't seem to find the site either.

It seems that the logic is embedded into the comments of the factor where you are isolating how much of the movement of VXX isn't due to the VIX index.

# Calculate the impact of the term structure (see more about the 'impact' at the link above). Basically,  
# isolate how much of the movement of VXX isn't due to the VIX index. As a handwavy model we'll  
# run a simple regression of the changes in the VIX to the changes in VXX, and use the  
# intercept term as an estimation of the 'impact.'  
class TermStructureImpact(CustomFactor):  
    # Pre-declare inputs and window_length  
    inputs = [yahoo_index_vix.close, USEquityPricing.close]  
    window_length = 20  
    def compute(self, today, assets, out, vix, close):  
        # Get the prices series of just VXX and calculate its daily returns.  
        vxx_returns = pd.DataFrame(close, columns=assets)[sid(38054)].pct_change()[1:]  
        # Since there isn't a ticker for the raw VIX Pipeline feeds us the value of the  
        # VIX for each day in the 'window_length' for each asset. Which kind of makes sense  
        # -- the VIX is the same value for every security.  
        # Since I have a fixed universe I'll just use VXX, one of my securities, to get a single series of  
        # VIX data. You could use any security or integer index to any column, but I'll use one of my  
        # securities just to keep things straight in my head.  
        vix_returns = pd.DataFrame(vix, columns=assets)[sid(38054)].pct_change()[1:]  
        # Calculate the 'impact.'  
        alpha = _intercept(vix_returns, vxx_returns)  
        out[:] = alpha * np.ones(len(assets)) # We do this so we can return a value for each security in the Pipeline.  
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Is there an easy way to modify this code to use open prices rather than close prices? I am unfamiliar with importing data.

You would change yahoo_index_vix.close to yahoo_index_vix.open based on the fact that the data has an open column. You can explore the data as demonstrated in the Example Notebook on the page for this data set: https://www.quantopian.com/data/quandl/yahoo_index_vix

Note, you'll be using the open value for this index for the previous day since the algorithm is accessing the data populated overnight. The open value for the current day hasn't yet been calculated nor delivered to Quantopian. Accessing of this data occurs at 8:45am.

Oh ok, thanks!

For clarification, it's not possible to access the current days VIX Index open after trading starts?

Correct. We're loading them externally from Quandl overnight.

Hi folks,

As of Thursday, July 7th. This algorithm will need to reflect changes to the way we load VIX: https://www.quantopian.com/posts/upcoming-changes-to-quandl-datasets-in-pipeline-vix-vxv-etc-dot.

In order to do that, you'll need to change

# Calculate the impact of the term structure (see more about the 'impact' at the link above). Basically,  
# isolate how much of the movement of VXX isn't due to the VIX index. As a handwavy model we'll  
# run a simple regression of the changes in the VIX to the changes in VXX, and use the  
# intercept term as an estimation of the 'impact.'  
class TermStructureImpact(CustomFactor):  
    # Pre-declare inputs and window_length  
    inputs = [yahoo_index_vix.close, USEquityPricing.close]  
    window_length = 20  
    def compute(self, today, assets, out, vix, close):  
        # Get the prices series of just VXX and calculate its daily returns.  
        vxx_returns = pd.DataFrame(close, columns=assets)[sid(38054)].pct_change()[1:]  
        # Since there isn't a ticker for the raw VIX Pipeline feeds us the value of the  
        # VIX for each day in the 'window_length' for each asset. Which kind of makes sense  
        # -- the VIX is the same value for every security.  
        # Since I have a fixed universe I'll just use VXX, one of my securities, to get a single series of  
        # VIX data. You could use any security or integer index to any column, but I'll use one of my  
        # securities just to keep things straight in my head.  
        vix_returns = pd.DataFrame(vix, columns=assets)[sid(38054)].pct_change()[1:]  
        # Calculate the 'impact.'  
        alpha = _intercept(vix_returns, vxx_returns)  
        out[:] = alpha * np.ones(len(assets))  

To this starting on July 7th

# Calculate the impact of the term structure (see more about the 'impact' at the link above). Basically,  
# isolate how much of the movement of VXX isn't due to the VIX index. As a handwavy model we'll  
# run a simple regression of the changes in the VIX to the changes in VXX, and use the  
# intercept term as an estimation of the 'impact.'  
class TermStructureImpact(CustomFactor):  
    # Pre-declare inputs and window_length  
    inputs = [yahoo_index_vix.close, USEquityPricing.close]  
    window_length = 20  
    def compute(self, today, assets, out, vix, close):  
        # Get the prices series of just VXX and calculate its daily returns.  
        vxx_returns = pd.DataFrame(close, columns=assets)[sid(38054)].pct_change()[1:]  
        # Since there isn't a ticker for the raw VIX Pipeline feeds us the value of the  
        # VIX for each day in the 'window_length' for each asset. Which kind of makes sense  
        # -- the VIX is the same value for every security.  
        # Since I have a fixed universe I'll just use VXX, one of my securities, to get a single series of  
        # VIX data. You could use any security or integer index to any column, but I'll use one of my  
        # securities just to keep things straight in my head.  
        vix_returns = pd.DataFrame(vix).pct_change()[0].iloc[1:]  
        # Calculate the 'impact.'  
        alpha = _intercept(vix_returns, vxx_returns)  
        out[:] = alpha  

The main algorithm source code will be migrated to reflect that change once the date comes. If you have any questions, please let me know.

Hi all, i just tried to run a back test on this algo unfortunately i have the following error message

Something went wrong. Sorry for the inconvenience. Try using the built-in debugger to analyze your code. If you would like help, send us an email.
ValueError: Shape of passed values is (1, 20), indices imply (7917, 20)
There was a runtime error on line 81.

anyone can help ?

Johann,

Please try cloning and running the algorithm now.

@Josh Payne :
I've cloned your notebook for getting VIX data from Quandl. But why just for 15 days? Can one get the data from any time?

Not sure about the 15 days statement but in Research, you can access any timeframe of data. Sometimes there are restrictions on how much data is printed out to your screen.

From your notebook I couldn't find the start date and end date. If I want to get the VIX data from 01/01/2017 to 12/30/2017, how to do that?