You may collect context.vix_hist and than calculate your indicator or create CustomFactor with your indicator, just like this:

from quantopian.algorithm import attach_pipeline, pipeline_output  
from quantopian.pipeline import Pipeline, CustomFactor  
from quantopian.pipeline.data.quandl import cboe_vix  
import numpy as np  
import pandas as pd  
# -----  
MA = 21  
# -----  
def initialize(context):  
    attach_pipeline(Pipeline(columns = {'VixClose':  GetVIX(), 'VixMavg': GetVixMavg()}), 'vix_pipeline')  
    context.vix_hist = []  

def before_trading_start(context, data):  
    output = pipeline_output('vix_pipeline')  
    pipeline_vix = output['VixClose'].iloc[-1]  
    pipeline_vix_ma = output['VixMavg'].iloc[-1]  
    context.vix_hist = np.append(context.vix_hist, pipeline_vix)

    if len(context.vix_hist) < MA: return  
    vix_ma = context.vix_hist[-MA:].mean()

    record(pipe_vix = pipeline_vix, vix_ma = vix_ma, pipeline_vix_ma = pipeline_vix_ma)  

class GetVIX(CustomFactor):  
    inputs = [cboe_vix.vix_close]  
    window_length = 1  
    def compute(self, today, assets, out, vix):  
        out[:] = vix[-1]  

class GetVixMavg(CustomFactor):  
    inputs = [cboe_vix.vix_close]  
    window_length = MA  
    def compute(self, today, assets, out, close):  
        close = close.ravel()  
        mean_close = pd.Series(close).rolling(MA).mean()  
        out[:] = mean_close.iloc[-1]  

Hi! Vladimir ,

Thanks for the quick response. It is certainly a workaround solution to get unconventional historical data . My problem is that I need years of data, not just days. This will likely impact the back test or any future processes in which I have to wait to collect enough data to get started. It would be nice if Q can provide access (like most other stocks) using data.history. Until then, it is an overhead I might have to live with.

Hi, Vladimir ,

I come up with this. Please comment:

from quantopian.algorithm import attach_pipeline, pipeline_output  
from quantopian.pipeline import Pipeline  
from quantopian.pipeline import  CustomFactor  
from quantopian.pipeline.data.quandl import cboe_vix

HIST_LEN = 50  #  sample length  
def initialize(context):  
    # Create our pipeline.  
    attach_pipeline(vix_pipeline(context), 'vix_pipeline')  
class GetVIX(CustomFactor):  
    inputs = [cboe_vix.vix_close]  
    def compute(self, today, assets, out, close):  
        vix = close.ravel()  
        for i in range(HIST_LEN):  
            out[i] = vix[i]

def vix_pipeline(context):  
    pipe = Pipeline()  
    vix = GetVIX(window_length = HIST_LEN)  
    pipe.add(vix, 'VIX')  
    return pipe  
def before_trading_start(context, data):  
    context.output = pipeline_output('vix_pipeline')  
    my_vix = context.output['VIX']  
    for i in range(HIST_LEN):  
        print (my_vix[i])  

Looking back at the original question "I need some historical VIX data to generate alpha signal to trade stocks. How would I go about it?". This is exactly the workflow/model that pipeline supports.

1. Fetch historical data
   
2. Manipulate data
   
3. Output a factor which associates an alpha or signal value with each security

The key is to manipulate the data inside the pipeline (in this case). Don't focus on getting the historical VIX out of pipeline. Rather, focus on moving the code which generates the alpha signal into pipeline. Maybe just a paradigm shift?

For example, a simple alpha signal may be "go long on stocks when the current VIX is less than the 10 day moving average of VIX, otherwise go short". This may be based upon the premise that VIX is generally inversely correlated with the market. How to code this signal inside of pipeline? Something like this:

from quantopian.pipeline.data.quandl import cboe_vix  
from quantopian.pipeline.factors import SimpleMovingAverage


vix_current = cboe_vix.vix_close.latest  
vix_ma_10 = SimpleMovingAverage(inputs=[cboe_vix.vix_close], window_length=10)  
long = vix_current < vix_ma_10  

A couple of things to notice. While the VIX dataset, like many of the macro economic datasets, is technically a 'slice' in that there isn't a value associated with each asset, one can combine it with regular datasets. Values are broadcast to each asset behind the scenes. Additionally, one doesn't always need to use a custom factor. In this case, 'vix_close' works fine as an input to a built-in factor such as 'SimpleMovingAverage'.

If more complex logic is needed beyond the built-in factors, one can always define a custom factor and put that logic into the 'compute' method. Generally, any logic that could be done on a dataframe returned by something like 'data.history' can just as easily (and much more quickly) be done inside a custom factor. Something like this.

    class VIX_Signal(CustomFactor):  
        # Get our inputs  
        inputs = [cboe_vix.vix_close, USEquityPricing.close]  
        # Set a window length to look back. In this case about 2 years  
        window_length = 400  
        def compute(self, today, assets, out, vix, close):  
            # Apply any logic over the input data  
            signal = ...some logic....  
            out[:] = signal  

Maybe include a more specific example if help is needed to move logic into a custom factor.

Another comment was made in the above posts "My problem is that I need years of data, not just days. This will likely impact the back test or any future processes in which I have to wait to collect enough data to get started". One doesn't need to wait to collect data to get started. The pipeline engine automatically grabs any history as needed.

The key is the paradigm shift to bring the logic into pipeline if large quantities of historical data are needed. Typically much simpler than trying to move the data out.

See the attached notebook for an example.

Good luck.

Dan,

I'm trying to reproduce your code in the IDE

# cboe VIX data pipe v4  
# First, we need to import the basic pipline methods  
from quantopian.pipeline import CustomFactor, Pipeline, factors, filters  
from quantopian.algorithm import attach_pipeline, pipeline_output  

# Also get the built-in filters and/or factors to use  
from quantopian.pipeline.filters import QTradableStocksUS, Q500US  
from quantopian.pipeline.factors import SimpleMovingAverage, Latest

# Finally get any data we want to use  
from quantopian.pipeline.data.quandl import cboe_vix  
# ----------------------  
W = 100  
# ----------------------  
def initialize(context):  
    attach_pipeline(vix_pipeline(context), 'vix_pipeline')     

def vix_pipeline(context):  
    universe = Q500US()  
    vix_current = cboe_vix.vix_close.latest  
    # vix_current = Latest(inputs=[cboe_vix.vix_close])  
    # vix_current = SimpleMovingAverage(inputs = [cboe_vix.vix_close], window_length = 1)  
    vix_ma = SimpleMovingAverage(inputs = [cboe_vix.vix_close], window_length = W) 

    pipe = pipe = Pipeline(  
        columns = {  
        'vix_current' : vix_current,  
        'vix_ma' : vix_ma,  
        },  
        screen = universe  
     )  
    return  pipe      

def before_trading_start(context, data):  
    output = pipeline_output('vix_pipeline')  
    current = output['vix_current'].iloc[-1]  
    mavg = output['vix_ma'].iloc[-1]

    print (current, mavg)  
    record(vix_ma = mavg, vix_current = current ) 

and get the error message:

UnsupportedPipelineOutput: Cannot add column 'vix_current' with term Latest([cboe_vix.vix_close], 1). Adding slices or single-column-output terms as pipeline columns is not currently supported.

It is working if I replace

vix_current = cboe_vix.vix_close.latest  

by

vix_current = SimpleMovingAverage(inputs = [cboe_vix.vix_close], window_length = 1)  

Please advise. .

@Vladimir

You got it! A simple concise approach to outputting a 'slice' or single column dataset is:

vix_current = SimpleMovingAverage(inputs = [cboe_vix.vix_close], window_length = 1)  

Why is this? Why the error 'UnsupportedPipelineOutput '? The basic issue stems from the definition of a pipe. A pipe by definition returns a dataframe. The rows are indexed by equities and the columns are whatever factors are defined in the pipe. The pipeline output associates values in columns to equities in the index. This works great for things like 'close price' or 'market cap' where each security has an associated unique value. However, the 'macro' datasets like VIX are just a series of data. The VIX on a particular day isn't associated with a particular equity. It's just a value. Outputting a single 'unassociated' value such as VIX doesn't fit the output definition of a pipeline.

So, if one want's to get the VIX value outputted from a pipe one needs to manually associate that value with an equity. To do this it's common to 'broadcast' that value across all equities. There would then be a column in the output dataframe with all the same values for all equities. Maybe not the most intuitive result but it fits into the pipeline construct.

How does one manually broadcast a value to equities? The explicit way is to create a custom factor. Maybe something like this.

    from quantopian.pipeline import CustomFactor  

    class VIX_Last(CustomFactor):  
        window_length = 1  
        inputs = [cboe_vix.vix_close]  
        def compute(self, today, assets, out, vix):  
            out[:] = vix[-1]

Notice the last line explicitly assigns the last vix value to all 'out' values.

However, the built in factors typically account for using 'slice' data as inputs. So, a little 'hack' ( instead of writing a custom factor) is to do exactly what @Vladimir did.

vix_current = SimpleMovingAverage(inputs = [cboe_vix.vix_close], window_length = 1)  

The built in functions such as 'SimpleMovingAverage' are smart enough to broadcast the result to all equities similar to creating a custom factor. So this one line of code could be used in place of a custom factor. A little less obvious but the same result. Personal preference which one to use.

Hope that all makes sense. See attached notebook showing the two versions of getting VIX and both return the same results.