How does one distill down the Pipeline to a single security?

Back to Community

posted Jun 5, 2017

I am a bit at a loss at the moment trying to figure out something that I am hoping is relatively easy for a more experienced Quantopian. I am trying to add a screen to my Pipeline to distill it down to a single security for troubleshooting purposes. I want to make sure that the behavior that I am seeing in a full backtest is correct but the logs are overwhelming to parse through so I'd like to get it down to one stock.

This is my Pipeline creation function:

def make_pipeline():

    # Base universe set to the Q1500US  
    base_universe = Q1500US()

    mean_close_50 = SimpleMovingAverage(inputs=[USEquityPricing.close], window_length=50)  
    # Factor of yesterday's close price.  
    yesterday_close = USEquityPricing.close.latest  
    pipe = Pipeline(  
        screen = base_universe,  
        columns = {  
            'close': yesterday_close,  
            '50_day_sma': mean_close_50,  
            'longs': yesterday_close > mean_close_50,  
            'shorts': yesterday_close < mean_close_50  
        }  
    )  
    return pipe

Any recommendations would be much appreciated.

Thanks in advance.

8 responses

Dan Whitnable

Jun 5, 2017

Take a look at the built in filter 'StaticAssets' https://www.quantopian.com/help#built-in-filters. It might be what you are looking for. Something like this


def make_pipeline():

    # Base universe set to a single stock. In this case AAPL.  
    base_universe  = StaticAssets(symbol('AAPL'))  

    mean_close_50 = SimpleMovingAverage(inputs=[USEquityPricing.close], window_length=50)  
    # Factor of yesterday's close price.  
    yesterday_close = USEquityPricing.close.latest  
    pipe = Pipeline(  
        screen = base_universe,  
        columns = {  
            'close': yesterday_close,  
            '50_day_sma': mean_close_50,  
            'longs': yesterday_close > mean_close_50,  
            'shorts': yesterday_close < mean_close_50  
        }  
    )  
    return pipe

Ali Rodell

Jun 6, 2017

Thanks for the response. That worked like a charm.

Just in case someone else is looking to do this you need to pass the argument into StaticAsset as a list, even if it is just a single entry.

For example:

base_universe  = StaticAssets([sid(24)])

Also you need to import it as:

from quantopian.pipeline.filters import StaticAssets

Thanks again!

Jerry Xu

Aug 19, 2017

thanks for the question and replies. I have one more question (maybe I can simply try it but it seems I always failed..:-), so just ask here. When I use USEquityPricing, can I use more than just "latest" (yesterday) price? can I get historic last five days or three days price? one of very simply but very effective filter is that, if a stock keep dropping N days with volume decrease, or, keep rising for N days with volume decrease, it always indicate a potential reverse in short term (swing trade) standard. Can anyone help to make this filter ?

Mustafa Tambawalla

Aug 19, 2017

Was looking for the same thing thanks dan

Jerry Xu

Aug 20, 2017

spend sometime look thru old posts here, figure it out how to get historic price and also how to set a range of security list (a rough example here in notebook). but the sid seems not working by looking up symbol.. how can I convert a symbol list into a sid list ?

from quantopian.pipeline import Pipeline, CustomFactor, CustomFilter
from quantopian.research import run_pipeline
from quantopian.pipeline.filters import StaticAssets
import math
import numpy as np

from quantopian.pipeline.factors import SimpleMovingAverage
from quantopian.pipeline.data import morningstar

from quantopian.pipeline.data.builtin import USEquityPricing

import datetime
import pytz
import pandas as pd

class SidInList(CustomFilter):
"""
Filter returns True for any SID included in parameter tuple passed at creation.
Usage: my_filter = SidInList(sid_list=(23911, 46631))
"""
inputs = []
window_length = 1
params = ('sid_list',)

def compute(self, today, assets, out, sid_list):  
    out[:] = np.in1d(assets, sid_list)

class CloseOnN(CustomFactor):
# Define inputs
inputs = [USEquityPricing.close]

# Set window_length to whatever number of days to lookback as a default  
# in the case where no window_length is given when instantiated.  
# This can also be set/over-ridden as shown below:  
# my_close_on_10 = CloseOnN(window_length = 10)  

#window_length = 2  
outputs = ['ret', 'latest', 'prv', 'threedayago']  

def compute(self, today, assets, out, close):  
    out.ret[:] = (close[0]-close[-1])/close[-1]  
    out.latest[:]=close[3]  
    out.prv[:]=close[2]  
    out.threedayago[:]=close[1]

class MarketCap(CustomFactor):

# Pre-declare inputs and window_length  
inputs = [USEquityPricing.close, morningstar.valuation.shares_outstanding]  
window_length = 1  

# Compute market cap value  
def compute(self, today, assets, out, close, shares):  
    out[:] = close[-1] * shares[-1]

def create_pipeline():
# Create a Pipeline computing the previous close factor.

ret,latest,prv,threedayago = CloseOnN(window_length = 4)  
mkt_cap = MarketCap()  
top_20 = mkt_cap.top(50)  


#nasdaq_100  
include_filter = SidInList(sid_list = (24,114,122,630, 67,20680, 328,14328, 368,16841, 9883,  337,   38650, 739,   27533,  3806,  18529, 1209,  40207, 1419,    15101, 17632, 39095, 1637,  1900,    32301, 18870, 14014, 25317, 36930,   12652, 26111, 24819, 24482, 2618,    2663,  27543, 1787 , 2696,  42950,   20208, 2853,  8816,  5530,  3212,    9736,  23906, 26578, 22316, 13862,   3951,  8655,  25339, 4246,  43405,   27357, 32046, 4485,  43919, 4668,    8677,  22802, 3450,  5061,  5121,    5149,  5166,  23709, 13905, 19926,   19725, 8857,  5767,  5787,  19917,   6295,  6413,  6546,  20281, 6683,    26169, 6872,  11901, 13940, 7061,    15581, 24518, 7272,  39840, 7671,    27872, 8017,  38817, 8045,  8132,    8158,  24124, 8344,  8352,  14848)) # 

p = Pipeline()  
p.set_screen(include_filter)  

p.add(ret, 'ret')  
p.add(latest, 'latest')  
p.add(prv, 'prv')  
p.add(threedayago, 'threedayago')  
p.add(mkt_cap, 'mkt_cap')  


#name_filter = (primary_symbol == 'SPY')  
#pipe.set_screen(market_cap_filter)  
#p.set_screen(name_filter)  
#base_universe  = StaticAssets([sid(24)])  

return p

results = run_pipeline(create_pipeline(), '8-19-2017', '8-19-2017')
results

Dan Whitnable

Aug 20, 2017

Probably don't use the custom filter 'SidInList' (and that's saying a lot because I wrote it). Use the built in 'StaticAssets' or 'StaticSids' filters instead. Use the former if you know the symbols. Use the latter if you know the SID. No need to really convert a symbol to a SID (or vice versa).

    from quantopian.pipeline.filters import StaticAssets, StaticSids

    universe_by_asset = StaticAssets(symbols('IBM', 'AAPL'))  
    universe_by_sid = StaticSids([24, 114])  
    universe =  universe_by_asset | universe_by_sid

Jerry Xu

Aug 21, 2017

Thanks, Dan! Just tried it, but it gave me error on this

---> 68 universe_by_asset = StaticAssets(symbols('IBM', 'AAPL'))
69 universe_by_sid = StaticSids([24, 114])
70 universe = universe_by_asset | universe_by_sid

Here is the entire code in the Notebook..

from quantopian.pipeline import Pipeline, CustomFactor, CustomFilter
from quantopian.research import run_pipeline
from quantopian.pipeline.filters import StaticAssets
import math
import numpy as np

from quantopian.pipeline.factors import SimpleMovingAverage
from quantopian.pipeline.data import morningstar

from quantopian.pipeline.data.builtin import USEquityPricing

import datetime
import pytz
import pandas as pd

def compute(self, today, assets, out, sid_list):  
    out[:] = np.in1d(assets, sid_list)

class CloseOnN(CustomFactor):
# Define inputs
inputs = [USEquityPricing.close]

# Set window_length to whatever number of days to lookback as a default  
# in the case where no window_length is given when instantiated.  
# This can also be set/over-ridden as shown below:  
# my_close_on_10 = CloseOnN(window_length = 10)  

#window_length = 2  
outputs = ['ret', 'latest', 'prv', 'threedayago']  

def compute(self, today, assets, out, close):  
    out.ret[:] = (close[0]-close[-1])/close[-1]  
    out.latest[:]=close[3]  
    out.prv[:]=close[2]  
    out.threedayago[:]=close[1]

class MarketCap(CustomFactor):

# Pre-declare inputs and window_length  
inputs = [USEquityPricing.close, morningstar.valuation.shares_outstanding]  
window_length = 1  

# Compute market cap value  
def compute(self, today, assets, out, close, shares):  
    out[:] = close[-1] * shares[-1]

def create_pipeline():
# Create a Pipeline computing the previous close factor.

ret,latest,prv,threedayago = CloseOnN(window_length = 4)  
mkt_cap = MarketCap()  
top_20 = mkt_cap.top(500)  

universe_by_asset = StaticAssets(symbols('IBM', 'AAPL'))  
universe_by_sid = StaticSids([24, 114])  
universe =  universe_by_asset | universe_by_sid  

p = Pipeline()  
#p.set_screen(include_filter)  

p.add(ret, 'ret')  
p.add(latest, 'latest')  
p.add(prv, 'prv')  
p.add(threedayago, 'threedayago')  
#p.add(mkt_cap, 'mkt_cap')  


#name_filter = (primary_symbol == 'SPY')  
#pipe.set_screen(market_cap_filter)  
#p.set_screen(name_filter)  
#base_universe  = StaticAssets([sid(24)])  

return p

results = run_pipeline(create_pipeline(), '8-19-2017', '8-19-2017')
results

Dan Whitnable

Aug 21, 2017

An irritating difference between the IDE (ie algorithm) environment and the research (ie Notebook) environment is the symbols method. The following code works in the IDE

symbols('IBM', 'AAPL')

but to use it in a notebook, one needs to add brackets

symbols(['IBM', 'AAPL'])

So, simply change line 68 to

universe_by_asset = StaticAssets(symbols(['IBM', 'AAPL']))

(note the added brackets) All should work fine.

You've successfully submitted a support ticket.

Our support team will be in touch soon.