Well, suppose that Q did require it. How would they enforce it? And what would it mean to be "using pipeline" in a substantive way? Presently, I'm playing around with using the Q500US and Q1500US universes, in conjunction with a pipeline market cap screen. However, the actual security weights are computed outside of pipeline; I'm probably not using pipeline as envisioned, with ranking based on mixing and matching of various factors, and then going long-short using weights output from pipeline.

Also, if Q had a silly rule that one had to use pipeline, then it seems that folks would just put code in to satisfy the rule, but not actually use pipeline. Seems like a non-starter rule, to me.

It would be interesting to know if anyone has used pipeline for a long-short factor-based algo that would be fundable at $5M-$10M (backtested to 2002). I've yet to see a canonical viable example posted, but then maybe I've missed it. I suppose if Q had one, they'd keep it confidential. That said, an existence proof would be nice to see.

You are of course absolutely right, Grant.

Would you be willing to post some code that uses Q500Us in combination with other non-pipeline methods (with all the secret-saus parts left out, of course)? On my part at least hat that would be greatly appreciated, as I have been trying in vain to find a way to use these universes in combination with non-pipeline ranking and ordering, for example.

Here's the pipeline part and a few other things (I think I captured everything). I call update_stocks(context, data) quarterly. Regarding the other stuff, let me think about how to formulate a separate post. I'm sure others could improve upon my hacked together framework.

import numpy as np  
import pandas as pd  
from quantopian.algorithm import attach_pipeline, pipeline_output  
from quantopian.pipeline import Pipeline  
from quantopian.pipeline.data import morningstar as mstar  
from quantopian.pipeline.filters import Q500US

def initialize(context):  
    # set_commission(commission.PerTrade(cost=0))  
    # set_slippage(slippage.FixedSlippage(spread=0.00))

    # parameters  
    # --------------------------  
    context.n_stocks = 200 # number of stocks  
    context.N = 5 # trailing window size, days  
    context.eps = 1.1 # optimization model parameter  
    context.leverage = 1.05 # gross leverage  
    context.pct_etf = 0  # ETF percent, 0 to 0.5  
    context.min_a = 0  # min. allocation, percent of 1.0/len(context.stocks)  
    # --------------------------  
    schedule_function(housekeep, date_rules.week_start(days_offset=1), time_rules.market_open())  
    schedule_function(get_weights, date_rules.week_start(days_offset=1), time_rules.market_open(minutes=60))  
    # schedule_function(housekeep, date_rules.every_day(), time_rules.market_open())  
    # schedule_function(get_weights, date_rules.every_day(), time_rules.market_open(minutes=60))  
    context.first_day = True  
    context.n_periods = 0  
    context.bad_data = [sid(27357),sid(27358),sid(5328)]  
    # Attach our pipeline.  
    attach_pipeline(make_pipeline(context), 'my_pipe')  
def make_pipeline(context):  
    market_cap = mstar.valuation.market_cap.latest  
    top_market_cap = market_cap.top(context.n_stocks, mask=Q500US())  
    return Pipeline(columns={  
            'market_cap': market_cap  
        },screen = top_market_cap)  
def before_trading_start(context,data):  
    """  
    Called every day before market open.  
    """  
    output = pipeline_output('my_pipe')  
    context.stocks_current = output.sort('market_cap', ascending=False).index.tolist()  
    context.stocks_current = [stock for stock in context.stocks_current if stock not in context.bad_data]  
    if context.first_day:  
        context.stocks = context.stocks_current  
        context.first_day = False  
def update_stocks(context, data):  
    context.stocks = context.stocks_current  
def housekeep(context, data):  
    leverage = context.account.leverage  
    if leverage >= 3.0:  
        print "Leverage >= 3.0"  
    record(leverage = leverage)  
    for stock in context.stocks:  
        if stock in security_lists.leveraged_etf_list: # leveraged ETF?  
            context.stocks.remove(stock)  
    # check if data exists  
    for stock in context.stocks:  
        if not data.can_trade(stock):  
            context.stocks.remove(stock)  
    num_secs = 0  
    for stock in context.portfolio.positions.keys():  
        if context.portfolio.positions[stock].amount != 0:  
            num_secs += 1  
    record(num_secs = num_secs)  

I have been trying in vain to find a way to use these universes in combination with non-pipeline ranking and ordering, for example.

What is the difficulty, specifically?

Many thanks for the code, Grant.

To answer your question, I basically did not have any idea how to construct a framework like the one you shared. Very kind of you!

Your welcome. If you have suggestions for improvement or add-ons, please post them. --Grant

Yo Tim, I was struggling with this for the longest time. It seems hard but it is not. Here is the link I used to get my pipeline running.

https://www.quantopian.com/posts/need-help-with-pipeline-just-a-beginner?utm_campaign=need-help-with-pipeline-just-a-beginner&utm_medium=email&utm_source=forums

Thanks,

Your Friendly Neighborhood, Quantdog

Thank you, Quantdog, the link that you quote seems to provide a good template for the general framework used with Pipeline.