Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Custom factors and TA.Lib

Hey guys, hope you are all having a great holiday season with your family and friends!
To start, let's say that I'm pretty much a noob in terms of python and the whole Quantopian IDE, but thanks to the lectures and tutorials provided I came up with something like this:

(You can focus directly on the Kama_ratio class, where the problem probably is)

from quantopian.algorithm import attach_pipeline, pipeline_output  
from quantopian.pipeline import Pipeline  
from quantopian.pipeline.data.builtin import USEquityPricing  
from quantopian.pipeline.factors import CustomFactor, AverageDollarVolume  
from quantopian.pipeline.data import morningstar  
from quantopian.pipeline.filters.morningstar import Q500US  
import pandas as pd  
import numpy as np  
import talib  
import math

class Kama_ratio(CustomFactor):  
    # Pre-declare inputs and window_length  
    inputs = [USEquityPricing.close]  
    window_length = 20  
    # Compute factor  
    def compute(self, today, assets, out, close):  
        print(close)  
        kama = talib.KAMA(close, self.window_length)  
        out[:] = kama / close 

class Growth(CustomFactor):  
    # Pre-declare inputs and window_length  
    inputs = [morningstar.earnings_ratios.diluted_eps_growth,  
              morningstar.asset_classification.growth_grade,  
              morningstar.asset_classification.growth_score]  
    window_length = 1  
    # Compute factor  
    def compute(self, today, assets, out, eps_growth, growth_grade, growth_score):  
        value_table = pd.DataFrame(index=assets)  
        value_table["eps_growth"] = eps_growth[-1]  
        value_table["growth_grade"] = growth_grade[-1]  
        value_table["growth_score"] = growth_score[-1]  
        out[:] = value_table.rank().mean(axis=1)  

class Momentum(CustomFactor):  
    inputs = [USEquityPricing.close]  
    window_length = 252  
    def compute(self, today, assets, out, close):  
        out[:] = close[-30] / close[0]  

# Final rank + longs and shorts  
def before_trading_start(context, data):  
    results = pipeline_output('factors').dropna()  
    ranks = results.rank().mean(axis=1).order()  
    context.shorts = 1 / ranks.head(200)  
    context.shorts /= context.shorts.sum()  
    context.longs = ranks.tail(200)  
    context.longs /= context.longs.sum()  
    context.security_list = context.longs.index.union(context.shorts.index).tolist()  
def initialize(context):  
    # Introduce Pipeline and name  
    pipe = Pipeline()  
    pipe = attach_pipeline(pipe, name='factors')  
    growth = Growth()  
    momentum = Momentum()  
    kama_ratio = Kama_ratio()

    # Add previous factor into Pipeline  
    pipe.add(growth, "growth")  
    pipe.add(momentum, "momentum")  
    pipe.add(kama_ratio, "kama_ratio")  


    # Screen Quantopian universe Q500US.  
    pipe.set_screen(Q500US())  
    context.spy = sid(8554)  
    context.shorts = None  
    context.longs = None  
    schedule_function(func=rebalance,  
                      date_rule=date_rules.week_start(),  
                      time_rule=time_rules.market_open(minutes=5))  
    schedule_function(func=cancel_open_orders,  
                      date_rule=date_rules.every_day(),  
                      time_rule=time_rules.market_close())  
    schedule_function(func=record_vars,  
                      date_rule=date_rules.every_day(),  
                      time_rule=time_rules.market_close())

# Recording what we want, ran by through schedule_function.  
def record_vars(context, data):  
    record(lever=context.account.leverage,  
           exposure=context.account.net_leverage,  
           num_pos=len(context.portfolio.positions),  
           oo=len(get_open_orders()))

# Cancels orders pending at close of the market  
def cancel_open_orders(context, data):  
    open_orders = get_open_orders()  
    for security in open_orders:  
        for order in open_orders[security]:  
            cancel_order(order)  
# Implementation  
def rebalance(context, data):  
    for security in context.shorts.index:  
        if get_open_orders(security):  
            continue  
        if data.can_trade(security):  
            order_target_percent(security, -context.shorts[security])  
    for security in context.longs.index:  
        if get_open_orders(security):  
            continue  
        if data.can_trade(security):  
            order_target_percent(security, context.longs[security])  
    for security in context.portfolio.positions:  
        if get_open_orders(security):  
            continue  
        if data.can_trade(security) and security not in context.security_list:  
                order_target_percent(security, 0)

Okay, so when I run this, I get:

AssertionError: real has wrong dimensions
There was a runtime error on line 53.

I suspect the problem is with the Kama_ratio class, because the algo runs perfectly well without it. I don't really care about KAMA, but I would like to understand what's happening for further algorithms. I know the code might also be sloppy, any pointers I will take.

Thanks everyone, your help is highly appreciated
Thanks James Christopher for the sample algo (https://www.quantopian.com/posts/long-short-pipeline-multi-factor)
Nicolas

2 responses

First off welcome!

You are correct, the problem is in your 'Kama_ratio' custom factor.

The parameter(s) passed to the compute function (in your case 'close') is a 2 dimensioned numpy array with rows and columns. The number of rows are the number of trading days you are requesting data for (ie the window_length). The number of columns are the number of securities being passed. This is normally every security in the Quantopian database but can be a subset of those if a mask is used when instantiating the custom factor (eg mask=my_mask).

The output of the compute function is a 1 dimensioned numpy array with length equal to the number of securities. This length is equal to the number of columns in the input data.

The problem(s) in your custom factor are 1) the talib.KAMA function expects a 1 dimensioned array as input and not a 2 dimensioned array and 2) it returns an array of averages but the factor ultimately expects only a single value for each security.

The solution is to iterate over each column of the inputs (ie each security), do your ta-lib function for that one security and set a specific value in the output array (eg output[column] = result). This isn't the most 'pythonic' way but it works.

A good place to start is to look at this excellent post by Scott Sanderson https://www.quantopian.com/posts/using-ta-lib-functions-in-pipeline.

Good luck.

Hey Dan,

Thank you for the accurate and quick reply. After printing some parts of my algo, I think I begin to understand the problem here. I will most definitely use the link you provided, which seemed pretty insightful.

Thanks again
Nicolas