Hi Tristan,
A custom factor only outputs one number, not the entire array of numbers. I think in this case, you would want to include more of your logic in the custom factor. For example, you could get the lows over a 10 day window, and then output the number of consecutive lows each stock had as the out. Then your filter could be for all securities where the lows was > 3, to get every company that had 3 or more consecutive lows in the last 10 days.

Let me know if that makes sense.

Ok, that make sense! I'll try to do it that way, and share my results (good or bad).

I'm running into a few issues:

1) Can I use the "window_length" value in my compute() function? I am getting an undefined error.

2) ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all():
USER ALGORITHM:41, in compute
if low[i-1] < low [i]:

class ConsecutiveHigherLows(CustomFactor):  
    window_length = 10  
    inputs = [USEquityPricing.low]

    def compute(self, today, assets, out, low):

        # Smart way to check lows  
        for i in range(-1,-window_length,-1):  
            pattern = 0  
            if low[i-1] < low [i]:  
                pattern = abs(i)  
            else:  
                break  
# Dumb way of coding this:  
#        pattern = 0  
#        if low[-2] < low[-1]:  
#            pattern=1  
#            if low[-3] < low[-2]:  
#                pattern=2  
#                if low[-4] < low[-3]:  
#                    pattern=3  
#                    if low[-5] < low[-4]:  
#                        pattern=4  
#                        if low[-6] < low[-5]:  
#                            pattern=5  
#                            if low[-7] < low[-6]:  
#                                pattern=6  
#                                if low[-8] < low[-7]:  
#                                    pattern=7  
#                                    if low[-9] < low[-8]:  
#                                        pattern=8  
#                                        if low[-10] < low[-9]:  
#                                            pattern=9

        # Return the number of times the pattern occurred  
        out[:] = pattern  

Thanks to Fawce for giving me the answer to #1 in the "quantopian-users" Slack chat.

You need to reference that variable like this: "self.window_length"

    # Smart way to check lows  
        for i in range(-1,-window_length,-1):  
            pattern = 0  
            if low[i-1] < low [i]:  
                pattern = abs(i)  
            else:  
                break  

Tristan,
Not sure if your loop does what you want, as all the if-statements in the loop are not nested,
while the big if statement probably does.

From a compact python expression point of view, here is another way:

-Use numpy arrays for vector operations
-Use np.diff to difference consecutive entries (Size of result array will be one less)
-Take a slice of the last window_length entries
-Create a True/False array based on the condition that the diff entry is > 0
-Use the all() method to see if all entries are True (ie. there is a run of positive differences)

#Example  
import numpy as np  
wl = 4                                                             #window length  
X = np.arange(10) *  arange(10)         # Squares of first ten integers(Test sequence)  
isRun = (np.diff(X)[-(wl-1):]>0).all()  #True if a wl run of diffs is >0 

Alan,

Thanks for the reply! I barely know my way around Python, so I appreciate the help. This is the second time I have seen diff() used to determine when a value changes relative to it's neighbor.

However, I believe your code only checks the condition when ALL days within the window_length are consecutively getting bigger. This is good, but I would like to rank stocks based on the number of consecutive days. I think I would have to encapsulate your logic inside a for loop and increment "wl".

for wl in range(self.window_length):                                                             #window length  
    isRun = (np.diff(lows)[-(wl-1):]>0).all()  #True if a wl run of diffs is >0  
    if isRun == True:  
         pattern == wl  
    else:  
         break  

Am I understanding this correctly?

Tristan,
Pretty much.
If you want the maximal length monotonic run at the end of the lows array,
then you want the first place, scanning backward, a False occurs in the np.diff(lows)>0 boolean array,

This is probably overkill, but it's hard to get all this niggly stuff right without actually running it in a debugger, IMHO...so here you go...
alan

#File:      max_lastrun_idx.py  
#Normal:    python max_lastrun_idx.py  
#Debugging: Run under ipython terminal with:  
#           ipython -i -c="%run max_lastrun_idx.py"

import numpy as np

#Debug On: uncomment tracer statements  
#from IPython.core.debugger import Tracer  
#tracer = Tracer()

def max_lstrun_idx(lows, window_length):  
    pat = 0  
    # Warning: Take care of window_length<=3 cases here.  
    for wl in range(2, window_length):                       #window length  
        isRun = np.array((np.diff(lows)[-(wl-1):]>0).all())  #True if a wl run of diffs is >0  
        if isRun:  
            pat = wl-1  
        else:  
            break  
    return pat

#Test  
#tracer()  
X = np.array([  0,   1,   4,   2,  16,  25, -10,  49,  64,  81])  
idx = max_lstrun_idx(X, len(X))  
print "Test array X=", X  
print "max_lstrun_idx(X, len(X))=", idx

# Alternative using np arrays  
wl=len(X)  
diff_gt0          = (np.diff(X)[-(wl-1):]>0)     # array of diff>0 bools  
false_idxs_diff   = np.nonzero(~diff_gt0)        # array of indices of <=0 bools  
max_run_start_idx = np.max(false_idxs_diff) + 2  # max of false indices (+2 to adjust for diff and python index conventions)

#Ans  
print "Longest Run Length=", len(X)-max_run_start_idx, "MaxRun=", X[max_run_start_idx:]


#Run result  
#>> python max_lastrun_idx.py  
#>> Test array X= [  0   1   4   2  16  25 -10  49  64  81]  
#>> max_lstrun_idx(X, len(X))= 3  
#>> Longest Run Length= 3 MaxRun= [49 64 81]

Alan,

Great stuff, I am going to try implementing your code next. In the meantime, I was trying to get around this by iterating through the assets, like this:

class ConsecutiveHigherLows(CustomFactor):  
    window_length = 10  
    inputs = [USEquityPricing.low]

    def compute(self, today, assets, out, low):  
        for a in range(len(assets)):  
            for i in range(-1,-self.window_length,-1):  
                pattern = 0  
                if low[i-1,a] < low[i,a]:  
                    pattern = abs(i)  
                else:  
                    break  
            out[a] = pattern  

However, I am getting this error when "update_universe" is run:

TypeError:  
There was a runtime error on line 73

def before_trading_start(context, data):  
    results = pipeline_output('MyPipeline').dropna()  
    update_universe(results)  

I did some debugging to see what object is created, and it is a Pandas frame with what looks like the right data. Here are a few rows:

pipeline_output('MyPipeline').dropna()  
DataFrame:  
                        num_of_lows  recent_returns  recent_returns_rank  
Equity(764 [OMX])                 9        0.039216                 5684  
Equity(4128 [JFC])                9        0.166261                 6904  
Equity(4580 [LUK])                9        0.046154                 5934  

Tristan,
Good catch on setting out one asset at a time...I didn't know how to do that,
as all the examples I had seen had been arithmetic operations that worked on whole columns of assets.
Seems like the comparison operators SHOULD work on columns.
Some points:
1. pattern needs to be moved outside the inside loop...otherwise, it keeps getting reset to 0.

class ConsecutiveHigherLows(CustomFactor):  
    window_length = 10  
    inputs = [USEquityPricing.low]

    def compute(self, today, assets, out, low):  
        for a in range(len(assets)):  
            pattern = 0  
            for i in range(-1,-self.window_length,-1):  
                if low[i-1,a] < low[i,a]:  
                    pattern = abs(i)  
                else:  
                    break  
            out[a] = pattern  

1. From the community post: "Ranked Universe & Long-Short Equity Strategy", you can try out your new factor....
   produce the column of low runs, then produce a rank column from that column, then sort it and take the top number.
   They are the assets ones that'll have your top runs.
   Here is the code I added to make it work.

def initialize(context):  
...
    consec_lows = ConsecutiveHigherLows()  
    pipe.add(consec_lows, 'consec_lows')

    # Rank factor 3 and add the rank to our pipeline  
    consec_lows_rank = consec_lows.rank(mask=top_2000, ascending=False)  
    pipe.add(consec_lows_rank, 'consec_lows_rank')  


def before_trading_start(context, data):  
    # Call pipelive_output to get the output  
    context.output = pipeline_output('ranked_2000')  
    # Narrow down the securities to only the top 100 & update my universe  
    context.long_list = context.output.sort(['consec_lows_rank'], ascending=True).iloc[:100]  
    update_universe(context.long_list.index)  
...

Wow, it feels great to finally figure something out! I couldn't have done it without the help of others. (Lots of thanks to Alan!)

I have attached a working backtest that uses Pipeline and a few CustomFactor classes. I have modified these classes so that they can be used for different input types (lows, highs, close, volume, etc).

class PercentChange(CustomFactor):  
    # Tristan Rhodes  
    # https://www.quantopian.com/posts/my-first-pipeline-how-to-find-stocks-with-consecutive-higher-lows-low-4-low-3-low-2-low-1  
    #  
    # PercentChange will calculate the percent change of an input over the n-most recent days, where n = window_length.  
    # This can by used for price inputs (low, high, close, open), volume, or even fundamentals (set window length for desired period)

    # Set the default list of inputs as well as the default window_length.  
    # Default values are used if the optional parameters are not specified.  
    inputs = [USEquityPricing.close]  
    window_length = 10

    def compute(self, today, assets, out, input1):  
        out[:] = (input1[-1] - input1[0]) / input1[0]

class ConsecutiveHigherValues(CustomFactor):  
    # Tristan Rhodes  
    # https://www.quantopian.com/posts/my-first-pipeline-how-to-find-stocks-with-consecutive-higher-lows-low-4-low-3-low-2-low-1  
    #  
    # ConsecutiveHigherValues will return the number of periods that the value has consecutively increased, leading up to the current period.  
    # This can by used for price inputs (low, high, close, open) or volume.  (Fundamentals don't usually change on a daily basis, right?)  
    #  
    # Set the default list of inputs as well as the default window_length.  
    # Default values are used if the optional parameters are not specified.  
    window_length = 10  
    inputs = [USEquityPricing.low]

    def compute(self, today, assets, out, input1):  
        for a in range(len(assets)):  
            consecutive = 0  
            for i in range(-1,-self.window_length,-1):  
                if input1[i-1,a] < input1[i,a]:  
                    consecutive = abs(i)  
                else:  
                    break  
            out[a] = consecutive  

Tristan,
Looks good...
I keep learning more about the pipeline stuff a bit at a time!
alan

@ Tristan.. how many how many consecutive lows... is this..? I want 3 to 5 days consecutive lows.. is that what is written in your.. algo..? thanks....

Chan John,

This is the line that says "more than 5 consecutive higher lows".

 #Screen out penny stocks and low liquidity securities.  
    pipe.set_screen((consec_highers > 5) & (sma_200 > 5.00) & (dollar_volume > 10**7))  

To accomplish what you want (3,4, or 5 days), you can modify that line:

 #Screen out penny stocks and low liquidity securities.  
    pipe.set_screen((consec_highers >= 3) & (consec_highers <= 5) & (sma_200 > 5.00) & (dollar_volume > 10**7))  

thanks... tristaN. appreciate it... ;)

Hi all,

I was trying to do the exact same thing and stumbled upon this thread. I settled on a pandas solution, which is less general than Tristan's final implementation. Given a window length it'll return a bool to indicate if there were consecutive lower-lows over the window period.

class ConsecutiveLowerLows(CustomFactor):  
    inputs = [USEquityPricing.low]  
    def compute(self, today, assets, out, lows):  
        low_change = pd.DataFrame(lows, columns=assets).diff()  
        negative_change = low_change.loc[1:,:] < 0  
        out[:] = negative_change.min(axis=0)