I sorted it finally.

Michael,

I'm very glad that you managed to figure this out! Would you be willing to post your solution? There may be community members in the future who come across a similar roadblock and could really learn from you.

class ExponentialRegression(CustomFactor):  
    [setup stuff]

    def compute(self, today, assets, out, close):  
        scores = np.empty(len(close.T)+1, dtype=np.float64)  
        for col in close.T:  
            [do something on col and add it to scores]  
        out[:] = scores 

Thats it. There is also a method in numpy apply_along_axis but after doing some tests I found it to be marginally slower for some reason.

Hope that helps others.

Thanks to this post, I had a hard time figuring out the same issue. I hope the compute api would be only passed in with each symbol's data at a time instead of the array including all symbols.

Hi Michael,

How did you put data into scores?

Thanks Michael, that helped me a lot.

@Yingzhong: I succeeded in adding data into scores by doing something like this, I hope that makes sense (note that I had to remove the +1 for the length of empty np array declaration). There's probably a better way of getting the index of the current column?

class CF1(CustomFactor):  
    inputs = [USEquityPricing.close]  
    window_length = 250  
    def compute(self, today, assets, out, close):  
        scores = np.empty(len(close.T), dtype=np.float64)  
        i = 0  
        for col in close.T:  
            scores[i] = Somecalc(col)  
            i += 1  
        out[:] = scores  

@yingzhong - exactly as Charles had illustrated above. Just use the i variable as an index. It's the fastest way. You can correlate the row to the assets variable later but in reality this is probably all you need to do in the function to pass back the calculated data.

Hi Michael, would you mind sharing the sorted exponential regression custom factor also?

@Kenneth P. its just the block of code at the top wrapped in the new loop style. There was no changes to that part. instead of scores.append I used the i pointer to do scores[i] = score although you might be able to get away with just adding them all in without the check.

Try it out in a custom factor and a simple algo that runs for 1 or 2 days and just print out the head of the close dataframe or the pipeline output to see how it works. Or use the debugger.

I ended up prototyping the factor in an ipython notebook first just using a simple numpy series object with simple values I knew were going to give me correct answers (think xUnit style testing. My background is financial systems development and you need to test things before you write them) to validate the factor logic worked before I spent the time to wire it into the algo on quantopian. (trust but validate...) :-)

@Michael, thanks. Unfortunately my implementation is still breaking at the definition of asset_returns with the same error as above, and I can't seem to figure out why.

    def compute(self, today, assets, out, close):  
        scores = np.empty(len(close.T)+1, dtype=np.float64)  
        x_index = pd.Series(range(self.window_length))+1  
        close_log_returns = np.diff(np.log(close))  
        i = 0  
        for col in close.T:  
            asset_returns = close_log_returns[:, assets] #breaking  
            slope, intercept, r_value, p_value, std_err = stats.linregress(x_index, asset_returns)  
            score = slope * np.sqrt(252) * r_value**2  
            scores[i] = score  
            i += 1  
        out[:] = scores  

IndexError: index 8414 is out of bounds for axis 1 with size 8408  

your looping on the transposed close.T but accessing the close_log_returns.

How is that going to work? They are the wrong shape and different objects. Better to loop over the object you are interested in...

        close_log_returns = np.diff(np.log(close))  
        i = 0  
        for col in close_log_returns.T:  
            asset_returns = do_something_funky_to_my(col) # assets is a wild goose chase, forget about it....  
            slope, intercept, r_value, p_value, std_err = stats.linregress(x_index, col) # is this not what you want?  
            ... then the rest ...  

You probably don't even need the asset_returns variable unless you are transforming the row again after taking the log normal returns, just operate on the col you are getting back from the loop on the transposed panel.

@Kenneth is that a solution for you?

We should put in a stack overflow voting system in here so people can say wether or not a reply is the correct answer or not. Would be nice to know people value the time and effort you put into giving free online tech support....