I suspect you are asking this question because you ultimately would like your algo to be sector neutral. That's a good goal as sector risk is a "common factor", i.e., it explains co-movement across stocks. Exposure to common factors lowers the effective breadth of a strategy and, even in modest amounts, can overwhelm any stock-specific alpha. A very common approach is to simply assume a beta of 1 to a stock's labeled sector, and 0 otherwise. The post here which introduced the optimization API does just that implicitly in the example of neutralizing sector risk. The {0,1} assumption will get you very far along to a realistic sector beta.

It is often helpful to be cognizant of the sector risk not just at the portfolio construction/optimization stage, but also at the alpha modeling stage. If you ultimately want the portfolio to be sector neutral, a good start is to de-mean your alpha (i.e., pipeline factor) by sector grouping.

I like breaking up my entire data frame into sectors in before_trading_start. Then I make sector by sector calculations. See link below:

https://www.quantopian.com/posts/sector-filter-template

Thanks both for the answers. I'm going to unpack this and play it back, as there's a lot of information in there.

Jonathan: First, I had always assumed sector neutrality was to avoid over-exposure to sector specific events, such as a cartel changing oil prices unexpectedly, causing companies in the energy sector to jump. But here you are relating it to breadth of the strategy, as in your blog post. Can you explain the "overwhelm any stock-specific alpha" part in more detail?

Second point is the assumption of beta of 1. This explains why it's so easy to make things dollar neutral using the Optimizer API, but there's no native support for calculating the sector beta of each stock. I'm guessing the performance gain from calculating another parameter (sector beta), and choosing a meta-parameter (look-back window length for the beta), in order to remove the exact beta from stock, is minimal. Assuming a beta of 1 is more or less the same as removing the cross sectional mean return within each sector (using Factor.demean(groupby=Sector()). This will over-adjust some stocks, and under-adjust others, but I guess the errors are noise, which is likely to cancel out?

Third point is to take the sector risk out at the alpha modelling stage. This hadn't occured to me before, and is a great insight. I don't believe Pipeline natively supports removing the cross-sectional mean return from a time series of prices and/or returns, but I'm sure this is straightforward to implement in a CustomFactor with numpy/pandas. I will post some code here.

Here's some code to demonstrate how to remove the mean sector return from a time series.

class NDayExcessHighLow(CustomFactor):  
    inputs = [USEquityPricing.close,  
              morningstar.asset_classification.morningstar_sector_code]  
    outputs = ['high','low']  
    #window_length = 21 # No default specified, do it in constructor  
    def compute(self, today, assets, out, close, sectors):  
        returns = pd.DataFrame(np.log(close)).T.diff(1,axis=1)  
        # rows are assets  
        # cols are time  
        # row index and column names are just integers  
        sector = lambda x: sectors[-1,x] # Most recent sector code only  
        demean = lambda x: x - x.mean(axis=0)  
        returns = returns.groupby(sector).transform(demean)  
        returns = returns.fillna(0.0)  
        prices  = np.exp(returns.cumsum(axis=0)).T.as_matrix()  
        # rows are time  
        # cols are assets  
        out.high[:] = prices[-1] == np.max(prices, axis=0)  
        out.low[:]  = prices[-1] == np.min(prices, axis=0)  

Hi Burrito Dan,

Would the .demean(groupby=Sector()) method achieve the same goal as the CustomFactor you posted? (https://www.quantopian.com/help#normalizing-results)

So:

from quantopian.pipeline.classifiers.morningstar import Sector  
factor.demean.groupby(Sector())  

Hi Seong. That would help with certain CustomFactors, but not my example. In my example, I subtract the cross sectional mean return at every point in the trailing window, then calculate some statistic on the trailing window. Of course, this may not be a sensible thing to do, hence not being natively supported by Pipeline!

@Dan, following Seong's suggestion, you could have your trailing window of sector demeaned returns like this:

ret = Returns(window_length=2)  
demeanret = ret.demean(groupby=Sector())  

my_factor = MyCustomFactor(inputs=[demeanret], window_length=90)  

@Luca, http://memeshappen.com/media/created/You-sir-are-a-total-legend-meme-27289.jpg