I found a workaround by creating the following CustomFactor, but it still has the drawback of having to load the data for the entire window_length.
Does anyone have an example of how to use the "today" parameter in custom factors?

class Price_on_day(CustomFactor):  
    inputs = [USEquityPricing.close]  
    window_length = 1

    def compute(self, today, assets, out, inputs):  
        out[:] = inputs[0,:]


# Pipeline definition  
def  make_pipeline():  
    universe = QTradableStocksUS()  
    price_1d = Price_on_day(window_length = 1)  
    price_2d = Price_on_day(window_length = 2)  
    price_3d = Price_on_day(window_length = 3)  
    return Pipeline(  
        columns={  
            'price_1d': price_1d,  
            'price_2d': price_2d,  
            'price_3d': price_3d,  
        },  
        # Set screen as the intersection between our filter  
        # and trading universe  
        screen=(QTradableStocksUS())  
    )

The 'today' parameter which is passed in the 'compute' method is simply the timestamp of the day which the pipeline is being run. It matches the level 0 index (ie the dates) in the returned pipeline dataframe when run in a notebook. I can't imagine too many uses for it. Perhaps something like this to conditionally set the output based upon the weekday?

 def compute(self, today, assets, out, data):  
        # check if today is a Monday or a Friday  
        if today.weekday() in [0, 4]:  
            do Monday and Friday logic...


        else:  
            do midweek logic...

Note that the date is the date the pipeline is being run so the data is from the previous day.

One could also compare the actual date, something like 'today before '2008-1-1' but that starts messing with lookahead bias etc.

Maybe someone else has a better use.

BTW, your method to get previous days data is generally the accepted method.

Thank you Dan for enlightening me on the 'today' parameter! I'll use the other method to get previous days data then.

@Frederic,

Pipelines are typically executed in chunks, allowing for data to be cached. Your pipeline shouldn't take a big performance hit if you have a long lookback window and only take the oldest value in it because most of the values will be needed for other days in the same execution chunk. This obviously becomes less true for very long lookback windows, but for anything less than a couple of months, the speed hit should be negligible.