Hi Joakim,

One reason why this could be happening is that MaximizeAlpha is receiving an empty Series as input (which defaults to object dtype). An easy way to verify this is to log the size of your input to MaximizeAlpha right before calling it.

You can guard against this error by checking if your pipeline output is empty (using Series.empty) before invoking MaximizeAlpha:

def rebalance(context, data):  
    # Retrieve alpha from pipeline output  
    alpha = context.pipeline_data.sentiment_score

    if not alpha.empty:  
        # Create MaximizeAlpha objective  
        objective = opt.MaximizeAlpha(alpha)  
    ...  

I got this code sample from the Getting Started tutorial.

Hi Ernesto,

Thanks for your help, I really appreciate it. Unfortunately your suggestion didn't fix my problem in this case. I'll see if I can reproduce it in a 'dummy' sample algo that I'll share either here or with Q Support, to see if that will help with troubleshooting.

@Joakim I'd be happy to help if you could share an algo (dummy or otherwise).

Hi Dan,

Great, thank you! Here's one based on the not-so-dummy L/S equity template algo. I've basically just tried to replace the Morningstar fields with the Factset fields for the 'value' factor (may not be entirely accurate, hence why it's 'dummy' :)).

In this case, I'm suspecting that it's due to 'dividing by 0', which may then default to an 'Object'? The reason why I think so is because my 'hack' of adding 0.0000001 to Enterprise value works in this case. If you (or anyone else) know of a good way to avoid risking dividing by 0 (which is oftentimes quite a valid value; 0 Long Term Debt for example is great in my view), I'd be all ears. Maybe winsorizing the final factor would help, but I don't want to winsorize too much.

Also, how do you troubleshoot this stuff? Do you print these values out to the logs? Hoping I'll be able to learn to troubleshooting errors myself so I don't have to ask for help in the forums each time.

@Joakim You were correct in tracing the problem back to dividing by zero. One way to avoid dividing by zero is to use a mask. In this case maybe something like this:

    entrpr_val_qf = fsf.entrpr_val_qf.latest  
    non_zero_entrpr_val_qf = entrpr_val_qf != 0

    entrpr_val_qf_win = entrpr_val_qf.winsorize(min_percentile=0.01, max_percentile=0.99, mask = non_zero_entrpr_val_qf)

That excludes any 0 values from the calculation and the algo runs without an error. I don't generally do this masking, but it seems good programming practice to always include something like this whenever dividing factors. Good catch!

Thanks Dan, super helpful as always!

Do you know if there's any way to use a mask based on dtype != 'object' as well? The reason I'm asking is because I sometimes get this error even for factors where I'm not at risk of dividing by zero. Maybe in the Pipeline 'screen' as more of a 'catchall' filter?

The error

TypeError:MaximizeAlpha() expected a value with dtype 'float64' or 'int64' for argument 'alphas', but got 'object' instead.

is typically because the series passed to 'MaximizeAlpha' is empty. Unless one is doing something fancy like changing what is being passed based on certain conditions, it will always be the same factor. This factor will always have the same dtype values UNLESS it doesn't have any values (ie it's empty). That empty series being presented to 'MaximizeAlpha' is what's causing the error.

One approach is to figure out why it would be empty (in this case a divide by zero) and fix that. A brute force approach (which I often use) however is to simply use a 'try:except' statement. This doesn't fix things but at least keeps the code going. Something like this

    try:  
        objective = opt.MaximizeAlpha(pipeline_data.combined_factor)  
    except:  
        log.info('MaximizeAlpha exception')  
        log.info(pipeline_data.combined_factor)  
        # can't really do the rest so just return  
        return

Take a look at the logs in the attached backtest. Notice the logged value of 'pipeline_data.combined_factor' when there is an exception. It shows:

INFO Series([], Name: combined_factor, dtype: object)

There's the culprit. The series passed to the 'MaximizeAlpha' method is empty.

Again, that doesn't fix it. It's best to now figure out why it would be empty when one is probably always expecting values.

Hope that helps Joakim.

So awesome, thank you Dan!!

One way I've been thinking about how to avoid dividing by zero (as it can oftentimes be a valid value, so I don't want to filter it out too much unless I really have to) is to 'move' a normalized zScore from a range between about -3 and 3 (roughly, winsorized by 1% on each side), to a range of say 1 to 3. This way there wouldn't be any risk of dividing by zero. I could also more easily multiply two factors, which doesn't really work if I just zScore them (e.g two negative zScores multiplied with each other equals a positive number, which is clearly not what I'd want).

Would you know how to do this, or if there's a better way of multiplying/dividing normalized factors, and/or if my above thinking is flawed in some way?

The mask is necessary on these two lines.
Resolves the root problem of longs & shorts both empty and therefore no pipeline output and no alpha for MaximizeAlpha to operate on.
This backtest is narrowed down more for the date where the problem was occurring and shows that it continues through it without error by adding that mask.

    ebit_oper_ltm_win = ebit_oper_ltm.winsorize(min_percentile=0.01, max_percentile=0.99, mask=universe)  
    entrpr_val_qf_win = entrpr_val_qf.winsorize(min_percentile=0.01, max_percentile=0.99, mask=universe)  

Perhaps try this, Joakim and see if it stabilises:

try:  
    context.pipeline_data = algo.pipeline_output('long_short_equity_template').replace([np.inf, -np.inf], np.nan)  
    context.pipeline_data.dropna(inplace=True)  
except Exception as message:  
    log.warn("pipeline_data error: {}".format(message))  
    return  

Firstly, replacing any inf values (undesirable values rather than errors from Pipeline) with NaN to be dropped from the set.
Secondly, error handling using try/except so the algorithm continues to run after skipping the session, and reporting the error for diagnosis.

ps: you may also add np.nan in the list to be replaced, and any values for replacement, for example:

df.replace([np.inf, -np.inf, np.nan], 0.999)  

Hope it helps.

pps: Thinking also that perhaps you may want to check this:

screen = universe  & alpha_factor.isfinite()  & ~(alpha_factor.notnan() | alpha_factor.notnull())  

Thanks guys, really appreciate your help!

@Karl,

Why is there essentially a double negative in your screen?

~(alpha_factor.notnan() | alpha_factor.notnull())

~ ('tilde' or 'not' right?) and then .notnan() or .notnull(), so essentially 'not' notnan or 'not' notnull?

I'm 99.9% sure you're correct, I'm just trying to understand the 'logic'. :)

Hi Joakim,

I take the assumption that alpha_factor.notnan() may not happen at the same time as alpha_factor.notnull() so by negating ( alpha_factor.notnan() or alpha_factor.notnull() ) would preclude either/both slipping through your alpha_factor.notnan() & alpha_factor.notnull() Boolean logic.

Hope it makes sense though I may have misinterpreted your intent :)

Try this and see if it helps you see what's going on

    #long_short_screen = (longs | shorts)  
    long_short_screen = ~fsf_value.percentile_between(10, 90, mask=universe)  # just some lows & highs, ~ means not  

With that change, there is output on 8/9.

Then toggle use_the_mask_that_works on line 87 off and on. Partial diff below...

In this first section, notice there are no longs or shorts. That means with the original long_short_screen, the dataframe is empty of course.

use_the_mask_that_works OFF

2011-08-09 07:00 log_data:327 INFO Rows: 6587  Columns: 8  
                               min              mean                max  
combined_factor                nan               nan                nan     NaNs 6587/6587  
        ebt_ltm      -2800180000.0     443824822.892      35156900000.0     NaNs 3886/6587  
        ebt_win       -121594000.0     339883491.163      11939300000.0     NaNs 3886/6587  
            fsf     -252.072043633               inf                inf     NaNs 3962/6587  
          longs              False               0.0              False  
             qf      -5274280000.0     7049287416.09     822583000000.0     NaNs 3212/6587  
         qf_win         -2280840.0     4940088680.93     167153000000.0     NaNs 3212/6587  
         shorts              False               0.0              False  
2011-08-09 07:00 log_data:342 INFO _ _ _   combined_factor   _ _ _  
    ... combined_factor highs  
                      combined_factor     ebt_ltm     ebt_win       fsf  \  
Equity(21 [AAME])                 NaN         NaN         NaN       NaN  
Equity(25 [ARNC_PR])              NaN         NaN         NaN       NaN  
Equity(31 [ABAX])                 NaN  19721000.0  19721000.0  0.034702  
Equity(37 [ABCW])                 NaN         NaN         NaN       NaN  

Notice the difference in fsf here ...

use_the_mask_that_works ON

2011-08-09 07:00 log_data:327 INFO Rows: 6587  Columns: 8  
                                min                mean                max  
combined_factor      -5.80677784964      0.119662267913      20.4170142151     NaNs 6249/6587  
        ebt_ltm       -2800180000.0       448632896.568      35156900000.0     NaNs 3886/6587  
        ebt_win        -145421000.0       550752342.018      12728000000.0     NaNs 6245/6587  
            fsf     -0.396482834994     0.0768264835087      1.69785455888     NaNs 6249/6587  
          longs               False     0.0256565963261               True  
             qf       -5274280000.0       7083996590.61     822583000000.0     NaNs 3212/6587  
         qf_win         338625000.0       11157241586.6     196765000000.0     NaNs 6079/6587  
         shorts               False     0.0256565963261               True  
2011-08-09 07:00 log_data:342 INFO _ _ _   combined_factor   _ _ _  
    ... combined_factor highs  
                      combined_factor      ebt_ltm      ebt_win       fsf  \  
Equity(33807 [IBKR])        20.417014  574936000.0  574936000.0  1.697855  
Equity(36243 [AGNC])        18.013224  509928000.0  509928000.0  1.505878  
Equity(6330 [RAD])           7.835089  234671000.0  234671000.0  0.693011  
Equity(32627 [GTU])          4.957366  156846000.0  156846000.0  0.463185  

Part of it is that without the mask, winsorize is operating on all 8000+ stocks.
Also, with this, down to 540 stocks instead of the 6587 above. I thought this would not be necessary.
screen = universe & long_short_screen
Not an end result answer, just hopefully some info and a test that can help.

Or a different route with 1678 stocks

def make_pipeline():  
    m = QTradableStocksUS()

    ebit_oper_ltm = fsf.ebit_oper_ltm.latest  
    entrpr_val_qf = fsf.entrpr_val_qf.latest #+ 0.00000001

    m &= ebit_oper_ltm.notnull()  
    m &= entrpr_val_qf.notnull()

    ebit_oper_ltm_win = ebit_oper_ltm.winsorize(min_percentile=0.01, max_percentile=0.99, mask=m)  
    entrpr_val_qf_win = entrpr_val_qf.winsorize(min_percentile=0.01, max_percentile=0.99, mask=m)

    fsf_value = ebit_oper_ltm_win / entrpr_val_qf_win

    combined_factor = (  
        # value_winsorized.zscore(mask=m) +  
        # quality_winsorized.zscore(mask=m) +  
        # sentiment_score_winsorized.zscore(mask=m)  
        fsf_value.zscore(mask=m)  
    )

    longs  = combined_factor.top   (TOTAL_POSITIONS//2, mask=m)  
    shorts = combined_factor.bottom(TOTAL_POSITIONS//2, mask=m)

    long_short_screen = m & (longs | shorts)

    pipe = Pipeline(  
        columns={  
            'ebt_ltm'        : ebit_oper_ltm,  
            'ebt_win'        : ebit_oper_ltm_win,  
            'qf'             : entrpr_val_qf,  
            'qf_win'         : entrpr_val_qf_win,  
            'fsf'            : fsf_value,  
            'longs'          : longs,  
            'shorts'         : shorts,  
            'combined_factor': combined_factor,  
        },  
        screen = long_short_screen  
    )  
    return pipe