Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Dynamic universe confounds series .iloc[-1]
Market Tech
edited

Requesting clues as to how to avoid issues when calculating talib metrics on securities that probably don't have data.

Running this code on a set of known securities with consistent data is no issue:

def HandleEntry(context, data):  
    closes   = history(DailyPeriods + 1, "1d", "close_price").resample("1w")  
    closes   = closes.dropna()  
    means    = closes.apply(talib.MA, timeperiod = WeeklyPeriods, matype = MAType.TRIMA)  
    sigmas   = closes.apply(talib.STDDEV, timeperiod = WeeklyPeriods)  
    zScores  = ((closes - means) / sigmas).iloc[-1]

But if set_universe or update_universe is used, undoubtedly there will be issues with data for some security and at "iloc[-1]" I'll get an index out of bounds error.

So, anyone know how to protect against or eliminate the bad apples from the series before this calculation?

Perhaps retrieve the calculated series first and manually iterate looking for and nixing the rotten rabblerousers?

3 responses
Seong Lee

I'm surprised that the 'closes.dropna() doesn't do it for you.

Is it possible to put in a try/except and skip over the ones with an index error?

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

James Jack

I use the following checks when using set_universe.

def Inspect(context, data):  
    HIGH = history(60, '1d', 'high', ffill=False)  
    HIGH = HIGH.dropna(axis=1)  
    Good_Sids = [Sid for Sid in HIGH if Sid in data]  
    HIGH = HIGH[Good_Sids]  
    # only use Sid's from the Good_Sids list from now on

This may be OTT but it tends to sort things out :)

Market Tech

[Updated]
@James J. Thanks. That cured the issue.

Attached new backtest.

[Original]
@Seong L. it turns out to be more complicated that I'd imagined. I can eliminate the bad securities from the dataframe using this code: 

    zScores  = ((closes - means) / sigmas)  
    zScoresIndex  = [item for item in zScores if len(zScores[item]) > 0]  
    if (len(zScoresIndex) == 0):  
        return  
    zScores  = zScores[zScoresIndex]  
    zScores  = zScores.iloc[-1]

This effectively builds a list of good securities and then uses the list, as a filter on the index, to subselect from the zScores dataframe. (Damn but this python stuff is tricky!)

The complication part is that now that this strategy I'm working out uses the update_universe, data delivered in history seems to go missing.

The pertinent parts are inside the HandleEntry method. Appreciate any analysis on this you can spare.

About Quantopian
Careers
Community
Events
Documentation
Academia
Lectures
Status
Facebook
Twitter
LinkedIn
YouTube
Terms of Use
Privacy Policy

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.