Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Fundamental data in pipeline not updated in 2008-2010
Julien D
posted Jan 19, 2016
Seeking Help Using Data Pipeline

Hi, I noticed that in 2008-2009 the fundamental data (for instance fields total_debt or total_debt_equity_ratio) are not updated quarterly while they were in 2007 and 2011 onward for all the SIDs I have checked (for instance Alcoa - ticker AA - SID 2). It seems like Canadian companies like ABX (sid 64) do not have quarterly data until 2013.

However, it seems that the data is quarterly in the backtester but I can't be 100% sure of that.

This makes historically backtesting around those period quite inaccurate. Please let me know what you find out.

Thanks!

5 responses
Josh Payne
Jan 19, 2016

Hi Julien,

I'm digging in a bit in our raw data and can confirm that some companies have that data fully populated and others do not. I'm not sure why but generally all companies don't have all metrics populated for them -- it depends on the type of company they are and what they include in their filings. I'll reach out to Morningstar to figure out if there is a better answer from them.

Thanks,
Josh

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Josh Payne
Jan 19, 2016

Note, one way I am spot checking individual securities and their values for total debt is with the following snippet in Research:

fundamentals = init_fundamentals()  
fund_df = get_fundamentals(query(fundamentals.balance_sheet.total_debt                                )  
                           #.filter(fundamentals.company_reference.primary_symbol == 'SWHC')  
                           .filter(fundamentals.company_reference.sid == 3766)  
                          ,  
                          '2016-01-01', '200m')  
fund_df.loc['total_debt'].plot()

Apple is the only one I've found so far with a long gap of no data.

Julien D
Jan 19, 2016

Thanks Josh for the quick follow up. Apple has a long gap because it did not have debt until very recently!

A good example though is Alcoa (ticker = A, sid = 2) as it has indeed no updated to fundamental fields in 2008 and 2009 .

Another one is Barrick Gold (ticker = ABX and sid = 64) which is Canadian so maybe a separate issue with processing 20-F (filed annually) instead of 10-Qs.

I checked on Bloomberg and definitely both have changes in total assets, total debt or total equity between 2008 and 2009. Most of the sids I have checked are not updated so I think it is a widespread issue.

Julien D
Jan 19, 2016

Also, as I don't see the same drop in coverage when I run a backtesting I am wondering if you call different sources?

Josh Payne
Jan 20, 2016

Yes, I can see the raw input of what MS sends to us before we transform it for use by the backtester. In the case of the backtester, we forward fill the data to account for when data isn't provided for one reason or another. So the backtester has the last "best known" data point. You can access the "as of" date for any field by appending _asof to the column name to assess the time period to which a data point is applied.

For AA, it looks like we're getting the right data to update the total_debt in time periods like 2007-2009 (see chart), but it's not being incorporated into the algo-facing data table for some reason. We're skipping some of the updates in our processing. I've submitted a bug to improve this.

Thanks for pointing it out!
Josh

About Quantopian
Careers
Community
Events
Documentation
Academia
Lectures
Status
Facebook
Twitter
LinkedIn
YouTube
Terms of Use
Privacy Policy

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.