Fundamental Data from Morningstar Now Available for Backtesting

This is great, really excited to try it out. Opens up a whole range of new possibilities. A few questions:

1) RE: before_trading_starts, can price history data be used in that function? Meaning - I'd like to create a universe only out of low beta stocks, for example (beta is not a fundamental field but it can be easily calculated from price history). Seems like the custom universe functionality here could apply to several use cases, not just fundamental filtering.

2) Any chance you can update the documentation to be more specific about update frequencies for the various fields? Meaning - which are quarterly, which are daily, weekly, etc? And for fields like P/E, which earnings are used to calculate it (I'm assuming last 4 quarters but different services use different values, e.g. Valueline's default is past 2 quarters plus next 2 quarters estimated). Seems nitpicky but in my experience with fundamental-based algorithms, it matters.

Simon Thornington

Awesome news! I can't wait to try out a backtest of a basic EBIT/EV, then we can get to work on replicating the Quantitative Value screen!

Matt, thanks for the questions and comments.

Both are things we can look to do in the future. Generally as folks begin to use these features more, we'll prioritize our efforts around how to flesh out these methods. So accessing pricing data would definitely fall in that category.

With respect to the documentation, we can definitely look into doing that. For now, you can also try to look at the "as_of" field for each metric. By as_of, I mean that there is an companion date field for each metric. So basic_eps also has basic_eps_as_of which indicates the end date for which the metric applies. So if it is a quarterly metrics, you'll see tell-tale as_of dates like 6-30 and 9-30 (the last day of each quarter). For more frequently updated metrics, the as_of dates will have more variety.

Disclaimer

By the way, I should introduce myself -- I'm a new product manager here at Quantopian. I started last month. My general sphere of responsibility is improving and expanding Quantopian's access to data for use in backtesting and beyond. If you have feedback for me on this front, I'd love to hear it. Feel free to message me or email me.

Disclaimer

Nyan Paing Tin

Dec 11, 2014

Hi Josh,

Overall, this is very exciting.... I have one quick question. Since get_fundamentals is called before the start of each trading day, wouldn't that limit flexibility or the scope of trading strategy. For example, what if someone wants to incorporate current EPS value on the day it is announced. I guess that problem is irrelevant if all the companies announced their EPS before 9:30 am. Am I missing something here?

Vyomakesh Sridhar

Dec 11, 2014

Hi Josh,

This is really great news ! Ive been waiting for Quantopian to come up with this. With this addition, I think Quantopian is one of the best platforms around for backtesting survivourship bias free trading strategies. Keep up the great work !

Dec 11, 2014

@Vyomakesh, Thanks!

@Nyan, we process updates from Morningstar on a daily frequency. This is typically done when the market is closed, in the evening EST.

Disclaimer

Dec 17, 2014

Is this data point in time?

Simon Thornington

Dec 17, 2014

Is the Morningstar data available in the research platform (which I still have not yet tried :/ ) ?

Dec 18, 2014

@Michasel S: yes it is point in time.

@Simon: It is not yet available in Research or Live Trading. Those are high on our backlog.

Disclaimer

Dec 31, 2014

Josh, Out of curiosity, how do you reconcile the filing date issue on morningstar? I was on the impression that filing dates are relatively recent with them.

Are you matching to edgar?

Hi Michael,

I was too brief with my answer on your point in time question.

We generate our date logic in the point-in-time database through a variety of approaches, depending on the data and it's age. In scenarios in the past, where a file date is not available to us, we approximate it based on the as_of date by adding 45 days to the as_of date. We describe this in the point in time section in the help docs:

Quantopian takes care of this logic for you in the backtester. For
data updates since Quantopian began subscribing to Morningstar's data,
Quantopian tracks the file date based on when the information changes
in Morningstar. For historic changes, Morningstar also provides a file
date to reconstruct how the data looked at specific points in time. In
circumstances where the file date is not known to Quantopian, the file
date is defaulted to be 45 days after the as_of date.

Disclaimer

Ok, that's probably close enough. That was my question, I didn't realize it was in the docs.

The 45 days approximation is definitely just that. We are definitely open to ways to improve that bit of course logic.

Thanks!

Disclaimer

I mean the only way to add it with morningstar is to reconcile it with sec data.

I mean the only way to add it with morningstar is to reconcile it with sec data.

Paul Jacobs

Feb 5, 2015

It looks like there's a small bug in the custom data graph for that algorithm:

    # track how many positions we're holding  
    record(num_positions = len(context.fundamental_df))

I don't see how that would track the number of positions. Shouldn't it be:
record(num_positions = len(context.portfolio.positions))

Blue Seahawk

Feb 5, 2015

Note securities can often be in .positions with 0 amount (shares).

Mark Segal

Apr 13, 2016

Any update on plans to include fundamental data for companies listed on non-US exchanges, TSX, FTSE, DAX etc.?

Regards,
Mark

Apr 13, 2016

Generally Quantopian supports the US equities markets (which includes ADR from other countries). Next up in the queue for us in terms of markets is futures.

Disclaimer

Chad Bonner

Apr 26, 2016

Can anyone explain what format the Morningstar file_date is in? It appears to be some number of units from some date but is just an integer.

For example:
Equity(48091 [VA])
pe_ratio 3.7998
file_date 14557536000000000

When will Morningstar data be allowed in active real time trading algorithms? I see messages in my back tests that states certain Morningstar data is not available for real time trading.

Most fields are available for use in live trading. There are a certain subset of fields (like shares_outstanding) for which we need to improve our data processing. As such we don't allow them in live trading until we've solved these problems. We're actively working to solve this problem currently.

Disclaimer

Behavioral Trader

Hi Josh, does the data set contain a pre computed Cash per Share?

If so, could you please give us an example of, for example, going long all companies which have Cash per Share greater than Share Price by X%?

Thanks in advance,
BT

EDIT: Please disregard this question. It was answered previously in this thread.

I would just love a better understanding of the morningstar data. Primarily if changes to data (such as cash balance) correspond with the exact date in which the company filed their 10-Q or 10-K. I am hoping this is the case, but I have not located any evidence to confirm, and it would be a big effort to analyze this in the backtester or in Research. Is there some documentation for the Morningstar data that I have not seen?

For my own reference:

Quantopian takes care of this logic for you in the backtester. For
data updates since Quantopian began subscribing to Morningstar's data,
Quantopian tracks the file date based on when the information changes
in Morningstar. For historic changes, Morningstar also provides a file
date to reconstruct how the data looked at specific points in time. In
circumstances where the file date is not known to Quantopian, the file
date is defaulted to be 45 days after the as_of date.

@ BT

I am running an algo that goes long stocks with Cash Per Share greater than Share Price. Finding about 300-600 exist in the top 4000 stocks per market cap. I will post it on your original thread after the back test is finished.

@Josh - I am particularly interested in the following fields: shares_outstanding & tangible_book_value_per_share to utilize in live trading, when would you anticipate availability? I am looking to launch my algorithm soon.

Thanks!

Thanks!

Hi all,
I'm loath to provide a specific estimate because this project (as most software projects do) has been very difficult to complete. I'd put it at weeks or a month or so.

All fields provided by Morningstar in the database are listed out here: quantopian.com/help/fundamentals if you're looking for information on the availability of specific metrics.

Thanks
Josh

Disclaimer

Jul 11, 2016

Hi Josh,

Thank you very much for the update!

Best
Adam

Hi Josh,

Can you provide a status update on the morningstar data availability for broker-backed trading?

We are in the process of rebuilding the database. It takes . . . a long time as it replicates our nightly processing of the data. So we're code complete. We just need to run through the processing and test the results before swapping over. This will make available shares_outstanding.

Just to be clear: Morningstar data is generally available in broker backed trading. It is just a subset of fields that measure companies on a per share basis that are ineligible.

Disclaimer

Hi Josh,

Thank you for the update.

So, per share fields are an issue? So, would tangible_book_value_per_share be ineligible? When would it become eligible?

Thank you,
Adam

Yes, that field would be impacted. I would recommend using tangible_book_value instead. The work to make all of these per share fields usable in live trading is large and we're at the outset of the project. I cannot commit to a timeframe.

Disclaimer

Kim Miller

Sep 9, 2016

Josh, Can you be more specific about point-in-time dating? Specifically, is there at date before which you approximate all restated as_of dates by adding 45 days to the as_of date? Also, can you point to the point in time section in the help docs? Can't quite find that section.

Sep 9, 2016

June 2014 is when we began processing files from Morningstar.

Here is the relevant section of the documentation but it doesn't expand much further than what you've likely read here: https://www.quantopian.com/help#ide-fundamentals

Disclaimer

Kim Miller

Sep 9, 2016

Thank you, Josh. Just to make sure I understand, let's say you have GE earnings:

6/30/10 Earnings released 7/15/10 as $10 / share
6/30/10 Earnings now REVISED 8/15/10 to $8 / share

In Quantopian today we would see the following?:

6/30/10 Earnings released 7/15/10 as $8 / share

Sep 11, 2016

@Josh - can you provide a status update on morningstar's shares_outstanding availability in live trading?

Thanks!

The reprocessing continues. It's had a few bumps in the road slowing us down but we are continuing to make process.

Disclaimer

Thanks Josh. Do you have a sense of when it will be completed? I have an algo I would like to launch, and this is the only dependency that is stopping me from launch. Thank you.

Kim Miller

Also @Josh, in addition to my question above, can you confirm if you do include RESTATED data in addition to originally reported?

Thanks so much.

@Adam, my guess is within a month or so (conservatively). Of course now that I've said that, I've probably jinxed it. We're processing and iterating through bug fixes as they arise. Fingers crossed.

@Kim, the answer to the revisions question is "sometimes". If the most recent data point for a metric/company combination is restated by Morningstar, we process that change and surface it. That applies to the timeframe after June 2014.

This is one of the reasons we're anxious to improve by re-architecting the underlying data source.

Regards,
Josh

Disclaimer

Thanks Josh!

@Josh - can you provide a status update on morningstar's shares_outstanding availability in live trading?

Thanks!

Jamie McCorriston

Nov 11, 2016

Hi Adam,

We recently fixed the shares_outstanding field in backtesting. We haven't yet unblocked it from live trading but we should be doing so shortly.

Jamie

Disclaimer

Dec 5, 2016

@Jamie

Could you provide an update on when the shares_outandinting field will be unblocked in live trading? I have an algo that I would like to launch that leverages that field.

Thanks
Adam

Jamie McCorriston

Dec 5, 2016

Hi Adam,

Unfortunately, I don't have a timeline yet. It's on the list, we just haven't to it yet.

Sorry for the inconvenience.

Disclaimer

Dec 6, 2016

Hi Jamie,

To be candid, I am frustrated. I have been asking about adding this data since April 2016, and have not had a response that indicates a timeline for actual release. (I keep getting shortly, or a month from now) I understand it is being worked on, but an understanding of how the process works and where it is on the list, and why it is taking so long would be helpful for the community to understand. I expect there are others with similar frustrations.

Thanks
Adam

Jamie McCorriston

Dec 7, 2016

Hi Adam,

I'm sorry that you've had such a frustrating experience. The big fundamentals project to fix shares_outstanding took longer than expected. The reason it hasn't yet been unblocked from live trading is because there are still some limitations to the dataset. For example, the data is only updated monthly by Morningstar, and it is not split-adjusted. As a result, we were hesitant to unblock it from live trading. Since then, we have reached out to some community members who have expressed an interest in live trading with shares_outstanding and we have come to the decision that we will unblock the field from live trading. The next step is to actually unblock it. I can't promise an exact timeline, but it's on the list.

Disclaimer