Trying to understand Quantopian's data sets

Back to Community

posted Jan 13, 2016

Hello,

I'm a new user at Quantopian, and am trying to understand the API. In particular I don't quite understand the data sets available. I'm particularly interested in value oriented investing, and by extension the Morningstar fundamental data.

Can I think of the Pipeline API as a means of creating a custom universe instance? For example, could I recreate DollarVolumeUniverse using the Pipeline API? What would this look like?
Is there a way I can plot a data set for a security over time, just to get a feel for the data? Can I plot out a histogram for a particular metric at a given point in time? Being able to do so would be great for getting a visual feel for the type and granularity of the data in the data set.
How often is the fundamental data updated? Every day? Week? Month? Quarter? Updating earnings every quarter might be OK, but updating market cap every quarter wouldn't work as well. Thus my question.
Is it possible to get (say) the top 20% of stocks by market cap at any given point in time? How about bottom 20% by P/E? I could write the math to do this myself (as I have attempted below) but I'm hoping there's a more efficient approach.
I tried writing an algorithm to fetch market caps for all securities... my Python isn't that great, so suggestions on basic Python would also be appreciated. The interesting thing is that the log statement at the end showed far fewer than the 8000 securities that Quantopian claims to track.

def before_trading_start(context, data):  
    """  
      Called before the start of each trading day.  
      It updates our universe with the  
      securities and values found from get_fundamentals.  
    """  
    ranges_df = get_fundamentals(  
        query(  
            # put your query in here by typing "fundamentals."  
            fundamentals.valuation.market_cap,  
            fundamentals.valuation.shares_outstanding  
        )  
        .filter(fundamentals.valuation.market_cap != None)  
        .filter(fundamentals.valuation.shares_outstanding != None)  
    )  
    caps = [ranges_df[stock]['market_cap'] for stock in ranges_df]  
    log.info("Number of market caps collected: " + str(len(caps)))

Log output with start date 2011-01-04:

2011-01-04before_trading_start:73INFONumber of market caps collected: 4778

Log output with start date 2015-01-04:

2015-01-05before_trading_start:67INFONumber of market caps collected: 5189

Thanks!

Sunil

2 responses

Josh Payne

Jan 13, 2016

Sunil,
Welcome! We're glad to have you poking around. Some quick answers.

Yes, it's possible to recreated dollar volume universe selection using Pipeline. I think fawce did that here: https://www.quantopian.com/posts/dollar-volume-pipeline
You certainly can plot securities on Quantopian. Check out the Research environment. It's preintegrated with our pricing data (and other data sources). Try plotting the returns from the get_pricing() call.
Fundamentals data is updated daily. Of course some metrics don't change but we process updates every day. So yes, you should see a new market cap every day. Similar to get_pricing, you can examine fundamentals data in Research using get_fundamentals().
Yes, this is possible but I'd recommend trying to use the Pipeline API for that purpose as it's more well suited for that use case. There are samples in the docs that outline how to access the Morningstar fundamentals data in the pipeline API (as opposed to the older get_fundamentals API).
Keep in mind that not all securities are covered by the fundamentals data. Things like ETFs aren't covered by that data. Morningstar fundamentals covers companies, not all securities.

Hope that helps!

Good luck,
Josh

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Sunil Mishra

Jan 23, 2016

Thanks Josh, just getting back into working with Quantopian. The pointers are very helpful.

Sunil

You've successfully submitted a support ticket.

Our support team will be in touch soon.