Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Working On Our Best Universe Yet: QTradableStocksUS

We just released an experimental version of what will become our best, most useful asset universe yet: the QTradableStocksUS. We are looking for feedback about the usability of the universe in your algorithms, its membership, and any technical issues.

Your strategy's foundation is its universe. You want a universe to be as large as possible, but contain only assets that match the platform capabilities. For Quantopian, that means assets for which we can find data reliably, that fit our risk management, that we can use in a multi-algorithm portfolio, and that we can trade at a reasonable transaction cost.

Everyone on Quantopian has taken advantage of the power and ease-of-use of one of our pre-defined universes, like the Q1500US. These universes provided a set of liquid, easy-to-trade stocks while excluding assets that have more difficult risk profiles like ADRs and ETFs. The QTradableStocksUS is an improvement over these universes in many ways:

  • The Q500US and Q1500US universes were arbitrarily limited in size. QTradableStocksUS has no explicit size limit, and generally has between 1600-2100 members.
  • The new universe has more effective screens to remove illiquid or otherwise untradeable stocks.
  • For companies with more than one share class, the new universe picks the most liquid rather than always picking the primary share.
  • The new universe is updated daily rather than monthly.

The new universe works hand-in-hand with the Quantopian Risk Model: all of the members of the QTradableStocksUS are covered by the risk model.

Once this universe is fully released (not experimental), this will be the preferred universe on Quantopian. We will ask everyone seeking an allocation to use this universe. At that time, algorithms that are already written using the Q500US and Q1500US will still be considered for an allocation, but we will eventually deprecate these older universes.

In the future we plan on applying other risk filters to improve QTradableStocksUS. We will remove acquisition targets based on an M&A data source. We hope to add information about short availability, and remove stocks that can't be shorted. We also plan on adding filters for stocks with pending major publicly known announcements from the FDA and other regulatory events. Each one of these data sources will make the universe's quality higher. Some of these improvements will be soon, and others will be farther in the future, after the universe is fully released.

One consideration when using the QTradableStocksUS at this time is the size of the universe. The universe is larger than the Q1500US, and many of our backend processes are optimized for pipelines that output no more than 1500 names. For now, you should restrict your pipelines to output 1500 stocks or fewer, or your backtest will run more slowly.

Here are the specific limits applied to the QTradableStocksUS:

  • Market cap: over $500M: This restriction eliminates many undiversifiable risks like low liquidity and difficulty in shorting.
  • Dollar volume: It is important that stocks in our universe be relatively easy to trade when entering and exiting positions. The QTradableStocksUS manages that by including only stocks that have median daily dollar volume of $2.5m or more over the trailing 200 days.
  • Prior day's close: If a stock's price is lower than $5, the bid-ask spread becomes larger relative to the price, and the transaction cost becomes too high.
  • 200 days of price and volume: If a stock has missing data for the previous 200 days, the company is excluded. This targets stocks with trading halts, IPOs, and other situations that make them harder to assess.
  • Primary/Common share: The QTradableStocksUS chooses a single share class for each company. The criteria is to find the common share with the most dollar volume.
  • ADRs, Limited Partnerships: QTradableStocksUS excludes ADRs and LPs.

Attached is a notebook that uses the QTradableStocksUS and explores the membership of the universe.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

30 responses

@Josh Payne,
Hi Josh. Very positive step. Thank you. I look forward to working with it ASAP.

Looks great, is possible to get data further down in time, so we can test more market cycles... Thanks!

Error while compiling, please help

5 Error Runtime exception: ImportError: cannot import name QTradableStocksUS

Code:
3 from quantopian.algorithm import attach_pipeline, pipeline_output
4 from quantopian.pipeline import Pipeline
5 from quantopian.pipeline.filters import QTradableStocksUS

Try from quantopian.pipeline.experimental import QTradableStocksUS

This is a great upgrade, thank you!

line 5

from quantopian.pipeline.experimental import QTradableStocksUS

worked!

Thank you Josh!

Hi @Josh,
May i suggest trying a slight change in the criteria for acceptance into the "Qtradable" universe:

a) Add the constraint that the stock is optionable, i.e. actually has exchange-traded options. Even though we are not working with options data, the fact that a stock actually has tradable options has a (generally desirable) impact on its characteristics.

b) Remove the Market Cap constraint and increase the minimum required dolar-volume liquidity instead.

This, plus item a) should help to ensure that stocks can be shorted, and avoids the problem that, with the Mkt Cap constraint, some fast-growing and heavily-traded but still small cap growth stocks that could be desirable additions to the portfolio are excluded just because their MktCap has not yet risen to a sufficient level. After all, if they can be shorted and they are sufficiently liquid, then the market cap constraint is both unnecessary and potentially more limiting than need be.

c) I completely agree with the attempt to keep Bid-Ask spreads and therefore transaction costs / slippage within reasonable bounds. However I'm not sure that the $5 price limit is the best way to do this. Of course bid-ask spread is a function of supply & demand for the stock and, at least based on the limited research that I have done on it , it appears that Dollar-Volume liquidity is actually a better indicator of expected bid-ask spread than is price alone. If a stock is highly liquid, heavily traded and therefore has a small bid-ask, then why exclude it just because the price is below $5? So as a further recommendation, may i suggest replacing the $5 minimum price limit with a more stringent dollar-volume liquidity constraint instead.

Be careful if you trade this universe in the contest. I had an algo which I tried to enter the most recent contest, and in its warm up, it blew the 1.1 leverage restriction. It uses the Optimizer with a target gross leverage of 1.0, but some of the stocks in this universe take some time to enter or exit positions, leading to intraday variability in leverage as the trades are made to hit the target portfolio. I solved it by halving my maximum position size.

Hi @Burrito Dan, thanks for the heads-up on this.
Actually i also found the same thing happened to me on my first 3 algos, all of which used the standard Q1500US universe. It's usually (but not always) the transient start-up behavior that blows the leverage above the target. I investigated by getting a custom output plot of a 10x zoomed up view of the % difference between target leverage = 1.00 and actual leverage, and then closely watching the size of the leverage excursions in detail as the algo runs. Code used:

lev1_warn_10xdpct = min(200, max(-200, 10*100*(leverage - 1.00)))  
record(lev1_warn_10xdpct=lev1_warn_10xdpct) 

What i found was that the leverage variation, even with Q1500US is dependent on:
a) the nature algo itself,
b) the target number of stocks (for a given algo a), and
c) the re-balancing period (for a given a) & b).

I found that, at least for all the algos i'm working with, setting nominal leverage = 1.00 occasionally blows out the actual leverage above 1.10 even with the Q1500US universe (haven't tried the new one yet), but using 0.90 nominal leverage results in maximum actual leverage (for my algos) of typically 0.92 and up as high as 0.96, well within the safe limits.

Thanks again, cheers, TonyM.

@Tony Interesting. Yes, I think there's a case for smarter order execution.

@Burrito Dan, i think you are right, but i don't yet have the python skills to implement it.

Actually what i was mainly thinking about when i wrote the earlier post to @Josh is the following:
Most people take the stock universe as a given and then put a lot of effort into stock selection within that universe. In my own personal trading (outside of Q, and i don't use python), i turned it around the other way and put a lot of thought & care into dynamic creation of the best (and potentially most profitable) stock universe, and then the decision regarding "which stocks" within that universe becomes less critical. So, in addition to the expanded universe that Josh is creating for us, actually there is a LOT of scope to take this a long, long way further forward in the form of appropriate dynamic filter functions applied on that universe. When doing that, it is useful that the initial universe (e.g. > Q1500US or the proposed QTradableStocksUS) should not be too restrictive.

Be careful if you trade this universe in the contest .... in its warm up, it blew the 1.1 leverage restriction.

I wrote this code because it appeared to me I was seeing a higher number of partial fills with QTradableStocksUS: https://www.quantopian.com/posts/qtradablestocksus-volume

I'm asking for your help with it, for someone to chart volume in research to two stocks there. Visually I'm expecting it will be clear that WMS has a narrow band of highs and PG is flat. Indicating that mean(), which was stated as being used for QTradableStocksUS can be improved I think by first excluding highs. I think the problem is simply that stocks that have both higher highs and lower lows are getting through. Or, using that code as a starting point, someone might find a brilliant approach and solution.

Beyond that, it's useful to look at extremes in solving problems: If all 78,000 minutes of volume for 200 trading days could be charted high to low, then I think what we need might be to look at a chunk at some point in the long tail, the mean of that limited section.

M&A, FDA, regulatory filters will be extremely useful. Ability to select an universe that automates that will be very helpful. Please update the post on when those updates to QTradeableUniverse happens along with a fast pipeline similar to Q1500US. Thanks!

Is it possible the set in the QTradeableUniverse a minimum marker cap other than 500M?
As I pointed in another post, the threshold should be dynamic (for example the bottom 25% percentile of the NYSE market cap).
This is expecially relevant for backtesing: 500M in 2017 is not the same as 500M in 2003.

For example O'Shaughnessy used in his book "What Works On Wall Street" a minimum market capitalization of $200 million (in 2007 dollars) and then to account for inflation, deflated back to 1926 getting the following thesholds:

06/30/1927 16,500,000
06/30/1937 13,608,000
06/30/1947 20,864,250
06/30/1957 26,592,000
06/30/1967 31,465,034
06/30/1977 57,246,000
06/30/1987 107,244,000
06/30/1997 151,401,260
06/30/2007 195,858,810
12/31/2009 205,652,370

Just to make another example, Wesley R. Gray and Tobias E. Carlise - as described in their book "Quantitative Value" - eliminate all stocks below the 40th percentile breakpoint of the NYSE by market capitalization. Using the data from the Kenneth R. French website, the thresholds become (data in $M):

2000-Jan 506.64
2001-Jan 653.76
2002-Jan 736.96
2003-Jan 664.45
2004-Jan 1028.66
2005-Jan 1251.59
2006-Jan 1418.86
2007-Jan 1584.92
2008-Jan 1408.05
2009-Jan 722.70
2010-Jan 1119.53
2011-Jan 1495.69
2012-Jan 1466.77
2013-Jan 1550.41
2014-Jan 1810.82
2015-Jan 1767.58
2016-Jan 1423.54
2017-Jan 1938.25

As you can see, the variability is high making the use of a fix threshold not the optimal choose.

@Constantino: Many thanks, those are valid points. Let us ponder a bit on that.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

This a bit like the "exam pass mark" problem. What to use as a cut off?

<3

It looks like some closed end funds have entered the QTradableStocksUS universe in recent time periods. I wouldn't think that would be the desired output, so I figured I'd mention it. See the attached notebook.

Hi Expert,

I modify the above program by just performing on one sector and it doesnt work. Any ideas?

def make_pipeline():  
    average_day_dv_200 = AverageDollarVolume(window_length=200)  
    market_cap = Fundamentals.market_cap.latest  
    price = USEquityPricing.close.latest  
    volume = USEquityPricing.volume.latest  
    sector = Sector()

    return Pipeline(  
        columns={  
            'AverageDollarVolume200': average_day_dv_200,  
            'MarketCap': market_cap,  
            'Price': price,  
            'Volume': volume,  
            'Sector': sector,  
        },  
        screen= QTradableStocksUS() & (Sector() == 101)  
        #screen=QTradableStocksUS()

    )

In the screen line at the bottom, try the following:

screen= QTradableStocksUS() & sector.eq(101)  

To convert classifiers and factors to filters, you have to use their .eq() method.

got it. Thanks

We will remove acquisition targets based on an M&A data source. ... We also plan on adding filters for stocks with pending major publicly known announcements from the FDA and other regulatory events.

...

200 days of price and volume: If a stock has missing data for the previous 200 days, the company is excluded. This targets stocks with trading halts, IPOs, and other situations that make them harder to assess.

I'm wondering if these filters block some non-price data strategies, for example:
1. Short stocks that had X trading halts in the last 2 years
2. Long IPO stocks
3. Long stocks of companies that are about to be acquired and for which sentiment is positive since the announcement

(I thought these are the type of strategies Q is actually looking for)

Thanks guys. Quick question, is there a difference between

from quantopian.pipeline.filters import QTradableStocksUS

and

from quantopian.pipeline.experimental import QTradableStocksUS

? I notice that the tutorials go back and forth with between these.

Hello Quantopian, I have an algorithm that performs significantly better on Q500US than on the full universe, how can i truncate the larger universe to look more like the Q500US ? Is it just a matter of filtering out the top 500 market cap ? Or is it the top 500 by average daily dollar volume ?

The intersection of the two universes should work:

universe = QTradableStocksUS() & Q500US()

Ah yes that seems like the best solution, thx!

I joined Quantopian specifically to create a program to simulate an ETF investment (TQQQ). Above it says the the required ‘universe’ for a submission excludes ETFs? So I can’t enter a contest? Why?

How do I get a QTradableStocksUS list for processing outside of Quantopian? I want to make a self serve dataset

Stocks get halted for pending news all the time. News hits, stock is unhalted, stock price reacts... and then a whole year has to pass before the stock is accepted again into the QTU again? Seems overkill.