Thanks Nathan,

There is some evidence that you may not have captured everything:

https://www.quantopian.com/posts/is-there-a-reliable-way-to-exclude-when-issued-securities-from-fundamentals-screens
https://www.quantopian.com/posts/what-do-symbols-ending-in-wi-indicate
https://www.quantopian.com/posts/how-to-filter-equity-types-or-classes-in-pipeline
https://www.quantopian.com/posts/cannot-access-fundamental-data-from-research-platform-500-internal-server-error
https://www.quantopian.com/posts/what-do-symbols-ending-in-wi-indicate
https://www.quantopian.com/posts/how-to-filter-exchange-id-in-pipeline

What was your methodology for establishing the list of exclusions? And is there a way to confirm that it is correct (e.g. maybe the research environment could be used to confirm that all of the undesirables have been removed)? There's probably a set of checks that can be run, to confirm that the universe has the desirable characteristics, and that nothing has been missed (e.g. due to coding errors by Morningstar). Do you have an alternate set of data that would allow you to check the Morningstar codings? There's no reason to think that their data are perfect, right?

Also, I'd asked Scott about the possibility of including a filter for stocks that have known bad data. Any thoughts on the feasibility of including such a list in your filter?

In the end, it'd be great to have a canonical base universe of equities, vetted by Quantopian, with clean data, to use for long-short equity strategy development. I gather that is the end goal here, but I can't tell if we are there yet.

You might also consider excluding recent IPOs. As your filter stands now, stocks could be admitted 1 day after their first trade after IPO (although maybe your high_volume_tradable implicitly handles this case?). If you go down this path, note that there are some bad data (see https://www.quantopian.com/posts/check-for-recent-ipos); there should be an associated internal ticket, as communicated to me privately (check with Jamie). I'd think that most long-short equity algos, unless explicitly dealing in IPOs, would want to let the dust settle a bit before including them in a portfolio.

Great, thank you.

This should be in the "Interesting" tab.

Very interesting. I think there should be a simple way, build into quantopian, to limit the securities to the active and tradeable ones. It seems silly for everyone to have to use their own set of filters, just to get a good trading universe. Right now there is a very high learning curve in my opinion, which could be lowered considerable by more quantopian predefined stuff like this. The ones who wish to use a broader trading universe can then define their own.

Btw is there a simple way to limit the securities to an index, like S&P500 or the danish OMXC20, which is the one where i follow the companies the closest?

Hi Nathan,

Any response to my feedback above? Any thoughts on how to filter out stocks with bad data (e.g. I think Q maintains an internal list of problem children)?

Can set_do_not_order_list be used within pipeline? And would set_do_not_order_list(security_lists.leveraged_etf_list) even be necessary with your framework above?

Just wondering if you are done, and have established an authoritative list of exclusions, for writing a typical equity long-short algo? To Allan's point, could it be "released" in some fashion (e.g. put into a github repository)?

Also, I'm curious...what if one just wanted ETFs or ETF-like thingys? It would seem if you can exclude them there should also be a way to include them, and exclude everything else, no?

Grant

I'm also desperately looking for a way to include only ETFs in a Pipeline screen unless there's a way to leverage Factors outside of pipeline that I've missed. Backtests are timing out if the only screen I have on is "dollar_volume > 10**7". I know how to select the ETFs I want AFTER the pipeline returns but therein lies the problem. I imagine that performance would be exponentially better if the pipeline is never run on the unwanted stocks in the first place.

As I was looking at this problem today, I found a solution for me that seems to speed things up:

screen = (dollar_volume > 10**7) & morningstar.share_class_reference.symbol.latest.isnull()

sorry i am so new to coding and finance. How would I say something like ev_ebitda = mstar.fundamentals.valuation_ratios.ev_to_ebitda < 14 I have been trying to figure out fundamentals in pipeline but i get all sorts of errors like,

AttributeError: Neither 'UserBinaryExpression' object nor 'Comparator' object has an attribute 'ndim'

or

AttributeError: Neither 'UserInstrumentedAttribute' object nor 'Comparator' object associated with UserInstrumentedAttribute has an attribute 'window_safe'

from what i understand of the above code, I don't need to build CustomFactor inherited instances right?

Thank you,
Tyler

Tyler,
Try using ".latest" to build comparisons using the latest value:

ev_ebitda = mstar.fundamentals.valuation_ratios.ev_to_ebitda.latest < 14  

I believe that should fix your issue.

thank you! it worked i just had to add the .latest and take out the .fundamentals! hahaaha,

Thanks a lot!

Hi Nathan,

I was specifically wondering - as far as the pipeline goes - is there a way to filter on beta? So far I've found methods to mostly calculate beta on a selected universe (example below). Also, wouldn't it just be inefficient to apply a vectorized calculation of Beta to a large group like that? But say that I wanted to use this as criteria for screening?

def _beta(ts, benchmark, benchmark_var):  
    return np.cov(ts, benchmark)[0, 1] / benchmark_var 


class Beta(CustomFactor):  
    inputs = [USEquityPricing.close]  
    window_length = 60  
    def compute(self, today, assets, out, close):  
        returns = pd.DataFrame(close, columns=assets).pct_change()[1:]  
        spy_returns = returns[sid(8554)]  
        spy_returns_var = np.var(spy_returns)  
        out[:] = returns.apply(_beta, args=(spy_returns,spy_returns_var,))  

For those reading this thread. Check out the built in filters: Q500() and default_us_equity_universe_mask().

https://www.quantopian.com/help#built-in-filters

thank you for this

Hi,
A question for Grant and/or anyone else who may know.

How can I exclude from consideration:
a) Just the Tobacco stocks (for example if I want an "ethical" portfolio) even if I don't know all their symbols?
b) Just a short list of very specific known stocks that I consider to be "inappropriate" for whatever reason?

thanks in advance for your help.