Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Quantopian Tutorial with Portfolio Rebalance Algorithm: Lesson 2 - Universe, Fetcher, and Schedule_function
Seong Lee
edited Feb 16, 2016

2/16/2016 This tutorial is out-dated, please view the new versions here: https://www.quantopian.com/posts/quantopian-tutorials

Welcome to the second lesson in the Quantopian Tutorial Series. In this lesson we extensively cover the universe of securities that will be in your algorithm, using fetcher to load in data to create your universe, and a portfolio rebalancing algorithm that uses schedule_function for minutely mode.

As a refresher, these videos covered the basics of the IDE.

Summary
We covered:
- Universe of Stocks: What actually belongs there and how you set your securities using multiple methods
- How to use Fetcher to set your Universe
- A daily rebalancing portfolio algorithm with schedule_function

Links
- Youtube: http://bit.ly/1swuNpW
- PDF Documentation and Guide: http://bit.ly/1Kvw03r
- First Quantopian Tutorial Series - Basics of the IDE: http://bit.ly/1wTmW7m
- Third Quantopian Tutorial Series - Basics of Fundamentals: http://bit.ly/16HhrNR

FEEDBACK
We're still building this out so I'd love to hear what you guys think. What was helpful? What wasn't helpful? What would you like to see next? Feel free to email us at [email protected] with your comments and suggestions.

** Credits to Jessica Stauth for writing up the algorithm found here

**EDIT:

On line 28 please use this new line instead: 

symbols = [s.symbol for s in sids if s != 0] # instead of [s.symbol for s in sids]
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

16 responses
Colin
Jan 19, 2015

You mentioned that the max allowed size of a universe is 200, but this does not seem to be the case. 

set_universe(universe.DollarVolumeUniverse(floor_percentile=90, ceiling_percentile=100))

results in a set of 802 securities outputted via
print len(data) Will it trade based on all of these and if not how does it choose a subset of 200?

Seong Lee
Jan 20, 2015

Hi Colin,

So the 200 max allowed universe size applies to minutely mode, but is a bit different when you're working in daily mode (with a larger allowed universe size)

To use the DollarVolumeUniverse, set the universe in the initialize
method. You can set a universe interval up to 2% for backtests in
minute mode or up to 10% in daily mode. Each percentile in the range
will result in ~80 securities, so the following example consists of
the ~160 securities with the highest dollar-volume volume, with
changes coming on the quarter end.

Let me know if you have any questions

Colin
Jan 20, 2015

Ah OK, thanks for the fast response, makes sense. Another issue I am wondering if you are able to reproduce - when I clone the algorithm you listed above it functions correctly, but if I create a new one and paste in the exact same code it never hits breakpoints set in my_universe.

Seong Lee
Jan 27, 2015

Hi Colin,

That is strange! I just tried copy/pasting from my algorithm into another one and it seems to be working for me.

Here's the screenshot: https://copy.com/DXphhjHrHeLZBBpj

Mind sending me a collab to your algorithm so I can take a closer look? [email protected]

Seong Lee
Jan 27, 2015

Hi Colin,

I suspect on the algo you just shared with me that you're running into a similar error. Do you mind trying your prior solution and checking the dates and see if that works?

Colin
Jan 27, 2015

That is what I thought as well (start date in backtester not matching the one entered in the csv file), but it is not the case. When the dates match up there is still no trading activity.

Seong Lee
Jan 28, 2015

Hi Colin,

The issue was that the date column in your FETCHER file started on a holiday. The best way to fix this is to make sure that your dates are on a trading day, not a holiday. In this case, I changed the date from 1/1/13 to 1/2/13 by adding in a df['date'] = '1/2/13' in your pre_func method.

If you don't mind, perhaps we could share snippets of your algorithm on this thread so that others who might run into the same problems might benefit as well?

I've also added in a 'if stock in data and 'price' in data[stock]' to make sure that we have data/volume for a security before trying to execute an order for it.

Thanks,
Seong

Colin
Jan 29, 2015

Sure that is fine with me. Thanks for figuring out the issue.

I've also tried backtesting with a start date that is after the one in my csv and no trading occurs.
-What is the best practice in setting a date for live trading? It seems based on this that it would need to be a future date since fetcher retrieves the universe from the file only once it hits the date.

A. Roy
Mar 11, 2015

Seong, I'm wondering the same thing as Colin, about the best practice in setting a date for live trading.

In other words, for a live trading algorithm, would you have to set df['date'] = datetime.today().strftime('%m/%d/%y') in your pre_func?

Seong Lee
Mar 11, 2015

Audrey,

That's a great question. Let me try setting that up in a live algorithm and will post my findings here.

Seong Lee
Mar 16, 2015

It looks like

    log.info("%r" % datetime.today().strftime("%m/%d/%y"))  
    df['date'] = datetime.today().strftime('%m/%d/%y')

Did in fact change the dates of the CSV file to the current day

Ueli Hofstetter
Mar 16, 2015

@Seong: Could you provide some more information about the maximum size of the universe in different cases. I mean how large can the universe be in case of daily data?
furthermore, I just tried to set the universe to 500 stocks using
"update_universe(fundamental_df.columns.values)" in the "before_trading_start(context)" method and I get the error message "UniverseExceedsSizeLimit: 0029 Universe size 500 exceeds limit of 200 for daily bars." so is it only possible to use a universe > 200 stocks when it is initialized from "initialize(context)"?

thanks in advance

Seong Lee
Apr 13, 2015

The latest version of the backtest that fixes a broken fetch_csv universe method.

If you're coming from an old version, on line 28 please use this new line instead:

symbols = [s.symbol for s in sids if s != 0] # instead of [s.symbol for s in sids]
Kevin Choi
Feb 25, 2016

Hey everyone, just out of curiosity how come we don't include a random walk with the fundamentals we've chosen in these graphs? We include a standard SPY benchmark and our algo, but I would think adding a random walk would also be important.

Chris Lonsberry
Mar 23, 2016

I got here from the tutorial notebook: " 2. Getting Started: Financial Modeling". But then at the top of this post it says:

2/16/2016 This tutorial is out-dated, please view the new versions here: https://www.quantopian.com/posts/quantopian-tutorials

So I follow that link to the main tutorials page where it refers users back here again as a more advanced tutorial. Can you please clarify the status of this post? Is it out-dated? More advanced?

Jamie McCorriston
Mar 31, 2016

Hi Chris,

Sorry for the confusion! I've removed the pointer from the basic tutorials to this one. This tutorial is indeed out of date, and we're working on replacing them with new 'intermediate' tutorials to follow the basic ones, all part of an effort to improve the learning experience for newcomers. We're working on creating written and video tutorials - stay tuned!

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

About Quantopian
Careers
Community
Events
Documentation
Academia
Lectures
Status
Facebook
Twitter
LinkedIn
YouTube
Terms of Use
Privacy Policy

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.