history API available in before_trading

Back to Community

history API available in before_trading_start?

posted Feb 27, 2016

When I run the code below, I get:

IndexError: index 0 is out of bounds for axis 0 with size 0  
There was a runtime error on line 7.

Does history() not work in before_trading_start()?

# Put any initialization logic here.  The context object will be passed to  
# the other methods in your algorithm.  
def initialize(context):  
    pass

def before_trading_start(context, data):  
    prices = history(30,'1d','price')

# Will be called on every trade event for the securities you specify.  
def handle_data(context, data):  
    # Implement your algorithm logic here.

    # data[sid(X)] holds the trade event data for that security.  
    # context.portfolio holds the current portfolio state.

    # Place orders with the order(SID, amount) method.

    # TODO: implement your own logic here.  
    order(sid(24), 50)

10 responses

Grant Kiehne

Mar 1, 2016

Any feedback?

Also, does the universe get updated immediately, upon a call to update_universe or at some later point?

Does before_trading_start run on the first day an algo is launched? It seems that one could start the algo just before the market opens and there would be no time to execute before_trading_start.

Josh Payne

Mar 1, 2016

I'd recommend using pipeline API for efficient access to pricing data in before_trading_start(). If you want minute bars for use with history, then you'll currently need to use history in handle_data during the trading day.

Generally, if you kick off a live trading algorithm, before_trading_start() will be run upon the start of your algo no matter what, regardless of the time.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Grant Kiehne

Mar 1, 2016

Why is history not accessible from before_trading_start? It would seem that one should be able to run update_universe which would immediately update data followed by history, all within before_trading_start. This way, the full 5 minutes of computation time could be used, if necessary, on the current set of securities. Or does update_universe not take effect until the first trades of the day are encountered? And maybe this is why history doesn't work in before_trading_start?

In any case, except for the first day of trading, the prior day's output of history can be stored in context, and used in before_trading_start.

Generally, if you kick off a live trading algorithm, before_trading_start() will be run upon the start of your algo no matter what, regardless of the time.

There's probably a corner case here that will fail. If a live algo is launched just before market open, what happens if before_trading_start hasn't completed before the first call to handle_data?

Josh Payne

Mar 2, 2016

The corner case is handled. In the case you describe, the code for the minute bars that occur in the past are run but orders are blocked from execution until the algorithm catches up.

Disclaimer

Grant Kiehne

Mar 2, 2016

So am I correct in my understanding of how before_trading_start() works? It just seems unnecessarily hobbled. My take on pipeline is that it is a rough screening tool, since it only supports daily OHLCV bars (well, the O doesn't work yet, right?). Using the output of pipeline, one could do finer analyses using minute bars, once the high-level screen is complete (mysteriously limited to 500 securities, which also seems unnecessarily restrictive). This can be made to work (I think) by using the universe from the prior day, storing the output of history in context, and then setting a flag to not update the universe. In this fashion, one can take advantage of the full 5 minutes of overnight computation time, and have signals ready prior to the open. I'll post an example, when I get the chance. It just seems awfully convoluted for something that should be supported natively.

The corner case is handled. In the case you describe, the code for the minute bars that occur in the past are run but orders are blocked from execution until the algorithm catches up.

That doesn't sound quite right. So the algo could be 5 minutes into the trading day, and then enter into a catch-up mode? Not sure I follow. If before_trading_start() is still running when the market opens, then shouldn't an exception be issued?

Grant Kiehne

Mar 2, 2016

Another approach would be to not allow algos to be launched within 5 minutes of the open, if they contain a call to before_trading_start(). This could get ugly, though, if you increase the time-out of before_trading_start(). I guess there's a more general discussion of how to handle overnight computations and the opening bell, since I have to imagine that you'll expand the scope as you improve the platform.

Is is possible/feasible to schedule the call to before_trading_start()? Or does it need to be run every day? Especially for backtesting, it'd be nice to run it only when needed.

Grant Kiehne

Mar 8, 2016

Hi Josh,

Please see attached and the line I commented out. It turns out that history() does work, but close_price must be used, rather than price. Is this a bug?

Grant

Josh Payne

Mar 8, 2016

iirc, the bug is actually that history() works at all in before_trading_start :) I don't believe we actually intended to enable it to work in the first place and it was an unintended consequence of our work with pipeline.

Fixing this one way or another isn't currently high on the priority but we've made note of it in the backlog.

Thanks
Josh

Disclaimer

Grant Kiehne

Mar 8, 2016

Thanks. Well, it would be more convenient to fully enable history() in before_trading_start(), assuming that update_universe() would take effect immediately. But maybe this is a non-starter for some fundamental reason I don't understand.

Grant Kiehne

Mar 9, 2016

Hi Josh,

Here's my work-around:

https://www.quantopian.com/posts/code-for-getting-minute-data-into-before-trading-start

Unfortunately, I don't think it integrates well with pipeline, since for a dynamic universe, context.stocks would need to be defined the prior day (or earlier).

After doing the coarse screening using daily data and pipeline, it'd be nice to be able to follow up within before_trading_start at the minute level. It seems it is doable, but with a 1 day minimum latency.

You've successfully submitted a support ticket.

Our support team will be in touch soon.