Feature Request: before_trading_start/after_trading

Back to Community

Feature Request: before_trading_start/after_trading_end events

posted

Hello,

My understanding of the current implementation of the method before_trading_start() is that it is rather limited since it can only be used to update the instruments universe.

In actual trading, it is common to update portfolio metrics and send orders to a broker overnight (i.e., between 4 pm EST and 9:30 am EST the next day).

Basically, what I'd like to see implemented is:

The ability to run the methods after_trading_end() and before_trading_start() at a specific time between the end of the current day's session and the beginning of the next day's trading session. For example, if I get custom data outside of Quantopian at 2 am EST I could run after_trading_end() at 2:30 am EST:

def after_trading_end(run_at=datetime.time(2, 30)):  
    # Get my custom data into Quantopian at 2:30 am EST  
    fetch_csv(url=my_url)  
    # do some other stuff

Access to the same data as during the trading day (i.e., pricing data, fundamentals, account, orders, etc.). For example, I would need to get the open orders in order to calculate the distance to stops and see where I stand for each position or strategy and my total portfolio risk. These methods should be similar to a data event. The only difference is that the trader chooses when these events are triggered.
The possibility to send orders to IB in after_trading_end() and before_trading_start() exactly like what I'm doing in real life. It is not doable in Quantopian at the moment and this is very unfortunate. One reason is to be found in a related topic where Alisa Deychman wrote (see discussion here):

Why do orders get cancelled at the end of the trading day, automatically?
This was built as a precautionary measure to protect live traders against wild price swings overnight.

I don't understand this choice. I know that Quantopian is a social platform, but socialist? If some traders don't know what they're doing, please let them learn it the hard way and do not limit those who've done their homework.

In IB, traders can specify the time in force and whether to allow the order to be filled outside of regular trading hours, or during the pre-open session. Is this a feature Quantopian is going to implement?

I know that we - the traders - are very picky when it comes to our trading tools. You folks at Quantopian get many requests and need to prioritize, I totally understand that. Adding support for futures, stats-arb and expanding the research platform is really cool. But closing the gap between what Quantopian can do and what we can do in real trading is IMHO the top priority. Quantopian's limitations are:

can't trade on the first minute bar
can't send market-on-open (MOO) and market-on-close (MOC) orders
need to cancel all open orders at 4 pm EST/can't specify time-in-force (e.g., DAY, GTC, pre-open/after-trading-hours)
can't run overnight events

That said, I do find Quantopian amazing for many more reasons not cited here. :)

I'd love to have your thoughts on this.

Cheers!

19 responses

Alisa Deychman

Alexis, thanks for your thoughtful comments. There are so many directions the platform can grow, and many of them are very exciting. The opportunities are pretty much endless - ordering mechanisms (MOO, MOC, overnight etc), data sources (sentiment, indicator, custom, etc), asset classes, IDE features, research features, additional brokerages, information discovery and more. Of course, we'd love to pursue all of these directions. But we need to focus, and our focus is on building our hedge fund. Our current work (as you mentioned, adding futures, stat-arb and expanding the research platform) is headed in that direction. Afterward, we'll re-evaluate and select the next group of priorities as we seek to build a complete platform to unlock your creative algos for the fund.

Cheers,
Alisa

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Alexis P.

Hey Alisa,

Hmm... apparently you're not getting my point, probably because I wasn't clear enough.

I am not asking you to go in every direction - quite the opposite actually. All I'm saying is that because of a flaw in its current design, the backtester behind Quantopian doesn't allow the trader to make overnight decisions and forces him to skip the first minute of every trading day.

Here's what the Quantopian would do everyday. He would:

wake up and start trading US equities at 9:31 am
stop trading and close all his open orders at 4 pm
sleep from 4 pm to 9:31 am on the following trading day

Does it look right to you? To me it doesn't. And adding futures and stat-arbs strategies won't help, because they are all built on top of this flaw.

Is there something wrong with my interpretation? I feel like I'm the only one raising this - major - issue!

Thanks!

Grant Kiehne

Hello Alexis,

My interpretation is that your request ain't gonna even get looked at until they get some other things done that hopefully will get them on a path toward revenue (or there will be no Quantopian--keep in mind that it is a start-up with $0 of revenue, to the best of my knowledge).

So unless you have an incredibly compelling argument that leads to scalable revenue, it ain't goin' nowhere. For example, I've been asking for a full implementation of a sorta supported module, CVXOPT, for awhile now (it was released 6 months ago), and they are just full out vectored in a certain direction (which could be a really great thing, since I'd like to see them succeed).

What, exactly, are you after here? Do you need a MOO order? Or something else?

Grant

Alexis P.

Hi Grant,

My argument would lead to a better backtesting/live trading engine, one that is closer to reality.

I'm not asking them to implement fancy stuff like CVXOPT, just fixing the initial flaws of the engine. Not making money isn't an acceptable justification, in my opinion, for not fixing the flaws of the product. What I'm after:

trading on the first minute bar
not cancelling orders overnight automatically
access to pricing data/portfolio/orders and ability to send orders in before_trading_start() method

Quite the basics, don't you think? Let's wait for Alisa's answer.

Thanks guys.

Grant Kiehne

Here's an example work-around, but there isn't much of an advantage. It illustrates that analysis can be done in before_trading_start(), but the resulting orders aren't processed until after the first minute of trading. The potential advantage is that if the computation takes awhile (but less than 50 seconds), it can be done ahead of time. Then, when the orders hit at 9:31, they'll be processed immediately by Interactive Brokers (since orders are sent asynchronously).

André Pajak

@Alexis - I've been thinking of requesting a function like your proposed after_trading_end for quite some time now. You have gone much further here. I second everything you've said.

There is a clear need for functions that would run after the end of each trading day, week, month, quarter, year, etc. - to readjust algorithm parameters for the next period, and to produce and print summaries. (There should also be a function that would run at the end of everything, ie. after a backtest ends or the algo is stopped by an error or manually. C++ has constructors and destructors, Java init and destroy, Python has __init__ and __del__, Quantopian has initialize and should have finalize, with the context object as an argument.)

We should also be able to send orders, via the order* functions, at any time (even from initialize), as we do in real life. These orders should then be processed during market hours, starting at 9:30:00.000 EST/EDT on the next trading day if the market is currently closed. MOO and MOC orders would be nice, but will require Q to write, test, and approve new code; removing unnecessary restrictions on when order can be called should be pretty easy.

Similarly, the (undocumented!) restrictions on when history can be called should be relaxed or abolished.

Grant Kiehne

I think that handle_data() gets called at 4 pm, after the market has closed, so in this sense, we already have after_trading_end(), but it is only 50 seconds long. See the attached backtest as an example.

You should be able to get pretty darn close to a MOC order by delaying your order within the last trading minute (e.g. 40 seconds or 10 seconds before the time-out).

Hacking a MOO order (I think) is impossible.

Market Tech

The obvious alternative is to use a different trading platform. If you got the money and the strategy(s) then the IB API is well known and straightforward to code against. Otherwise, we've all read the writing on the wall, Quantopian is looking for very specific, narrowly defined set of algorithms with which to pad their strict ideas of an ultra-conservative hedge fund. Although your requests make sense in the actual world of an open trading platform, they have no bearing on what the Q supports or delivers.

Alexis P.

We should also be able to send orders, via the order* functions, at any time (even from initialize), as we do in real life. These orders should then be processed during market hours, starting at 9:30:00.000 EST/EDT on the next trading day if the market is currently closed. MOO and MOC orders would be nice, but will require Q to write, test, and approve new code; removing unnecessary restrictions on when order can be called should be pretty easy.

If orders sent overnight (i.e., when market is closed) are processed at 9:30 am (and not at 9:31 am), then it is close enough to a MOO for me.

You should be able to get pretty darn close to a MOC order by delaying your order within the last trading minute (e.g. 40 seconds or 10 seconds before the time-out).

Quite likely. When I get the chance, I will compare these filled prices to an actual strategy I'm trading that has over 600 fills using MOC in IB. That's my point of reference, I'll keep you posted.

The obvious alternative is to use a different trading platform.

True, but despite its current flaws, I find the Quantopian platform really fantastic and very flexible.

Grant Kiehne

If orders sent overnight (i.e., when market is closed) are processed at 9:30 am (and not at 9:31 am), then it is close enough to a MOO for me.

Did you mean "processed at 9:31 am (and not at 9:30 am)"?

Aside from using fetcher, I'm not aware of any way to send in orders overnight. And with fetcher, the orders wouldn't be sent to IB; they'd just sit in your external file until it gets uploaded automatically, with the orders going out to IB at 9:31 am at the earliest.

Alexis P.

No, I was assuming that this flaw was fixed.

Grant Kiehne

I see. So it sounds like if you miss the first minute of trading, it would be bad for your strategy? Hmm. Well, maybe there is a crazy hack whereby you would stop your Quantopian algo, enter the orders overnight (manually or automatically), and then restart the Quantopian algo? The automated approach would probably require writing a script to click the stop button, scraping the algo log to capture the orders, submitting the orders to IB using your own code, and then re-launching, using more automated browser navigating & clicking.

When the algo is stopped, I'm not sure that the log output persists. If not, you'd need to scrape it while the algo is still running.

I'm assuming that the Quantopian algo doesn't reach out to IB overnight and cancel all open orders when it is re-started.

One would have to wonder if the Q folks have a Python API to hook up to IB, but just haven't published it? It would make all of this hacking a lot easier (but maybe security would be an issue and there'd be more of a risk of licensed data being stolen?).

Just some thoughts...I know they are far from ideal.

Alexis P.

I really appreciate your efforts to help Grant, but it just doesn't make sense to even consider such a complicated approach IMO. It is the platform that must adapt to trading reality, not the other way around; these arbitrary limitations need to be removed.

Let's wait for Alisa's answer about all this.

Market Tech

Wait? Why wait when we can speculate. Hmmm, Quantopian altering the basic operation of their trading mechanics... Someday. Maybe. Not.

Alisa... Alexis is knocking... maybe you should private message her, you know, cuz all this errant public forum speculation is just, well, speculation.

Alisa Deychman

Hi Alexis, sorry about the wait, I had missed your earlier reply and thanks for the ping. And it looks like I read your reply with a different understanding :) You're right that:

the backtester behind Quantopian doesn't allow the trader to make overnight decisions and forces him to skip the first minute of every trading day.

This would require a significant rework of our current architecture, to support overnight trading and within the first minute. It's not a simple task. As I mentioned, our current prioritization is to build tools to support better algos for the fund. We are comfortable with this current structure because there are many minutes in the market hours to make decisions and derive signals. And we've seen many algos with success. Of these two requests, MOO will likely be the first candidate built. We're not there yet, but perhaps further down the road.

Disclaimer

Alexis P.

@Alisa:

We are comfortable with this current structure because there are many minutes in the market hours to make decisions and derive signals.

Wow, wow, and wow! This is worrisome that Quantopian was built on shaky ground and that no effort is going to be made to fix it. What else are you guys willing to compromise?

And we've seen many algos with success.

Of course. You can probably skip a Monday or two per month and still find working algos. It is failed logic.

@Market Tech:

The obvious alternative is to use a different trading platform.

Any suggestion? QuantConnect seems to be an interesting alternative. Apparently the platform doesn't suffer from the arbitrary flaws that Quantopian finds legitimate to keep. But it's in C#...

Market Tech

@Alexis P., Some passing advice; you might want to unsaddle from the high-horse you seem to be riding around upon. There's room for only so many indignant personas here and we, myself being one, do not care for competition (roguish grin here).

As I mentioned above, depending on your bank and time availability, there are some great alternatives. Primarily IB TWS connectivity has some solid options. Join the Yahoo group TWSAPI and find out. There's more than a decade of experience in that group. If you've got specific execution ideas and need information and help that's where to find it. I don't know which members are attached to the Q, but I'm sure they're there. Most developers in the group have written their own trading systems; lots of useful advice there. So ground up coding using the TWS API is your first option. Pick your language, they're all available.

Don't want to code your own system? Have a gander at this list:

http://www.ampfutures.com/platforms/

Probably half or more of those support IB. And if you've got other brokers in mind, well, you'll find many of those platforms are broker sponsored systems.

And of course if IB is your broker of choice you can check out their list of optional platforms...

https://gdcdyn.interactivebrokers.com/Universal/servlet/MarketPlace.MarketPlaceServlet (center click, follow the buttons).

Lastly, suffer with the Q's inadequate offering, so as to leverage their free data. Build your system using their platform, and in the mean time construct a parallel system using zipline, IbPy and a platform proxy of your design. When you're done you can take your Q strats and run them in your own system with all the extended functionality you chose to add. I've predicted that someone is already doing this and will soon offer their solution to the community -- soon. But that's just more speculation...

James F

Quantopian, when will you support trading at the market open (9:30am) instead of 9:31am, which is currently the case?

There is significant opportunity lost by not allowing to execute at the market open as prices can change tremendously during the first minute. In addition, evaluating pre-open prices (extended trading hours) of various instruments prior to market open is important.

Jas Chimni

folks, I read through some of the comments on this post. I wanted to know if there is a way to execute orders() during pre-market or after-hours.

use case: LinkedIn stock price surged 49% pre market after Microsoft acquisition. is it possible for me to buy the stock right after the announcement?

Please let me know. If we want to level the playing field with big wall street hedge funds, we should be able to execute trading orders anytime of the day.
With regards to sourcing the signal, I will need to either subscribe to one of the data feeds available on this platform or build a custom one on my own.

I am personally willing to place a bet on the stock market that the moment acquisition news hits the "news wire", buy the stock or stock options with the hopes that the algorithm will beat the market before the bid/ask jumps from $135 to $190 (in LinkedIn's case)

You've successfully submitted a support ticket.

Our support team will be in touch soon.