Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Disaster preparedness @ Q

This was submitted into another thread just 4 days ago. I think that in view of Friday’s DDoS attacks, it deserves a “promotion”.

From the viewing angle of real-time live trading, the requirement to submit Good-Til-Canceled (GTC) orders is rated the highest! GTC is the life-boat that may save our live portfolios.

Once in a while there will be a disruption of the connection between Quantopian and the broker’s server (see http://status.quantopian.com/).

Such loss of communication will require the user to re-enter Login Credentials in order to re-connect / re-launch the algorithm. This may take time, especially if you are not sitting next to your workstation daily.

Should such a disruption happen, GTC orders are the only way to liquidate positions, autonomously by the broker, before the algorithm owner initiated the re-connection (or even before being alerted by Quantopian that such outage happened).

With GTC orders, users shall keep such liquidation orders at about 5% below current price. Then, regularly whenever each asset price shifts (up or down) update such GTC liquidation orders. Working in this manner is the only feasible watchdog to save us from holding undesired positions which would have been sold by the algorithm (had it been connected to the broker during the disruption).

Prevention is better than cure: having such kind of orders stationed
in the broker’s system, will keep all of us safe in case of trouble.

Technology might fail; for responsible live trading, you can't do without Good-Til-Canceled orders.

Thank you in advance.

9 responses

On https://www.quantopian.com/posts/big-news-for-the-quantopian-community-managing-external-capital , Fawce mentions:

The selected algorithms will connect to a prime broker through an order/execution management system (O/EMS) at Quantopian.

Presumably, there will be an overlay of protection against infrastructure failures, since shortly, they hope to be dealing with $250M in capital (times leverage), from which they can profit directly. They'll need to figure this out; maybe the solution will be across-the-board, applicable to retail traders, as well. Not a new problem in the world of institutional algorithmic trading, I'm guessing.

Hi Grant,

I do hope this will be processed soon. I’ve noticed you were discussing this very issue with Fawce a couple of years ago:

Thomas, Grant,

Great catch, thank you.

The orders are Good-Til-Canceled. As we have been working on our live
trading interface, order management has emerged as the biggest
discrepancy between our simulator and live trading. We need to add
limit and stop orders, time in force, access to the current open
orders, and order cancellation. Please let us know if you find other
holes!

thanks,

fawce

Back in 2012, Fawce indicated orders were GTC. A couple of years later, live orders are only good for the day. Users and investors alike unknowingly risk their life savings. This should have been already rectified.

The idea behind 'low frequency' algo trading is that it gives you the essential peace of mind. You do not have to stay tuned every minute of the trading day; just program your robot and let it do keep an eye on your assets for you.
A non-trustworthy system is VERY FINE for developers and for ‘funny-money’ accounts. However, before commencing live deployment of capital, you will need a reliable system structure .

Unlike developers, investors should investigate robustness first. Only then they would check the algorithms' financial claim to fame.

Have a great week.

Hello Mook,

It is very true that servers sometimes go down, connections to brokers go down, and even brokers themselves go down. A wise investor should prepare for that.

If I understand your post correctly, you want to prepare for that using liquidation orders, also known as stop-loss orders. Quantopian does support that with our current system.

What you can do is place your liquidation orders every morning when the market opens. Those orders (stop or stop-limit) will rest with the broker. If, for some reason Quantopian goes down, the orders remain effective. Of course, at any time and for any reason, you can place additional orders through your brokerage. I believe that process entirely fill the need that you're describing.

We don't have any plans to support GTC orders in the near- or medium-future. For the foreseeable future we will rely on the algorithms to place any necessary orders on a daily basis.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Dear Dan,

Thank you very much for taking the time to look into this matter.

Almost every exchange related investor is working with some kind of a broker. I do acknowledge that those entities are having technical difficulties from time to time.

However, at this moment I wish to focus solely on Quantopian. Primarily because it is in your hands, and also because it is one of the youngest (and hence, not as mature) and because it might (by force majeure, design flaws, etc.) add another layer of risks which I wish to curtail (I trust this is your prime interest as well).

As an investor I do not wish my hard-earned portfolio to navigate the
"big blue" with a DINGHY on board.

I’d rather see it cruising with a LIFEBOAT.

THE DINGHY: The Good-for-the-Day stop loss mechanism you have indicated is generally fine. However, it does not provide the level of security investors usually consider essential. Had the liquidation condition happened following service disruption and prior to the daily end of trade, it will function perfectly. Such a “dinghy” can “save” assets, only on the first day of disruption, at the time of trade. Once the trading hours are over, had a disruption occurred, the Good-for-the-Day orders are all gone and hence should not be considered as a reliable mechanism for saving portfolios.

THE LIFEBOAT: However, in order to be really really safe, we need to respond to the following case:

The liquidation condition would have been triggered on a trading day
following the identification date of a disruption event, and prior to the
disruption event’s resolution.

In such a case, only Good-Til-Cancel (GTC) orders that “will rest with the broker” may save our assets.

EXAMPLES: Plenty! I just wish to avoid saying “If I had more time, I would have written a shorter letter.”…

IMHO, indicating that “If, for some reason Quantopian goes down, the orders remain effective” may have a misleading interpretation.

From the portfolio safety point of view, relying on Good-for-the-Day orders is a bug, not a feature…

I (we?) need a reliable watchdog to autonomously respond to service disruptions.

Thanks again for addressing this important matter.
I hope Q will re-think R&D priorities and I am looking forward to continue this very significant discussion with you and the community.

Hi Mook,

If I understand Dan's solution correctly, there would be no way to have protective orders in place at the market open, since all orders would have been canceled after the prior close. I think this means that since the market opens at 9:30, and you could not submit the orders until 9:31, you'd be exposed to loss for a little over a minute per day (since there would be a slight delay in getting your orders to the broker). This assumes that your algo was able to connect to the broker, however. To cover this risk, I suppose you'd need to enter orders in overnight manually to the broker? Is this feasible? How would the algo handle open orders when it starts up the next day? Could they be queried from the algo, even though they were entered manually?

Grant

Hi Grant,

Thank you very much for your message. I will try to address the issues one at a time.

ONE:
On its own, a protective (Good-for-the-Day) order placed by the algorithm is a significant feature.

This can be processed at market open or at any other time. But this feature is not reliably providing any cure to service disruptions.

TWO:
What I am looking for is a protective order which be placed on the broker’s system by the algorithm. Such an order may be updated as long as the algorithm can access the broker (prices might change either way, and the algorithm should respond accordingly).

Had a disruption occurred, only Good-'til-cancelled (GTC) orders may contribute to the portfolio safety.

THREE:
”you'd need to enter orders in overnight manually to the broker” - Right! Given Dan Dunn's response this is what you'd do "in the near- or medium-future". And, also this is exactly what I am trying to avoid!

If one needs to attend the algorithm daily (in order to make it safe)
then the entire service, provided by Quantopian, is alarmingly decayed.

IMHO, supporting GTC orders is critical. It will turn Quantopian into a thriving robust service, in line with the interests of any rational, risk averse investor.

Hi Dan -

I'm still trying to digest your comments above, particularly:

Of course, at any time and for any reason, you can place additional orders through your brokerage.

What are the mechanics of this? I was under the impression that once an algo is running, a user would need to stop it in order to place orders manually (or with a different API). And then what happens when the algo is re-started? If this is at all feasible, is there guidance on when it would need to be done?

The process would look something like this:

  1. Market close. All open orders cancelled.
  2. User stops Quantopian algo and connects his own script and submits orders (or performs this step manually).
  3. User disconnects his script.
  4. User re-starts Quantopian algo.
  5. Market open. Orders placed in step #2 are still in place at broker (and presumably visible to the Quantopian algo, even though it did not place them).

How are you handling this for the 15 black-box algos you've funded at an average of $100K each ($1.5 M total)? As I understand, there is no requirement to code risk management into the algos, and even if there were, the issue raised by Mook would be extant. Are you trading them through an IB account that allows you to overlay protections? But then this would seem to be pretty blunt, since if you don't know the mechanics of the algo, how are you approaching it?

We don't have any plans to support GTC orders in the near- or medium-future.

Does this include algos selected for the fund? Seems kinda risky, no? At 6X leverage, you'll have $1.5B swirling around, if Point72 puts up $250M.

Not trying to stir the pot here. The facts, as they swirl around in my head, just don't quite add up.

Hi Dan -

Another comment is that you might provide some context to your statement:

We don't have any plans to support GTC orders in the near- or medium-future. For the foreseeable future we will rely on the algorithms to place any necessary orders on a daily basis.

The question has come up before, and I recall that the explanation on the forum by Quantopian was to protect users by cancelling orders at the market close, since conditions could change overnight. Is this the basic rationale? Or is there something about the mechanics of your system that requires it? Or maybe it is hard/expensive to implement at IB? Something else?

This has been a bit of a mystery to me. Perhaps you could explain your approach.

Hi Grant,

Thank you for your messages. I certainly share your concerns. Prior to deployment, everybody should.

Pertaining to “conditions could change overnight” -- I do like Dan Dunn’s response: “What you can do is place your liquidation orders every morning when the market opens”. Generally it will serve your objective with no difficulties.

Unfortunately it does not handle dormant algorithm situations; as a designer, or investor, nobody will give you assurances that a dormant algorithm will never happen.

Hence, designers should prepare for such situation. Investor’s just need to ask designers: “have you considered…”

The smoothest method to overcome such mishaps (as in any other critical real-time system) is a watchdog mechanism. Usually a watchdog timer is used; in our case we need one watchdog order per each portfolio asset.

I’ve taken the liberty to adapt the Wikipedia definition of Watchdog, to the Quantopian environment:

The watchdog is used to detect and recover from computer malfunctions. During normal operation, the algorithm regularly resets the watchdog order to prevent it from triggering. If, due to a hardware fault, communication or program error, the computer fails to reset the watchdog order limits, the order will be executed. The execution is used to initiate a corrective action. The corrective actions typically include placing the portfolio in a safe state until normal system operation is restored. Liquidation or other actions may be dictated by the algorithm designer, whatever the designer considers as the safest state for the portfolio.

BTW, watchdog orders are Good-‘Til-Cancelled. Such orders are stored in an auxiliary system (the broker's).

For as long as the broker and exchange are functioning, whatever happened to the national grid, the internet, or the Q server, will not even scratch such orders.

I hope Q will soon demonstrate a trustworthy method to enable designers to keep investors away from turning a transitory loss of control, into a CALAMITY

Have a great weekend!