It's definitely possible to do this, if your computations are optimized.
If you're running into timeout errors, it's likely that you're using loops instead of vectorizing your computations with numpy/pandas.

A sample skeleton of what it sounds like you're trying to do would be this:

1. Gather the past 2 minutes data into a pandas Dataframe
   
2. Do dataframe-level comparisons to select the relevant securities.
   

3. Do your orders/cancellations with the .apply() method of your indices

   Now with your slippage models, you might run into problems with ordering (for backtesting anyway).
   Not sure if this answers your question, but cancelling orders are done asynchronously -- sent immediately.

Also the comparisons of whether an order has been completed can be found in the Order object under the .filled attribute.

You can view the source code for Zipline at the github repository.

"but cancelling orders are done asynchronously -- sent immediately"

Why not add order type logic here then? How do you expect one to code something half resembling real trading, if OCO (one cancels the other) orders for stop loss and profit taking are not supported?

Dear Trader,

Quantopian does not robustly support all styles of trading - for instance, if there is a strategy that depends on low-latency order placement and replacement, we aren't a good match. That said, I'm quite confident that we do support many "real trading" families of strategies. For instance, with the addition of the Pipeline API and the performance improvements that came with Quantopian 2, we enable a wide range of stat-arb long/short strategies with as many as 1000 or 2000 liquid names evaluated in the strategy.

We also are regularly adding to our capabilities. We listen to the community and engage with industry experts to identify areas that we should tackle next. Expanded order types (OCO, one cancels many, and other more exotic types) come up sometimes. So do requests for trading futures, options, foreign exchanges, order on market open, expanded machine learning capability, etc. We sort through them all and work on them one or two at a time. We value your feedback on which is most important.

On your pipeline timeout problem: Please contact [email protected] with any specific algorithms that are timing out. I keep a careful eye on our error rates, and they haven't moved much since pipeline came out. If you have specific algos that aren't getting run then we'd like to investigate why.

Back to your original question about how "much" an algo can do on Quantopian. The answer is "it depends." I think of it as a multi-dimensional space. One dimension is how many equities you're looking at; how many orders you place; how many positions you hold; how much computation you do per minute; how many history calls you do; how liquid your equities are (price lookups for illiquid securities take longer than ones that are regularly traded); and more. If you write an algorithm that maxes out all of those axes, it will not run. But if you write one that maxes out one axis, but is very fast on the other axes, it will run fine.

The only way to know for sure is to write it and test it. If it doesn't run, bring it to our attention. We can see if there is a fix or optimization available.

Thanks for your thoughts.

Mr Dunn, thank you for your comment.

As I mentioned above, the proposed algo should examine 4-5K stocks, compare their current price and the price (OHLC) from the prior minute, create up to 200 (just a guess) new buy/sell orders per minute with a profit taking and stop loss bracket, and time stop closing order, the 3 of them in "one cancels all others" fashion. All open orders should be closed at the end of the day if no targets are filled - the time stops will have a max of market close minus a few minutes. It should also check the price of at least one index (or index ETF) and make a decision using that input as well.

Depending on what comparisons the algo will make to the prior minute OHLC, it could need to make up to 20K price comparisons per minute. EDIT: it probably needs to check volume as well, so make that 40K comparisons per minute.

Depending on the algo criteria, it could create up to 200 new buy/sell orders per minute - this is just a guess for argument's sake. It could need to cancel up to 400 orders per minute (the one left over from the OCO and the time stop, or both from the OCO), and execute up to 200 closing orders, if everything goes right and profit is taken.

This is definitely NOT meant to be a low latency algo, but a relatively large scope market making algo, (better term for it is risk taking) on a minute basis.

Is this type of trading supported on Quantopian?

Thanks,
BT

Hi BT,

I suggest writing some skeleton code to test the computational capacity. My hunch is that hundreds of orders per minute will work (easy to test). And if you can vectorize your price/volume computations, then perhaps your 40K comparisons per minute will fly (kinda depends on what you mean by "compare").

Can you mock something up, without nitty-gritty coding? Then you could get a feel for the limitations.

Grant

Hi Grant, from what I read so far, the order logic (one cancels all) is not supported, so this means that the rest of the code and its performance are meaningless. When and if such orders are supported, it will make sense to think about this sort of algo. It is too bad, really. BTW, I was able to code this on TD Ameritrade Strategy Desk but it is no longer supported.

So why did TD Ameritrade Strategy Desk drop the feature? Just curious.

Sorry, they didn't drop just the feature, but they dropped the entire platform. I don't know why exactly....they dropped the Strategy Desk retail platform (not to be confused with their Stategy Desk group, for insititutional advisors), and they do not offer backtesting or auto trading any longer. Hence my running around to find another platform for automated trading.

You could consider virtualizing the order processing, so that it all takes place within the algo. For the one-cancels-all, you'd have a set of pending orders (not submitted to the broker). You'd pick the one most likely to be profitable, submit it, and drop the rest.

"You'd pick the one most likely to be profitable, submit it, and drop the rest. "

It does not work like this in real life. You need to have active pending orders resting on broker's servers so that if one hits, the rest are canceled. You need to be able to set this up via Quantopian, and push it through to IB, or whatever broker. IB is more than capable of accepting such orders but Quantopian is not capable of sending those orders, from what I have read so far. Also, Quantopian is not capable of supporting this order cancellation logic in back testing, but as you suggest, it might be possible to do with some complex code, which is out of my domain of expertese.

Yeah, that's kinda what I figured. It's code that runs on the IB server that you're after.

"It's code that runs on the IB server that you're after. "

Grant, not at all.

This is not some "special IB server code". It is basic conditional order logic. Other orders get much more complex than the one I am asking about. Your most rudimentary online broker lets you use this type of order. Quantopian does not let you use it in backtests, or for real orders. This is a major issue in my opinion. Quantopian will disagree, I am sure.

MT -

If you read https://www.quantopian.com/about, you'll see that Q is not really a retail trading outfit. My understanding is that they want to set up a $10B fund, based on licensing the IP of its users (or maybe retaining a cadre of them full-time, at this AUM level). So it is apples-to-oranges to compare Q with "your most rudimentary broker." It is also my understanding that IB wouldn't be the best choice at this scale. At some point, I recall one of the Q folks mentioning that they'd move to a so-called prime broker. And one could imagine that they'd even port algos over to other platforms, better suited for institutional-style trading. I have to think that this all plays into sorting out how many IB order types to support at this point.

I think they'd be better served by working on the slippage model, at this point, since I keep hearing it is way off.

Just my two cents...

Trant, IB does prime brokerage, and it can be a great choice, since they excel in trading technology (prime brokerage is not "so-called", but a requirement for large funds). The limitations are not in the IB side, but on the Quantopian side. Quantopian needs to support rudimentary order types, such as the stop loss and profit taking OCO conditional order, if it wants to be taken seriously. No matter who the prime broker ends up being.

I agree with you on slippage, but I am disillusioned on what can and can not be done on Quantopian, that is all.

PS: If they hire, it will be some propeller head in China or India, where they can use purchasing power parity to pay as little as possible, but WHY should they pay if they can get all this intel for free already? (It is a rhethorical question)

I speculate that they are NOT after a market making algo, given the supported order types.

Let's consider this thread closed, until and unless quantopian adds complex order types. Thanks for all your notes and opinions.

@ Lotanna,

Is there a good example of using the .apply() method for ordering? Just looking to be pointed in the right direction.

Thanks

Frank

Hi Frank,

I noticed there aren't very many examples of using the .apply() method for ordering, so I went ahead and wrote one.
It actually implements the one-cancels-the-other (OCO) bracket logic that BehavioralTrader was wondering about.
(Sorry in advance if there's a bug in the code.)

A lot of this implementation isn't specific to Quantopian, but rather numpy/pandas. Here's a link to the documentation page on .apply().

The modules really show their strengths when it comes to computations like this one.
If you have any questions about the implementation or the .apply() method, feel free to ask.

Cheers,
Lotanna Ezenwa

Found a bug.
Updated to use cost_basis as opposed to price.