Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Dealing with partial fills / low liquidity? (Sync, async, order modification pains)

I'd like to make my algorithm survive a situation when it hits a low-liquidity security. I will, of course, check volume and use 2.5% slippage defaults to be on the safe side, but I expect that sooner or later I may hit a bad volume print and end up entering an order for a very slow stock. How would you recommend dealing with such situation? Would you just abort the algo and manually exit, or have the algo exit?

The problem I see is that without callbacks it's very hard to deal with partial fills and synchronization of market/limit and stop order sizes.

Example: Let's say my algo placed an order to buy 100 with a stop loss sell 100. Due to low liquidity, after time X (end of day, or say, 5 minutes after the order is placed), the algo detects the following:

  • 34/100 of the market order is filled
  • 12/100 of the stop is filled

actions:
1. log alarm
2. cancel market order
3. reduce stop order size to the filled: cancel stop order, place new stop order for the remaining size (34-12 = 22)
at the next bar (after new orders have been sent out):
4. check that the cancellation of the market and stop orders have happened (alarm if not?)
5. since there may have been more fills between the initial fill observed and cancel orders (e.g. we got 36/100 and 15/100 before both got cancelled), I need to adjust the stop order size again (cancel stop of size 22 and place new stop of size 21)

It's just pretty clumsy and hard to do, considering that after placing new orders one needs to wait for the next bar, verify positions and order states and adjust again and again. It seems that having order modifications and partial fill events would help to code around things like that easier.

I am sure this problems comes up in any algo which tries to cancel a partially filled order and keep it working with a stop (without exiting right away).

What would you recommend? Am I overthinking this?

12 responses

These are all real and tedious problems, but in the 100 share range, actual executions should be much, much faster than quantopian's simulation. The real annoyance is early in the day, not being able to place orders at all for things which have not yet traded.

FYI the above problems keeping stops and limits synchronized with positions is why I gave up on algos which depend on that sort of granularity with quantopian.

I see. Makes sense. (I used 100 as an example)

I think it's extremely unrealistic, and I'm starting to feel like there is something going on here. I'm a trader and I often trade penny stocks with over 1 million in volume breaking new highs. I get in with limit orders, sometimes as large as 30,000 shares. I exit, maybe a penny or two with a limit order and I can make a coupled thousand a day with this strategy. NEVER have I ever encountered a situation where I couldn't get out of a position. When you write algorithms against this platform, you can't get out period. How is that realistic if I'm trading the same stocks daily and I'm trying to write an algorithm that doesn't fill ever!!!!

To add to what L.Williams is saying I too find it odd.When I was trading manually I didn't have issues getting filled as I've been trading with an account that is $10,000 or less trading stocks that have a quarter of a million in volume. Despite not having trouble getting fills when trading manually I'm having trouble getting fills when backtesting.

Williams, have you tried paper trading to see if the issue persisted even in paper trading ?

I was able create two algorithms that operate on low float stocks. The first I don't care to share, until I've made myself rich. The second however, doesn't seem to ever be able to obtain fills going long, and logic for this is as follows:

  1. Set slippage to 0.0
  2. Filter for stocks that gap up at open greater than 5%, with a price between $1 and $10. This will ensure that you catch the biggest mover on a regular basis.
  3. Filter by volume, and select the top four, ensuring an average daily volume over the last 10 days of at least a million.
  4. spread your funds equally across the selected stocks, and go long, setting a stop loss of 9%, and a target of 100%, and ultimately closing 5 minutes prior to close.

Now here is what should happen:

  1. About 10% of the time stocks lose 9% or more.
  2. 10% of the time, you should see huge gains on at least one stock.
  3. Most of the times you will lose, but the large gains should offset the small losses

Here is what actually happens:

  1. Orders never fill at a reasonable price, even when it's clear that stocks close much higher then they open. It doesn't matter if you use a limit order or marker order.
  2. Even when you encounter stocks that finish higher than open, you seem to always end up with a loss, or orders that just don't fill.
  3. You end up losing everything.

I've actually given up on trying to go long on these stocks, simply because I can't get a reasonable fill, not because I can't write a good algorithm. And if this was truly leveling the playing field, I would be able to get my fill. Do I think this is reasonable? No. My honest opinion? I think it's done purposely, but of course I can only speculate. However, I feel that corruption is everywhere, especially when we're talking about the illusion of leveling the playing field. This will never happen, when monster funds have their systems on the same network as the exchange, and can see your orders, and execute prior to you. Think about it? Why would the 1% allow individuals like us from the other 99%, to create the sort of algorithms that can generate the sort of capital that would make us all rich. All the algorithms on here that you see, move at a snails pace. Nevertheless, they left a gaping hole they forgot to cover, and I fully intend to exploit it. What this platform has done, for which I will always be grateful, is provide a framework for me to research the probabilities of movements associated with specific conditions, allowing me to build algorithms based upon extreme fear and greed.

By the way, watch out for leverage. I've noticed that if you don't put a cap on this, money will pop out of thin air and you will rejoice thinking that you've found some great exploit. I don't know why the platform is engineered this way, but it's worth nothing. I typically use record(leverage=context.account.leverage) to ensure I'm using no more than 2 for swing trading algorithms, and 4 for day trading.

I wouldn't be surprised if you're right regarding it being done purposely unfortunately, there is an overall incentive in general to have members trading larger cap stocks as the purpose of Quantopian is to devise strategies for million dollar accounts not small accounts trading small caps. A deterrent in place(such as excessive slippage) to further incentivize traders to trade larger caps would make sense as it's inline with the overall purpose of the platform.

As for leverage I actually messaged help. The solution they offered which actually worked for me was adding to leverage stored in a variable for the amount I ordered.

see bolded below:
leverage =context.account.leverage
for s in context.security_list:
if context.portfolio.cash > 0:
if leverage <=1:
order_target_percent(s,0.10)
leverage+=0.10

I hear you, but I've never been a big fan of handcuffs. I like your leverage solution as well. I simply set my condition to:

if context.account.leverage == 0:
order_target(s,shares,style=LimitOrder(cp))

Now here is what should happen:

L. Williams,

Why do you think that "should" happen? Do you trade this way live and it works? Also if it works live then it doesn't matter what the backtester tells you, you can program your algorithm to execute the trades live and they will function the same way. You can also set the backtester's slippage model to execute better, not just setting slippage to 0 but by setting the volume limit manually to 1.0 so the backtester uses all availible volume.

Precisely. There are an abundance of variables, but I can build an account this way through momentum trading. The only missing variable here is the news, but for the most part the criteria I provided isolates the same. I estimate that the problem may lie in the minute threshold. In real trading, when handling low price stocks, a minute is an eternity. Usually when I place an order, it gets filled in seconds. Here, when you place and order, it get fills in the next minute. And in back testing, assuming you don't set your slippage, it may fill over several minutes or not at all. This behavior absolutely infuriated me when I first started coding on here, as I didn't think it really emulated real trading. I think Leigham is correct is that the way the platform is set up, it's actually biased towards large cap stocks. It's not that you can't make money doing that. You absolutely can. However, the greatest gains and losses you can incur will always be with low cap stocks. There is absolutely no question that the biggest moves emulate from stocks between $1 and $10 on a regular basis. And my research has confirmed that stocks in this range that exhibit five times or more of their normal daily volume on upward gaps are typically your big daily winners. So yes, I do trade this way, but my point is that if you follow the back tester, it would deceive you into thinking you're doing something wrong, when in reality you may have done something brilliant. That is NOT leveling the playing field. It's leveling your development community.

And lastly, before I forget, aside from the next minute fill delay, I can't trade pre or post market. As a trader, I have made some of my biggest moves during these times, and there are times if I see news early, I will actually get into the pre-market at a limit order, an get out before or at market open, depending upon how wide the gap may have grown. This is because momentum usually builds prior to exploding or collapsing out of the gate. I can't do either of these things here, and it is a huge draw back to trading low cap stocks where getting in an out at opportune times is imperative. It is also a hindrance to trading large cap stocks. For instance, if a stock like Nike or Amazon misses earnings and there is a panic sell off, I would like to put a bid in to either double my position, or buy at the discounted price. I can't do this either. Alas, the force is just not with me.....