Kevin, not sure exactly what you're seeing. But when a security isn't traded in a given bar (minute bar or daily bar), then the price information is forward-filled from the previous bar. So I'm not sure what would be causing you to see a large drawdown.

We don't have bid/ask data, only as-traded.

If you have a hedged position and one leg trades while the other doesn't, you show a drawdown because system is calculating a loss based on the stale price. When the untraded piece of the hedge starts to trade again, you see a corresponding gain on the hedge equal to the drawdown so it has no impact on the final gain/loss calculation. However the algorithm has been penalized for the drawdown that in fact never actually occurred.
For example, suppose you had a simple (pointless) position of SPY and another S&P 500 tracking index that had low volume and only traded every other day. Assume both perfectly track the S&P 500. You put on an equally weighted long position in one and short the other. The max drawdown should be 0, however if the S&P 500 fell by 3% today, when the second stock didn't trade, Quantopian would record a 3% drawdown, even if they both went back to exactly matched tomorrow with a total gain/loss of 0. This is an admittedly extreme example to make the point, but it is a big issue nonetheless. Even liquid stocks can go several minutes without trading (witness the SPX futures AM settlement process!), in which case any hedged position is inaccurately recording phantom drawdowns every time one side of the hedge loses money in a minute that the other side of the hedge doesn't trade.

That's a very interesting point. When I brought it up here, I think other people on the team had recognized it, but it was a new concept to me. It does highlight the difficulties of modeling illiquid securities!

Changing the underlying infrastructure to use bid/ask instead of trades isn't a trivial change - it would be a big one. We'd be re-writing much of our core code. If we get a lot of requests in this direction, though, it's something we'd work on.

Wouldn't even using the mid be problematic too? What if no orders at all entered the book for the illiquid security for the illiquid period? The ffill would be pressing the mid forward (as the synthetic trade price) and you'd still get your hedge imbalance. This issue may just be part of the risk of trading. What if you thought you'd close your pairs trade out on the period where security B had zero bid/offers? But maybe you didn't know that. You get out on security A, and try to close B and,,,, you're legged. Just part of the risk I'd have to say.

I agree it would be a big change to incorporate bids and asks, although my experience is that midpoint can get you a much different result from last trade so its worth doing eventually. I'd definitely rather see options and futures first, however you really will need to go to bid/ask in order to do options right because there are just so many that they're almost all technically illiquid securities. I know the data is out there because its the standard in academic research, but I can also see the impact it would have on your codebase. Just wanted to get the limitation out there, maybe another use can come up with a better way to solve the issue that doesn't require bid/ask data. Thanks again for your responsiveness and openness to suggestions.

Hi Kevin,

I understand your hedging example, but I don't understand how you could be seeing "dramatic phantom max drawdowns caused by a stock not trading" when dealing with a single security. If the price is forward-filled, wouldn't the drawdown be zero from the last trade?

Grant

The price filled forward on the stale quote doesn't reflect the actual market price that minute; it could easily be outside the current bid/ask, for example. The hedging security, on the other hand, is reflecting the real market price that minute. This difference between the stale carry forward price and the current hedging stock price is reflected as a drawdown, even though the drawdown instantly goes away the next time the stale security trades and prints a price. If you have a bunch of these type of hedges it could show a big max drawdown, at the open for example, even if it was all due to stale pricing. Its easy to dismiss this as an illiquid stock issue, however if you look at AM settled index futures like SPX you can see that even a large number of S&P 500 stocks don't have their first trade until sometimes hours after the open. As a result, the SPX settlement price, which is based on the first traded price of every stock in the index, is often substantially different than the opening index price and in some cases doesn't match any price the index traded at that day.

Imagine two securities that ought to have the same price, arbitrage should keep their quotes in line, even if one or the other never trades. If one is actually executing that arbitrage, then one's portfolio value is better calculated as the immediate liquidation price, buying at the ask and selling at the bid. If one is (erroneously) valuing the portfolio based on the last traded price, and the last trade is several hours ago, you'll have an imaginary drawdown. I used my excellent artistic skills to draw an example in MS Paint:

http://imgur.com/YGsTHXf

Note this problem appears in reverse when looking for mean-reverting spreads or baskets using last trade data: those "drawdowns" then appear as profitable departures from the mean, but actually they are completely untradeable, since you can't execute the infequently-traded leg against its last trade, you need to use the current quote. And this is of course what lead me to make my own system in the first place.

Simon, I'm thinking that image needs some sort of Andy Warhol award...

So it the fact that ffill is pressing the stale last trade price forward, and a system using such prices would be unaware that said price is in fact stale, that a primitive system would make decisions (using inaccurate data) that would lead to inappropriate trades. A smart system would monitor the age of the prices, realize that the arb it's about to make may be dirty due to stale data, and either skip the trade, or alert the master chief trader such that they could take action against this sluggard of a illiquid security.

It sounds like, and I've known this intrinsically for years, that forward filling prices is a hack used effectively to fill in minor gaps in data.

When we download data from various sources we have a limit to the number of periods we will forward fill in preparation for storage and use in back tests. When the data is then used in a strategy, the unfilled gaps, actual missing time stamped bars, trip up the historical market data provider module, giving the strategy a heads up that "Hey, no data for the period under test for this security."

Knowledge of the gaps that have been ffilled, the widest gap, and the total number of gaps, might be useful to gauge the fragility of a hedge or arb setup.

A related problem with mean reversion trading based upon on last traded prices is simply mistaking bid-ask bounce for spread movement. If the inside spread of the spread/basket is the quite wide (not uncommon), one might mistakenly fit the model to last trades by effectively assuming the mean is the midpoint and every departure from that midpoint is an excursion to be traded. The reality is that the basket itself has a bid/ask that one must execute at, and any excursion which is not well beyond this is not tradeable unless market-making. I will draw a picture of that in a moment too.

EDIT: Picture: http://imgur.com/1WPMwFD

Simon, are you actually able to build such a nuanced system within Quantopian? It would seem that the pairs and basket arb ops you're describing are fleeting in nature. And I do mean fleeting, zip - they're gone! To build such a system, in my old environment, we'd need to consume tick data, or at least build it to test real time and tune as it worked the synthetic spread.

Correct, these are all issues that cannot be solved within this framework here, at the moment. I'm just pointing them out because if folks go forward with these sorts of systems here, they won't realize it's not going to work until they start trading with real money. Even QuantConnect tick data wouldn't help, since it's also only last trade, as far as I know.

If doing trades using infrequently-traded ETFs or stocks, these issues can even percolate all the way up to daily data...

I guess the moral of the story is, around here, stick to liquid and continuously-traded assets!

Market Tech,

On https://www.quantopian.com/posts/thinly-traded-stocks-why-no-gaps, Fawce gives an example of how to detect forward filling:

# http://seekingalpha.com/article/123969-five-thinly-traded-stocks-with-plenty-of-upside  
def initialize(context):  
    context.stocks = [sid(3722),sid(33054),sid(16812),sid(5107),sid(1675)]  
def handle_data(context, data):  
    for stock in data:  
        if data[stock].datetime < get_datetime():  
            log.info("no trades in {s}".format(s=stock.symbol))  

Also, with the history API, forward filling can be toggled on/off.

Grant

@Grant, thanks. Useful to know.

Was I was referring to was that some filling is OK, but not too much. That's a tough one to incorporate here it seems. Although I suppose one could do that date test and keep track; manually forward filling X number of periods before you throw up your hands and say "Uncle", gap too big.

Hi Dan,

Regarding bid/ask, if your vendor could provide them, wouldn't it just amount to adding some additional fields to the history API? To see the bid/ask values for every bar, a "clock" such as SPY would be added to the universe. You would still fill orders using the trade data.

Grant