Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Missing Minute prices?

Are missing minute prices simply an indication that no trade has occurred and we can forward fill those?

1 response

When requesting minute price and volume data, either in an algo or in research, most of the fields will not return data for minutes when no trades occurred. Typically the result isn't nan, but rather simply no row or index for those particular minutes. No data is an indication that no trades occurred during that minute. However, most of the methods allow for a field called price. This will automatically forward fill the close price. The data.history method in algorithms allows for this field as well as prices and get_pricing in notebooks.

A small footnote. All rows will be returned when requesting the price field with no missing values. If one were to request both price and close then close may have nan values when there were no trades. This is because the price field forces all rows to be returned.

So, to forward fill or not to forward fill? That is the question. The bottom line with forward filling data is you are adding fictitious data. You are adding 'trades' that didn't really exist. You are changing the data. This isn't always a bad thing but do be very aware of how this may impact other calculations (and potentially trade decisions).

As an example (albeit an extreme example), take a stock which had three trades during the day - at open, 10 minutes after the open, and at close. It traded for $1, $2, and $1 respectively. The returns would be 100% and -50%. The average price would be $1.30. With the huge swing in returns it would be a highly volatile stock. Now, let's do those same calculations using forward filled prices. There would be 9 prices at $1, 380 prices at $2, and 1 price back at $1. The returns would be almost all 0% (387 at 0%). The average price would be much higher almost $2. This would be considered a very un-volatile stock with almost flat returns. There really isn't a right or wrong, and this is almost a silly extreme example, but do be aware how forward filling changes things. Forward filling smooths out the returns and skews the average prices (among other things).

It's a good observation to notice minute values may be missing, and it's very important to explicitly consider how those will be handled. I personally was frustrated working through a 'bug' in some code the other day. The first 30 minute return was being calculated as close[30]/close[0]. If either the open or the 30 minute close price was missing this wouldn't return a value, and moreover, would therefore drop the stock from my 'universe'. It took awhile to understand how I started with 100 stocks but then sometimes had only 90. I forward filled the close prices and then had the expected 100. Not to say forward filing is always the answer but it does have it's place.

A final note, it may be wise to simply drop any assets from one's universe if the number of missing rows is above some threshold. Perhaps 20%. This would ensure only 20% of 'fictitious' data is being introduced by forward filling. Moreover, having a lot of missing data implies that a stock is very thinly traded. Backtest results for thinly traded stocks cannot be relied upon to model real world trades.

Anyway, good question!

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.