1-1. Analysis of Normal Data¶

Below is an introduction to the language and tools for analysis of normal data that will be used in the rest of this notebook. Our normality test is the Jarque-Bera, and we will be using the size of its test statistic as a sample-size adjusted measure of how close to normal the distribution of our data is. The null hypothesis of the JB test is that the data are from a normal distribution. In frequentist statistics, we can never accept the null. If the p-value is zero, smaller JB test statistics indicate greater degrees of normality, while still being decidedly non-normal. If the p-value is non-zero, and fluctuates with the test statistic as the test is run multiple times, then the test has failed to reject the null hypothesis. This is pretty good evidence, but by definition not proof, that our data are at least quite close to normal.

When normal data are tested on, we see a chi-squared distribution of both our test statistic (first return value) and p-values (second return value). (Well, they just appear random here. If you were to run that cell dozens of times it would show the distribution). This is to be expected, as the null hypothesis can never be confirmed via a frequentist test.

Here, the test statistic is massive and the p-value is 0. This is indicative of highly non-normal data combined with a high sample size. Greater sample sizes -> greater test statistics. When comparing test statistics across datasets, it is vital to keep sample sizes consistent. This also shows that our test is robust where the human eye may not be; at a first glance that t-distribution appears pretty normal, but the test shows otherwise. Similar to the Jarque-Bera test for normality, we will be using the Augmented Dickey-Fuller test for stationarity, and the Breusch-Pagan test for heteroskedasticity.

Stationarity is the property described in the introduction. Time series data is stationary if its probability distribution does not change over time, meaning that its mean and variance are constant. This is an assumption in much of statistical analysis, including the vast majority of machine learning models.

Heteroskedasticity is a... difficult to define property. It describes a situation where the variance of a variable varies with respect to another variable... yea. Perhaps a better illustration would be comparing the distribution of annual incomes for people differing in age. The majority of 17 year olds are minimum wage workers, with perhaps a few outliers, but largely centered around one value. 50 year olds, on the other hand, can range from minimum wage workers to the highest paid CEOs in the world. The former having low variance, the latter having much greater. Here, one could say that income is heteroskedastic with respect to age. In a financial context, we would aspire to work with homoskedastic data, as this property would provide further evidence for stationarity (constant variance).

Finally, autocorrelation (or serial correlation) is the correlation of a signal with a lagged version of itself. Pricing data is highly autocorrelated (today's price is closely related to yesterday's), whereas returns may be less so. The existence of autocorrelation makes machine learning on time series data extremely difficult or impossible. Thus, minimizing it will improve the power of our resultant machine learning algorithm.

2: Financial Data Structures¶

The chief theory of this notebook is that different representations of pricing data exhibit different statistical properties. This is non-obvious, as there is only one true historical path the price has taken, but consider that each bar (observation of prices) can vary wildly depending on how you define it. Minutely or daily bars can vary quite a bit in information contained; some minutes/days have very little price movement or shares traded, while some have wild swings in price and high volume. Different amounts of information are contained in each bar, so we would expect that statistical properties would vary quite a bit between bars. One example of this problem is that the majority of a day's trading activity occurs within a few minutes of the open and close. Minute bars are too low frequency here, and fail to show all of the detail, while being too high frequency around noon when far fewer trades occur.

An alternative is volume bars, indexed instead by the # of shares traded. This solves our problem of varying frequency (more bars are generated with more trading action), however comes with another problem. If 1,000,000 shares are bought of a stock worth \$0.33, not much is actually being invested. If that same stock rises or falls sharply in price, then 1,000,000 shares could mean much more interest is being given to the stock. The volume bar model fails to take this into account, and leads to inconsistency in statistical properties across price changes in the stock. A solution to this problem? Dollar bars, indexed by the amount of value traded (number of shares traded * price per share). This should, theoretically, contain all pertinent information about price and trading activity in a consistent manner across time.

There are other interesting bar types, particularly those described in Advances in Financial Machine Learning. Information driven bars are designed to "sample more frequently when new information arrives to the market." These are outside of the scope of this notebook, but largely focus on creating a bar when price/volume/value changes diverge from the expectation in a bar. This helps solve the problem that all of our bar types are susceptible to creating many observations for slow, sideways trading (though volume/dollar bars at least attenuate this a bit).

We expect the more complex bar types to better exhibit the above described properties (normality, stationarity, homoskedasticity, autocorrelation). Perfect normality, stationarity etc are not expected, however the closer our dataset is, the stronger our machine learning algorithm will be.

3: Implementation and Analysis¶

Now let us use our analysis of normal data (with added stationarity testing) on the aforementioned data structures. Before we can begin, however, we need to construct our bars. Ideally we would use tick data (base data where each transaction is indexed alone), but Quantopian's minutely pricing information should provide a good base to build on. Let's get into it.

3-1. Time Bars¶

Bars indexed by time intervals, minutely, daily, etc. OHLCV (Open, High, Low, Close, Volume) is standard. Taken straight from Quantopian.

Some notable behaviour here, particularly that the returns appear more or less centered at 0 with fairly consistent variance. At the start of 2018 this changed a bit, with far greater volatility exhibited in returns. Stationarity implies that our observations have the same statistical properties (mean and variance) over time, so we have some problems there.

The appearance of outliers make the results visually confusing, but the extreme test statistics imply our data are highly non-IID. Note, the very high sample size here inflates the test statistic for the JB test; when using test statistics to compare relative closeness to satisfying the null hypothesis, the sample sizes must be the same. This will become important later, as we compare these results to those for other bars' returns.

3-2. Tick Bars¶

Ticks refer to individual transactions on the market. Each tick involves some number of shares being traded hands, at a certain time and price. Bid and ask prices of the paired orders are also included, as well as the exchange the trade ocurred on. This is the highest frequency dataset possible, and is useful for constructing other bar types as well as high-frequency analysis. Unfortunately, this dataset is also very rich and often expensive and/or rife with errors, so finding a good source is vital. We do not provide this data on the platform, however there is a strong analysis of different bar types from Lopez de Prado's book, inclusive of a link to a free short-term sample of tick data. We can use Quantopian's minutely data to closely approximate other bar types over longer periods of time, with some tricks.

3-3. Volume Bars¶

Bars indexed by total volume, with each set # of shares traded creating a distinct bar. We can transform minute bars into an approximation for volume bars, but ideally we would use tick bars to maintain information for all parameters across bars. Let's set a target bar volume equal to the maximum minutely volume in our dataset (otherwise we'd have to split minute bars, which we can't). Then, we construct our bars in such a way as to minimize the distance from this target:

1. Combine minute bars while maintaining the time of the first and final minutes, adding volume of bars together, and keeping track of the high and low over the interval.
2. At each minute, check if the difference from the target volume would be minimized by adding or excluding the next minute. Select the optimal option.

This is meant to prevent the scenario where your target volume is 100K shares, and 2 bars of 99.9K shares follow each other. Adding the bars would result in essentially twice our target, while this strategy would result in 2 volume bars much closer to the target. This regularizes the volume of our bars.

Below is an implementation from Michael Matthews in this forum post, that runs much more quickly (and elegantly!) than the above, at the cost of accuracy.

We can see that the volatility of volume around our target is much greater in the vectorized solution. Both implementations suffer from look-ahead bias (a solution would involve setting the volume target as some multiple of past volume, with an exception case if one minute's volume exceeds this target), and both accurately model price changes. One runs extremely quickly, one more accurately approaches our volume target. If you are incorporating volume bars into your research, pick whichever you prefer. We will be continuing with the slower solution for this analysis.

Little information was lost in our sampling method (on long timescales at least); the price behaviour remains clear.

That's pretty spiky, but decidedly centered at our target. Theoretical worst case volume bar multiples should be 0.5x and 1.5x our target, less if we use a higher multiple of max volume. Let's see if we constructed this properly.

That's pretty good! Our outliers are about 30% from our target, but the mean is spot on and almost the exact same amount of bars surpass our target as those that fail to meet it. Using tick data would largely eliminate this inefficiency, but this approximation seems pretty good. Let's look at returns, as this is where we want to analyze the statistical properties.

We do still see the variance changing over time, but more spread out. This may actually prove problematic in our stationarity tests, as a greater proportion of our data exhibits 'abnormal' variance. The tradeoff is that it may better describe the trends in that generally volatile time period. Ah well, let's see.

This is a bit more convincing, but has a significantly smaller sample size than our minute bars so we can't directly compare test statistics. Let's see how this compares to dollar bars.

3-4. Dollar Bars¶

Bars indexed by value exchanged (number of shares traded times price they were traded at). Again, time bars are poor for this, and we must approximate the price for each order as well. Let's use the maximum daily dollar value of all trades as the target value to create a new bar.

A similar vectorized solution could also be implemented for dollar bars, but let's continue with the more constrained implementation

Looks good here, little long-run pricing information lost.

Pretty similar to volume bars, perhaps a bit more squiffy outside of that early 2018 range. Let's quantify that.

Our outliers are a bit further from our target (~38%) but the mean value and ratio of under:over seem excellent, so this is probably about as good as we can get.

This may look more wishy-washy, but look carefully at the y-axis. Our returns look decidedly more constrained, but still exhibit strong changes in variance.

The JB test statistic here is smaller than that of our volume bars, indicating more normal data. However, the sample size is also smaller so that might explain it. Let's normalize our sample sizes and compare again.

4: Comparing Results¶

In these frequentist tests, the p-value (0.0 or nan in the examples above), is the probability with the given test's distribution of generating a test statistic at least as extreme as that derived from our data, assuming the null hypothesis is true. We will be looking at relative test-statistics to determine how extreme our result is, as opposed to the typical method where the p-value is used as a confidence measure. In all of these the null is rejected, but the test-statistic tells us approximately how close to our null hypothesis we are. Frequentist tests are not typically used in this way, but it is a common strategy in Lopez de Prado's work.

Jarque-Bera Test: A goodness of fit test of whether sample data have the skewness and kurtosis of a normal distribution. The null hypothesis is that our distribution is normal. For a given p-value, lower values of our test statistic represent decreasing confidence in the rejection of the null, i.e. a greater degree of normality.

Augmented Dickey-Fuller Test: A frequentist test with the null hypothesis that some unit root is present in our data, indicating a lack of stationarity. The test statistic is negative, with a more negative result indicating a stronger rejection of our null hypothesis, i.e. less indication of stationarity.

Breusch-Pagan Test: A chi-squared test with the null hypothesis that homoskedasticity is exhibited by our data with respect to the given variable. Generally, high values of the test-statistic support a theory of homoskedasticity, which is ideal in financial data, showing that our returns' variance is more constant with respect to time.

To summarize, the null hypotheses are all firmly rejected (p-values of 0 or nan [Chi-Squared distributions like that of the BP test asymptotically approach but never reach 0, so the test outputs a nan instead]), but at different confidence levels. The JB test statistic decreases for each of our constructions, with lower values indicating a greater degree of normality. The ADF test statistic decreases (or grows in the negative direction), indicating a more firm rejection of the null hypothesis of a unit root and a better argument for stationarity in our data. The BP test statistic grows, which in general indicates a greater degree of homoskedasticity.

In the three properties tested for above, and this quick graphical representation, we see that our increasingly complex bar constructions are progressively better candidates for a machine learning model. This may seem subtle, but a strong foundation is vital to producing models that pick up on real patterns. Thinking back to the strange analogy for facial recognition across sci-fi human evolution, we need to pick up on what is true for humans now, and humans in millions/billions of years. Just looking at a bunch of faces at one point in time will fail in this, but isolating key factors in millions of faces over long spans of time will allow a clever model to pick up on consistencies. This is our task, and without data transformations like these it is likely an impossible one.

5: Next Steps¶

Perhaps the most important change that could be made is using tick data as our base structure. This would allow for smaller volume thresholds while maintaining the information in our bars, increasing our sample size and reducing the variability of volume/values between bars. Incorporating information-driven bars as described in Advances in Financial Machine Learning would be interesting as well, but again relies on tick data. Additional bar types might be useful, so posing novel ideas is encouraged!

Additionally, testing for serial correlation as an additional statistic would be useful, however this notebook was already extremely cluttered. Implementing the statistical tests locally, specialized for financial data if possible, would also improve the consistency and power of our conclusions. Additionally, the very idea of using statistical testing in this way is a bit... non-standard. Additional research must be done on the robustness of this strategy in general.

Furthermore, there are still several necessary components to a thorough machine learning platform. Fractional differentiation, as opposed to simple 1-differentiation (prices -> returns) will further allow us to balance our ration of information:stationarity. The triple-barrier method will allow us to label observations for both trade direction and sizing. Finally, the methods in scikit-learn must be adapted to work with our other methods and data structures.

This will result in a model that can make trades, though more work must be done to lay the groundwork to effectively backtest it, nevermind researching an actual signal to trade off of. Unsurprisingly, it is fairly complicated to create a machine learning trading algorithm, but we can at least scratch at the surface and lay the groundwork/software for the research process.

We will be continuing this internally, however any input from the community is highly welcome (Michael's vectorized volume bar solution in particular was excellent, sorry we didn't end up using it but from a programming and runtime perspective it certainly put the one presented here to shame!)

Good luck researching!