Hi Tony,

I'm glad you brought our discussions on avoiding bad trades (periods) and back it up with your own study to this thread. I concur with the results of your findings that most of algorithmic trading systems have a tendecy to over trade mainly because they have not factored in avoidance of nonviable trading periods into consideration. I have arrived at the same conclusions in a previous study of SP500 index using Amibroker platform that has Maximum Favorable & Maximum Adverse Excursion features. A substaintial portion of short term time horizons have non viable trading opportunities due to thin spreads that is only eaten up by friction costs. Entering such periods is, more often than not, going against the predictive grain.

These have many practical implications in prediction model design. One approach is after identifying the viable and nonviable tradeable opportunites through a mechanism such as Tony just described, you could use this information to detect when it is OK to enter/exit a trade and when to avoid one. This can have a positive impact in designing systems that have target/output variable that wants to mimic actual trading outcome with trade efficiency.

Hi James, thank you. This all ties in very nicely with our favorite topic of "regimes". Here are a few more comments from me, based on work that i am doing right now "as we speak".

1) I also use AmiBroker as my main experimental trading platform, as i now have many years of experience with its AFL language which IMHO is more powerful that that of any similar alternative platforms such as Metastock, Tradestation, EasyLanguage, MultiCharts etc and, at least for me, is a lot easier and more intuitive than python for the specific context of trading financial price data series. Yes AmiBro does have built-in MFE & MAE, but irrespective of whatever platform one uses, MFE & MAE are actually very easy to program and use for anyone who has read John Sweeny's book and understands just how useful and powerful those concepts really are as part of general trading system development.

2) Identifying non-viable periods for trading is not really very difficult and there are many different indicators that one can use for this. One good place to start, for anyone who isn't doing this already, is with Al Brooks concept of "Barb Wire", as something to be avoided as much as possible in trading. Avoid it or else you will almost certainly get whipsawed & cut!! With a little creativity, his concept (as per any of his 4 books) can easily be extended and improved further.

3) What i found pleasantly surprising is the extent to which non-viable / BarbWire periods, as identified by appropriate minimum-lag indicators that you can easily devise & apply for "real time" work, ALSO provide quite good short-term predictions of future behavior, as can be confirmed with look-ahead analysis. What this demonstrates very nicely is that, in the context of market regimes, if we define "non-tradeable regime" carefully and well, then it really does have quite a good degree of persistence. Much better in fact that that of uptrends or downtrends for example.

4) What you say, James, about thin markets & spreads may be true, but i think is not really the key issue. The sort of "non-tradable" or at best only marginally tradable times that i am talking about occur even in very liquid stocks with small spreads. It is simply a part of real-life market behavior that occurs a surprisingly large amount of the time, but which most people, for some reason, just don't bother to think carefully about!

5) I suggest that anyone who is NOT using some sort of "regime pre-processing" like this BEFORE going into ML (or whatever else they use) is probably wasting a lot of the potential power of their predictive tools in trying to generate predictions on that part of the input data that is identifiable in advance and is best simply left out altogether as being "non-tradable so just avoid it". Then the trader's favorite ML etc tools can work more effectively by allowing them to focus on just what is potentially tradable, rather than confusing them with other periods of market data (regimes) that are not.

6) Notwithstanding 5) above, i expect that most people will simply ignore this and keep on doing the same sort of stuff as they have always been doing ... and getting the same sort of results as they have always been getting. Oh well, good luck to them!

So yes, in conclusion, i agree absolutely with your statement that: "This can have a positive impact in designing systems that have target/output variable [that wants] to mimic actual trading outcome with trade efficiency" (or perhaps that wording should be "efficiently"?).

Safe trading!
All the best, T :-))

Hi Tony,

The sort of "non-tradable" or at best only marginally tradable times that i am talking about occur even in very liquid stocks with small spreads.

That is exactly what I intended to imply. With a short term time horizon, price movements are very noisy, more so if you factor in transaction costs. One has to extract, identify then segregate periods where there is enough volatility that causes spreads to widen to call it a mean reverting trading opportunity or low volatility scenarios where there is successive breach of recent highs or lows to form an upward or downward short term trend to call it a trend following trading opportunity or what you call "non-tradeable regime" wherein there is low or normal volatility but small spreads that it is rendered or viewed as merely flat or consolidating where chances of being a profitable trade either side your on will be very small if not none.

What I meant by "trade efficiency" is the market timing metric used by some platforms to measure how efficient a trade entry/exit is vis a vis the short term future horizon (i.e. if I entered long today and exited 10 days from now at its peak, my trade efficiency score is 100!).

Cheers!

Tony, James, I agree with you both. In my own research, way back, when looking at all the trades taken, the top 5% accounted for about 80% of the profits. The rest could have been dropped as not that useful. But, when you dropped them, you would also lose much of your 5% simply because they originated from the same trade triggering mechanism.

I even had one strategy where I reintroduced the losing trades in order to increase the overall profits. In order to maintain your 5%, you might have to take the other 95% as part of the cost of achieving the 80%. And since that 95% still produced the other 20% in profits. The problem is somewhat mitigated, you did more work, for sure, but you still got that 5% making the 80% of the profits.

Nonetheless, I do think that 10 days is really too short a predictive interval. However, you could use the dull periods as some price consensus where no one big enough is ready to commit or initiate a move in one direction or other. And when you see volume expanding with price, it might be your cue to piggyback along for a short duration momentum ride.

Thanks Guy, I hear you, understand, and largely agree. I think the trick is to try to get better with understanding the details of differences between continuous periods of non-tradable regime vs the interspersed isolated bars that don't give useful trades either. They are not necessarily similar.

This approach of focusing attention on what doesn't work seems somewhat counter-intuitive, but it leads to the formulation of the problem in a different way that i think has the potential to be more efficient for solution in ML.

As an analogy, and despite the fact that many (most?) people don't spend enough time thinking about pre-processing (and i don't just mean de-noising) of data, i spent years working in the oil exploration & production industry, watching Geologists & Petrophysicists naievely dump data into NNs while saying that "the machine can correctly pre-process the data itself", and then wondering why their results were not so great. It makes no sense to give any sort of ML the task of trying to struggle with figuring out something for itself, especially in the context of noisy data, when an operation (for example differencing or taking ratios), can be done far better as part of intelligent human pre-processing first, leaving the machine to spend its efforts looking for more subtle features. So i see this idea of working with the not-so-good trading results first as part of good pre-processing, in which i am definitely a strong believer. I think we are more or less on the same track so far.

With regard to your comment that "10 days is really too short a predictive interval", this is an interesting topic for consideration in the context of ML, and of course has significant practical implications. I understand where you are coming from, and let me start by saying that i am also a strong believer in the importance of fundamentally driven trends due to many factors such as macro-economics, government interest rate policies, time delays in bringing new mines & factories on-stream, effects of inventory buildup & drawdown delays, etc, etc. HOWEVER while it is definitelt true that many trends do unfold over time-frames much longer than 10 days (and sometimes much longer than 10 months), nevertheless the question remains as to what is the best way to trade them? At one extreme we have a sort-of "buy and hold the trend" approach, and if one uses that, then i would agree with your comment, but in many cases it is preferable to take a series of short-term "trading chunk bites", all in the same direction, out of longer term trends rather than to sit and ride the pullbacks. What is best in this regard tends to be not only market-specific, but also varies over time (typical non-stationarity effects) even within single markets. There is a general tendency to believe that markets have become more Mean-Reverting in character over time, and if this is indeed true, then the idea of shorter (e.g. 10-day chunks within longer trends) does make sense. I didn't make that statement without some careful consideration & research first. Nevertheless, sure, the timeframe mentioned is indeed largely a matter of personal preference. Another related aspect of this is also the fact that the reliability of any form of ML results tends to degrade the further out in time one looks, and so this interval is to some extent an effort to find a useful compromise. I know people who run futures trading funds and believe that anything longer than a 5-day "prediction horizon" with any form of ML is completely unrealistic nowadays.

I'm with Guy -- often we enter these pointless positions because we cannot discern between them and our winners. I've often thought the solution would be to combine disparate signals, and where they agree you'd have a higher confidence the stock will go somewhere. I'd never really considered the inverse approach -- that there'd be some way to identify the efficient stocks and filter them out. If you could predict stocks that won't move, you could make a fortune selling straddles. For this reason, I doubt it is so easy to separate the wheat from the chaff.

Hi Viridian,
-- Yes, combining disparate (in the sense of orthogonal) signals always makes sense.
-- Predicting Moving vs Not moving becomes more difficult the longer the time horizon.
-- "... make a fortune selling straddles" ? No, i don't think so. The market makers control pricing to make sure that you generally can't!
-- Not-so-easy doesn't mean not worth trying. If you do some experimental analysis, you may find (as i have) that if you use a Markov chain approach, then it is easier (i.e. transition probabilities are higher) for predicting "not moving" than for predicting trends or directional movement, which is the way that most people approach it. But honestly, i'm not intending to try convince anyone else here to whom the idea does not appeal.

It's definitely a very appealing idea. Can you share a before and after example of the results you've achieved by cutting the fat ala Markov chain?

Just had a critical situation in my family & flying overseas now. I will get back as soon as i can. Tony

Tony, my prayers are with you in your time of crisis. Take care my friend. James.

@Viridian Hawk,

Luc Prieur posted awhile back his attempt at identifying market regimes through Hidden Markov Chains and OneClass SVM in this post: here