Hi Tony,

i'm sort-of new on Quantopian and not using candlesticks here (at least not at the moment), but I did play around a lot with candlesticks in my own personal trading. My conclusion was and still is that candlesticks are a fantastic visualization tool, and I always use them as the display format in charts when I'm trading. However the problem and shortcoming with candlesticks is that they are purely descriptive in nature, i.e. qualitative rather than quantitative. Maybe once-upon-a-time people could successfully trade on a purely qualitative basis, but I think those days are probably over, except for a small handful of exceptionally skilled discretionary traders.

My personal opinion, and I'm sure most people here at Quantopian would share the same view (after all it is called QUANT-opian) is that success in modern trading requires a QUANT-itative approach, and the Quantopian platform has some fantastic tools for exactly that. So, IMHO, if you want to do well at trading, and you want to use a candlestick approach, then you will almost certainly have to introduce a strong numerical element as well, which goes far beyond the traditional candlestick approach.

You will probably also have to introduce a comprehensive nomenclature / classification system that is significantly different to the classical Japanese one. I have worked on it with the idea of taking it into an algorithm but, for reasons that you will understand if you try for yourself, in order to be adequately comprehensive for real trading, even the simplest system becomes almost unmanageable once you get to 3 or more bars!

If you have read all the standard candlestick type books and you want to read something good about trading WITHOUT using a numerical approach, then I would definitely suggest the 4 books by Al Brooks. He uses candlesticks, one moving average, and trend-lines for support & resistance, and that's all. After reading & understanding his books & his method, I tried using it live day-trading the ES futures contract for several months at the end of last year, but just as an experiment, because actually I don't normally trade futures much and I don't really like day-trading at all. (It's too exciting, and I don't want my trading to be for excitement). Yes, it worked, following the ideas in his books, but the hard part is getting to the point of really understanding what Brooks is on about.

Al Brooks books (all 4 of them) have very similar titles: "... Price Action ... Bar-by-bar ... for the Serious Trader". At first I found them difficult reading because of his particular writing style. In fact honestly, I found them to be four of the most dull, boring and tedious books that I have ever read and it was a big effort to keep going ...... UNTIL I started to get what Brooks is really on about, and then at that point my opinion changed completely. I re-read them all again and, on the second time around I found them fascinating, to the point of feeling like I couldn't put them down. And then after that, I read them AGAIN! Yes, seriously, I'm not kidding!

What I realized was that Al Brooks, after many years of trying all sorts of approaches, has devised a simple yet profound minimalist approach to his own discretionary trading. Is it possible to "algorithmize" what he does? I don't know. I have tried. It's not easy and I have not succeeded (yet). I have asked around among some very good traders, including the person who originally recommended Brooks books to me in the first place, to find out if they have or even if they know of anyone who has managed to "algorithmize" Brooks ideas, and the answer is "no".

However, if you are skilled at coding in python (and I'm NOT) and you really want to try a qualitative candlesticks approach that incorporates Al Brooks' ideas, then I'm certainly up to working on it with you .... on the proviso that first you read all of Brooks 4 books carefully, cover-to-cover, at least once each ... if you can endure it ;-))

Cheers, very best wishes,
TonyM.

Hi TonyM, thank for your positive feedback. I know a bit about python and now i am writing my own python code and use the IB API for conducting backtest (Quantopian does not support forex spot yet and also unable to change the trading hour because forex is trading almost 24 hour except New York time 17:00 - 17:15). Since accessing forex market data at IB is free, i try to test by tracking engulfing pattern (e.g. one bullish bar and then immediately engulfing by a bearish bar and then I short). However, the result is really disappointing. i can not get an average positive return. So i am a bit doubtful on candlestick pattern.

Hi Tony C,

You write: "... engulfing pattern (e.g. one bullish bar and then immediately engulfing by a bearish bar and then I short). However, the result is really disappointing".

Don't give up on it. If you want to use this pattern (and actually it's not a bad one to work with for looking at the end of up-trends and then going short at the tops), I think you need to do a whole lot more than just identify the existence of the pattern. Here are some potentially useful suggestions for you, based on my own research & experience:

1) As this is only a small pattern (2 bars), you need to carefully consider the overall CONTEXT in which the pattern occurs.
2) Do you have any other indicators that would tend to suggest that the uptrend is weakening BEFORE the pattern occurs?
3) Is the open or the high of the 2nd bar (black) at an identifiable resistance level?
4) How long, in proportion to the total range of the black bar, is its LOWER wick (or tail or shadow) between the low and the close?
5) Is the low of the black bar at an identifiable support level?
6) Where is the close of the 2nd (black) bar with respect to the low and/or the close of the first (white) bar?
7) Use some sort of metric (measuring unit) such as the length of the black bar, and/or the ATR of the past N = ?? bars as the basis for comparison of all your measurements
8) Take these factors, as well as any others that you think might be relevant (and there should be quite a lot), and then design a QUANTITATIVE scoring system for evaluating the "Quality" of all patterns of this type that you find.
9) See if you can find any useful relationship(s) between the score (from item 8 above) and the "success" that could potentially have been obtained with a subsequent short immediately after the black bar.
10) Can you find a "threshold" score such that any patterns that you see but which don't (at least) achieve that score should NOT be traded?

Good hunting, best wishes,
Tony M.

Hi Tony C,
Well, as I expected, the general level of enthusiasm for this topic doesn't seem to be very high does it, with no-one else posting here (yet) except you & me. Maybe a lot of people here at Quantopian consider anything with a thread title like "... Purely Using Traditional Candlestick Patterns" is something akin to Gann & Voodoo ;-))

Actually its not a bad little problem . I worked on a few things like this years ago using Neural Networks. The problem with NN (at least the old ones that I was using) was their tendency to memorize rather than generalize, and I'm sure that some of the modern ML (such as SVM) could do a much better job, although I don't have time to start getting into that right now. However the 10 point summary in the post above gives a framework for how to do it.

What I can do easily is write good working code in AFL to search for viable patterns and generate some initial statistics, so I did this last night for you (and for anyone else here who might be interested). I present it at the end of this post and if you use AmiBroker software then you can just copy & run it. Unfortunately I am not good with python, but maybe you or someone else can translate it into python.

I ran it on my personal database of about five and a half thousand securities (futures, FX, US, Canadian, Australian and HongKong stocks) with data going back to 1980. What I found is that on the 5-point scale of strength that I used (see above and the code below) the average score is 2.9 for tops & 2.6 for bottoms, and that on average a potentially viable engulfing pattern signal corresponding to a genuine turning point occurs between about 0.9 and 1.1 times per year (using EOD data bars), with similar frequencies for tops & bottoms. I did this as a statistical data gathering exercise and did not do the additional work to take it forward to evaluate P&L results that might be expected if this were implemented as an actual trading system.

What we can infer from this so far is that actually useful engulfing pattern signals are rather rare, typically only about one per year for each of tops & bottoms for most securities. If you are trying to trade more than that, then I'm not surprised that you aren't having very good results. The reason is that there just aren't all that many good signals, and if you are not trading GOOD signals, then its probably not a good idea, especially if you have only been looking at a small number of securities. What MIGHT work however is to apply the strategy to a large basket of securities, e.g. at the very least several hundred, and then you might just get enough good entry signals to make it viable.

Anyway, if you want to take it further, my code (in AFL language) is below. If you (or anyone else) can translate it to python and then post back here, I would be delighted.
Cheers, best wishes, TonyM.

AFL code follows:
//------------------------------------------------------ // // EngulfingPattern_TopsBots // TonyM. Created 26 Oct 2017, Updated 26 Oct 2017 // //------------------------------------------------------ // // Basic setup SetChartOptions(Mode= 0, Flags= chartShowDates);
SetBarsRequired(-2, -2); // Require ALL past and future bars -- this turns OFF AmiBro's quickAFL for script/DLL to properly execute (AmiBro v5.20 +) and to ensure correct display.
// i = BarIndex(); // zero-based bar indexing.
imax = BarCount-1; // imax = LastValue(BarIndex());
// Oprev = Ref(O,-1);
Hprev = Ref(H,-1);
Lprev = Ref(L,-1);
Cprev = Ref(C,-1);
M = (H+L)/2;
P4 = (O+H+L+C)/4;
MA5 = MA(P4, 5);
MA10 = MA(P4, 10);
MA21 = MA(P4, 21);
MA5prev = Ref(MA5, -1);
MA10prev = Ref(MA10, -1);
MA21prev = Ref(MA21, -1);
// // Candlesticks BodyTop = Max(O, C);
BodyBot = Min(O, C);
Body = BodyTop - BodyBot;
UpperTail = H - BodyTop;
LowerTail = BodyBot - L;

// Pre-existing Trend direction TrendDir = sign(sign(MA5-MA5prev) + sign(MA10-MA10prev) + sign(MA21-MA21prev));
// // Min 5-bars prior to potential Top / Bot Top5bars = IIf(H>= Ref(HHV(H, 5), -1), 1, 0);
Bot5bars = IIf(L<= Ref(LLV(L, 5), -1), -1, 0);
TopBot5bars = Top5bars + Bot5bars;
// // Engulfing Pattern Exists EngulfingTop = IIf(H> Hprev AND L< Lprev AND Cprev> Oprev AND C< O, 1, 0);
EngulfingBot = IIf(H> Hprev AND L< Lprev AND Cprev< Oprev AND C> O, -1, 0);
EngulfingPattern = EngulfingTop + EngulfingBot;
// // Valid signals for Engulfing Top / Bot iEngulfTop = IIf(TrendDir== 1 AND TopBot5bars== 1 AND EngulfingTop== 1, 1, 0);
iEngulfBot = IIf(TrendDir== -1 AND TopBot5bars== -1 AND EngulfingBot== -1, -1, 0);
iEngulf = iEngulfTop + iEngulfBot;
// // Pattern Test score iTestScore[0] = 0;
iCountEngulfTops[0] = 0;
iCountEngulfBots[0] = 0;
TestScoreSumTops[0] = 0;
TestScoreSumBots[0] = 0;
AvTestScoreTops[0] = 0;
AvTestScoreBots[0] = 0;
Nyears[0] = 0;
for (i=1; i 0)
AvTestScoreTops[i] = TestScoreSumTops[i] / iCountEngulfTops[i];
}
//
if (iEngulf[i]== -1) // Bullish Engulfing Bot
{
iCountEngulfBots[i] = iCountEngulfBots[i-1] + 1;
if (C[i]> C[i-1])
iTestScore[i] = iTestScore[i]- 1;
if (C[i]> H[i-1])
iTestScore[i] = iTestScore[i]- 1;
if (C[i]> M[i])
iTestScore[i] = iTestScore[i]- 1;
if (UpperTail[i]< LowerTail[i]) // want small upper tail
iTestScore[i] = iTestScore[i]- 1;
if (UpperTail[i]< Body[i])
iTestScore[i] = iTestScore[i]- 1;
TestScoreSumBots[i] = TestScoreSumBots[i] - iTestScore[i];
if (iCountEngulfBots[i]> 0)
AvTestScoreBots[i] = -TestScoreSumBots[i] / iCountEngulfBots[i];
}
// Cumulative average stats
AvEngulfingTopsPerYr[i] = iCountEngulfTops[i]/ Nyears[i];
AvEngulfingBotsPerYr[i] = -iCountEngulfBots[i]/ Nyears[i];
} // //------------------------------------------------------ // Plot (0, "", colorBlack, styleThick);
Plot (TrendDir, "TrendDir", colorTeal);
//Plot (TopBot5bars, "TopBot5bars", colorRed); //Plot (EngulfingPattern, "EngulfingPattern", colorBlue); Plot (iEngulf, "iEngulf", colorBlack, styleThick);
Plot (iTestScore, "iTestScore", colorRed);
//Plot (iCountEngulfTops, "iCountEngulfTops", colorBrown); //Plot (iCountEngulfBots, "iCountEngulfBots (dashed)", colorBrown, styleDashed); Plot (AvEngulfingTopsPerYr, "AvEngulfingTopsPerYr", colorBlue);
Plot (AvEngulfingBotsPerYr, "AvEngulfingBotsPerYr (dashed)", colorBlue, styleDashed);
Plot (AvTestScoreTops, "AvTestScoreTops", colorBrown);
Plot (AvTestScoreBots, "AvTestScoreBots (dashed)", colorBrown, styleDashed);
// //------------------------------------------------------ // Filter = (1== 1);
AddColumn(AvEngulfingTopsPerYr, "AvEngulfingTops/Yr", 1.2);
AddColumn(AvEngulfingBotsPerYr, "AvEngulfingBots/Yr", 1.2);
AddColumn(AvTestScoreTops, "AvTestScoreTops/5", 1.2);
AddColumn(AvTestScoreBots, "AvTestScoreBots/5", 1.2);

Hi Tony Morland,
Thank for your detailed response.
My engulfing algorithm is already running in certain score system. For example, for engulfing bullish pattern, i am based on the following score (i just show some examples below). i use forex GBPUSD below.
- if the 1-hour bearish pattern close is the loweset in the last 12-hour, add 1-score.
- if the current bullish bar is touching the previous bearish open for conesecutive 20 minutes and if sma(five 1-minute bar) > sma(ten 1-minute bar) > sma(fifteen 1-minute bar) > sma(twenty 1-minute bar), add 1-score (because it seems the bullish impulse is pretty good, i guess)
- if the current bulish bar is touching the previous bearish open for conesecutive 20 minutes and during the process it does not touch the bottom of the previous bearish bar, add 1-score. Also, set stop as previous bearish bar bottom and then gain as (currentprice - stop) * 1.5
- calculate the "candlebodylength" as abs(open - close) / (high - low). if the bearish bar candlebodylength is higher than candlebodylength.mean(last tweleve 1-hour bar), add 1-score.
i have 4 attributes above, and so i set "create position" if the score >= 3
something similar ...
My pain point is : the algorithm (if i run .. say 2016 with positive gain..the same did not work for 2015 .. or 2014..i am afarid that i am now falling into the so-called data overfitting). That is, pure engulfing pattern is statisitially not possible for getting a positive expected gain.
And I believe that is also why at Quantopian i seldom see people talking pure candlestick trading pattern.

Hi Tony C,
I didn't know what instruments you were watching or what timeframe you were trading on, so I used EOD bars on everything that I have in my own database. I think it is at least a reasonable assumption that the results I had should more-or-less be similar on a bar-by-bar basis, irrespective of the different timescales, so when I wrote "1 year" you can interpret that as 252 bars on your own timescale. If we assume that FX trades 23.75 hrs/d and your bar-size = 1 min, then that's 1,425 bars/d, which equates to 1425/252 = 5.65 years in my experiment, and my observed average of 0.9 to 1.1 genuine Engulfing signals / yr for EOD equates to an estimated 5 or 6 raw signals per day on your timescale BEFORE applying any scoring threshold.

Your scoring system is different to the one I used, but let's just compare. You require 3 out of 4 points for a "pass" on your system, and I required anything higher than the average score (which came out to be 2.9 out of 5 for tops & 2.6 out of 5 for bottoms) for a "pass" on my system. I didn't do any histogram of score frequency, but if we assume the scores are approx evenly distributed between minimum=1 & maximum =5, then that would imply about 2.75av / (5-1)range = about 0.64 times the 5-6 raw signals per day, or at most about 3 or 4 valid trade entries/day or a maximum of about 750 to 1,000 trades/year. Is that at least somewhere close to what you actually get?

When I looked at this with my EOD data, my first thought was that there probably would not be enough signals or trades to do anything that is statistically meaningful at all. However, even though I know I have made some rather rough assumptions in extrapolating from my experiment (lots of different markets, long time-scale) to your situation (one instrument, short time-scale), I would nevertheless expect at least some similarity in the results on a per-bar basis. If you are getting more that 750 trade entry signals / year, then that is certainly enough to make meaningful conclusions and hopefully avoid over-fitting. I'm not sure, but maybe your conclusion that: "... pure engulfing pattern is statisitially not possible for getting a positive expected gain" is not necessarily correct, and the reason that you: "... seldom see people talking pure candlestick trading pattern" at Quantopian might just be because they don't believe in candlestick patterns, and haven't even looked at them. Don't give up on it yet!

May I suggest a little more investigation.
I will describe it, but you will have to do it because I don't have the data in my own database, and I don't yet have the skill in python to write it in Quantopian. However, if you are willing to try it, here is what I suggest:

1) Take your entry scoring system rules and convert them all to equivalent 1-minute bar rules, so that you can think carefully about what you are doing on a bar-by-bar basis rather than a time basis. Forget about the stop for now and let's just compare how many potential trade ENTRIES each of our scoring system gives.
2) Now take my entry scoring rules (which are all on a bar-by-bar basis) and convert them to python in your system using 1-minute bars.
3) Check if your entry scoring system gives you something close to the same number of entries as mine does, and are they in the same places?
4) If not, then let's figure out why.
5) Maybe by combining your entry scoring system & mine, we might even come up with a revised set of rules that are better than either.

I don't personally watch the GBPUSD and haven't traded FX for a few years, but here are some questions to think about:

• Do you think it is possible that there may be any political, macro-economic or other factors that may have significantly altered the behavior of GBPUSD in 2016 so that it really might be behaving now in a way that is different from 2014 & 2015? I don't know, but I just pulled up a ten-year chart and the very first thing I see is that from late 2008 all the way through to mid June 2016, GBPUSD was consistently constrained within the range 1.400 to 1.700 but the on 23 Jun 2006 something significant happened, GBPUSD took a big drop and since then has never again traded above its former support level of 1.400

Without doing any analysis or testing at all, just from a visual inspection of the chart, the general "character" of GBPUSD looks subtly different to me before and after the low in mid-April 2014.

I would suggest not doing any sort of parameter optimization in your system. Leave it as it is. Maybe the "problem" is not your system per se, but rather that there could actually be a real change in the character of GBPUSD itself (for reasons that I don't know and haven't thought about).

Maybe you could try generating four different sets of results:
1) for the period from start 2009 up to mid 2014,
2) for the period from mid 2004 to 1st June 2016,
3) 1st June 2016 to 1st July 2016,
4) 1st July 2016 to now.
Then compare and see what you find, and lets go from there.

Also, can you put a tag on this thread that says "FX" and see if we can attract some attention from other people who trade any pairs with GBP.

All the best, TonyM.

Hey, I didn't read the whole thread, but in case it wasn't pointed out, you don't need to reinvestment the wheel. Talib already had all the classic candlestick pattern detection, same as is used in lots of charting software.
I didn't find any consistency using them with equities- - underperformed benchmark. But that's the great thing about th e algo approach. Instead of relying on superstition, you can run analysis, figure out what is statically a did and what works. As pointed out, my hunch is that combined with other signals they can be useful.

Hi Viridian, i didn't look at TAlib, and i'm not surprised to read that you "didn't find any consistency using them". I spent many many hundreds of hours on this years ago, doing careful statistics but without the benefit of such a nice platform as Quantopian. My conclusion was very much in line with your ending comment that "Combined with other signals they can be useful". The candle patterns themselves are only a very small part of the whole picture, and what is critical is the specific context in which they occur. The difficulty lies in defining this "context" in a way that is both concise enough and also sufficiently broad to be useful.

Hello Tony Morland, I am back now (i was busy in the last few weeks). I do really feel your heart and sincerity from you. Take a look at this link. I basically use GBPUSD (forex) and run backtest from 2011 to 2017.

Basically i try to capture engulfing pattern and the following four attributes need at least a "score" of 3 or above. If i see four "1", i double my bet. The python code is also there (very dirty). If you or anyone were interested further, let me know. I will be glad to elaborate further.
.
{'nottouch': 1, 'candlebody': 1, 'lookbackmaxmin': 0, 'part2sma': 1}