My List: The Seven Sins in System Testing

QUESTION: Anyone have some more sins to add?

§ Calculation errors §

1.Data quality
If your data contains “fake prints” or has not been adjusted for splits, this would certainly negatively affect your trading decisions from historical Peaks and Troughs based on the wrong values. Especially if the traders who moves the markets (e.i institutional investors and large hedge funds) with big money make their trading decisions with correct values from professional data feeds.
Solutions: Use professional data feed

2.No broker commissions
If broker commissions were not included, that would mean that 50 percent of all strategies that is based on “ping-pong” trades between Long and Short positions would be successful. Meaning that if your strategy is showing a negative result, just reverse the strategy and you will end up with the same result but with positive numbers.
Solution: Using 0.2 + 0.2 percent commission on each trade

§ Statistical errors §

3.Illiquid stocks
The reason why technical trading system is working comes from how you interpret the statistical probabilities from mass psychology, crowd behavior and numeric cognition. Your work becomes similar to the work of an actuary at an insurance company. The work is to mathematically evaluate the probability of events and quantify the possible outcomes. In any statistic calculations more samples and population make the probability theories stronger and more valid. If a handful of people are trading a penny stock back and forth, that will not make the best environment to successful use theories of mass psychology.
Solutions: Use instruments with more than 20 millions USD in average daily turnover

4.Short periods of data
If you are only testing your strategy against data from only a couple of years during a consistent bull market like we had from 94-88, 03-07 or the present since 2011 it is obvious that the market conditions where a lot different than during the bear markets after dot-com bubble and the last subprime crisis.
Solution: Using only stocks with consistent data from 1996 that includes 2 major bear markets and 3 major bull markets and check that the gains are evenly spread during each year during that time

5.Small sample of stocks
It makes sense that having data from more symbols gives less variation (and more precision) in your results. It is a lot higher probability that your trading strategy is working on all S&P 500 stocks if the strategy is working and tested on the S&P 100 stocks then if it is only working and tested on the Dow 30 stocks, or even worse only have been optimized to work on the S&P 500 index or the Dow Jones Industrial Index.
Solution: Using at least 200 global stocks from various exchanges with highest daily turnover

6."Buy and Hold" bias
The stock market is in some way a reflection of how the economy in general is developing. As long as the major economies in the world have a general positive development in GDP together with an increasing volume in the monetary system the numeric value of the stocks tend to go up. Especially if you also take in to account that all the exchanges around the world is no longer are holding stocks of companies that gone broke and been delisted. That is why almost any buy indicator has been successful if you also have a sell strategy from your long holdings to be more than 10 – 15 years. If your enter signal would be, buy if it is Monday and sell after 15 years, that would be a great trading strategy but you would also be sitting on “dead money” for long, long periods of time.
Solution: Only use list of stock with negative performance under used time period with Long Strategies and vice versa

§ Programing Errors §

7.Look-ahead bias
Some back testing software let you use values from indicators that in reality has been established after the time where the trade took place. One of the most notorious indicators is the ZigZag function and its sub functions. On the other hand it can be used to an advantage if you know how to code it and you want to make experiments how great the maximum gains would be if you could manage to consistent make perfect entries and exits in the short term.
Solution: Double check with graphical simulation and lists of all trades

I love this list. Thanks for sharing it with us! Part of the reason I love it is that Quantopian has built-in features that help you fall into these traps.

Your algo runs with prices from Nanex's NxCore trade feed - a professional data feed. As you know, nothing can stop all bad prints, and your code should still have protections where it is sensitive to them.
By default Quantopian includes commissions.
Quantopian has a more elegant solution to manage backtesting with illiquid stocks. Our slippage model makes sure you can't order stock that isn't traded, and limits you to 25% of a given minute's volume.
We don't have quite the date range that you'd like, but we do have data since 2002. You do, always, have to take care that your algo is manageable in many different market regimes.
We have two different ways for you to manage this. You can pick broad ranges of stocks by either filtering on criteria in our corporate fundamental dataset or by using our dollar-volume universe to choose a market cross-section.
I think this one is very similar to the previous one, with the twist of de-listed and bankrupt companies. Our backtest data includes all companies that traded since 2002, including the ones that have gone bankrupt.
This one is my favorite because Quantopian handles it so beautifully. Because we have an event-driven backtesting system, fundamentals, price, and volume data are fed into the algo at the "right" time, and never before. Your algo can never make a decision based on future data because, simply, it doesn't have the data! (The exception to this is fetcher. When you bring fetched data into your algo, you have to make sure you're managing the dates properly. But all data that is natively available within Quantopian can be used without fear of look-ahead bias.)