Back-testing: A Useful Tool or "Financial Charlatanism"?

A Boston based company, pioneer in risk modeling led by a founder with a deep understanding of algorithmic trading periodically holds seminars on interesting topics. On Thursday, October 22, 2015 at 11:00, Northfield Information Services is holding a one hour, free webinar on the topic Back-testing: A Useful Tool or "Financial Charlatanism"?, I've copied the abstract below. I think it is key for anyone interested in backtests to listen in on this webinar, especially the Quantopian staff, it will help you understand the limitations on your approach to investment idea generation.

Full disclosure, I am completely independent of Northfield and not a current customer of their products.

Back-testing is the widely used practice of simulating an algorithmic
investment strategy. While essentially everyone involved in
quantitatively driven investment methods conducts back-tests, it is
widely accepted that simulated investment results achieved "in sample"
are at best only a very weak indication of results to be expected in
"out of sample" experience.

In this presentation, we will describe the causes for the minimal
validity of back-tests, and suggest methods to mitigate the problems.
We will discuss the current day implications of material from seminal
studies by Kahn and Rudd (1995) on the relationship of past and future
performance, Kahn (1997) on common statistical errors in investment
tests, and diBartolomeo (1999) on the conceptual and philosophical
limitations of back-tests. The final portion of the presentation will
be devoted to a detailed exposition of how practitioners can limit the
risk of "overfitting", based on the mathematical framework of Bailey,
Borwein, de Prado and Zhu (2014).

Like any scientific endeavor, algorithmic trading must be approached rigorously - ideally through the scientific method. The scientific method provides a framework for testing a hypothesis after making observations of surroundings or events. In the context of trading, it provides a way to test the exploitation of repeated patterns. Like any scientist who observes their surroundings and is interested in explaining them, theories are created about why something happens and those theories are tested. Because the financial markets do not obey any laws of physics, they cannot be described perfectly by any model (or backtest). In physics, or other hard sciences, events can be observed and hypotheses can be tested and theories proven. This is not true of trading but it does not discount the the fact traders still need to test their hypotheses.

Algorithmic (or quant) trading has been around for a very long time (e.g. Thorp in the 60s/70s doing convertible bond arbitrage). Back in the day, the markets were so inefficient that hypotheses could be tested and executed with very strong odds of success. The edge has since largely disappeared which only means traders must employ more sophisticated tools to find that edge. It also means that new and advanced methods for testing hypotheses must be employed. Enter Quantopian.

I'd venture that a large portion of people that try algorithmic trading do not have the wherewithal and rigor to properly test their observations. Despite this, tools like Quantopian seek to make it as easy as possible. Quantopian is a profit-seeking enterprise that has invested money in their platform to allow people the opportunity to rigorously test their ideas if they so choose. I don's suppose Quantopian much cares what people do with the results of those tests. At a minimum, it at least provide a learning experience to those interested in a challenge.

No one serious about algorithmic trading (or trading for that matter) will ever claim backtesting is a panacea that will accurately predict results of their idea. Anyone who does is lying or selling a bullsh*t system to novices. Backtesting is however, a critical tool to test hypotheses. It doesn't take a presentation from a company selling "alpha" models to remind me of the limitations of testing observations through backtesting.