Great point Market Tech, I'm glad you brought this up. That is why I backtest on a individual yearly basis ( Jan - Dec 2003 , and so on). ETF Replay has a simple , but great metric for their backtest engine," Periods outperformed/underperformed benchmark (%)" . I don't think it's a good idea to run an algo that outperformed the benchmark by less than 65 % . I wonder if Quantopian could include this in the list of metric , or perhaps there 's away to code it .

Although your figure illustrates 'the high watermark', I may suggest forecasting the equity curve within 90% certainty using GARH/ARIMA and raise an alert if 90% is broken, then a hard stop at 75%. It's been on my to do list for quite some time.

Note: the concept: http://statsmodels.sourceforge.net/0.6.0/generated/statsmodels.tsa.arima_model.ARMAResults.plot_predict.html

In the eastern world there are funds that work like this: advertise and begin investing other people's money with some strategy. If it loses 10% they stop. The company folds, and everyone employed gets the sack. If that isn't a hard-stop, I don't know what is.

But you raise a very important point. The psychological effect of a crater in the equity curve is a big problem. Winning too much can also be a problem: What I Learned Losing a Million Dollars by Jim Paul (its a fun read). Wish I had a good answer.

It is really fun running backtests through 2013, less through 2008. I like the question a lot and also the way you present the underwater elements. So far i have found it is better to start algorithm validation across different market periods AND at various start times within those periods. This helps isolate market bias (e.g. 2013's bull charge) and stock selection bias (e.g. the lucky pick). Moving forward i'm looking to build return histograms to compare performance between other algorithms, alg configurations, and ETFs to help in this decision making process.

2cents: A quick look at the graph appears to show a long only strategy. In falling markets, the algorithm looks weak but I can't tell if it is by design (e.g. let it ride) or a weakness in its shorting strategy.

Great question and I actually have a real live case with an algo that performed well in backtests, but when I put it live I looked at a hole of 25% (2500$) after a few weeks. I discovered that one of the indicators that I used didnt work in the live environment while it did work in the backtest environment. Still trying to figure out why and what. What I actually planned to do was to put a hard stop on the maxDD of the backtesting and pull the algo.... that is not what I did.... I let it run with limited capital to learn why and backstopped my positions with options. As the blotter does not see the options in the account but does see the PnL my metrics are now completely screwed.

Below a screenshot:
1. I know the algorithm lost about 2500, 25%... Topline says 35%.. the graph says its 41.4%
2. Same trouble with the cash, top says 9.8K and below 5.6
3. The topscore gave a reading yesterday for the first time.... the backtest gave a few....

One of the things I discovered is the un-reliability of the target_percent functions. It does not take in account wether the cash is settled, so the algo wanted to execute 5 trades but only did 2 or 3... which meant that the hedging was broken and the portfolio was weighted to one side.

Next algo I put live:

1. I'll bake in the absolute DD limit discovered while backtesting, a kind of "Oh Shit" trigger.
   
2. I'll be disciplined to not trade other stocks in the account that are not visible to the blotter.
   
3. I'll run a paper algo and a live algo at the same time and monitor if they deviate
   
4. have more feedback in the log output to monitor the performance and the levels of the indicators. Just because an indicator works in backtesting does not mean it works in Live.... unfortunately
   

@ Market Tech,

One problem is that there's lots of talk about risk management, hedging, etc. but what's the price folks are willing to pay for security? If I'm gonna shut down my algo after it drops X% shouldn't I also shut it down after it goes up X%? To take a personal example, if my winning algo had shot up like a rocket after being launched with real money, would Quantopian have started wringing their hands, saying "Oh no! It's even worse than we'd anticipated!" Probably not. And say we have another bubble like the dot.com one of recent memory (I'm old enough to have lived it), and Q has investors wanting to jump on board the party wagon. Will they turn them away, forgoing the revenue from fees, in favor of conservatively managing risk?

There's a whole industry around taking advantage of the psychology of risk-reward. Hmm...is the industry called Wall Street?

Grant

@Grant: There's a fundamental asymmetry between underperformance and outperformance and it has to do w/ confidence in your model and backtest. If your strategy is doing poorly, one of the reasons could be normal variance but others can be things like a mismatch between the backtest and live trading, a bug in your code, significantly larger than expected slippage, market impact, overfitting historical data, etc. If your strategy does better than expected, the reason is unlikely to be anything but variance or perhaps a favorable regime (in which case, you would expect a simulation/backtest run over the production days to do well as well).

So the reason to be concerned about underperformance compared to overperformance isn't just 'I like money,' psychology of risk-reward or the principle-agent problem but rather, because underperformance is more likely to be a result of some fundamental problem that won't auto-correct even if you let the system run for a long enough time for the variance to cancel out.

To the specific original questions, Jonathan Kinlay just wrote a post about using Monte Carlo simulation to identify expectation boundaries and when strategies deviate below the bands, you pull the algo:

http://jonathankinlay.com/index.php/2015/04/strategy-still-working/

EDIT: more interesting links on this topic:

http://epchan.blogspot.ca/2012/04/life-and-death-of-strategy.html
http://www.amazon.com/gp/product/B00JUUZU2E/ref=oh_aui_search_detailpage?ie=UTF8&psc=1

The image I posted above was extracted from a popular recent post here regarding using sentiment in a research project. I used that specifically to call out the potential dichotomy of reviewing back test results vs living the account impacts of real time trading.

Quantitative analysis, in general, tends to abstract away the money. "They're just numbers" seems to be the theme. Quant's enmesh themselves in the deep math and projections and wow factor of their algorithms but often fail to step back and examine a P&L curve with a mind to fathom the true gut twisting quandary of trading one's hard won, harder lost nest egg. No matter how advanced your algo may be, or how sexy your P&L curve looks, when it gets down to actually weathering the market storms -- you don't know how big a wave you'll take until it nearly drowns you.

I'm happy to have read all the thoughtful, useful comments. In the end trading is a human endeavor. It's such a strange mix of statistics, math, charts, reports and... personally held cash money.

I find the only way to deal with it is to work diligently on the systems, be aware of biases, and don't obsess over daily results. It helps to have assets spread over many different classes.

Frankly, my discretionary investing skill sucks. As bad as algos can be, they are better than me investing in RSX and sticking with it through a 50% drawdown, which I am still in haha.

@Market Tech: I am with Simon you need to do your Homework and check the Volatility of your Algo. I personaly set my mental MAX DD 10% it is a good start however if the Stocks that I am using have a slow ATR then set it lower like 7~8% as I read in this Strategy

http://en.wikipedia.org/wiki/CAN_SLIM

Using Average True Range can help you to establish your hard STOPS. However I think that it comes back to preference. Just remember the effort that you need in order to get back your original capital after DD

By the way How did you added a picture with-in the post?

EG

There "used" to be a link on the New Post page which took you to the markdown cheatsheet:

Mark down cheat sheet

! followed by brackets [] followed by parens ( link goes here)

Homework -- yeah, done a fair amount of that in my time. Now if I could just turn of the male / human switches in my mind...

"On Being Certain" -- Robert Burton, has some great insight on risk vs timid. There may actually be genetic predisposition when it comes to how one approaches risk. It's worth reading.

Perhaps germane:

Against the Gods: The Remarkable Story of Risk
by Peter L. Bernstein
Link: http://amzn.com/0471295639

I read it awhile back. One thing I recall is that the whole intellectual concept of probability is fairly new on the scene--the idea that outcomes, even though uncertain, can be quantified.

-- Grant, reading it now. Thanks for the recommendation. I'd have sent you a private, but alas, you'd rather we not.

Market Tech,

You're welcome. I have no problem communicating privately, when it is appropriate. I'll send you my e-mail address for future reference.

Grant