I work in developing algorithms and evaluating the significance of the backtesting results. Namely in developing algorithms the primary objective should be to remain skeptical of the results. As you said there are many factors that can skew the results, leaving many individuals to toss money into a strategy that they fitted. As far as your concerns go:

1) Leverage: please take a look at some code Daniel Sandberg and I worked on to address this. I started a framework to address this in a Long only portfolio, and Daniel extended it to support L/S. I am really hoping to see this evolve with the community, because I agree that the unrealistic strategies on this forum (while fun) have no merit in the real world.

2) Lucky Trades: Strategies can definitely be inflated by a single unrepeatable trade, to combat this I often will look at my equity curve and check for huge spikes. These tell me my algo got lucky and took advantage of a few unrepeatable events in history. Optimally I want a smooth equity curve that informs me that my idea is valid across all market conditions. Granted this is merely a qualitative method to approach this. Furthermore you can look at the trailing returns of your strategy vs the benchmark and calculate the percentage of times you surpassed it, this should also help to remove outlier performance. Alternatively you could look at the descriptive statistics to winsorize your results to get a better picture of your equity curve, essentially this is just removing trades that performed some threshold above average. Also I do like to look at event studies of my strategy to see if my idea has any kind of merit by seeing the average of all my trades over some length of time

3) I kinda disagree with you here... The idea should be to scale your trades with your account value, if you have a $1B account does it make since to trade the same lots as a $100k account? If your account is growing you should be increasing your trade size, this is how you achieve pseudo exponential growth in your equity. While a $100K will be sensitive to $10K equity at risk, the same dollar value risk is not going to make much of a percentage difference in the $1B account.

I would also like more freedom in what kinds of plots I can get out of the backtest. Namely additional subplots, as well as the individual securities with my entries and exits.

I am not saying the current backtesting curve doesn't help. I am just pointing out that it is not easy detect the issues mentioned above.
2. The plain benchmark backtesting graph does't help understand this. I would rather see a graph that tells me how consistently my signals beat the benchmark in 80 out of a 100 trades(by even 2%) rather than it show that I have beat the benchmark by 150% over all even though this lead was established 5 years back during a 1 year period where the algorithm performed exceptionally better than the benchmark and then just tracked it before and after it.
3. I don't disagree we should be scaling our trades with the account value in a real trading algorithm implementation. Its just that when analysing and compraring how the trading signals performed, it should not be impacted by the size or growth/contraction of the portfolio during the backtest period. Instead I would prefer if we bet the same excact amount from the begining to the end of the backtest period so that we can understand how the bets performed during the entire period.

Sarvi

Regarding #3, as you guys mentioned, it's also an issue comparing it to the backtest. For example: My algorithm trades all the money it can while not overspending. Let's say my algorithm has a 50-day warmup period. During that time, the value of my portfolio remains flat since I own no shares, but the S&P has fallen 5%. So when my algo starts trading, it already has an advantage over the benchmark. If my algo performs similar relative to the benchmark from here, then my graph will look better compared to the benchmark graph because the average slope of each graph will be proportional to the portfolio starting size which is 5% more for my algo. Not to mention, obviously my algo starts 5% farther ahead than it should. This can also occur if my algorithm just gets lucky early on then moves similar to the S&P. I think a lot of people have seen this, and it is misleading since they see diverging benchmark returns and algo returns lines that make a strategy look nice even though it has average performance.

That said, this is a problem that can be mitigated within code, and it is up to people to do that themselves right now. Quantopian could add tools or features to automatically avoid problems like this, but I think another option is having users build frameworks that other users can easily include in their algorithm. I'm not sure what the most elegant solution is, I think there are a few good routes to take, especially with something very flexible like Python.

@Sarvi,

Let me reconvey my advice in regards to comparing your performance to the benchmark, as I think I might have not explained this properly. I am saying the general shape of the equity curve is a good qualitative measure that you should first look at. Here you can spot any events that you backtest luckily took advantage of that resulted in an inflated return by looking for huge spikes. Next you need to actually evaluate the frequency in which you beat the benchmark, you can look at the performance breakdown numbers in the backtest page and tally this up by hand. But better yet why not just automate the process. Simply code up a solution, it would not be hard to keep track of the 252 day rolling compounded returns for both your portfolio and your benchmark. From here with each new calculation add a tally if the % portfolio > % benchmark and there you can show if your portfolio consistently beats the benchmark. Of course you don't have to use 252 days but rather whatever is applicable to you.

I still am not on the same page as you on #3 however, as you could simply just look at the average return on a trade and disassociate from the dollar value. Furthermore you might want to step out of quantopian and use an event profiler to see if your strategy has any merit, this will graphically show you how the average trade performed including the deviations.

@Gus I would love to see an additional forum page for users to share code regarding the infrastructure of their backtests as opposed to a strategy, I think this would provide an avenue for those who might not be likely to share their strategy to share code that addressed caveats and gotchas within the library. Then each user could implement tidbits of code into their personal framework that would improve the information gained in their backtests. Additional useful feature would allow users to set a default environment in importing their framework(s) whenever they create a new algorithm