Hello Peter,

I think I see your point. In the attached backtest, I jotted down the SPY closing price for 4-10, and then ran the backtest, starting 4-11. The SPY return for 4-11 is 0.32% whereas the benchmark return is 0.4%. On https://www.quantopian.com/faq it states "The benchmark is an ETF designed to match the S&P 500." Is the ETF SPY? If so, why don't the returns match exactly?

I think that you are pointing out that for the example above, starting the benchmarking on 4-11 doesn't make sense, since the earliest an order can be submitted is 4-11, to be filled on 4-12 at closing. The earliest the position could be closed out would be 4-15 at closing. So, it seems that the earliest a gain/loss could be realized would be 4-15, right?

To be consistent with the trading, it seems that the benchmark should start two trading days after the start of the backtest (two minutes, when running on minute data). In the example, it would start on 4-15, using the 4-12 and 4-15 closing prices to compute the benchmark return for the first day when a gain/loss could be realized in trading.

Or am I missing something here?

Grant

Hello Grant,

I'm not sure of the details but I think we both see a slight problem. Maybe the end of day 2 is an 'unrealised P&L' return i.e. measurable but not actual?

P.

For the first day of a backtest, the benchmark return is calculated on the opening price; on subsequent days, the return is calculated on the previous day's close.

There's definitely an argument to be made that investing in the benchmark has the same limitations as the algo, i.e. it can't actually "own" stock until the close of the 2nd day. There's an argument that the current behavior is correct - it's modeling what would happen if you just "did the benchmark." We don't do any slippage or commissions on the benchmark, either. The benchmark, as implemented, is more conceptual than a literal, exact comparison. If those differences look like they are substantial we should update the benchmark implementation.

Thanks Dan,

Seems to me that the benchmark should represent an ideal hypothetical investment in the S&P 500 (using SPY as a proxy), without slippage and commissions. So, during the first tic (#1) of the backtest, the hypothetical order would be submitted and filled in the next tic (#2) at the closing price, If a pretend sell order is submitted during tic #2, then the hypothetical gain/loss could be realized at the end of the final tic (#3).

Then, it's a matter of deciding whether to assign the gain/loss to tic #2 (when the hypothetical decision was made to sell) or tic #3 (when the virtual transaction was completed).

Or am I thinking about this incorrectly?

Grant

I'm not sure there is a right or wrong answer on this one. I think it can go either way. I think I'd lean towards either a hyptothetical (no transactions, slippage, and completely start-to-finish calculation) or a grittily realistic (bar-based, transaction, slippage, etc). I'm not inclined to have a mix of the two.

In general, I think I have two bigger fish to fry anyway:

1) Fix the benchmark to be value, not price
2) Make the benchmark configureable

Thanks Dan,

What do you mean by "Fix the benchmark to be value, not price"?

Also, regarding drafting the new benchmark, I'd recommend a process along the lines of https://www.quantopian.com/posts/draft-proposal-of-new-data-history-api, to get user input.

Grant

Hello Grant,

Discussions a few months ago suggested that the benchmark does not include reinvested dividends and as a result the benchmark is several percent lower each year that it should really be. Over the (almost) 12 years of Quantopian data this could be an 'under-reporting' of benchmark returns of ~ 30% - 40% so all algos look a lot better than they really are.

P.

Ah...thanks Peter.

Some background here: https://quantopian.com/posts/question-regarding-the-benchmark

P.