tiger,

This may help you see the difference between EX-DATE and PAYABLE DATE.

Thanks Vladimir for the explanation !

I have identified the source of the errors---
For benchmark, the backtest system always uses the ADJUSTED closing price;
For the algorithm backtesting, it uses the ADJUSTED closing price, but EXCEPT the period between EX-DATE and PAYABLE-DATE (usually 10 - 50 days). During such period, the cash value in the account was not accrued or did not have compensation of the dividend amounts in advance to match up the true lookahead net capital value.

If the algorithm bought the equity before the EX-DATE, and 'unfortunately' sold it during the days between EX-DATE and PAYABLE-DATE, the backtest will show PERMANENT LOSS of the dividend value ! This could considered as a Bug that influences any algorithm that holds ETF or stocks which have many times of dividend or large amounts of dividend.

For example, Microsoft (MSFT) had dividend of $3.08 in 2004 Q4, which was traded in the $28-$30 range during that time, the EX-DATE and PAYABLE-DATE was 2004-11-17 and 2004-12-02, let's see what would happen if the algorithm bought MSFT before the EX-DATE and hold it through this period. As shown in the attached backtest (Buy&Hold MSFT vs. Benchmark MSFT), there was a huge loss of 10% suddenly after the EX-DATE, which might trigger StopLoss orders for many algorithms.

While this bug exists, many algorithms will be appeared to be underperformed and unstable due to the sudden change or jump in the backtest. There are two suggestions could be considered: (1) ALWAYS uses ADJUSTED price for the algorithm as the same as the benchmark; (2) During the days between EX-DATE and PAYABLE-DATE, uses UNADJUSTED price with compensation for the cash value in the amount of dividend values, but this solution may also trigger StopLoss orders for some algorithms.

Quantopian should take notice of this issue, or the backtesting system might give inaccurate numbers calculating many parameters like performance, sharp ratio, volatility, etc., and further mislead the users when they take effort to design and optimize the algorithms.

Thanks.
Tiger. S.

Hi Tiger -

This is behaving as expected. I'll walk you through some of the design decisions and let's see what you think. Also check out the help section on dividends which covers this in additional detail.

I added some record() statements to your algorithm to add custom variables to the graph. You can see that cash starts at $2974, drops to $.90, and then up to $300.90 when the dividend is paid out. Then, look at the price and returns lines (Tip: if you click "cash" in the legend, it's a toggle that removes the cash line and lets you zoom into the remaining custom lines). In the price you can see it drop by almost $3 on the dividend date. But the percent change line doesn't show a 10% drop - the returns for that day are about 1.5%.

Unpacking what that means:

• The cash of the dividend is added back to the portfolio on payment date (the cash jump)
  
• The simulation is always using the as-traded price of MSFT. That's why we see the price of MSFT drop.
  
• The simulation is always using fully adjusted historical prices. That's why even though the MSFT price dropped, the percentage change from the previous day doesn't reflect the price drop - it's using an adjusted price to do the calculation.

If you have built a sensible stop-loss for your MSFT position, it isn't going to be triggered. The MSFT price changes are fully adjusted.

I think what is throwing you off is that your day-to-day portfolio returns are dropping temporarily. The good news is that the simulation is correctly telling you what your cash position is. The bad news is the non-intuitive drop in portfolio value. I think a future improvement might be to include some accounting for a pending dividend payment. You point about calculations like Sharpe being affected by this portfolio value change are valid. In most cases the effect of dividends is negligible; this example is the exception to that.

As for the benchmark, it does indeed use adjusted prices. The backtester is simulating your portfolio, with all the ugly sausage making of cash payments and slippage costs. The benchmark is idealized and hypothetical - it is not constrained by transaction costs or liquidity or other real-world constraints.