So the backtest in Quantopian is attached to this post and this is the output from my backtester --> https://dl.dropboxusercontent.com/s/gk3bzt7wmdc4bbm/Sector_Rotation_test.jpg

As you can see, Quantopian indicates the algorithm yields a 150% premium over SPY and my backtester indicates barely 20% over the same time period. There are a number of discrepancies between the two that I am trying to rectify and I'm cataloging my results here for any insight and also to share my conclusions:

1) Quantopian handles dividends "realistically" meaning that price data is used in the backtester and dividend payments are credited to cash. My backtester expects the price data to be dividend adjusted. Right now, this is my leading suspicion for the difference in results and the implications, if this is the cause, would be AMAZING. Note that dividend adjusted data was used to rank the sectors and so the transaction log SHOULD (i'll get to this next) be the same. This would be mean that 'dividend adjusting' data significantly alters the performance, i.e. the adjustment is not accurate to reality. (ON A SIDE NOTE, I stand by prior posts I've made that dividend adjusted data must be used to code technical-based algos. I'll continue to use dividend adjusted data from my other data feed until it is available here.)

2) My backtester allows the user to perform the ranking using closing data and then purchase at the closing price. I understand this is a source of look-ahead bias, technically, but my feeling was that using the NEXT day's closing price was probably farther from the actual execution price I would get than the same day's closing price.... I'm going to change this in my program.

3) In my Quantopian algorithm I buy on day 0, liquidate the entire portfolio on day 63, and then buy new securities on day 64. In my backtester securities are liquidated and repurchased on day 63.

I can't find any other discrepancies and, as I said, points 2 and 3 shouldn't cause THAT much of a difference (the cost basis and sale price change by a day on either end). At this point, I'm thinking dividends are the issue.

Will keep you all posted.

Daniel, that's a fascinating algo. I spent 5-10 minutes looking at it. It looks very well coded. I couldn't see any of the "common" errors like over-borrowing or other things that falsely inflate returns.

Of your 3 possible sources of error, I'm most concerned about #2, simply because I know least about how you're generating your rankings. For instance, one possible way to err would be to say Q2 = April 1, but Q2 is composed of data from April 1 to June 30. When you're importing data you have to be extra careful that that your Q2 rank data is only used within the algorithm on July 1 or later.

I don't have a strong feeling either way about #1. #3 feels like a non-issue, but I agree it's possible.

Absolutely fascinating. I hope it holds up to more scrutiny!

Me too! If my backtester is wrong and Quantopian is right then I actually have a viable trading strategy.

So... I'm still trying to deconstruct both backtesters to find the difference. One thing I noticed in my most recent backtest (below) on lines 3 and 4 of the backtester were transactions made on 04/07/2002 (a Sunday) and 07/04/2002 (Independence Day, markets are closed). This is probably not causing the differences obviously but any idea why trades were placed when the market was closed?

NOTE: Different dates are displayed in the position value tab. Maybe a bug in the transaction log?

Gah. I think you're running into the same bug you reported before, cropped up in a different screen.

When I clone and run your algo, the log output looks right.

2002-01-03handle_data:117DEBUGBuying most undervalued stock Security(19659 [XLP]) with score of -0.9473006  
2002-04-05handle_data:117DEBUGBuying most undervalued stock Security(19658 [XLK]) with score of -0.9180112  
2002-07-05handle_data:117DEBUGBuying most undervalued stock Security(19658 [XLK]) with score of -0.9425241  

What's happening in the transaction tab is that we're taking midnight UTC (which is the timestamp on minute bars) and casting that incorrectly to Eastern (which is why the transactions show at 1900 or 2000, depending on daylight savings time). The trades are happening on the "right" day, but they're being displayed in the transaction tab on the previous day. I don't know how I missed that when I was debugging it the last time!

Bottom line: that's a UI bug that we will fix, but the backtester itself is trading on the right day.

No worries - I figured that this wasn't the cause of the difference in returns. And I appreciate your response earlier regarding your words of caution about look ahead bias. I've been quite careful about this and do not think this is the issue. I'm reasonably certain that the issue is using dividend adjusted data and I think you folks are doing it correctly. I'm trying to find an example, though, of a substantial difference in returns between using dividend adjusted data (and only looking at the price difference) and using price data with the actual dividends credited to cash..... I'll report back.

Got something else of interest, possibly more fundamental....

So - let's forget about the algorithm and just look at the benchmark, SPY in the backtest I post below, which runs from Jan 3, 2002 to July 12, 2013.

On Jan 3, 2002 the closing price of SPY was 116.84 and on July 12, 2013 the closing price of SPY was 167.51 - a return of 43.37%. The 43.37% (Yahoo data) is very close to the value calculated by Quantopian, 45.5%. However, SPY also paid $26.997 per share in dividends over the same time period, meaning that the actual buy-and-hold return of SPY would be 66.47%! And this is without reinvesting dividends!

If we use the dividend adjusted price of $93.44 for Jan 3, 2002 we get a return of 79.3% up to July 12, 2013 (this would be the approximate return if dividends were reinvested).

Have I miscalculated here? If not, my recommendation moving forward would be 1) at the very least have the benchmark include it's dividends in the calculated percent return, the way it is included in the algo and, eventually, 2) provide the user the ability to reinvest dividends automatically (both in the benchmark and the algo).

Daniel,
Thanks for this very well written strategy. It gave me a lot of inspiration.
For the reinvestment problem, I think the best way to do it is to reinvest the dividend received manually. Quantopian will look at the dividend as cash increase in your portfolio, and you can access this number as "context.portfolio.cash"

Here is a very short sample of buying SPY and spending all the dividend to re-invest in SPY.

@Jiaming - It is interesting that the reinvested dividends beats SPY by 40% when dividends are reinvested but let me be clear - the ~43% return of the benchmark does not include dividends AT ALL! That is the price return only! This is misleading at best and possibly a big oversight.

I think Daniel's right. We covered it over in this thread. Just putting the link here in case someone else is following this thread.

I'd like not only to clone your algorithm (this is my first visit/attempt - and I've done it!), but also to expand/change the ETF universe. I have no clue how to work about this idea (using Fetcher, create my own CSV file). Could you please give me some hints? Thank you.

Andrzejek,

You can include up to 100 specific securities in an algorithm. For example:

context.stocks = [(your list of securities goes here)]

Grant

Yes, but as long as they are not included in the ("Fetch") CSV file, they are omitted in the analysis.This is my understanding, maybe I'm wrong. I ran my clone algorithm several times (with context.stocks expanded) and only "XL_" ETFs were included in the final rapport. Thank you for your response.

@Andrzejek - thanks for taking an interest in my algorithm! Unfortunately the ranking system that I used is not open source. It is a proprietary algorithm developed by me on my firm's behalf. There may come a point where we publish it but that decision would have to be approved by our CEO and we would probably publish it in a formal white paper (I promise I'll post a link on Quantopian if that ever happens).

There is an issue with the algorithm - or more specifically just a concern I have... we have been using in-house software to test the algorithm and have been getting slightly different results than we get in Quantopian. I am working on figuring out the difference between the two. The biggest road block at the moment is an issue with fetcher's NAN filling routine. (https://quantopian.com/posts/fetcher-problems and https://quantopian.com/posts/help-with-fetcher-and-filling-nan-values) Once I resolve this issue I can use Quantopian to read my trade log from my other backtester and verify that the algo generates similar if not identical returns in both programs.