Hello Branko,

If you run on minute data, you can buy at the opening price by submitting at the close. See attached for an example. If you run the code and look at the Transactions Details and the log, you'll see how it works.

Grant

I have two concerns about this algorithm, one of which I am myself trying to sort out:

The first is that the data feed in Quantopian is not dividend adjusted. Therefore, a stock that pays a large dividend will see a substantial drop in share price that ostensibly reflects poor performance but in reality is just the dividend.

The other concern is simply that my understanding of true mean reversion is that is takes a longer period of time than momentum strategies. However, I can say that it does seem that stocks that plummet sometimes bounce back (hence the saying "even a dead cat bounces")

Grant,
Thanks for the input. I'll work on formatting my algo to take minute data and calculate the open and close values based on the time. I'll post my results soon. Not sure if I'm safe as saying open is 9:31 and close is 16:00. If the minute data for some stocks is incomplete where such a time doesn't exist for one or several days, I could be computing the price change over several days. Unless someone knows of an easier way.

Daniel,
Good point. Is there any way we can determine if a dividend has been paid? Maybe one of the libraries has dividend ex-dates and amounts for stocks. Or if we can get a list somewhere external and access it through fetcher. In that case, we can exclude those or at least adjust for the dividend.

@Branko - I'm working on something to mitigate the dividend problem right now but presently this is what is written in the API

Dividends are not relayed to algorithms as events that can be accessed by the API; we will add that feature in the future.

I have a subscription to a data feed that provides dividend adjusted data and I can generate a .csv of adjusted prices. I'm going to need to fetch each csv to read the dividend adjusted data and use that data to determine my trading strategy. I'll post the algo that reads and handles the .csv when I'm done with it. I'm hoping that dividend adjusted prices will be available from within the API at some point but I don't think anytime soon.

By hardcoding a list of stocks that make up the current index, you've introduced survivorship bias. Try it with a set_universe....

Don't know if I got it exactly right, but here's an attempt at using minute data. Doesn't look too promising, but I could be doing something wrong...

@Simon - I used the set_universe you had suggested

Every time I tried running it for a few years it would stop after some time and never finish. Am I running out of memory perhaps?

Doesn't look too promising, but I could be doing something wrong...

Until dividend payments are taken into account, practically no algorithms on this site are accurate. Dividends must be handled, especially with a mean reversion type of algorithm.

Hmm I haven't looked into it -- is there no way to detect a dividend ex-date the morning after? It's probably sufficient to just ignore those signals...

Maybe I missed something in the API but this is the only information I saw:

Dividends are not relayed to algorithms as events that can be accessed by the API; we will add that feature in the future.

On a side note, I'm use a subscription data feed to dividend adjusted data to do my calculations and then purchasing as per the Quantopian backtester.

Yeah, perhaps I will try to add bloomberg support to zipline.

Yes, I understand the issue with the dividends. I'm working with what I got. I may be able to review the transactions and see if any of them were actually impacted by dividend shocks or if the signal was due to a dividend ex-date.

@Branko - We're looking into that performance problem, sorry about that.

@Daniel - I think you're overstating the issue somewhat ;) I completely agree that we need to implement dividend notices in the API, and that the backtester will be improved when we do. But there are still plenty of valid, interesting algos that can be built and tested without it.

@Dan - Okay, you may be right that I'm overstating. As I think you know, I've been working on my own backtester for the past 9 months and presently an algo that was performing reasonably well in my backtester is performing EXCEPTIONALLY well in Quantopian's backtester. You'd think this would make me happy but the inconsistency is more alarming than anything. I'm working on tracking down the differences but I feel like I'm spinning my wheels.

@Daniel, how can we help?

@John - Thanks alot and let me apologize for being grouchy. Quantopian is an amazing accomplishment and contribution to the community. I will keep you posted --- whatever is causing this discrepancy is going to be VERY interesting....

@Daniel,

There are three potential sources of discrepancy we've discussed here, which I wanted to share in the hopes they trigger insights for you:

1. We model dividends without modifying the price history, so the dividend bearing stocks will typically have a price drop on the opening of the ex-date. This can cause false signals from moving averages and other trailing window calculations.
   
2. We omit special dividends, because of a hole in our logic for handling the ex-date versus the pay-date. Our code assumes the ex-date is always before the pay-date, but part of the specialness of special dividends is that the pay-date comes before the ex-date.
   
3. Different models for slippage and commissions. Our models are configurable, and you can effectively turn them to zero impact, but I just wanted to see if you had compared your system's models on these costs versus Quantopian's.

If none of that helps, at least you'll know we are cheering for you to figure this out!

thanks,
fawce

I think this conversation has unfortunately derailed into an airing of grievances.

@John
Any idea why I'm having trouble with minute data? Is it unrealistic to expect I can work with a large set_universe over several years in minute data? It's also painfully slow when run.

I'm only really evaluating anything on market open and market close. I'm just using minute data as a work around.I have a return statement to try and minimize unnecessary calculation but not sure if that's helping too much.

On a side note, more flexibility as to the time periods would be a good feature (not just minute and daily). Perhaps open/close, or hour time intervals. Maybe a wrapper similar to batch_transform.

Finally, is there a fast way to sell all holdings? That would be a useful function. I tried iterating through the portfolio.positions dictionary but sometimes I get a non-sid key. I can try to recreate if this shouldn't be the case.

@John - Thanks for the insights. There is also a discrepancy in the prices of the securities when using end of day data, but the differences seem to be negligible for the most part and, by exporting the prices from Quantopian into my backtester, I was able to determine this was NOT the cause of the problem. I have slippage and commissions set to zero on both backtesters. The difference in handling dividends might be the problem but I'm a little skeptical. Essentially what I'm seeing is a drop in '08 in my backtester but not the same drop in Quantopian - doesn't necessarily make sense that is dividend related.

@Branko - no grievances here. I'm huge fan of Quantopian but, as a professional money manager, it is my responsibility to ensure that code is exactly correct. A false signal in an algo can result in huge losses in real life. I can't just say "it all averages out".

Daniel -- is it possible that the quantopian slippage/execution rules have your algo buying only very small amounts of something in 2008 (when some ETFs were just getting listed and therefore only traded 2000 shares a day), whereas your backtester assumes you get complete fills (to your detriment)?

I've had that problem in the past...

@Daniel - can you create a minimalist reproduction with a tiny backtest duration? For example, just one day? At that scale, it would be feasible to pick through the individual transactions for differences.

@Branko, thanks for the notes, I'm going to clone and investigate. I'll report back soon.

thanks,
fawce

@Simon - I checked that the transactions matched but not the volumes. That's very interesting..... I'll have to look into it more

@John - I'd love to post it, but 1) I'm not sure you'd see anything wrong with it at the outset and 2) I need to get permission from our firm's CEO before sharing anything that might be considered proprietary. I might go that route and post it (this algorithm isn't actually that great in my opinion and I'd love to share it but it does seem to be reasonable for conservative clients and rules are rules - it isn't a day trading algo either)

@Branko - Sorry for hijacking your thread! As far as liquidating all your positions, you can create a series that holds all your sids and loop thru those instead of portfolio.positions or you can throw in the "try" keyword so that the non-sid key exception is caught (I actually don't know if it does get caught. I prefer sticking all my sids in to a series at the outset and then just write a Go_To_Cash function.)

@Branko, I did a quick test to see if the set_universe call was the source of the trouble, but I found that the performance was similar with that commented out. I also tried just commenting out the majority of the securities in the SP100 definition, and saw dramatic improvements to performance. That lead me to thinking the algorithm code may be driving very non-linear runtime with the securities. A few of us took a closer look at the algo code, and we thought you might want to:

• reduce the number of orders placed each minute -- it appears that you
  are placing at least one order in every security in the current
  universe on every minute. Is that intended? Each of the open orders will be evaluated on every loop against the current trade bars, to see if orders can be filled.
  
• see if you could reduce
  the runtime of get_min_keys, since it is looping and is called from
  within handle_data.

Can you take a look?

thanks,
fawce

See attached backtest with minute data.

The algorithm takes 5 biggest losers overnight (throughout the day) and holds them throughout the day (overnight). It uses the set_universe.

The algorithm took forever to run (over an hour). I imagine it's because of the minute data and the large universe. It only trades on open and close though.