Hi Grant,

It's curious that you get nans, those should be cleaned by default. They might be a result of using a universe, can you try without that?

In terms of speed, you might be interested in recent efforts to refactor the batch_transform. This leads to a 100x speed-up: https://github.com/quantopian/zipline/pull/136

Also, I think it might be a good idea to add a get_data function to quantopian that just returns the DataPanel. That way you wouldn't have to create a single-line function all the time.

Thomas

Thanks Thomas,

I'll see if I can reproduce the problem with a definite list of sids, however, I'm interested in using set_universe.

Glad to hear that you are working on speeding up the batch transform...please keep me posted.

Regarding adding a get_data function to Quantopian, I wouldn't put it at the top of the priority list. In fact, I'd rather not return the entire DataPanel, but just the data I need for a given algorithm, in a numpy ndarray (or another data structure directly useable in the algorithm).

By the way, perhaps you guys are already thinking along these lines, but it'd be nice if we could eliminate some/all of the loops in Quantopian. For example, the order function could accept vectors of sids and corresponding number of shares, rather than having to loop though the sids. This is pretty much the paradigm in MATLAB and it tends to result in more readable code.

Grant

Hello Thomas,

If the algorithm spits out NaNs and I capture the offending sids, I can re-run it, listing the sids explicitly (see attached). Then, the backtester automatically resets the dates of the backtest and avoids the NaNs (as best I can tell). I figure that securities are popping in and out of existence, or maybe the set of securities delivered by set_universe is changing, and NaNs result.

I suppose I can just test for NaNs and skip the tic if there are any in the data...or I can eliminate columns from the data matrix that contains NaNs (perhaps the best approach).

Grant

Thanks for checking that.

Yes, the problem are actually not the nans -- there is no way to avoid having missing data (e.g. if a security didn't trade on that event). By default, however, the batch_transform already removes those (using datapanel.dropna()) so I was surprised that it didn't do that in this case. In any case, I opened an issue here: https://github.com/quantopian/zipline/issues/140 to get to the bottom of this.

Thanks for the repro example Grant. That was immensely helpful. I don't know what's wrong yet, but I sure can make it happen here.

Following up on this problem led me down a rabbit hole of a bunch of bugs and code changes, but in the end I think they're all irrelevant. I mention them only to tell you why it took me so long!

I think the NaNs you are seeing are expected. I took your first shared example, for instance, and tracked down the NaNs I saw there, The cause? A two-month trading suspension of RGA because of a spin-off activity from MetLife. It feels like a one-off, but there are one-offs everywhere!

When we built set_universe we tried to avoid NaNs because they were a pain. We just can't avoid them, it seems - they are everywhere.

You probably have read the line in the help document where it says "If a stock is not present for the full quarter (due to an IPO, merger, bankruptcy, or other corporate action) it is excluded from the universe for that quarter. This behavior will be modified in the future to minimize any survivorship bias." We were trying to avoid NaNs, but we knew we couldn't forever.

I think your early question, "what is the recommended way of dealing with them?" is the key one. I don't have any specific advice for you yet. Let's see what other quants do with the problem, and we'll see how to add it to the product.

Thanks Dan...Grant