Backtest. Still getting error, even though I think I'm removing DVMT from the list of stocks to be traded.

Backtester shows no trade on 2016-09-07, but research platform does.

1969-12-31 19:00 PRINT 2016-09-07 00:00:00+00:00
2016-09-08 08:45 PRINT 2016-09-07 13:31:00+00:00 NaN
2016-09-07 13:32:00+00:00 NaN
2016-09-07 13:33:00+00:00 NaN
2016-09-07 13:34:00+00:00 NaN
2016-09-07 13:35:00+00:00 NaN
Name: Equity(50242 [DVMT]), dtype: float64

Hello Grant,

It looks like your algorithm holds an EMC (sid(2518)) position on 09/07/2016, which is the pay date for a stock dividend that pays out in DVMT (Dell). This causes the algo to acquire an unexpected position in DVMT and when it calls order_optimal_portfolio with no target weight for this asset, the optimizer tries to sell out of the position. However, the backtester doesn't have trade data for DVMT until 2016-09-08, so it thinks DVMT isn't tradable and rejects the attempt to close the position.

A temporary fix would be to filter out EMC from your tradable universe either by removing it from the pipeline output in before_trading_start, or excluding it from your pipeline using the built-in filter StaticAssets. Adding a more general solution to the optimization API is on our list of issues to tackle.

The get_pricing function actually uses a different data feed than the one used by the backtester and the pipeline API. We are currently working on migrating get_pricing to use the new data feed, so the datasets match in both environments.

Thanks Ernesto -

I never would have figured that one out!

It sounds like the trick might be to also apply the data.can_trade test to stocks in the currently held portfolio, since sometimes one ticker can magically convert into another one overnight. Or I guess one would still be hosed, since universe=todays_universe only applies to the stocks being added--everything else gets dropped automatically under the hood.

Why does the optimization routine not apply the data.can_trade test upon selling? Could the test be added? It'd be a one-line change, right? Or is the optimization API on the back-burner? Or maybe you are batching up a bunch of changes? Is it on github? Maybe I could submit a request there?

Hi Ernesto,

Any feedback on a fix? Seems really straightforward.

Thanks,

Grant

We resolved this problem by changing the way we model EMC's transaction. Rather than model it as a cash dividend and a stock dividend, it is now modeled as a regular delisted security. Your algorithm gets an all-cash payout at the closing price, just like we do for delistings. Of course this isn't a perfect representation of the transaction, but it is a good analog on the important aspects of calculating returns.

None of the can_trade options you suggest are available; the problem occurs well before it gets to the orders being placed. The optimization routine can't construct an optimal portfolio when the portfolio includes a security with an unknown (and unknowable) price. My favorite quote about software development: "Seems really straightforward."

Thanks Dan,

What about live trading at IB? If I'm thinking about this correctly, if EMC were held the prior day at IB, then it would magically be converted to DVMT at the open the next day, right? And it would appear in the portfolio. Wouldn't there still be a problem, under live trading with a broker? There would be no price until the first trade, so presumably, the optimization API and ordering would choke, right?

Yes, there is still an open bug about this particular case.

If you are integrated with a broker, and if your portfolio holds an equity that is listed on a major exchange but has never traded, and you run the optimization API, the optimizer can't handle it and will throw an exception.

Thanks. I suppose it tries to look up the price, but there is no key; the sid doesn't exist, or something like that. Maybe a brute-force try-except? Anyway, for me, the broker use case is not of concern presently; just thought I'd point it out, since it came to mind.