Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Why does order_optimal_portfolio "over buy"?

I'm new to algorithmic trading and am trying to learn the Quantopian platform. To test my understanding so far, I tried creating a very simple algorithm that just picks the top 10 stocks by volume and does a weekly rebalance using order_optimal_portfolio with a TargetWeights objective where each of the 10 stocks are equally waited at 0.1. To my surprise, when I backtested this simple algorithm, order_optimal_portfolio always "over buys" by a smallish amount, resulting in a negative cash balance. I've read through many of the tutorials and most of the very long help page, and I can not deduce why I get this behavior. Can anyone provide any insight into this?

Thanks!
Ryan

4 responses

Hi Ryan, I've noticed that also, and have been lazy enough that I haven't stopped what I'm doing to nail it down.
You can probably find the answer by the route below and if you're inclined to let everyone know, that'd be great.

This will show detail of what's happening with the orders. In the past it could be done with a single copy & paste.
Just copy all of the code at track_orders() here and paste at the end of initialize().
Then 'Build Algorithm' to run your code.
Look at the logging window and then if you want to, you can copy/paste info from there for easier reading.
Since this is order_optimal_portfolio, some filling of orders can happen immediately instead of waiting for the next minute. The extra step is to capture order id's on ordering and let track_orders know about them, like this:

    ids = order_optimal_portfolio(  
        [ .... ]  
    )  
    for o in ids:  
        context.trac[o.id] = 0  

If there's an initialization error, then try running track_orders(context, data) once also at the start of initialize(), or simply context.trac = []. (If higher adoption were to ever occur I could rewrite that just a bit to not store id's at the top level of the context variable and make the whole thing better-behaved and easier to interpret).

Couple of things in more detail:
- It is possible that the negative cash you mention happens quite a ways into the backtest. The overall logging window could reach its limit before that happens or it might require quite a bit of scrolling to get there. If so, you can set a date to start recording using 'start' like ['2012-02-22'].
- Another way the logging window might hit a limit is daily, from your 10 stocks (if there are partial fills). If that happens, you could reduce to 5 stocks and see if the negative cash still occurs, changing from your .1 to .2 of course. You can make that dynamic with 1.0 / len(stocks) in your code.
- I copy the output from track_orders to Notepad++ where highlighting one of the order id's (last on each line) highlights all of the others also. Or doing a 'Find' for a symbol shows a list of all lines where it appears. And so on.
- The second-to-last item shown is cash. If there's a ton of output, you can 'Find' space-minus_sign for when cash drops below zero.
- Notice 'log_cancels' in the options, happens to be off by default. Probably would just make noise and isn't needed in this case but if you want to see any order cancellations, just set that to 1 to turn the option on.When there are cancels at the start of day, it means they were cancelled at the end of the previous trading day.

I may have over-delivered in the description above but in the past in pointing people to track-orders it is often followed by silence so I thought maybe people are confused by it, too many unknowns causing them fears or something. I decided to try something new this time.

My guess would be that you will find the following all true:
- Initial cash is high enough that even though they are top volume stocks, they are still hitting minute-level limits per the default slippage model.
- TargetWeights is ordering such that everything would be beautiful, ideally.
- The high dollar value of stocks being ordered is often generating partial fills.
- There are days when it so happens that reductions in stock allocations called-for (sells for long, buyback for short) do not fill as fast as increases or opens (new positions).

Hope you'll let us know how it turns out. If I've made mistakes above, it could get confusing, if so, don't hesitate to ask.

Thanks

When do you record the portfolio size? A common setup is order on open, record portfolio positions on close. This will cause slight deviations in the portfolio value when measured.

When you buy you have to buy whole number of shares, you can't buy fractional shares. There will always be some rounding error when converting from percentages in target weights to whole numbers (of shares).

Thanks for the replies! @Blue Seahawk, thanks for those details, it'll take me a little time to try to work through those ideas! Some clarifications:

  • My observations were not a result of calls to record or log. I was simply looking at the transactions and positions in the full backtest. Immediately following the very first order, it showed a negative cash balance.
  • I understand that it is (usually) not possible to buy whole shares and have the cost exactly match the cash available in the portfolio, but my expectation would be that numbers of shares would be rounded down such that cash doesn't go negative but gets as close as possible to zero.