What happens if we decrease all the weights by 1/10th and increase the starting capital from $10mm to $100mm? You'd think the effect on portfolio construction would be negligible. However, this is not the case. It doesn't order anything, because it clips away any weights below some 0.01%.

Using the old order_target... functions instead of order_optimal_portfolioand it works exactly as one would expect.

There's something wrong with order_optimal_portfolio. How is anybody constructing diversified portfolios with a smooth gradient of portfolio weights if the function simply tosses all the positions weighted lower than 0.01%? 0.01% times ~2000 QTU stocks means that up to 20% of an intended portfolio ($2mm out of a $10mm book) could be unaccounted for. That's not an insignificant contribution!

In case run_optimization() might shed some light on it.

The magic number is .0001. If the absolute value of a weight is less than .0001 it will effectively be 'rounded' to zero. That's what's causing the behavior you are seeing.

Before getting into the 'why' let's look at the 'what'. For a $10M portfolio, the order_optimal_portfolio method will effectively not open any position with an absolute weight less than .0001 or $1000 (.0001 x $10,000,000). It will also close any existing position with a value less than $1000. For a stock that trades at $20 that's 50 shares. That of course is a percentage. For a $1M portfolio that minimum is only $100.

Why is this? This was introduced to 'fix' the issue of the optimizer leaving only a few shares of a stock in one's portfolio. This issue had the undesirable consequence of potentially leaving a stock, which may no longer be in the QTradableUniverse (QTU), in one's portfolio. Having too many stocks for too long not in the QTU can disqualify an algo from the contest. So, now the order_optimal_portfolio method will close any positions with a portfolio weight less than .0001 to avoid this. The behavior now also acts as a behind the scenes 'housekeeper' closing (or never opening) very small positions.

This minimum weight is actually implemented as adding an extra penalty on these target-zero positions. It isn't a "hard constraint" that a position can never be under .0001. If the optimizer cannot find any other solution to satisfy all the constraints there is the potential for some positions to be weighted less than .0001. It effectively acts as a "soft constraint".

I made a change to the algo above which "clips" all the weights to a value a little more than .0001. This ensures all the target weights are above that magic number and do not get 'rounded' to zero. The algo now runs as expected. The number of securities ordered by order_optimal_portfolio is generally equal to the number of securities returned by the pipe. There are times where the portfolio positions are greater than the pipeline however, these are due to close orders not completely filling by the end of the day.

Hope that helps explain the sometimes baffling behavior of the optimizer.

Awesome explanation Dan, as always. Thank you!

Thanks, Dan. I agree that a $1000 position in a $10mm portfolio is insignificant. However, once you have 600 $1000 positions disappearing, it starts causing unintended consequences with the gross leverage and also some effect on reducing the amount of diversification present.

I tend to use the opt.CannotHold(set(context.portfolio.positions) - set(qtu)) constraint to make sure I'm not holding any non-QTU positions. I also make sure their weights are set to 0.

Thanks for the clip example code! That'll probably help. I'll just round up to the cutoff for weights that are close enough.