Hi Abhijeet -

I'm not entirely clear what the Optimize API is doing. As you say, it "respects...the external constraints placed upon the algorithm" but what does this mean, precisely? For example, say this code is executed:

order_optimal_portfolio(  
  objective=TargetWeights({sid(24): 0.5, sid(5861): 0.5}),  
  constraints=[],  
)

Would you send the new weights to the broker, get some data back, complete the optimization, and then re-submit a new set of weights? Or does the broker get the new weights, apply another layer of optimization, and then send the orders on their merry way into the market? Or maybe since you know the universe of stocks the algo is using, the status of each stock is known prior to running the Optimize API (i.e. the algo is getting a real-time minutely (or more frequent?) feed of data from the broker, upon market open, for its universe)?

Also, what happens if order_optimal_portfolio is called multiple times within one trading minute (for both backtesting, paper trading, and real-money trading)? On the other post, you say "Optimize accounts for open market orders when determining target positions, unlike the order_target family of methods which only account for filled orders" which implies that if my algo calls order_optimal_portfolio more than once in a trading minute, for each subsequent call, any orders generated by prior calls will be considered completely unfilled (i.e. effectively not yet submitted)? Or are open orders only accounted for properly if order_optimal_portfolio is called at most once per trading minute?

Hi Grant,

This all happens before the order goes out to the broker. In a multi-strategy portfolio you have to consider the other algorithms as well as per-algo limits. In your example, your 50% AAPL portfolio might breach some risk constraints (e.g. maximum position concentration), or if you were to want to short the stock, it might not be available to short. The optimize call knows these constraints behind the scenes and will provide a portfolio that is actually feasible.

Hi Thomas -

So what happens if opt.calculate_optimal_portfolio is used during backtesting? Will it incorporate the additional constraints? And if so, where does it get the data (e.g. availability of shorts)? Why not expose the data?

Still a lot of details missing here on the mechanics of the mysterious external constraints. No big deal, I suppose, but one can waste a lot of time by being confused (and contacting Q support) when complete documentation will clear things up at the outset.

@Grant: No additional constraints are applied during backtesting, only the ones you provide. This only applies for a fund-algorithm. We try to make as much of that data available as possible. Usually we do a first cut to where we can use it internally and then productize it properly to make it available to the larger Q community, which takes some time.

The global constraints are not really anything a user needs to be concerned with, especially not during algorithm development, as they don't apply there. The optimization tries to get as close to the target portfolio provided by the algorithm as possible.

Thanks Thomas -

Hmm? Wouldn't that result in backtests that could be significantly different from real trading? Or once you identify candidate algos for the fund, you would apply the external constraints and run a backtest?

Also, what about my question regarding multiple calls to order_optimal_portfolio within a trading minute? Does the algo now track pending orders within a trading minute and account for them?

This all happens before the order goes out to the broker.

Just curious how this works. If you are getting a feed of short availability, for example, and it is minutely, then might it be kinda stale? In theory, the order could be transmitted to the broker up to ~50 seconds later.

1) Yes, applying global portfolio level constraints to an algorithm's portfolio can/will cause that algorithm's performance to deviate from a run without global portfolio constraints. Portfolio construction is an interesting technical problem.

2) If you make multiple calls to order_optimal_portfolio with the same parameters, it is a no-op. If you change the parameters, it will assume the previous orders aren't filled. I can't think of a good reason to do that in the current environment, but we want to support it potentially for the future.

3) Short availability is another interesting technical problem. Nobody can ever guarantee they will fill orders. Orders sometimes fail to be filled. We can work to make it less frequent and be more resilient when it does happen.

Hi Dan,

If I'm following correctly, when I run a backtest (and paper trading), only the constraints defined in my code are applied to the Optimize API (the "constraints provided by the author"). There could also be "external constraints placed upon the algorithm" which would not apply for ordinary algos. However, is it correct that external constraints would be imposed on a fund algo? And as a user, I would then see the effect on backtests? Or perhaps the external constraints would be applied, but only Quantopian could see the effect on the backtest? Or would external constraints only be applied at the point when the algo trades real money with the prime broker (i.e. only out-of-sample real money trading)?

Also, I'm still not clear on "Optimize accounts for open market orders when determining target positions, unlike the order_target family of methods which only account for filled orders" and when orders are considered pending versus submitted in backtesting and real-money trading. Under backtesting, presumably all orders are considered unfilled until the next whole trading minute, and the same would apply with your prime broker, assuming that you'll still be getting minutely feedback. Presumably, any open orders (partial fills left over from the prior trading minute and orders already submitted within the trading minute) will be taken into account by the Optimize API, and that order status is only available minutely, for both backtesting/paper trading and real money trading with your prime broker.

By the way, why not just use the name of your prime broker. Eventually, I'd think that as part of engaging institutional investors, you'd have to reveal it, per regulatory requirements, no?

Q - Thanks! This has saved me hundreds of lines of code.

Attached is a most simple algorithm as a starting point.

It would be refreshingly transparent and interesting if Q could "open source" how the "global optimization" problem is being approached, both in selecting algos for the fund that will be accretive and applying the "external constraints" when executing the fund trading. I suppose it is understandable that the details would be kept confidential, but it is another opportunity to teach quantitative finance to the masses.

Having updated a bunch of code to use order_optimal_portfolio() I have found that it does not appear to exit positions before entering new ones.

Say I have 1 stock (MU) that has weight 1.0 with MaxGrossExposure(1.0). It fills.
Some ticks later I want to exit MU and replace it with INTC with weight = 1.0

With MaxGrossExposure=1.0 I would expect that MU would be exited and then INTC would be filled.

Unfortunately that doesn't seem to be happening. See for example April 27th in the attached backtest.

If I really have to manually sequence my calls to order_optimal_portfolio() to manage leverage then most of the code I just got rid of will have to come back.

Other than manually checking, is there a way to ensure the sequence of orders?

I used the new Optimize API in two of my contest entries, however the 2 year contest backtest results are significantly different from the research backtest results (7% vs. 58%). The set up is to place all orders at 9:32am and close all at 3:56pm. In research mode, this implementation is done correctly. I understand that this could change in contest/live trading environment depending on many factors such as whether orders are filled accordingly as intended by the code, short order inventory, etc. As mentioned... "Optimize handles these kinds of situations by getting you to an achievable portfolio that's as close as possible to the originally desired portfolio, but also respects both the external constraints placed upon the algorithm and the constraints provided by the author." In my example, close as possible, is not achieved by optimize api. Short order inventory is probably why my short orders at filled partially at one minute intervals, this I understand. However, buy market orders are also partially filled at minute intervals, this baffles me because these buy orders are not large and these are very liquid stocks. What would cause these buy market orders to not be filled instantaneously as there are in the research environment? These are what's causing the huge discrepancies in my results and very misleading in my choices of algorithms to field in the contest. I need more guidance on how to avoid these pitfalls.

@ Dan Dunn,

You said and I quote above:
1) Yes, applying global portfolio level constraints to an algorithm's portfolio can/will cause that algorithm's performance to deviate from a run without global portfolio constraints. Portfolio construction is an interesting technical problem.
3) Short availability is another interesting technical problem. Nobody can ever guarantee they will fill orders. Orders sometimes fail to be filled. We can work to make it less frequent and be more resilient when it does happen.

My personal experience as stated in prior reply regarding huge discrepancies in contest and research backresults makes me want to to scrutinize the details of how optimize api works. For example, does Q know apriori short inventory of stocks from broker and if so, is this info incorporated in the optimize algo. If it doesn't, then the stock selection process is hit or miss even if you pipe in Dollar Volume to get highly liquid stocks. Also, if you factor in other constraints like leverage/exposure, market neutral, sector neutral...is partial filling of shorts due to lack of short inventory adjusting the overall optimization process, i.e, which takes precendence?

Is it possible to 'catch' the positions the Optimize API is suggesting along with their weights inside your algorithm so you can then use that data to help your performance?

See calculate_optimal_portfolio on the help page:

Returns:

optimal_portfolio (pd.Series) – A Series containing portfolio weights
that maximize (or minimize) objective without violating any
constraints. Weights should be interpreted in the same way as
current_portfolio.