@Grant, that is a great idea. It would give you the ability to provide moving and adapting targets to the Optimize API. Make its boundaries flexible and even provide it with tracking functions. Add some leeway or constrain these functions based on some market data or other sentiment functions. I would like to see that too.

If I understand correctly, the idea is to run calculate_optimal_portfolio each day and save the resulting weights. Then, calculate the exponentially weighted moving average of those weights and use that "weighted" weight as the actual target weight.

One idea is to store each days weights in a dataframe. One row for each day. One column for each security. Use the 'append' method (it gracefully aligns columns and adds any as needed). Then simply use the 'ewm' method to get the EWMA of that data frame.

May need to delete rows of the dataframe if running a long backtest. Also need to tune the ewm method to get the desired weighting. Take a look at the attached notebook for a start...

Thanks Dan -

Yes, that's the basic idea. I should be able to get something running, based on your example.

Note that exponential smoothing is recursive and only needs the last value. So there's no need to store multiple days of values in a dataframe. There's a start-up issue, however, in that the smoothing won't kick in until enough time has passed for the smoothing (filter) to do it's job. One fix for this would be just to hold off trading for a period of time.

The right way to handle the start-up issue would be to get a trailing window of alpha values out of Pipeline at the start of a backtest/live trading, but last I heard, this is not possible, correct?

@Dan, that is a great start, impressive. Thanks.

How about having another EMA function to direct the output of the last_ewma_weight_series in the desired direction. As in, start with the last_ewma_weight_series, and try to go to: last_ewma_weight_series ± ϵ, where ϵ is trying to override the objective without overshooting its boundaries.

The purpose being not only to adapt to the objectives, but also trying to partially provide some outside control. You still need to normalize the weights before and after otherwise you might easily overshoot the constraints, not to mention leveraging limits, beta and dollar neutrality. You want to somehow gain some control the thing and stay within the boundaries, but not necessarily on the middle mark. Kind of giving some leeway, but in the direction you want instead of having the Optimize API making the decision for you.

Any idea as to how CPU-intensive this could be on say 200-300 stocks with a 30-90 day lookback?

Hi Guy,

Nice idea. That might be another way to do the smoothing. Basically, take the output of the optimizer (from the prior period) and combine it with the new alpha values (current period) and run it though again, in an iterative fashion. The degree of smoothing would be controlled by the relative weights of the old optimizer output and the new alpha values (similar to the recursion relation for exponential smoothing). In this sense, the new alpha values would be an adjustment to the existing portfolio weights, versus a whole new set of values for the optimizer to crunch on, potentially leading to unproductive turnover.

@Grant, yes. The Optimize API in its strict current boundary definitions is doing a lot of unnecessary churning. Providing more blurry lines to expand or contract those boundaries based on our perception of where the market is going could help reduce this excessive churning, reduce trading costs and even increase overall profitability. This leeway is needed in order to gain better control over what we want to do.

Because of the weighing constraints, if one weight of one stock changes from one period to the next, the consequence is that all stocks will have their respective inventories readjusted. And no one, as yet, has demonstrated to my satisfaction that it was economically justified to do so. The objective evidently being to reduce white noise trading.

@ Guy -

Not that you don't have to use the constraints. For example:

objective = opt.TargetWeights(pipeline_data.combined_alpha)

order_optimal_portfolio(  
        objective=objective,  
        constraints=[],  
        )  

For the new contest, given that the judging is based solely on the volatility-normalized return, this might be the best choice, assuming that all of the constraints are met without engaging the fanciness of the Optimize API.

@Grant, yes, agree. Note that I have nothing against the Optimize API, nonetheless, a lot of its trades are the result of white noise. Trading white noise is unproductive, if not a waste of resources. If white noise triggers a trade, it should be considered as an allocation error, meaning that it should not have occurred for whatever its reason. We could allow this if it were exceptions that happen once in a while as part of the game, the software, or the trading environment. But, when it exceeds 30% of all trades, it should be considered too much. While we are allocating resources to white noise trades, we are not allocating it to potentially better trades.

Presently, I do see the Optimize API as a “black box” which gives me the impression I have little real control. I do not have access to any of its internal functions.

However, based on your first post and @Dan's notebook where he proposes a way to gain some control over the API, I find it more interesting. I work a lot with objective functions, my aim has always been to maximize profits first, even use leverage if I found it applicable. This often ignoring constraints for low volatility, low beta, even low drawdowns. As long as they were within acceptable limits it was OK. However, a basic requirement was: the strategy has to last a long time, and not fizzle out right out of the gate.

I view this “trading game” as a CAGR game where our objective should be to reduce the doubling time as much as we possibly can. And this implies a long-term vision of things with an eye on the long-term CAGR, which I do not see in the API. Having outside controlling functions might just help to guide the API where I want it to go.

Here's an example of one approach. Will post with TAU_SMOOTH=3 next.

    TAU_SMOOTH = 1 # smoothing time constant (days)  
    ALPHA_SMOOTH = 1 - np.exp(-1.0/TAU_SMOOTH)

    weights = opt.calculate_optimal_portfolio(  
        objective=objective,  
        constraints=constraints,  
        )  
    # smooth weights  
    weights = ALPHA_SMOOTH*weights  
    weights = weights.add((1-ALPHA_SMOOTH)*context.weights,fill_value=0)  
    context.weights = weights  
    # order using smoothed weights  
    objective = opt.TargetWeights(weights)  
    order_optimal_portfolio(  
        objective=objective,  
        constraints=constraints,  
        )  

Example with longer smoothing time constant, TAU_SMOOTH=3.

Note that in the above example, one could replace context.weights with the current weights of the portfolio. Is there a built-in function that will return the required Pandas Series? Or would I need to write my own?

@ Guy -

Regarding the Optimize API and churn, I think the fundamental problem is that everything is wonderful when there is no cost in transactions (commissions & slippage)--optimize at will. However, as soon as one adds a cost for managing risk, then it becomes a more complicated problem. I'm getting the sense that the Optimize API is a bit too naive, in that it assumes negligible transaction costs. For example, the short_term_reversal risk is the 14-day RSI, so managing that risk will result in a lot of churn, since it would be expected to vary on a relatively short time frame.

@Grant, yes, your analysis is right on. Following your last strategy contribution, I extended the trading interval to 6 years, made no change to the code. Saw the strategy's CAGR melt down to about 2.1%. It was not surprising since: E[R(p)] = r_f + β∙(E[R(m)] - r_f) – β∙E[R(m)] = r_f. Implicitly saying that the outcome of this type of strategy should result in something close to the risk-free rate.

The merit of the strategy is that it is getting a low volatility equity line (almost flat) with its corresponding low drawdowns and near zero beta. It should be considered a real plus. There are applications for that kind of behavior.

However, at a 2.1% CAGR, one should start considering if it is worth the effort to do all that work, even if it is a machine doing it. In terms I like to measure, this strategy has a doubling time of 33.3 years! And this is if it does not deteriorate more as you continue to increase the trading interval.

Note that in its tear sheet, the Cumulative Return on Logarithmic Scale graph shows no real alpha either. As a matter of fact, it has an increasing negative alpha, and will continue to deteriorate. This strategy simply underperform its peers. And, this might be viewed as, say non-constructive, but, it is nonetheless designed to do so.

In my last post, I was generous when saying about 30% of trades were the result of noise rebalancing. That number is much higher. Every day, all the positions are affected simply because one of the stock price moved, and this has a cascading effect over the entire portfolio over the whole trading interval.

I do not see how, anyone could leverage this script to 6x. It simply would not even cover its leveraging costs. One should look closer at what this strategy is really doing and why it is doing it.

You have this moving blob of price variances where you want to extract, by trading, something that should result in: F(0) + Σ(q∙Δp) > B&H. And all you get is: F(0) + Σ(q∙Δp) ≈ r_f.

I would say: “Houston, we have a problem”.

Regarding another of GK's points: "the sense that the Optimize API [...] assumes negligible transaction costs"

To find out, since Optimize attempts to fill right away, only partial and unfilled orders are visible the minute after. To calculate commissions, an extra step, save the list of order id's returned by order_optimal_portfolio and add their get_order(order_id).commissions in succeeding minutes when fully filled using status.

@Guy you are spot on in stating "Every day, all the positions are affected simply because one of the stock price moved, and this has a cascading effect over the entire portfolio over the whole trading interval."

The 'churn' this creates generates transaction costs, and turnover, which need to be considered in ones strategy. The idea of 'smoothing' the optimize results doesn't exactly minimize this 'churn' it simply changes it. Smoothing the results is analogous to airbrushing a picture. It does a good job for some things (eg my picture on Facebook) but perhaps isn't the best tool for others (eg that x-ray of my broken foot). In the latter case details are important.

@Grant is correct that the "Optimize API is a bit too naive". We just need to make sure that we aren't similarly naive when using it.

So, what I do to minimize churn is to use the 'Frozen' constraint. In general, anything that has a gain less than the transaction cost is 'frozen'. Something like this.

    # Create a TargetWeights object using desired weights  
    # This will become our ordering objective  
    equal_weight_objective = opt.TargetWeights(context.output.weight)  


    # Next 'freeze' any securities that we don't want to trade  
    # Here we simply put a min and max gain limit  
    MIN_GAIN, MAX_LOSS = .005, .005  
    for security, position in context.portfolio.positions.items():  
        context.output.at[security, 'gain'] = (position.last_sale_price / position.cost_basis) - 1  


    freeze_these = context.output.query('(gain < @MIN_GAIN) & (-gain < @MAX_LOSS)').index.tolist()  
    freeze_constraint = opt.Frozen(freeze_these)  

    # Finally, execute the order_optimal_portfolio method  
    order_optimal_portfolio(objective = context.equal_weight_objective, constraints = [freeze_constraint])  

The above is pretty simplistic, and more thought should be put into accounting for short positions, but shows the basic idea. Go through whatever logic one has to open, close, and rebalance positions. Then, determine what held securities (if any) to keep 'as is'. Then simply add those securities to a 'Frozen' constraint. The optimize framework is elegant in that this logic can be independent of one another.

In a similar fashion, the 'CannotHold' constraint could be used if one wanted to close a position entirely.

@ Dan -

I agree regarding the smoothing (which is a recursive low-pass filter); it just puts things on the right time scale (which is handy). What I'd like to do is use the TargetWeights objective with a constraint on the forecast profit of the portfolio adjustment. If the profit will not exceed a certain level, then no update is done (or perhaps only an adjustment for risk is made). Is this feasible?

@Dan, yes. Agree. We need to gain control over what the API does. And by setting some “min and max gain limit” as you propose, would indeed reduce the churn. It also has added benefits.

Your “CannotHold” constraint would also be an add-on. Like you do not want stocks that do not move, or are illiquid, that somehow might have made the list from its fringe for some reason or other.

A “DoHold” constraint under certain conditions could also be an add-on. It would permit to override the API settings, also reduce churn and maybe increase profits.

Stocks with rising prices need not have their inventory reduced because they moved or some other stock moved. We are used to protect ourselves with stop losses, but here we are putting on some stop profits instead and hoping to achieve more. Is there not a contradiction in there.

It is why what you and @Grant propose makes a lot of sense. Override some of the weaknesses, alleviate the turnover, reduce trading expenses. There will probably be a cost: slightly higher volatility and drawdowns, but still manageable.

It looks like the FactorExposure constraint might do the trick (if I understand it correctly) for setting a profit range. It has the same form as the dot-product in Section 4.2 of https://arxiv.org/pdf/1206.4626.pdf, which sets a lower limit on the expected return.

Have to play around with it when I get the chance. It would be an interesting exercise to formulate the classic OLMAR paper using the Optimize API. Should be doable.

class FactorExposure(loadings, min_exposures, max_exposures)

Constraint requiring bounded net exposure to a set of risk factors.

Factor loadings are specified as a DataFrame of floats whose columns are factor labels and whose index contains Assets. Minimum and maximum factor exposures are specified as maps from factor label to min/max net exposure.

For each column in the loadings frame, we constrain:

(new_weights * loadings[column]).sum() >= min_exposure[column]  
(new_weights * loadings[column]).sum() <= max_exposure[column]

Parameters: 

    loadings (pd.DataFrame) -- An (assets x labels) frame of weights for each (asset, factor) pair.  
    min_exposures (dict or pd.Series) -- Minimum net exposure values for each factor.  
    max_exposures (dict or pd.Series) -- Maximum net exposure values for each factor.  

@Grant, the paper you mention has extraordinary numbers. Performance levels going through the roof. I remember reading that study a few years back.

There is only this one thing. Their next paper did add some consideration for commissions. And it was sufficient to almost totally destroy whatever alpha they thought they had “found”.

I think it is why we should always account for frictional costs even if they are at times just estimates of what they could be. At least, we could make them higher than they really are in order to force us to do better under adverse conditions. IMHO.