With a monthly snapshot I'm seeing HERO as an outlier in pnl as of 12-2015 at -40k (with next nearest CONN at -3.3k). It delisted 6-2016.
At the dip toward the end, PVAC shows up at -152k when CONN is -5.4k. PVAC is short while the price goes from 5 to 49.
Maybe verify data with any splits are ok on both of those and then experiment with modifying the factors and try other factors.
If factors for example had PVAC going long instead of short, that would have been great.
Stocks on both ends of the pnl win/loss spectrum tend to be short more than long.

As a temporary diagnostic test just to see the difference it would make you can tell optimize to ignore those two stocks:

constraints.append(opt.CannotHold([symbol('HERO'), symbol('PVAC')]))  

Thanks for taking the time to take a look at this crazy backtest.

I definitely need to play around with the constraints. Also, probably having some kind of stop loss on the shorts would make sense as right now the model just picks stocks once per month and does nothing until a month later.

Shorts inherently will have bigger negative tail risks and lower positive tail outcomes than longs (unlimited loss, limited gain vs unlimited gain, limited loss for longs). Risk control methods to limit large losses on shorts usually pay off over time...

Right now, I have to say that I'm not yet that comfortable with the optimize() API. I'd prefer to first develop an equal weighted strategy that works with order_percent(), and only after I can see that it works try to get it working with optimize().

Hi Mattias,

You're running into a few interesting problems simultaneously.

As others have noted in the thread, your two big drawdowns are caused by short positions in penny stocks (HERO and PVAC), that experienced major spikes upward while your algo held a short position. That should be surprising, however, because your algorithm is trying to filter down to stocks with prices greater than 5 dollars during its universe construction, so your algorithm probably wasn't meant to be holding those positions in the first place.

Your algorithm ends up holding shorts in those penny stocks because of how the optimizer handles positions that are in your portfolio but not in the vector of alphas you passed to MaximizeAlpha. The optimizer needs to assume some alpha value for the assets that dropped out of your universe, because we need to calculate a new target portfolio, which might have to include those assets (we can't simply exit those positions because you might have, say, a MaxTurnover constraint). The current implementation assumes an alpha value of zero for assets that weren't provided to MaximizeAlpha. That assumption works out poorly for your algorithm, because all of your alpha values are positive (as a side note, generally speaking you want negative alpha values for your expected shorts and positive alphas for your expected longs), so the optimizer ends up using the assets that have dropped out of your portfolio as hedges against your other positions. Needless to say, this isn't the behavior anyone wants or expects. We're currently investigating ways of making MaximizeAlpha handle these cases better.

The other problem you're running into is that PVAC was actually delisted for about a year during the period where your algorithm held it. Normally the backtester would automatically liquidate the delisted position, but we don't currently model the case where an asset is listed, gets delisted, and then subsequently gets re-listed (the backtester just sees this as a long period of zero volume and no price changes). When PVAC re-lists, it does so at a totally different price, likely because the company re-capitalized and/or experienced other corporate actions that we don't have because they occurred while the company wasn't publicly traded.

I've attached a notebook with a more in-depth analysis of the issues I just outlined. The notebook also makes use of the pretty wide variety of tools available for investigating backtests in research, which hopefully makes for an interesting case study.

Wow! Thank you Scott! You spent a lot of time on this!

I have never used the backtest tearsheets before, but I can immediately see their huge value in this scenario. I'm going to study your notebook carefully to use it as a template in the future. In addition to the largest long and short positions of the backtest period, it would be cool to have a table that could give me individual positions responsible for largest P&L moves (they are usually correlated but not exactly the same as largest positions). The rolling Fama single factors chart is unbelievably cool! Really useful to identify which factors are working and when they are working!

Now, I understand why things started going wrong when I removed the zscore() in order to make my quadratic manipulation. In the workshop last week, one of the people in my table commented to Delaney that Rank and Zscore were serving the same purpose in terms of normalizing the values and were not both really necessary in this case. In reality, the Zscore() call was used as a quick way to transform half the rankings around a 0 mean with half the population with negative values...

While I understand that optimize() needs to keep track of old universe values to effectively get rid of them after they are no longer part of the universe, I think it would make a lot of sense to have a constraint that can force optimize() to only have active positions in stocks that do not have an absolute 0 vector passed into MaximizeAlpha().

Thanks again for spending so much time to respond to my query.

Best,

Mattias

While I understand that optimize() needs to keep track of old universe values to effectively get rid of them after they are no longer part of the universe, I think it would make a lot of sense to have a constraint that can force optimize() to only have active positions in stocks that do not have an absolute 0 vector passed into MaximizeAlpha().

My current thinking on this is that MaximizeAlpha needs to have a separate (much more strongly-weighted) term in its optimization to account for the desire to exit positions in assets no longer in the the trading universe. Right now, the core of MaximizeAlpha builds a CVXPY objective by doing, roughly:

cvxpy_objective = cvx.Maximize(new_weights.T * alphas)  

There's really no way in this formulation to express a desire to not hold an asset. Better would be something like:

expected_alpha = new_weights.T * alphas  
# Penalize continuing to hold positions in assets that aren't in our alphas.  
penalty_coefficients = [0 if asset in alphas.index else LARGE_NUMBER  
                        for asset in current_weights.index]  
non_universe_penalty = cvx.abs(new_weights.T) * np.array(penalty_coefficients)  
cvxpy_objective = cvx.Maximize(expected_alpha - non_universe_penalty)  

The idea here is to use the non_universe_penalty term to penalize the magnitude of any held position for which an alpha wasn't supplied. If LARGE_NUMBER is sufficiently large relative to alpha values (doing this correctly probably requires re-scaling of alphas, which is safe as long as you just multiply every alpha by the same constant), then in practice the penalty operates like a constraint unless incurring the penalty is necessary to satisfy a constraint.

Holy cow. Scott, that notebook is amazing.

@Scott, that is really a great notebook. Thanks.

Hi Scott -

generally speaking you want negative alpha values for your expected shorts and positive alphas for your expected longs

I haven't digested all of this, but I'd assumed that the Optimize API could be fed normalized alphas that are not demeaned. But I think you are saying if the input is not demeaned, then stocks that should be reduced in weight will be highly weighted (since you are setting the alpha values of stocks not in the universe to zero)? Is this the correct interpretation?

Is a work-around simply to demean the input to the Optimize API?

Also, what does order_optimal_portfolio do? I'd expected that it would attempt to close out any positions of stocks not in the current universe, with a separate set of orders:

for stock in context.portfolio.positions.keys():  
        if stock not in universe:  
            if data.can_trade(stock):  
                order_target_percent(stock,0)  

But apparently not. I guess I'm unclear how stocks get dropped from the universe without code similar to what I have above.

Wouldn't you simply run the optimize API on the current universe only, and then close out any positions not in the universe? Or is the paradigm that changes in the universe shouldn't dictate changes in positions?

If one wanted to assure stocks dropped from the universe are actually dropped, would it mean running a separate routine, prior to the final optimize API run, to close out positions of stocks no longer in the universe?

Regarding the listing-de-listing-listing issue, when a stock gets listed, shouldn't it be assigned a new SID? Wouldn't this solve the problem? If a stock gets de-listed, then when it is re-listed, I'd think a new SID would be appropriate. Even if it subsequently trades under the same symbol, as soon as it disappears, within Quantopian, one has no visibility of it. It is effectively a new issue when it gets re-listed, right?

Injecting stocks into the pipeline from within order_optimal_portfolio with zero alphas can have a lot of unintended and random consequences. Please fix this issue or alternately expose a flag to only rank pipeline and not add positions to it (which should be default behavior everybody expects).

Not sure why in the code we are adding existing positions to optimize routine. If we do optimize on just the pipeline stocks and add a routine at the end to close out any positions that are not in pipeline, it should achieve the intuitive and expected behavior of order_optimal_portfolio.

It all depends on what one wants to achieve with the Optimize API and order_optimal_portfolio. Looking at https://blog.quantopian.com/a-professional-quant-equity-workflow/, there is an element of risk management in this step, so even though the universe changes, it may be too risky to make a dramatic change all at once. Optimization might require holding onto stocks no longer in the universe (but then how does one eventually exit them definitively?). However, this could end up being belt-and-suspenders, since presumably Q is cobbling together the orders from various Q fund algos and applying another layer of constrained optimization, where they could apply such risk management, and also their prime broker could do it, too, depending on what functionality is included in the Execution step on https://blog.quantopian.com/a-professional-quant-equity-workflow/.

I'd say keep the backtester straightforward and intuitive, and push the business of how to close out positions no longer in the universe to the fund construction and execution steps.

What everybody wants and expects should be supported in a code path via the default behavior or through a flag.

We rather deal with the problem of ranking stocks in the pipeline according to our algorithm and getting a resultant portfolio based solely on what we ranked, rather than have to deal with the complexity of non-pipeline stocks lingering around in the portfolio for reasons we cannot possibly comprehend or deal with in our ranking algorithms.

Here is an example strategy that illustrates why it will be problematic to optimize by merging positions into pipeline using zero alpha.

Say your pipeline filter is for stocks > 5 equity price plus some other filters
Your alphas/pipeline weights are 1 day returns
You enter short 10 of the smallest returns and long 10 of the largest returns
If the stocks from prior day positions fall below 5 then they are filtered in the pipeline and not ranked
If optimizer sets zero alphas to them (<5 priced prior day stock positions) then on days when there is up market and all stocks in the rest of your universe are up we will be entering shorts on positions from prior day that are < 5 and zero alpha (meaning lowest weight). That will be contrary to the algorithm strategy and no way to work around it.

Hmm I always thought order_optimal_portfolio() would liquidate all positions not in the weights vector.

@ Scott -

Within CVXPY, wouldn't it simply be a matter of adding hard constraints? For example, x[0] == 0, where x[0] is the weight for a stock that should not be held? Wouldn't this force zeros where they should be, or result in an exception if a solution can't be found? It just seems like it would be a more definite approach, unless you are seeing the need to provide users with the ability to set the LARGE_NUMBER you describe above?

@Grant

I haven't digested all of this, but I'd assumed that the Optimize API could be fed normalized alphas that are not demeaned. But I think you are saying if the input is not demeaned, then stocks that should be reduced in weight will be highly weighted (since you are setting the alpha values of stocks not in the universe to zero)? Is this the correct interpretation?

The answer to this question depends on what constraints are being applied.

In general, the behavior of MaximizeAlpha is that, given a vector of alpha values, a[0], a[1], ..., a[N], we find a vector of new target portfolio weights, w[0], w[1], ..., w[N] that maximizes a[0] * w[0] + a[1] * w[1], ... a[N] * w[N] while respecting any additional constraints that have been supplied.

Some observations that fall out of that mathematical definition are:

1. Without any additional constraints, MaximizeAlpha can't produce a portfolio at all, because there's no upper bound on the function it's trying to maximize.
   
2. With just a MaxGrossLeverage constraint, MaximizeAlpha will allocate all of its capital to the asset with the largest alpha magnitude, going short if the sign of that alpha is negative, and going long if the sign of that alpha is positive.
   
3. With just MaxGrossLeverage and PositionConcentration, MaximizeAlpha will proceed through the assets in order of alpha magnitude, allocating the maximum-size long or short position to each, depending on the sign of the alpha value. One implication of this is that if you use ranks as your "alphas" you'll always end up with a long-only portfolio with these constraints.
   
4. With MaxGrossLeverage, PositionConcentration, and DollarNeutral constraints, the behavior of MaximizeAlpha ends up being roughly what you'd get from an algorithm like: "Choose the N largest and N smallest assets by alpha value and allocate the maximum-size long to the largest assets and the maximum-size short to the smallest assets." This is where we run afoul of MaximizeAlpha's current behavior of assuming an alpha value of zero for assets with no alpha provided. If all the provided alphas are greater than zero, then the "N smallest assets" are all the zeros that we inserted for the assets that weren't provided, which makes them the most attractive targets for short positions.

As you add in additional constraints, the behavior becomes more complex to describe succinctly. One intuition that I find helpful is to think of each constraint as defining one or more "budgets" that positions contribute to at different rates, and to think of the alpha vector you provide to MaximizeAlpha as defining the amount of "value" you get from holding a unit of each asset. The goal of the optimizer when using MaximizeAlpha is to produce the portfolio with the largest amount of value that still respects all the budgets impose by the constraints.

One thing that's important to stress is that the sign and magnitude of alphas matter, because they determine how attractive MaximizeAlphaconsiders each asset as it's deciding how best to make use of its budgets. In particular, assets with positive alpha values are attractive to hold, assets with negative alpha values are attractive to hold short.

The best way I've found for building intuition about how the optimizer responds to different inputs is to set up simple test cases in a notebook (like the ones in the middle section of the notebook I posted above) and see how the result of the optimization changes as I change inputs.

A few people in the thread have asked why the optimizer doesn't just close out positions in any assets not provided in the alpha vector. In general, we can't do that because there may be constraints (e.g. a user-provided MaxTurnover constraint) that prevent us from completely closing out an existing position. The penalty term version I've outlined above has the same behavior as a hard constraint in all the cases where it's possible to do so, but it also makes a best effort in cases where completely closing out the target positions would conflict with another hard constraint.

Thanks Scott -

One thing that's important to stress is that the sign and magnitude of alphas matter

It seems like you'd want a standard preprocessing step for alphas (consistent with what is done in ML). For example:

from sklearn import preprocessing  
import numpy as np

def preprocess(a):  
    a = np.nan_to_num(a - np.nanmean(a))  
    # return preprocessing.robust_scale(a) # security violation  
    return preprocessing.scale(a)  

Lots of alternatives listed here:

http://scikit-learn.org/stable/modules/preprocessing.html

(although perhaps not all available yet on Q)

The penalty term version I've outlined above has the same behavior as a hard constraint in all the cases where it's possible to do so, but it also makes a best effort in cases where completely closing out the target positions would conflict with another hard constraint.

It would seem that the scale of the penalty term would matter (e.g. for a sufficiently large penalty, I'd think that the turnover constraint wouldn't work on non-universe stocks). How would users know how to set it?

Seems overly complicated. The universe shouldn't be changing that frequently anyway, so its contribution to turnover should be minimal. It would be more intuitive simply to close positions no longer in the universe.

Is it possible to detect if user has set MaxTurnover in constraints and only do special case handling (of adding positions to alphas) if it is set? Is it possible to demean alphas if adding positions to alphas with alpha=zero?

I'd think that the turnover constraint wouldn't work on non-universe stocks

Constraints are unconditional requirements that must be satisfied by the new portfolio, so no change in the implementation of the objective will ever cause the optimizer to emit a portfolio that violates a constraints.

What does matter in the context of a penalty term is that the scale of the alpha-maximization term in the objective is smaller than the scale of the "close out alpha-less positions" term. The easiest way to achieve that is to ensure that the scale of the alpha-maximization term is fixed by re-scaling alphas to fall between -1 and 1. There are a couple ways you could imagine doing that, but the only semantics-preserving rescaling is to divide all the alphas by the absolute value of the largest-magnitude alpha, which is safe because the w that maximizes dot(w, a) also maximizes dot(w, a * N) as long as N is a positive constant.

How would users know how to set it?

My expectation is that users shouldn't have to set it, unless they're doing something unusual. The goals is for the behavior of MaximizeAlpha to be what most people in this thread already assumed, which is that positions in assets with no alphas provided are always closed out unless doing so is impossible. The only case where scaling the penalty term would matter is if you want fine-grained control over what happens in the tiny percentage of cases where closing out the undesired positions isn't possible

Is it possible to detect if user has set MaxTurnover in constraints and only do special case handling (of adding positions to alphas) if it is set?

There are more cases than just MaxTurnover that can prevent us from closing a position. In general, an arbitrarily complex combination of constraints can conspire to make it impossible to close an existing position, and the only robust way to find that out is to do the full optimization (or something morally equivalent to it). To take an extreme example, you could encode a Boolean Satisfiability problem as a set of Basket constraints.

Is it possible to demean alphas if adding positions to alphas with alpha=zero?

I don't think it would be appropriate for an objective to automatically de-mean alphas. Consider, for example, that providing all-positive alphas will yield a long-only portfolio in the absence of a DollarNeutral (or similar) constraint, but the same alphas de-meaned will yield a long-short portfolio.

Wouldn't the median (or mean?) alpha be a better choice for positions than an arbitrary number like zero when there is no expectation that algorithm developers demean their alphas prior to calling order_optimal_portfolio for their long short algorithms?

I'm glad that my original post created this interesting debate, when clearly the reason my backtest created these "anomalies" was because my constraints were diagonally opposed to each other - "Go right! No, Go Left!" (my alpha vectors were all positive, and I simultaneously set constraints for a dollar neutral portfolio and a Beta between -0.1 and +0.1). It does make sense that Alpha vectors should not be "a preference ranking" but should instead be "a real expected Alpha over a given time horizon". There should however be some optional constraint that prevents non-pipeline output from being in your portfolio - Who's in control of this algo I just wrote? Is it optimize() or is it me!? ;)

I have to say that this is a completely new way of thinking about a trading system's construction and it's hard for old trading school guys like me to adapt (my current live models have big arrays to keep track of state, and they are all based on conditional triggers causing a Buy, a Sell, a Take Profit or a Stop Loss signal for each individual asset) .

I can immediately see the edge that optimize has over the "classic signal trigger methodology": the trigger methodology is by definition focused on building a portfolio at the micro level without regard to the macro portfolio level. For example, a momentum strategy on the Q500US with moving average triggers could very well buy 30 energy energy stocks on the same day because of a big oil price move - this probably wouldn't make much sense on a risk/reward basis for the entire portfolio. Optimize() would clearly allow you to cap your risk more efficiently given certain constraints rather than creating sector position limits using "classic signal trigger methodology". I'm curious: is there a constraint that would allow you to combine MaximizeAlpha and "MinimizePortfolioVariance" so that you could combine an Alpha vector with a portfolio co-variance ranking? (i.e.: if 2 assets have a similar Alpha vector, Optimize() might as well try to overweight the asset which is more likely to act as a dampener on overall portfolio volatility).

Interesting observation @Mattias. I wonder if you could just combine the alpha vector and co-variance weightings with some type of averaging?

You could, for example, just multiply the alpha vector by the covariance weightings for a final portfolio output that optimize could use.

Hi Cheng,

It was nice to meet you at QuantCon Singapore. You showed me during the Hackathon that you can read Python code faster than I can read English! I was very impressed how you could read code structure and debug someone else's code in the blink of an eye...

Since my last post, I thought a little bit about this issue and it's a really complicated problem (which I'm sure many smart guys have already tried to tackle - as it would seem like a natural thing one would want to optimize). I remember Delaney briefly mentioning the complexity of a similar optimization problem during the workshop.

While I'm sure there are various manipulations that could be done to the Alpha vector using co-variances, I don't think your suggestion would work. An asset with close to zero covariance to all the assets in the universe and a very high expected Alpha should be a pretty sought after asset for any strategy's long book, but multiplying them together would give such an asset a middle close to 0 ranking, when it should get a very high ranking.

One either has to use 1/ a covariance matrix with all the other assets in the universe over a given time frame (that's a seriously big matrix to optimise a theoritical Sharpe ratio), or 2/ one has to iterate through each of the single covariances with each permutation of the portfolio minus each asset (you get a different covariance for each permutation of weights in the n-1 portfolio universe).

It would appear that there are 2 different ways to approach this problem: A/ one optimises the matrix mentioned in 1/ above to find out what weights would have produced the highest Sharpe Ratio using historical co-variances and Alpha vectors instead of historical performances, or B) one uses a recursive technique to approximate a historical weight "sweet spot". As Approach A is seeking to find an exact optimization of the past to make an approximation of the future, I would have thought that Approach B will be just as good at estimating what will be the best weights for the future. Depending on the number of iterations required to get a decent approximation, Approach B could also have much less computational requirements than Approach A, for a similar end result.

For approach B, I would probably try to use Fisher Black's "combination of 2 risky assets" formula (Variance_2assetportfolio = ((x1^2)(s1^2)) + (2*x1*x2*r12*s1*s2) + ((x2^2)(s2^2)) where x is weights, s is standard deviations and r is correlation). I would start with an equally weighed portfolio and then parse through the portfolio multiple times optimizing weights one at a time. I would suspect that even after 5 iterations of parsing through the entire portfolio, I would have a pretty good setting for my weights...

Great Point. The approach I mentioned fails to account for the correlation between the alpha vector and covariance, and will likely lead to lackluster results. That said, the approach B you mentioned seems very sound since it looks at both the variances and correlation in both vectors.

Let me know if that works out for you!

In case you haven't seen this yet, this post talks a lot about portfolio diversification techniques that could be useful: https://www.quantopian.com/posts/hierarchical-risk-parity-comparing-various-portfolio-diversification-techniques

Thanks for sharing this post. I had missed it.

Hi Scott,

Regarding the non_universe_penalty term you mentioned earlier in a post, I had an idea yesterday that we could vary the non_universe_penalty in such a way that positions that have been longer in the portfolio and not in current alpha vector get higher penalty than those positions that have been added recently (and not in alpha vector). This would ensure starvation problem doesn't occur while satisfying the max turnover constraint. Similar to how some caching mechanisms work getting rid of the oldest cache item to free up space for a new item.

Thanks,
Leo