Hi James,

What's the rebalance frequency of this algorithm? The QTradableStocksUS can change from day to day, so if an algorithm doesn't rebalance frequently, it can hold a few names that have fallen out of the QTU. That said, your turnover looks to be a in a good spot, and your position concentration is very low, so I don't think it's likely that this is the problem.

To debug this, I'd recommend finding the date which drops to 94.7%, and look at the names held which aren't in the QTU to see if there's a pattern. Unfortunately, I can't do this by cloning the notebook since the backtest wasn't publicized. If you'd like, you can create a support ticket with permission to see your notebooks (you don't have to give permission on the algo/backtest), and I can try to figure out what's happening.

@Jamie, thanks for the prompt response. I just posted a question on this at @Alisa thread and copy below:

I want to ask, based on your above statement, if I had an open position on IBM and the next day it was dropped from the QTU list, the algo will then try and close that position, how will the algo account for this transaction, as IBM now as in or out of the QTU list? I assume that a new QTU is updated before start of trading day and therefore has precedence over subsequent transaction during the day. Question is how a dropped stock from QTU list accounted for as it closes its postion during the day? This could be part of the problem if not accounted for properly, I think. Can you please check this.

This algo has daily rebalance. I am wondering if it is an accounting problem?

@James: The QTU % metric is checked at the end of each day, so that shouldn't be the issue. This really sounds odd. Any chance you'd be willing to share the notebook? If not, I can try to send you a modified version of the notebook you shared to give a helpful report about non-QTU positions held. Let me know.

@Jamie, I understand that QTU % metric is checked at the end of each day but you didn't answer my question on how the closing transaction of a dropped stock accounted for , is it still in the QTU list when it closed its position? If it is then this is the proper accounting. If not , this is where the problem is. So you have to check the precedence logic first and then check the accounting of the closing transaction.

James, the QTU is updated in the pipeline each morning.

I assume that a new QTU is updated before start of trading day

This is a correct assumption. The QTU is updated every day.

... and therefore has precedence over subsequent transaction during the day. Question is how a dropped stock from QTU list accounted for as it closes its postion during the day?

This question is harder to answer. In theory, the answer is 'yes', but dropping non-QTU stocks from the portfolio is ultimately controlled by the algorithm. If an algorithm checks the result of the pipeline each day, and the pipeline screens based on the QTU, the algorithm should close out of positions that fall out of the QTU.

Let me know if I'm misunderstanding the question.

@Jamie, I think you are misunderstanding my question. Let me try to put it in simple terms, a complete trade transaction cycle involves opening a trade and closing a trade. Say I open a trade on Monday to buy 100 shares of IBM which is on QTU list that day. On Tuesday before trading starts, QTU updates its list and IBM is taken off that list. At start of trading day, Tuesday, since IBM is off the list, algo will try to close that position ASAP. When it closes the position, how is IBM accounted for, it is still in the QTU list ? If it is, then it is acounted for correctly. If the closing transaction of IBM is accounted for as Non-QTU then the accounting is incorrect and this is reflected incorrectly also in the tally of QTU% metric at the end of the day. Hope this clarifies the question better.

James, that helps, thank you. Transactions are not accounted for in the QTU% metric. That metric only looks at positions held at the end of the day. In your example, if all 100 shares of IBM are closed on on the Tuesday, then they will not be included in the QTU% metric on Tuesday.

One issue could be partial fills. Going back to the example you provided, if only 90 of the 100 shares were closed out by EOD Tuesday, then you'd have 10 shares of IBM counted as non-QTU on Tuesday EOD.

That said, based on the position concentrations reported in your tearsheet, I don't think this is the problem. A QTU% of 94.7 suggests that there are 25+ tickers (at ~0.20% of the portfolio each) being held on a particular day outside of the QTU, which seems unlikely. I say 'unlikely' instead of 'impossible' because I really can't say for sure what's happening here without more data. If you don't feel comfortable sharing your notebook with our support team, I'll start working on a notebook for you to run that might help gather some data on what's happening.

(As always, let me know if I'm misunderstanding still!)

@Jamie, I'm OK to share the notebook. How do I do that? Just email support and attach notebook?

Thanks, James. You'll have to open a ticket by clicking 'Help' > 'Contact Support' at the top of the website. A modal will pop up where you can choose to grant permission for us to view your notebooks (grants temporary access).

Hi James -

I note your specific returns are negative. While this algo would be valid for the contest (once you fix the QTU error), my understanding is you'll need positive specific returns to be eligible for a Q Fund allocation.

Hi Grant,

Thanks for the info. I'm wondering what constitutes common returns. If it is the equivalent to the buy and hold returns of SPY, then that will be a very tall order for an equity market neutral strategy in a long term backtest analysis . It's possible for a market neutral strategy to beat market returns for one or two years but have yet to see one that consistently beat it in a 10-15 year evaluation period, Most industry reports I've seen on equity market neutral strategies use the 3 month Treasury Bills as their benchmark which is consistent and more in line with the strategy's expected returns and purpose.

I'm wondering what constitutes common returns.

My understanding is that any returns attributable to the Q risk factors will be categorized as "common"--what's left is called "specific." It's not a matter of beating SPY or referencing 3-month T-Bills as a benchmark, but rather steering clear of whatever set of formulae Q is using to attribute returns to what they consider common sources, versus unexplained sources (to get an allocation, you would still need to provide an explanation, in words, not code).

Since presumably Jamie is listening, I will whine that they have not published enough details on the risk model and its implementation. Namely, the underlying code should be published, along with a "white paper" describing the theoretical underpinnings and empirical evidence for its ability to control risk and generate profit for the Q fund.

The irony here is that all of this "risk management" may be irrelevant if too much leverage is applied. I just finished a book that discusses LTCM in one of its chapters. If the "risk management" is just a story so that excessive leverage can be applied, then it is just smoke-and-mirrors.

Regarding the Q version of the Sharpe ratio and using 3-month Treasury Bills as a benchmark, there is an extensive discussion here which lead nowhere, so I won't repeat it. I will note that this Q-sponsored example does use SHY as a benchmark:

 # Set benchmark to short-term Treasury note ETF (SHY) since strategy is dollar neutral  
    set_benchmark(sid(23911))  

This, I think, is an isolated example. Generally, I have not seen Q reference SHY or any other risk-free rate benchmark. It does seem that if the goal is to generate uncorrelated returns, then using SPY as a benchmark would not make sense ("beating the market" is not the game here).

It would be interesting to understand if this type of hedge fund historically has a significant edge over SHY or if individual funds tend to "mean revert" to the risk-free rate. Of course, what really matters is what the return is to the investor, so the return should be the ratio of money returned to the investor divided by money put up by the investor (i.e. accounting for any 2/20 type expenses).

Hi Grant,

Thanks for enlightening me on common returns. So common returns, correct me if I'm wrong, is the sum of their assumed style returns, i.e., short term reversal + size + value + momentum + volatility = common returns. Anything over or under the common returns of the algo's total returns is specific returns. What could specific returns then be, anything? If you noticed I bolded assumed style returns because that is what it is, a modeling assumption. I read somewhere that for short term reversal, Q uses the 14 day RSI. The above notebook example is an interesting case because the Stat Arb is a pure mean reversion strategy that would fall under the short term reversal category. Before I ran this strategy, I thought it would not pass this short term reversal constraint of no more than 40% but it did! As I am trying to understand how a pure mean reversion strategy like Stat Arb can pass the short term reversal constraint, it occured to me that could this faster pace mean reversion strategy stealthly gone under the radar of a much slower pace RSI14? Something to ponder upon. And here is where your call for code and whitepaper can come in handy for better understanding and testing its rationale and validity.

Also as I've mentioned before the traditional use of equity market neutral strategies by industry is as a 5-10% allocation to a multi-asset portfolio to temper overall market risks and based on empirical reports, it does its purpose effectively. However, I read somewhere that Q's intended use of this strategy is to leverage it 5-6 times in its Q hedge fund. While this may work in a low interest environment, it can quickly fall apart in a high interest environment as not only does the cost of borrowing increases but also the disparity in the spread on income from cash and interest in borrowing becomes wider. The last two days of market correction is attributable to two main factors: perceived overpriced market and perceived increase in interest rates, a lethal combination. A prelude to the perfect storm? Another LTCM?

I think common returns also include the sector exposures, but I'm not entirely sure.

Hi James -

Just curious - did you use the Optimize API to control the risk? You'd have something like this in your code:

    risk_model_exposure = opt.experimental.RiskModelExposure(  
        context.risk_loading_pipeline,  
        version=opt.Newest,  
    )  
    constraints.append(risk_model_exposure)  

Hi Grant,

No, didn't use any of that. As I mentioned to you before, I use Occam Razor's methodology which gives it a minimalist list of constraints to use in order to pass all contest requirements. The pitfall of using optimization with too many constraints is the uncertainties of its behavior due to too many degrees of freedom. It gets confused and ends up with suboptimal results !

FYI, there is an ongoing issue with the QTU constraint and I'm still sorting it out with Jamie. Hopefully, we can get it sorted out before Feb. 16 contest.

Well, you are pretty close on the short_term_reversal and so you might want to put in some risk control if you are planning this to be a contest entry.

I think I'm having the same issue, but from a different angle. I have an algorithm has a weekly rebalance using a pipeline screened by QTradableStocksUS(). And the one test that fails in the check is the same tradable universe check.
"FAIL: Minimum investment in QTradableStocksUS of 81.40% on 2016-07-26 is < 90.0%."

I have two ideas for other ways to go about this if others are having the same issue.
1) Lower the threshold to account for the daily removal in stocks from the daily-changing QTradableStocksUS, or;
2) Make this filter only apply to the buying of stocks (i.e. 90+% stocks bought or shorted must be from the QTradeableStocksUS).

One problem may be that the backtester can't always exit positions. For example, even though I'm using a "tradable" universe, I still get a lot of messages like:

Your order for 49603 shares of CSTM has been partially filled. 48247 shares were successfully purchased. 1356 shares were not filled by the end of day and were canceled.  

This suggests that even if one is desperate to close a position, due to the risk of it not being in the QTradableStocksUS, the backtester may choke (unrealistically, as I understand, since there should always be a price at which a position could be closed, I'd think).

The risk should be measured in dollars, not count of stock tickers. Is the constraint applied by counting the number of tickers held? If so, perhaps the constraint definition should be changed to measure the exposure in dollars?

The other weirdness here is that this rule kinda forces one to run a scrub every day to dump positions of stocks that are no longer in the QTradableStocksUS, but I'm not clear if this can be done without rebalancing the entire portfolio. Is there a way with order_optimal_portfolio to execute trades on individual stocks?

I understand the intent behind the "Trade Within QTradableStocksUS: >= 95%" but the implementation seems a bit of a hack at this point.

@ Jamie -

Perhaps you could shed some light on how one can close positions of stocks that are no longer in the QTU, without re-balancing the entire portfolio. It would seem that for algos that trade less frequently than daily, it would be a good idea to schedule a daily function that checks for QTU inclusion, and drops stocks that are no longer in it (without re-balancing the portfolio). Since the only way to place orders is through order_optimal_portfolio it is not clear how to accomplish this. Is the trick to use:

calculate_optimal_portfolio(objective, constraints, current_portfolio=None)  

and define current_portfolio to be only the stocks that are to be dropped? With target weights of zero for all of the stocks in current_portfolio would a set of market orders go in to close the positions, without generating other orders?

Of course, when the QTU inclusion management orders go through, the leverage will drop, but that would be fixed the next time the whole portfolio is re-balanced. In theory, I suppose that the QTU inclusion management scheme could include a constraint/check on leverage, but your 0.8X minimum is pretty generous, so I wouldn't expect a problem (unless the QTU is turning over a lot...how much does it change?).

Hi Grant,

Let's wait for Jamie's official answers. In the meantime, I have been trying to resolve this problem with Jamie via support channel and at least for my case in above Stat Arb algo I was able to find a workaround although it is not 100% foolproof, it worked in shorter runs (2-3years) but still falters in the 11 year backtest that is why I was hestitant to share this workaround right way till I found the absolute solution.

Here's what Jamie and I agreed on, for MaximizeAlpha construct, the closing routine on order_optimal_portfolio for newly dropped stocks works fine but for the TargetWeights construct, which is used by my algo, somehow does not work properly. My solution was to create code that ensures that newly dropped stocks from QTU list are closed first before the new set of weights is pass on to order_optimal_portfolio for rebalancing. The code is inserted before opt.TargetWeights:

    # Sell any positions in assets that are no longer in QTU list.  
    for security in context.portfolio.positions:  
        if data.can_trade(security):  # Work around inability to close de-listed stocks for QTU.  
            if security not in context.stocks:  
                to_close = [security]   # securities to be changed  
                ids = order_optimal_portfolio(  
                    objective   = opt.TargetWeights(  
                    pd.Series(0, index = to_close)  # the 0 means close them  
                ),  
                constraints = [  
                    opt.Frozen(  
                        set(context.portfolio.positions.keys()) - set(to_close)  
                    )  
                ]  
            )  

This code solved my problem on my latest 3 year backtest, however when I ran it through the 11year backtest above, it faltered sometime in 2015 and gave me the following errors:

InfeasibleConstraints: The attempted optimization failed because no portfolio could be found that
satisfied all required constraints.

The following special portfolios were spot checked and found to be in violation
of at least one constraint:

Target Portfolio (as provided to TargetWeights):

Would violate Frozen([Equity(32770 [DEI]), Equity(16389 [NCR]), ...]) because:
New weight for Equity(32770 [DEI]) (0.0) would not equal old weight (-0.00124713995777).
New weight for Equity(16389 [NCR]) (0.0) would not equal old weight (0.00127471453249).
New weight for Equity(38921 [LEA]) (0.0) would not equal old weight (-0.0012673465914).
New weight for Equity(2 [ARNC]) (0.0) would not equal old weight (0.00124764231739).
New weight for Equity(6161 [PRGO]) (0.0) would not equal old weight (0.0012355562442).
New weight for Equity(4117 [JCI]) (0.0) would not equal old weight (0.00114328492707).
New weight for Equity(4118 [JCP]) (0.0) would not equal old weight (0.00120445963172).
New weight for Equity(2071 [D]) (0.0) would not equal old weight (0.00110161134441).
New weight for Equity(4120 [JEC]) (0.0) would not equal old weight (0.0011990730669).
New weight for Equity(14372 [EIX]) (0.0) would not equal old weight (-0.00124497410696).
... (841 more)

Current Portfolio (at the time of the optimization):

Would violate Frozen([Equity(32770 [DEI]), Equity(16389 [NCR]), ...]) because:
New weight for Equity(49458 [MSG]) (nan) would not equal old weight (nan).

Empty Portfolio (no positions):

Would violate Frozen([Equity(32770 [DEI]), Equity(16389 [NCR]), ...]) because:
New weight for Equity(32770 [DEI]) (0.0) would not equal old weight (-0.00124713995777).
New weight for Equity(16389 [NCR]) (0.0) would not equal old weight (0.00127471453249).
New weight for Equity(38921 [LEA]) (0.0) would not equal old weight (-0.0012673465914).
New weight for Equity(2 [ARNC]) (0.0) would not equal old weight (0.00124764231739).
New weight for Equity(6161 [PRGO]) (0.0) would not equal old weight (0.0012355562442).
New weight for Equity(4117 [JCI]) (0.0) would not equal old weight (0.00114328492707).
New weight for Equity(4118 [JCP]) (0.0) would not equal old weight (0.00120445963172).
New weight for Equity(2071 [D]) (0.0) would not equal old weight (0.00110161134441).
New weight for Equity(4120 [JEC]) (0.0) would not equal old weight (0.0011990730669).
New weight for Equity(14372 [EIX]) (0.0) would not equal old weight (-0.00124497410696).
... (841 more) There was a runtime error on line 144.

I am still trying to find the source of this error and when I do, I'll share the complete fix.

for MaximizeAlpha construct, the closing rountine on order_optimal_portfolio for newly dropped stocks works fine but for the TargetWeights construct, which is used by my algo, somehow does not work properly.

I've seen some evidence of this, as well, but haven't dug into it. So is there officially a "bug" with TargetWeights? And is it in conjunction with order_optimal_portfolio or calculate_optimal_portfolio (or both)?

For what it's worth, below is the code I'm playing with. It includes a recursive exponential portfolio smoothing function, which incorporates the entire portfolio history, from the start of the backtest. So, it needs to be able to drop tickers that are no longer in the QTU. I'm not entirely sure it is doing this, given that I don't code for it explicitly.

I find the Optimize API documentation to be a bit abstract and convoluted, frankly. I've read it over a number of times, and I keep saying "Now what does that do, again?"

For example, say I use objective = opt.MaximizeAlpha(pipeline_data.combined_alpha) followed by:

weights = opt.calculate_optimal_portfolio(  
        objective=objective,  
        constraints=constraints,  
        )  
objective = opt.TargetWeights(weights)  
order_optimal_portfolio(  
    objective=objective,  
    constraints=[],  
    )  

Is it equivalent to objective = opt.MaximizeAlpha(pipeline_data.combined_alpha) followed by:

order_optimal_portfolio(  
        objective=objective,  
        constraints=constraints,  
        )  

Here's the code I'm toying with, that includes a turnover constraint and the exponential portfolio vector smoothing:

    if context.init:  
        weights = opt.calculate_optimal_portfolio(  
        objective=objective,  
        constraints=constraints,  
        )  
        context.init = False  
        context.weights = weights  
        objective = opt.TargetWeights(weights)  
        order_optimal_portfolio(  
        objective=objective,  
        constraints=[],  
        )  
        return  
    constraints.append(opt.MaxTurnover(MAX_TURNOVER))  
    try:  
        weights = opt.calculate_optimal_portfolio(  
        objective=objective,  
        constraints=constraints,  
        )  
    except:  
        constraints = constraints[:-1]  
        print 'Turnover constraint dropped.'  
        weights = opt.calculate_optimal_portfolio(  
        objective=objective,  
        constraints=constraints,  
        )  
    context.weights = context.weights.loc[pipeline_data.combined_alpha.keys()]  
    context.weights = context.weights - context.weights.mean()  
    context.weights = context.weights/context.weights.abs().sum()  
    weights *= ALPHA  
    weights = weights.add((1-ALPHA)*context.weights,fill_value=0)  
    weights = weights - weights.mean()  
    weights = weights/weights.abs().sum()  
    context.weights = weights  
    objective = opt.TargetWeights(weights)  
    order_optimal_portfolio(  
        objective=objective,  
        constraints=[],  
        )  

Until the InfeasibleConstraints mystery above is resolved, to keep the backtest running, just wrap my first example you're utilizing (and added to) with try except here like the third example, that works for me. Decided to put that together below. By the way in fixing that error, would be good to remove the duplicate content and add more decimal points beyond 0.0. Maybe you can apply this to both of them for 5 decimal points: `%.5f' % the_float_variable

If data.can_trade(stock) is false, is that always a delist when operating on pipeline output when not using sid() etc? I think so.
Then the context.stocks check handles any stocks that have simply fallen out of QTU without being delisted I guess.

    # Close any positions no longer in QTU list.  
    to_close = []   # securities to be changed  
    for security in context.portfolio.positions:  
        if data.can_trade(security):   continue   # delists only?  
        if security in context.stocks: continue   # assuming all are QTU  
        to_close.append(security)  
    try:  
        ids = order_optimal_portfolio(  
            objective   = opt.TargetWeights(  
                pd.Series(0, index = to_close)  # the 0 means close them  
            ),  
            constraints = [  
                opt.Frozen(  
                    set(context.portfolio.positions.keys()) - set(to_close)  
                )  
            ]  
        )  
        for i in ids:  # log any ordered to close  
            o = get_order(i)    # order object  
            s = o.sid   # including filled as a head's up in case partial  
            log.info('{} {} {}'.format(s.symbol, o.filled, o.amount))  
    except Exception as e:  
        log.info(e)  

Hi @Blue Seahawk,

Yes, the inspiration for the code came from you, sorry if I forgot to mention to it in my above post. I will try your suggestions and see what happens. I also think Q team should come out with an official fix for TargetWeights construct. Thanks again, Gary!

I also think Q team should come out with an official fix for TargetWeights construct.

What is the problem, precisely? I'll see if I can reproduce it.

Grant,

Checking investment in tradable universe...
FAIL: Investment in QTradableStocksUS (2nd percentile) of 94.70% is < 95.0%.

If you useTargetWeights construct in Optimize API, sometimes it doesn't close out the open positions of newly delisted stocks off the QTU list which ultimately leads to failure in this test.

For testing scenarios I think we need some security examples that are fine with MaximizeAlpha and not TargetWeights. I'm editing, removing some that I thought were like that but may have been mere delists ("expired assets").

By the way thanks James for pointing this out.
And for your comment, I edited mine to make it more clear that my code was only part of what you were doing there.

Here's an example, with a switch to toggle between two different, but presumably equivalent ways of applying the Optimize API. At first-glance, I don't see a major difference:

ORDER_OPTIMAL = True # True for order_optimal_portfolio, False for calculate_optimal_portfolio  

2013-07-01 16:00 WARN Your order for 241 shares of FOXA failed to fill by the end of day and was canceled.  
2014-10-20 09:35 WARN Dropping expired assets from optimization universe:  
['Equity(36628 [GTAT])']
End of logs.  

Total Returns
21.39%
Benchmark Returns
74.36%
Alpha
0.06
Beta
-0.08
Sharpe
1.12
Sortino
1.69
Volatility
0.05
Max Drawdown
-5.5%

ORDER_OPTIMAL = False # True for order_optimal_portfolio, False for calculate_optimal_portfolio  

2013-07-01 16:00 WARN Your order for 242 shares of FOXA failed to fill by the end of day and was canceled.  
2014-10-20 09:35 WARN Dropping expired assets from optimization universe:  
['Equity(36628 [GTAT])']
2014-10-20 09:35 WARN Dropping expired assets from optimization universe:  
['Equity(36628 [GTAT])']
End of logs.  

Total Returns
21.38%
Benchmark Returns
74.36%
Alpha
0.06
Beta
-0.08
Sharpe
1.12
Sortino
1.69
Volatility
0.05
Max Drawdown
-5.5%

Hi Grant,

The issue is not with ORDER_OPTIMAL = True # True for order_optimal_portfolio, False for calculate_optimal_portfolio

but with objective = opt.MaximizeAlpha(pipeline_data.alpha) versus objective = opt.TargetWeights(pipeline_data.alpha)

With MaximizeAlpha, the closing of recently dropped stocks is handled properly by order_optimal_portfolio. However with TargetWeights, order_optimal_portfolio does not ensure closing therefore it ultimately leads to a failure in the QTU test.

@ James -

O.K. I've attached an algo in which I compute the inclusion in the QTU every day:

def recording_statements(context, data):  
    num_positions = len(context.portfolio.positions)

    record(num_positions=num_positions)  
    record(leverage=context.account.leverage)  
    n_QTU = 0  
    for stock in context.portfolio.positions.keys():  
        if stock in context.QTU['beta'].keys():  
            n_QTU += 1  
    if num_positions > 0:  
        pct_QTU = n_QTU/num_positions  
    else:  
        pct_QTU = 0  
    record(pct_QTU = pct_QTU)  

This algo is trading weekly, and shows that one can end up with no stocks in the QTU, in the days between portfolio updates. Next, I'll post the same algo, except trading daily, to show that the problem goes away, mostly (at least with the standard MaximizeAlpha constraint, and using order_optimal_portfolio directly). Note that there is still one drop-out...

What this says, I think, is:

1. The QTU is turning over a lot--whew!
   
2. Algos need a separate check on QTU inclusion, to drop stocks not in it, between portfolio rebalancing (which won't necessarily work anyway, as the algo below illustrates...)
   

Here's the version that re-balances daily.

Hi Grant,

I ran a tearsheet and evaluate algo notebook on the weekly rebalance and it passed the QTU test.
Checking investment in tradable universe...
PASS: Investment in QTradableStocksUS is >= 95.0%.

Your calculation of daily pct_QTU is probably out of sync with the weekly rebalancing. Try weekly pct_QTU instead.

@ James -

I thought the QTU check is to be performed daily, regardless of transaction frequency? Every day after the close, the composition of the portfolio will be checked, to see that >= 95% of the stocks are within the QTU, as defined prior to the open that same trading day (i.e. the current QTU). I could do the calculation differently in my algo, but it wouldn't be consistent with my understanding of the rule.

Yes, Grant, your understanding regarding daily calculation of QTU is consistent with the rules. Your formula to calculate daily pct_QTU seems correct, so I don't really know where the error in results of daily pct_QTU is emanating from.

Why don't you combine the weights with to_close on every iteration? So you call TargetWeights only once but set the weights of to_close to zero.

@Aqua Rooster,

Supposedly, order_optimal_portfolio should handle closing out newly dropped stocks from QTU in one call as it is updated daily. Put another way, any stocks not in the algo's new order list including those that were recently dropped by the updated QTU list will have an order 0 to close out its position. For MaximizeAlpha construct this seems to work fine but for TargetWeights construct it doesn't seem to work properly, i.e, sometimes it closes same day, sometimes it closes x days after, sometimes it doesn't close at all and just left hanging in the portfolio as an inactive position and sometimes it is ignored as a non QTU and acts as if is tradeable although it is classified as a non QTU. In other words, the behavior of TargetWeights as it handles the QTU updates is unpredictable and the source of this unpredictability is still unknown. I have supplied @Jamie the notebooks and code (minus secret sauce) to try and figure it out.

In the meantime, my workaround code inspired by @BlueSeahawk in modifying-individual-positions-using-order-optimal-portfolio-frozen above although crude seem to fix the issue for short runs but still falters in very long runs.

@Blue Seahawk, your suggested try and except code works great at keeping the backtest running and logging errors as it goes. Great work, thanks again. But like you said until these InfeasibleConstraints are resolved, it is still unstable.

So what is the requirement? It looks to me like the notebook may be calculating the investment, versus the count of stock names. I thought the requirement was to have >= 95% of the stocks in the portfolio, after trading closes for the day, in the QTU. The total investment in dollars doesn't matter. Have I misinterpreted?

# Tradable Universe  
    print ''  
    print 'Checking investment in tradable universe...'  
    positions_wo_cash = positions.drop('cash', axis=1)  
    positions_wo_cash = positions_wo_cash.abs()  
    total_investment = positions_wo_cash.fillna(0).sum(axis=1)  
    daily_qtu_investment = universe.multiply(positions_wo_cash).fillna(0).sum(axis=1)  
    percent_in_qtu = daily_qtu_investment / total_investment  
    percent_in_qtu = percent_in_qtu[5:].fillna(0)  
    percent_in_qtu_0 = percent_in_qtu.min()  
    percent_in_qtu_2 = percent_in_qtu.quantile(0.02)  
    if percent_in_qtu_0 < TRADABLE_UNIVERSE_0TH_MIN:  
        min_percent_in_qtu_date = percent_in_qtu.argmin()  
        print 'FAIL: Minimum investment in QTradableStocksUS of %.2f%% on %s is < %.1f%%.' % (  
            percent_in_qtu_0*100,  
            min_percent_in_qtu_date.date(),  
            TRADABLE_UNIVERSE_0TH_MIN*100  
        )  
    elif percent_in_qtu_2 < TRADABLE_UNIVERSE_2ND_MIN:  
        print 'FAIL: Investment in QTradableStocksUS (2nd percentile) of %.2f%% is < %.1f%%.' % (  
            percent_in_qtu_2*100,  
            TRADABLE_UNIVERSE_2ND_MIN*100  
        )  
    else:  
        print 'PASS: Investment in QTradableStocksUS is >= %.1f%%.' % (  
            TRADABLE_UNIVERSE_2ND_MIN*100  
        )  
        constraints_met += 1  

Hi Grant,

Excellent point! My understanding of the requirement is, as you stated, by the count as opposed to what Q code is calculating:

    daily_qtu_investment = universe.multiply(positions_wo_cash).fillna(0).sum(axis=1)  
    percent_in_qtu = daily_qtu_investment / total_investment  

This may well be the source of where the instability of the TargetWeights construct is emanating from.
The requirement was to have >= 95% of the stocks in the portfolio after each daily update of the QTU list.
I agree with you, Grant, it should be measured more accurately with the count vis a vis the requirement.
As to why, Q chose to measure it in dollar terms, they have to explain because I suspect that the instability is
coming from this inconsistency.

I have seen this operational flaw before in the beta neutral vs. dollar neutral redundancy argument as articulated here beta-constraint-in-risk-model-totally-unnecessary. Two different constraints with similar purpose trying to achieve the same objective gives rise to the question of which has precedence and/or dominance? Occam's razor solution will shave off, the one that is unnecessary.

In the above case, you have two different calculations of measurement of the requirement, >= 95%. There are both correct depending on what perspective you're looking at. What would Occam do? Occam will choose the by the count measure because it is the simplest calculation to make!

@Grant, @James: The QTU% is measured in terms of portfolio value. As stated in the official contest rules, an algorithm must have >= 95% of its capital invested in stocks in the QTU. Portfolio value is used because the rule is meant to measure how much of an algorithm's capital is invested in the QTU.

Regarding the TargetWeights objective, as James mentioned, we discovered that he needed to apply Blue's solution to close out of positions that drop out of the QTU. This is unrelated to how the QTU% is measured - non-QTU names were not being dropped no matter which way you measure it. I don't yet have a better answer as to why this is happening, but I will try to get one. I don't expect to have an answer this week, but I'll see if I can get some answers soon.

@Jamie,

My interpretation of the QTU universe rule was based on here a-new-contest-is-coming-more-winners-and-a-new-scoring-system

Constraint Threshold Computation Window Tool
Use order_optimal_portfolio to Order Must use. Every day. Optimize API
Trade Within QTradableStocksUS >= 95% Every day. QTradableStocksUS

Apparently, this threshold was not explicitly stated above as opposed to the that of official contest rules. So Occam apologizes for misunderstanding and not reading the official contest rules:) Is there any compelling reason why Q chose dollar value vs. number of count ?

The wording in the post was vague. Occam is most certainly forgiven.

The QTU% rule is meant to encourage algorithms to invest the vast majority of their capital in QTU stocks. If the QTU% were measured as number of stocks held that are in the QTU instead of as % of portfolio value, it would be possible for a contest algorithm to invest the majority of its portfolio value in non-QTU names. For example, I could hold 2000 single-share positions in the QTU, and then 100 names outside the QTU to take up my remaining portfolio value. This would result in my algorithm being 95% invested in the QTU (by name count), when in reality, the majority of my investment is outside of the QTU.

The portfolio-value based measurement can also be thought of as a position-weighted value (position_in_QTU_boolean * weight_in_portfolio), which is consistent with the other criteria in the contest.

@Jamie,

Thanks for the clarification. Will wait for further updates on the TargetWeights issue. I am thinking of switching to MaximizeAlpha construct for upcoming contest as even my supposed workaround fix still seem unstable and might crash in the middle of a good run.

Used @Grant's algo above. # Backtest ID: 5a802f48f20a9e4118d01037

The following chart shows a relatively subdued equity line. Nonetheless, the outcome is more than double @Grant's results. Only changed numbers seen as pressure points. No alteration to the program logic was made. Switched to daily rebalancing instead of weekly.

This strategy trades too much (43,680 trades), even if the turnover is recorded as less than 5%. All I see is account churning. Everyday, it rebalances almost all of its inventory on what should be considered mostly market noise.

It does shorts, but is very poor at it: 42% success rate. Whatever was thought to be predictive for shorts might definitely need some re-engineering.

Evidently, and the word is not used haphazardly, it will break down going forward. This strategy is not designed to have a sustainable CAGR.

If the test was done weekly as in @Grant's version, what would be the outcome? Easy to answer:

This should raise more questions. One of which is: is it really worth it to do it that way?

Is there a simple algo that illustrates the potential problem with the TargetWeights objective and maintaining the a portfolio that is >= 95% in the QTU (by dollar value...thanks for the clarification)? For example, say one just held all stocks in the QTU equal weight, and updated the portfolio daily. Would the problem manifest itself?

Hi Grant,

Here's an algo that illustrate the problem in TargetWeights objective and maintaining the a portfolio that is >= 95% in the QTU.
The notebook that follows gives you the details. Hope this helps.

And here's the notebook on above algo...

Thanks James -

Well, I guess the question is why would one expect to stay within the QTU, point-in-time? I can see several potential impediments:

1. Turnover constraint. It depends on how it is implemented, but if the TargetWeights objective doesn't ignore the turnover constraint when exiting stocks that are no longer in the QTU, then it will be a problem. In the limit that the turnover constraint dictates a minuscule change in the portfolio, stocks will linger. Given that the turnover in the QTU on a daily basis is significant, one could end up holding stocks that would need to be dropped to maintain the QTU inclusion level for the portfolio.
   
2. The backtester doesn't realistically handle closing positions, as I understand. Even when trading within the QTU, and within the 5% position constraint, I still see orders being cancelled at the end of the day, not being filled.
   
3. There is no explicit prioritization in exiting positions for stocks no longer in the QTU--it is all mixed up with the optimize API. To make it a priority, for example, at the open, one could submit orders solely to clear the portfolio of the non-QTU stocks. A strict priority would be for all of those orders to close, prior to running the optimize API, such that it is only dealing with a current portfolio of QTU stocks. One could also set a more forgiving threshold, cancelling any unfilled orders, and then running the optimize API (which I guess would again attempt to close out the non-QTU positions).
   
4. Conceivably, there could be an optimize API QTU participation constraint, which would result in a failed optimize result, if it could not be met. Not sure if this is doable within the mechanics of the optimize API, but it would give explicit control over this algo constraint.

Of course, if Q continues on their trend of releasing closed-source, black-box key API modules (e.g. QTU, optimize API, risk model), it'll become increasingly difficult for users to aid in understanding and troubleshooting such problems. I guess open-sourcing code hasn't been all that helpful to them over the years...

Hi Grant,

Regarding your points (2) & (3), code below is the temporary workaround fix. It is placed before the objective function TargetWeights to give precedence in closing newly dropped stocks from QTU list before being passed to Optimize API:

    # Sell any positions in assets that are no longer in our target portfolio.  
    for security in context.portfolio.positions:  
        if data.can_trade(security):  # Work around inability to sell de-listed stocks.  
            if security not in context.stocks: #QTradableStocksUS():  
                to_close = [security]   # securities to be changed  
                try:  
                    ids = order_optimal_portfolio(  
                        objective   = opt.TargetWeights(  
                        pd.Series(0, index = to_close)  # the 0 means close them  
                    ),  
                    constraints = [  
                        opt.Frozen(  
                            set(context.portfolio.positions.keys()) - set(to_close)  
                        )  
                    ]  
                )  
                #for i in ids:  # log any ordered to close  
                    #o = get_order(i)    # order object  
                    #s = o.sid   # including filled as a head's up in case partial  
                    #log.info('{} {} {}'.format(s.symbol, o.filled, o.amount))  
                except Exception as e:  
                    log.info(e)  

I said temporary workaround because in a long backtest run I encountered point (4) and received this error:

InfeasibleConstraints: The attempted optimization failed because no portfolio could be found that
satisfied all required constraints.

The following special portfolios were spot checked and found to be in violation
of at least one constraint:

Target Portfolio (as provided to TargetWeights):

Would violate Frozen([Equity(32770 [DEI]), Equity(16389 [NCR]), ...]) because:
New weight for Equity(32770 [DEI]) (0.0) would not equal old weight (-0.00124713995777).
New weight for Equity(16389 [NCR]) (0.0) would not equal old weight (0.00127471453249).
New weight for Equity(38921 [LEA]) (0.0) would not equal old weight (-0.0012673465914).
New weight for Equity(2 [ARNC]) (0.0) would not equal old weight (0.00124764231739).
New weight for Equity(6161 [PRGO]) (0.0) would not equal old weight (0.0012355562442).
New weight for Equity(4117 [JCI]) (0.0) would not equal old weight (0.00114328492707).
New weight for Equity(4118 [JCP]) (0.0) would not equal old weight (0.00120445963172).
New weight for Equity(2071 [D]) (0.0) would not equal old weight (0.00110161134441).
New weight for Equity(4120 [JEC]) (0.0) would not equal old weight (0.0011990730669).
New weight for Equity(14372 [EIX]) (0.0) would not equal old weight (-0.00124497410696).
... (841 more)

To make the backtest continue to the end, I added " except Exception as e: log.info(e) " , to ignore error and just log it.

I am with you on your frustrations with closed-source modules. The above case clearly illustrates that there are still "holes" in their framework implementation. If this were truly a "crowd-source" effort, it would help if everything is open source as the "crowd" themselves would help sort out and resolve issues like these.

I am particularly not sold on the idea of "style risks" as they are unnecessary risks that neither needs to be mitigated nor diversified because the market as a whole is non-stationary and non-linear, exposed to all kinds of imagineable risks. There is also the matter of wrong measurement of these conceived risks specially if they are static. Take the case of short term reversal risk which is measured by a 14 day RSI, a static measurement in a non stationary series of returns. Clearly, it is flawed because in high volatility periods, the phase of returns are fast while in low volatility periods which implies an ensuing trend, the phase of returns are slower. If the measurement were adaptive and in tuned to the phase of returns (period of RSI is adaptive in reponse to current phase), then there might be some validility to the measurement of short term reversal. Again, on the bigger picture, are all these risks factors actually achieving their desired and designed objective or are they, in fact, churning out sub optimal results because it is overly choking potentially higher returns in exchange for a very rigid and stringent desire to contain risks?

There are many "Mysteries of the Q" (to be elucidated in an upcoming YouTube series).

Regarding the risk model and style risks, since it's introduction, there has been almost complete radio silence on the topic of how it works in detail, why it is a net good thing, and if this particular formulation is effective. I'm guessing this may be what is used under the hood:

https://www.quantopian.com/lectures/factor-risk-exposure

Part of the motivation, as I've heard it, is that Q wants to filter out less-profitable "common" factors (i.e. not something sophisticated customers would pay a premium for). It was noted by the author on https://www.quantopian.com/posts/enhancing-short-term-mean-reversion-strategies-1 that short-term reversal is very common, and so I guess it makes sense that it is not the sort of strategy that one would want if the goal is to show that the crowd is finding alpha gems, over what a traditional brick-and-mortar hedge fund would be finding. In the long run, if the 1337 Street Fund is correlated with every other hedge fund out there, then it will not have differentiated itself.

So I gather that Q wants to offer sophisticated investors a unique product in the equity market neutral strategy space. This "product" differentiates itself from others by design through a strategy framework with a desired objective that has been subjected to different kinds of returns / risks thresholds with the intent of extracting uncorrelated returns while at the same time containing different types of identifiable risks through risk mitigation and diversification techniques. Q designed the "product" but the crowd sourced authors "produced" it through Q's allocation scheme extracted primarily through its contests winners.

Bottom line, we are but mere "worker bees" and must bow to the Queen Bee. We can drop something in their suggestion box or give out constructive criticisms with the intent of improving the product and just see how it goes. Is the lure of getting an allocation worth the effort given this kind of relationship?

I would like to see Q post a notebook of their model algo, preferably one that had received an allocation with a long backtest such as the one above that has passed all their contest requirements. At least this would give us some guidance as to what kind of long term returns and risk levels are to be expected with this unique "product".

Well, somebody must know something we don't, since Q has gotten a boatload of money from VCs over the years--at least $50 M, if not more, from its inception in 2011, has been invested (e.g. see https://angel.co/quantopian). Not sure how these VCs think, but I have to wonder if they look for some sort of decent pay-back in a finite amount of time (e.g. X% in 5 years), but maybe sometimes it is more of a long-term strategic investment.

As far as I can tell, Q has completely shut the door on any relationship with users other than for them to be worker bees. The fundamental problem, of course, is that the business would need to be completely restructured from the ground up, for the crowd to have any role in running the show. This was never in the cards from the get-go, despite all of the initial "let's take on Wall Street together" hoopla.

Q has never posted anything remotely feasible as a viable algo--I wouldn't hold your breath for an example. My guess is that anyone in the industry familiar with the sort of long-short equity algos Q is cultivating knows the answer.

VC's make a lot of bets and it's bet on Q is just a drop in the bucket in the overall scheme of VC's business model. There are normally contract covenants to comply with on a time bound basis and also exit clauses normally attached to these investments that is shrouded with confidentiality agreements which includes among other things, non disclosure to the public or media. Having said that, we may never know if a VC pulled out their investments or not.

I don't want to pre empt judgement, that would be speculative on my part but reading between the lines, I see signs that not everything is hanky dory.

] feasible as a viable algo

Quantopian Risk Model In Algorithms is the best that I know of right now but if started six months ago is -2.5%.

I wonder if it might be possible to build stops limits into optimize, I was able to greatly improve the algos in the lecture series in part using those. Would my improvements have held up with different start dates? That's the thing facing all of us.

One of the challenges is the $10M requirement so we often gravitate into large dollar volume stocks and many of us are focusing on the same stocks to some degree. Then we also necessarily extend our trading into stocks with far weaker signals for less investment in each, more neutral. I could be wrong but my impression is that we could find a lot of alpha in factors at a threshold that screens down to 50 stocks and then need to change that threshold bringing in an additional 100 stocks with weaker signals to pass constraints.

Fairly confident we'll eventually get there somehow, with adaptability, I admire the hero investors backing Quantopian.

Lots going on in this thread, but as I recall, there was a potential bug found in the TargetWeights objective, correct? If so, was there ever a help ticket/bug report logged? If so, what is the status?

Hi Grant,

Yes, I filed a help ticket/bug report about two weeks ago. Worked with Jamie on this issue via pm and he confirmed the existence of the problem. I submitted a temporary fix but not stable in long runs. Jamie is still working on the possible source of the problem. Here's the latest feedback from Jamie about a week ago:

Regarding the TargetWeights objective, as James mentioned, we discovered that he needed to apply Blue's solution to close out of positions that drop out of the QTU. This is unrelated to how the QTU% is measured - non-QTU names were not being dropped no matter which way you measure it. I don't yet have a better answer as to why this is happening, but I will try to get one. I don't expect to have an answer this week, but I'll see if I can get some answers soon.

@ Jamie/Q support -

Obvious, but if you'd open-sourced the Optimize API from the get-go, users could simply have a look for themselves and help identify the source of bugs (as they have over the years). Is there a reason it is not open source? There would seem to be only upside.

Checking back on the status of a potential bug in TargetWeights. Is there, in fact, a verified bug?

Following up yet again...is there a bug? If so, is it being worked?

Sorry for being obnoxious, but I do have a potential use case...

Anyone?

I got feedback from Q that there is no "bug" in the implementation of TargetWeights. My interpretation is that there won't be any changes on their end, so I'll dig into the work-arounds described above.

@Grant,

That's quite an unsettling feedback from Q. I filed the help ticket/"bug" report and last response I got was they are still trying to figure out the source of the problem. I guess if their engineers can't find the source and/or solution to the problem, they mark it as a "no bugger"! It's probably in the low end of their priority list considering their new challenge with regards to Morningstar's major revamp of fundamental data. I think they should do more in the area of checking for data integrity / consistency of feed from their data vendors particularly in composing and filtering QTradableStocksUS. I noticed a few threads popping up questioning the integrity / consistency of some fundamental data coming from Morningstar. While not directly Q's fault, it is incumbent upon them to do due diligence of data received from data vendors before they release to us "worker bees". As the old computer addage says, GIGO - garbage in, garbage out.

@ James -

Yeah, I'm not sure what happened here. Looking above, it sounded like Jamie would look into it, and report back. Maybe I'll write a dirt-simple algo (e.g. equal weight QTU, updated daily) to see to TargetWeights misbehaves.

@Grant, @James: I think we had a bit of a miscommunication internally. Technically, the TargetWeights objective is behaving as originally intended. However, there is a particular case that James ran into where TargetWeights should probably behave differently. There are some tweaks to the objective that we've identified could be made to improve TargetWeights and fix the unexpected behavior that James encountered. Unfortunately, this is not something that we are working on right now. It's currently on our to-do list, but it is not at the top of that list. In the meantime, I recommend using the MaximizeAlpha objective, or the FrozenConstraint workaround.

@ Jamie -

Thanks for the update. I had been tinkering around with TargetWeights and then came across this thread, and started to wonder if some wonky behavior I was seeing was due to my not implementing it correctly, or a "bug" (call it what you will).

Do you have a concise description of the issue, and why the workaround is required?

By the way, if you were to just open-source the optimize API (not necessarily making it compatible with zipline, at this point), you could refer me to the associated issue--easy-peasy. I was told that you would eventually open-source the optimize API. Why not just put it out now? Maybe it has to be scrubbed for proprietary stuff? Or it becomes another support issue, with folks asking questions? I guess there must be a logical reason it has not been shared.

@ James -

Can you articulate the "bug" in the implementation of TargetWeights? I've gone several rounds now with Q support both here and offline, and have yet to get a description of the problem, and the recommended work-around. Maybe I'm being impatient, but I'd think if there is an internal report, the findings could simply be copied and pasted here. Not sure why this is so hard...guess they are busy with other things.

@Grant,

The issue with the implementation of TargetWeights is, in some cases, it does not close positions of recently dropped stocks in the QTU universe as it is suppose to under the order_optimal_portfolio routine. Since these positions remain open and are now considered non QTU, it causes the algo to fail the >= 95% QTU test.

In my post to you dated Feb 14, 2018 above, I attached both the backtest and notebook to illustrate the problem. At the bottom of the attached notebook is a graph that tracks daily %QTU composition and it shows that it goes below the >=95% threshold as a result of newly dropped stocks in QTU not being closed under the TargetWeights construct.

I did come up with a workaround inspired by @BlueSeahawk in modifying-individual-positions-using-order-optimal-portfolio-frozen and posted it above. It forces the algo to close the open positions of these newly dropped stocks out of the QTU before calling TargetWeights and order-optimal-portfolio routines.
Hope this helps.

@ James -

Thanks. I'll take a look. Just curious, if TargetWeights is used without constraints, is there still a problem? In theory, if I understand correctly, it should behave effectively like order_target_percent and attempt to close all positions with 0% weights, but maybe not?

Here's a first cut at troubleshooting. Probably unrelated, but I got an odd nan error, too.