Hi Rob,

Regarding your comments on data mining, in my opinion, Quantopian makes it very challenging to do the kind of response surface and sensitivity analysis you presumably did manually by adjusting parameters and re-running backtests. The backtesting engine supports running a large number of backtests in parallel, but the support for varying parameters programmatically and then post-processing the results needs to be improved. For example, say I want to carry out a study of the effect of a single parameter. I would run N backtests, noting the backtest_id for each, and the corresponding parameter value, which I would have set manually. I would then enter these data into the research platform to retrieve the backtest results for analysis. I would then need to sort out how to analyze the N backtests (presumably I could leverage the existing pyfolio tool, applying it iteratively, and then collating the results).

One solution would be to provide support from within the research platform to be able to run backtests in parallel, with programmatic control over backtest parameters. I think making it a lot easier to "slice and dice" backtest results systematically over a space of M parameters would go a long way toward addressing the data mining issue you raise. If the platform readily supported a workflow that included running N backtests over the M parameter space and some plug-and-play analysis tools, then the "point solution" approach could be avoided. There's an operational cost in terms of supporting the running the N backtests in parallel (users might tend to run more backtests), but I figure it must be close to the cost of electricity to power the servers, at scale (i.e. we are using commodity cloud computing).

Regarding your comments about the hedge fund industry capital limitations, from http://www.hedgeweek.com/2016/10/20/244960/hedge-fund-industry-capital-rose-third-quarter , we have:

Hedge fund assets rose to USD2.972 trillion, an increase of USD73.5 billion from the prior quarter. The 3Q16 hedge fund asset level eclipses the previous record of USD2.969 trillion set in 2Q15.

It would seem that if a true zero-risk arbitrage opportunity is out there, they will come up with the capital to profit from it, one way or the other (I'd read that prior to the mortgage crisis debacle, home mortgages were effectively leveraged up something like 50X, so Wall Street knows how to print money when they need it). It is kinda hard to believe that they wouldn't come up with the money, in the limit of zero risk.

However, for large index rebalances like the annual Russell rebalance, the anomaly, although much smaller than it used to be, still seems to exist.

Perhaps the reward is not high enough to justify the risk. In other words, the cost of coming up with the capital doesn't justify the expected reward, with risk taken into account.

Grant, instead of trying to optimize parameters for a backtest, you should be evaluating your strategy in pieces. For example, you could easily vary parameters of a pipeline programmatically in research, and then run the results through alphalens to evaluate the predictive ability of your alpha signal.

You can't run a parallel search, but it's at least automated, and you can simply copy and paste your pipeline from research to an algorithm. I've attached a simple example notebook of what that might look like.

The notebook performs a parameter search on an alpha factor in the last cell, which should solve the problem that you're running into. I didn't do the optimization step, I'll leave that up to you.

P.S. I used the alphalens tearsheet definition to figure out which functions to call.

Here is a short notebook that accompanies the "Driven to Distraction" backtesting post.

Bit of a rudimentary question if you don't mind - Can we think of long PEAD as a momentum bet and shorting PEAD as a mean reversion bet? Then your algo is choosing momentum on crowded earnings dates and mean reversion on sparse earnings dates.

Thanks for sharing the strategy Rob. Here is a quick effort to enhance:

@ Jamie -

Yes, the fundamental way to avoid data mining is to know that if I do X I'll get Y (at least on a statistical basis). So, I'd throw the question back to you. It sounds like Rob used backtesting to explore the parameter space. You seem to be suggesting that he could have used alphalens. Is this the case? Do you have an example for his particular strategy? Or does one need to get all the way to the backtest stage?

My point was that if it is difficult/impossible to map out the complete response surface of a strategy, then there will be a tendency to overfit/mine data. One solution is to make it really easy to get the big picture. Define M parameters, run N backtests, and presto, one can get a sense if an overfit "point solution" has been selected, or if there is actually no dope-smoking going on.

Anyway, no need to hijack the discussion here, to point out the obvious. There's a cost-benefit calculus on your end, but unless those data are brought into the discussion, it is a one-sided dialog.

Response surface and backtest 1001 different parameters. Yes, I worked with a proprietary backtesting engine for years which made this very easy to do and graphically very easy to see.

Even if you avoid an obvious peak of profitability, MAR or whatever your chosen risk metric I fear it made little difference to forecasting the future profitability if the strategy. You whole "response surface" can change unrecognizably after a few years.

No solution there I fear although it is certainly a helpful process in forcing you to realise what absurd difference can result from minute parameter change.

Hi Rob,

A couple more quick comments:

• I suggest as a baseline, normalizing to a leverage of 1.0. My understanding is that when evaluating algos for potential investment, Quantopian does this. Having a leverage of 2 (or whatever) is arbitary (Quantopian is talking about using 6X leverage for the hedge fund anyway).
  
• I just noticed your use of a "hedging instrument" (SPY). I've gathered that this is discouraged for algos that would be candidates for the hedge fund. What happens if you don't hedge? Or hedge in some other fashion, without using a hedging instrument?
  
• What if you have a mix of longs and shorts? Your code doesn't appear to handle this case:

If only long or short postions, hedge other side with SPY  
    if(len(shorts)==0 and len(longs)>0):  
        order_target_percent(context.SPY,-1)  
    elif(len(longs)==0 and len(shorts)>0):  
        order_target_percent(context.SPY,1)  
    else:  
        order_target_percent(context.SPY,0)  

• It would be interesting to put this strategy into the context of the type of algos Quantopian might be wanting to add to their portfolio. There must be something wrong with it, or it wouldn't have been posted (in a private communication, Dan Dunn made it clear to me that Quantopian would never post anything that would work). What's wrong with the strategy, vis-a-vis the Q hedge fund?
  

Grant,

Thanks for your thoughtful comments.

• The point you make (in your first reply) is valid that when hedge funds see opportunities like the annual Russell reconstitution, they do have the ability to move capital. One possible counterargument, though, is that the index rebalancing business requires some specialized knowledge (for example, merely predicting the float adjusted shares in the index is quite complex) so a fund that specializes in stat arb may be reluctant to redeploy large amounts of capital in the annual Russell rebalance without having the expertise. And because the amount of capital required for these types of trades shoots up at discrete periods of time, the funds may run into leverage constraints. Also, I didn't intend to imply that this trade is risk free.

• I should probably have used a target leverage of 1.0 instead of 2.0 so it's consistent with Quantopian's preferred leverage ratio. It's a very simple change to make in this algo, which equally weights all longs and all shorts. For example, change order_target_percent(security,1/len(longs)) to order_target_percent(security,0.5/len(longs)) and do the same for shorts and the possible SPY hedge.

• I should explain the section of the code that you refer to that checks whether the algorithm holds only longs or only shorts, and if so, hedges with SPY. It is theoretically possible that the algorithm finds PEAD stocks that cross the buy threshold but finds no stocks that cross the sell threshold (or the other way around). In this case, to be truly dollar neutral, I would sell an equal dollar amount of SPY against the long-only positions or buy SPY against short-only positions. With the parameters I use, there are only a handful of days where this actually happens. You can see those days by inserting a log.info() message inside that portion of the code. If I had left this section of the code out, it would have had very little effect on the performance of the algorithm. However, if you run the algo for other sets of parameters, like more extreme quantile cutoffs or shorter PEAD holding periods, there will be more days where the algorithm wants to hold only long positions or only short positions. So I wanted to generalize the code to handle this possibility.

Fawce,

That is correct that the long PEAD trades (on crowded earnings dates) are momentum trades and the short PEAD trades (on sparse earnings dates) are mean reversion trades. The one caveat I would make is that there are cases where a stock declines despite beating earnings expectations (because of weak forward guidance, a revenue miss, analyst expectations not representing the market consensus, and a variety of other possible reasons) as well as cases where a stock advances despite missing earnings expectations. An interesting variation on this algorithm would be to sort stocks into quantiles based on the stock price reaction to earnings rather than on the degree to which it beats Zacks's consensus estimates. Another variation is to use Estimize's consensus estimates rather than Zacks's.

I just want to point out (as I suggested should be pointed out on every backtest that does this), the results are ZERO commisson and slippage.

https://www.quantopian.com/posts/request-add-disclaimer-to-backtest-graphs-without-commissions

    #: Set commissions and slippage to 0 to determine pure alpha  
    set_commission(commission.PerShare(cost=0, min_trade_cost=0))  
    set_slippage(slippage.FixedSlippage(spread=0))  

Here is the original algo with slippage/commission on.

Hi Rob:

Thanks for sharing insight on incorporating price action rather than earning beat/miss alone. As you said, there might be other nuances on the earning release.

One question, you argument about limited capital. Does that apply to both long and short PEAD? If you run a long/short hedge fund, wouldn't you be able to apply a higher leverage than a long only fund, i.e. more access to capital?

A side note regarding achieving dollar/beta neutral by using SPY. An alternative is using sector ETF which can hedge out sector performance issues.

Why is there that when I clone the Algo and run it, I get vastly different results then what is shown above? Anybody else has the same problem?

/Luc

Any one care to answer or try to clone the algo and see if they get the posted performance?

Thanks,

@luc,

New backtest is defaulted to new slippage model, 5 basis points fixed, 10% volume limit. Believe above algos does not account for slippage nor commissions.

James,

Thanks for answering. I still don't get it. The algo (the first posted one in this thread) has already the slip and commission set to zero:

set_commission(commission.PerShare(cost=0, min_trade_cost=0))  
set_slippage(slippage.FixedSlippage(spread=0))

And I still don't get anywhere near the same results. There are no warnings that this algo is using deprecated api (althought it does not use the optimize api).

Any thoughts?

@luc,

I may be wrong but I believe the New Backtest automatically defaults to the new slippage model and overrides other/previous settings to ensure it complies with contest rules. You can try it under the old backtest by adding /old after the backtest id number.

James,

Thanks for the suggestion. I added /old to the backtest and it just shows the same results but with the old format. The performance is the same. I could not find anywhere in the documentation anything related to the custom commission and slippage being overridden.

This seemed like a well performing algo. Too bad.

@luc, @James

If you set custom slippage and commission, they should take precedence over the default, regardless of the backtest screen.

~It's possible that the different result is coming from the fundamental data overhaul that we did in Oct. 2017. The changes included many data corrections which can have a compounding effect on a long backtest like the ones on this thread. Can you attach a copy of your clone so we can see if anything else is different?~

EDIT: It's unlikely that the fundamental data changes actually had an impact here, since they only affect the Q500US constituency, not the actual signal.

@jamie,

Here is my back test. Same exact code as the original post.

/Luc

/old is just a different (prior) view of underlying backtest returns.

For old results you will have to force slippage to the Volumeshareslippage something like here
https://www.quantopian.com/posts/documentation-question-about-slippage