SIDE NOTES:

1. The concept behind this experiment is to test the hypothesis that given perfect knowledge of the market's next day returns as represented by SPY, wouldn't one be making a killing under the Quantopian long/short algo model by simply extracting the stock universe's correlation with SPY and trading with it.

2. The fetcher file 'Perfect' contains the sign SPY's next day returns, 1 for upday and -1 for downday.

3. We Optimize each individual's stock correlation with SPY times the direction of tomorrow's SPY (perfect prediction). For example, if tomorrow's SPY returns is positive, a buy, highly correlated stocks will be longs and lowly correlated stocks will be shorts under the constraint of Dollar Neutral and vice versa.

4. I've tried different correlation lengths(5,21,126,252) but all results are still negative. This backtest is set at 252.

Below is results of the Quantopian framework:

Folks, let's analyze what happened here. I really want to impress my neighbor!

Why is it that with perfect knowledge of what is going to happen to the market (SPY) the next day, Quantopian's long/short algo model with all its constraints not churning out positive returns?

Is my correlation hypothesis wrong? Or is that the stock universe price movements is so highly correlated to SPY, that it renders the Dollar Neutral leveraging useless?
Or am I just nuts!?

@Karl, this experiment is actually related to our discussion in the DNN and Beyond thread. Made your proposed changes above. Slight improvement but still negative returns.

@Karl, alpha -0.04 beta =-0.04...Here's the notebook of tearsheet....

@Karl, here's new NB with round_trips=True.

@Karl, thanks for your feedback. Yes, my first suspect is that correlations when optimized into the constraints results in picking the same 50 longs / 50 shorts. Still puzzled though. I'll try your suggestion regarding alphalens,

Hi James,

As Karl suggested, a good exercise would be to analyze your algorithm's signal using Alphalens. The attached notebook will help you get started, and you can learn more in our Factor Analysis lecture. To re-run the notebook you will need to upload your Perfect SPY data file to the /data folder in Research, and change the file name passed to local_csv as required.

Ideally, a ranking model will assign alpha values such that assets in higher quantiles exhibit high cumulative returns, while returns for assets in lower quantiles remain low. Based on the quick analysis you will find in the attached notebook, we can see this is not the case for your algorithm's signal. This is even before applying any slippage or commissions to the model, or applying portfolio and risk constraints. At this point you would want to re-check your investment hypothesis as Karl also mentioned.

I hope this helps.

@Ernesto, thanks a lot, this alphalens analysis helped a lot. It seems to me that based solely on this alphalens analysis, that the correlation hyphothesis with regards to having perfect knowledge of SPY next day returns does not hold.

I ran another test, this time just taking the top 5% percentile and bottom 5% percentile of correlations with SPY and trading them with equal weights without running them through Optimize subject to constraints. While it turned profitable still a big disappointment. Here are the results:

@Karl, thanks for your suggestions. I gave up on the correlation hypothesis, however, a 21 day stock beta seem to give some ray of hope tested with the new risk model, no slippage, no commissions. Here are the results:.

However, same as above but with the new slippage model, it falls apart.

@Karl, I have found the correct way to trade the SPY DNN. It does not work with the highly hedged long/short model, as tested using 252 day correlation optimize or fixed percentile. Reasons it doesn't work are (1) the gains on one side is offset by the losses of the other side (Dollar Neutral effect) and (2) correlation is a linear measurement that doesn't accurately measure the 'real world' nonlinear relationships. So by removing Dollar Neutral, Sector and Market Beta constraints in Optimize while holding Maximum Exposure to 1, Position Exposure to 1% and accounting for commissions and new Slippage Model, we now have a boilerplate strategy. It is not within the Quantopian model framework but that is neither here or there. Here are the results:

@James: Thanks for this thread...always good doing Gedenken-Experiments on large systems!
I ran the tear sheet, and looked at the daily holdings....funny thing...
- Each day, the algo holds either all longs, to the tune of all the cash, or all shorts, to the tune of 2x the cash!
Turnover is huge...
I can only surmise that this is due to the [email protected] telling you which way SPY is moving tomorrow, and OptimizeApi acting on that in an "I'm All In" way!
-This points up another issue: Quantopian's OptimizeAPI is a black box.

@Ernesto: Thanks for the nice notebook! I ran it at higher frequencies:1,5,10,21,44,64, and see great seperation of the top&bottom quintiles from the 2-4'th quintiles, which I can't figure out why that happens.

@Alan, I'm glad you enjoyed the experiment as much as I did. As @Karl pointed out, I disabled the Dollar Neutral constraint because the experiment shows correlation or stock beta to market as proxy for alpha does not work and counter intuitive. The takeaway here is that the stock universe generally moves together with the market, so with having perfect knowledge or a high accuracy prediction of SPY, one just needs to trade with the direction of the prediction and not against.

The thing that stands out with me is volatility which is a little on the high side at 17%. Volatility is both a blessing and a curse. The highest returns can be achieved during high volatility periods, if you trade the on the right side but risky. This confirms the high risks/ high returns premise. Low volatility generally implies an ensuing trend, catching the right side is very profitable and less risky. Volatility at 17% in this experiment represents the market risk. Beating the market risk is in itself an alpha but this may be a tall order. In the context of Q fund framework, low volatility, low drawdown and high stability, this is more profound. The relationship between returns and volatility is such that to get a higher return one has to give in to a little increased volatility. My take on Q's focused on minimizing risk isvthat it might be an overkill that constricts the potential for higher returns. It seems to me that their focus in risk management outweighs that of maximizing returns. This maybe is due to their ultimate strategy of trading a number of uncorrelated low return /low risk algos and leveraging it. This may well work in a low interest environment but I question whether it will hold in a high interest scenario.

@Karl - Yes, I don't understand the high turnover, yet may simply be the rapid movement of @James DNN SPY signal is causing an asset to be long one day and short the next...perhaps a short ExpWgtedMovAvg applied to the DNN would help...or intraday as you mention?

@James - Trying to understand/paraphrase your takeaway...which is always the most important part of Gedenken-Experiments...
If I have a perfect SPY prediction, and my trading universe approximates the SPY-universe, then by weighting my asset purchases by the sign of the SPY movement times the size of beta correlation, I'll get a return stream that approximates trading SPY by it's returns-sign prediction!

I modified you code to use your perfect SPY prediction stream, and cut down the trading universe to only SPY...to see what the DNN perfect prediction does...for some reason, over the last two years, you "only" get a 5x return over the benchmark SPY, whereas your one above gives much higher returns...any ideas why?

@Alan and @Karl, the high turnover is just the direct result of perfect one day head SPY prediction and mirrors SPY movements on a daily frequency. Moving Average will introduce some lag and will degrade performance but smoothing it with a Kalman or Wavelet filter might work with minimal degradation.

@Alan, your modification to reduce the trading universe to approximate the 500 constitiuents of SPY will not approximate its return perfectly because (1) the 500 components of SPY is not weighted equally and (2) the weighting is now governed by Optimized subject to constraints given. The last two years has lesser returns due to lesser volatilty which reflect a trending, in this case, upward market. With perfect 20/20 hindsight, you'll make more returns in high volatilitiy periods than low volatilitiy periods.

@Alan, James and Karl: I also enjoy Gedanken-Experimente a lot :). What about this one: we have the perfect SPY-file as the result of the red pill (see above). I assume, that at the end of James' cure, there was a degrading effect and predictions became worse up to randomness?

What if we would run these algos on files differing from 100% perfection? I strongly believe that AI will get there in a short timeframe.

@Frank, glad you enjoyed this Gedanken-Experiment. Actually, I ran this experiment to test if Deep Neural Network predictions will fit the Quantopian long/short equity framework. It doesn't because the Q framework is not about prediction accuracy but rather about discrepancies between long and short entries in hopes that the net effect is a low profit, low volatility which could be compounded in the long run. The stock universe generally moves together with the market (SP500), so a filtered down basket of stocks equally halved between longs and shorts will generally tend to cancel out each other but not exactly to zero. So finding these small positive (profitable) discrepancies is the objective in this long/short strategy. More like a slow and steady flow.

If you want to see how a DNN prediction with only 56% directional accuracy on SPY traded with a basket of 100 stocks could look like, click here DNN And Beyond