Quantopian open scoring factors ranks

Oct 4, 2015

Correlations cannot prove anything.

Simon,

I will agree with yours statement in only one case:
Correlations cannot prove anything if Pearson Correlation Coefficient is near zero.
But if it goes to 0.35 or -0.35 you have 2:1 statistical advantage.
If it goes to its extremes near 1.0 or -1.0 it shows that between factors exist almost functional relationship.

All that your correlations show is that, amongst the population of algos, those doing relatively well in paper trading have worse relative stability.

Your argument rests on a key assumption: that the algos worth keeping are those with good relative ranks in the other measures. You could just as easily assume the converse, and throw away all the other metrics.

In fact, the truth is probably that the algos which have both stability and good metrics are rare, so rare that they do not affect the correlation statistics (if they exist at all).

Simon,
Sometime ago you agreed that stability factor promote looses and doing nothing.

Simon Thornington
May 4, 2015

Hmm okay sorry, I see the problem now! That does seem strange, and I agree with you, it seems odd that that would have made it to the top ten.
https://www.quantopian.com/posts/how-stable-is-stability-calculation

Did you change your mind?

Nope I still disagree with it, but correlations don't show anything...

Some more samples.
This algo was ranked 132 in October 2015 Quantopian open. To my mind it deserve to be somewhere among top 10.
It was killed by stability factor.
Do you see any bad metrics except stability?

If you do not trust in Pearson Correlation Coefficient than
look at performance of best 50 by stability here.

You may compare killed by stability to best by stability.

Sortino with Stability scatter chart.

Your data actually supports the opposite of what you are arguing; the stability factor is something uncorrelated with the other return metrics, and therefore measuring something completely different, which is good - it's good to have orthogonal features.

The fact that algos which have good stability have bad returns and vice versa might just be a reflection of the algos, rather than the metrics. Stability will reward algos which do nothing, or which steadily lose money, which is exactly what it is supposed to do...

I struggle to understand your vendetta about these (and other) minutiae of the contest.

Data actually supports what was at the beginning post : stability factor has negative correlation to five major scoring factors
and not something uncorrelated with the other return metrics.
This is not vendetta.
Part of objective function for 10B fund is not minutiae.
But I like the way you are thinking especially last words:
Stability will reward algos which do nothing, or which steadily lose money, which is exactly what it is supposed to do...

Yes, but who cares, it is not supposed to correlate, it's value is in its orthogonality. You are falling for a classic black swan fallacy; just because all the algos in the algo pool are such that X and Y are negatively correlated does not mean it's wrong to search for things that have both X and Y, it just means that what you have gotten so far is not what you want. The solution is not to drop Y as a criterion, it is to shift the population towards what you want.

To take your argument to its logical conclusion, if the merit of a scoring factor is in its correlation with other scoring factors, the solution would clearly be to score based on Sharpe, Sharpe*2, Sharpe*3, Sharpe*4, Sharpe*5, and then congratulate yourself for having five perfectly correlated scoring factors. This is ridiculous, so clearly the merit of a scoring factor is not in its correlation with other scoring factors, or in it's anticorrelation with other scoring factors. You could reasonably say that a scoring factor's correlation with other scoring factors would be better off closer to 0.

Simon,

I agree with You could reasonably say that a scoring factor's correlation with other scoring factors would be better off closer to 0.
What do you think about this kind of procedure for choosing factors:
Factor1 will be the one with highest positive correlation to AnnRet in our case it is Sharpe.
Than I may add factor2 only if it has closer to 0 correlation to factor1 and positive correlation to AnnRet.
If I find factor2 than I may repeat the cycle looking for factor3 from all possible and so on.
If we follow this rule the only two factors will be chosen Sharpe and BetaSPY.

Do we really need 7 if 5 of them substantially correlated?

Well, there's definitely a flaw in a simple average of features which are all colinear/correlated, in that the score response is highly sensitive to things which appear multiple times, especially in denominators. I don't know the best way to find a bunch of orthogonal features which measure what you want in an algo though...

Perhaps simpler features combined into a linear model, which is pre-fitted to maximize the Sharpe of a hypothetical combined portfolio. That might have to be an iterative process between fitting the model and re-ranking the algos to join the hypothetical portfolio

The features might include:

Annualized Returns
Annualized Downside Volatility
Annualized Downside Median Deviation
Max Drawdown
Max Drawdown Length
Stability
Correlation with other algos
Beta to SPY

Those will be correlated, but at least each is bringing something new to the equation a little more than Calmar + Volatility + MaxDrawdown + Sharpe + Sortino, which are all very interdependent.

I agree with this "Those will be correlated, but at least each is bringing something new to the equation a little more" ether.
More of that I will add some more positively correlated to Annualized Returns:
Omega ratio
ROC expectancy.

But Stability for what?
Just to attract possible customers by name?
Than I will definitely recommend Quantopian to license yours Stability definition for orthogonal thinking customers:
"Stability will reward algos which do nothing, or which steadily lose money, which is exactly what it is supposed to do..."