That score is based on 2 days of trading data. There's nobody who thinks that a Sharpe of 36.69 is the last word on that algorithm. You'll find that the contest scoreboard is very noisy in the beginning. Lots of people have a good day.

We don't give out prizes for having one good day. That would be like declaring the winner of a marathon based on their first 1/8 mile. We give out prizes after you've put together a lot of good days. The question will be, what is his Sharpe after trading out of sample for 6 months?

Dan,

This what you said:

We give out prizes after you've put together a lot of good days.

But I can not find any metrics in scoring system which counts good days.

May be there is a reason to add something like this

good days/(good days+bad days) = portion of good days  

or probability weighted ratio of gains versus losses for some threshold return target

SUM(IF(Returns > RiskFreeReturn, Returns - RiskFreeReturn,"")) / -SUM(IF(Returns < RiskFreeReturn, Returns - RiskFreeReturn,""))  

to make scoring system more balanced?

First thank you Dan for your reply but this just leads to more confusion for me about what the "Leaderboard" is all about. How can I see who is actually leading in the standings, you know determine who the next contest winner will be? On Kaggle, the leaderboard indicates who is leading, often using a public testing set and while there are changes in rankings after the private test set is applied, the metrics of the rankings are the same.

And to Vladimir's observation, the scoring system used to construct the leaderboard may need updating. From what you wrote, perhaps the "leaderboard" should be renamed, "The Newest Algorithm with Positive Returns Board".

Would you be so kind and send me a pointer to documentation or give a brief overview that helps me understand how algorithms are ranked? Many heartfelt thank you's in advance.

Hi Sally,

I would urge you to read the contest page available at https://www.quantopian.com/open

You will find that the seven performance criteria that generate the final score for each entry are neatly mentioned there. I believe that the contest is constantly improving, and rules are updated every month to ensure that. Have a look at https://www.quantopian.com/posts/august-contest-rules-update-new-prizes-staying-hedged-going-longer

And yes, the Leaderboard will be volatile during the first few days. The 6-month contest has been kicked off to address that.

Thank you AD for your helpful comments. I read and now re-read the contest page you referenced and while it was helpful, it brings up all sorts of questions most of which I will have to think through for myself.

Correct me if I am wrong, and I'll admit my maths are a bit rusty, but the one month contest uses paper trades for a one month period or approximately 20 trading days. Doesn't that mean that my sampling error at the 95% confidence level is something like 22%? So each stat listed should have a +- 22% range?

A second question is on how the 7 criteria were chosen. One might expect many of them to be correlated since Sharpe ratio, annualized returns and volatility and Sortino ratio all use returns while beta and return stability are quite similar. I've seen these metrics used in marketing and comparing funds, but worry about using them for investment decisions, which is what the contest is - an investment decision on selecting an algorithm. Would anyone use these seven criteria for selecting stocks? If not why use them for selecting algorithms?

The six month contest sounds interesting. 120 trading days still leaves a 10% sampling error. No wonder many investors demand a three year track record using real money before considering an investment.

Justin Lent in https://www.quantopian.com/posts/bug-in-consistency-score explains this briefly, "It's important to think about the metrics used for the contest ranking in a more holistic manner. No single metric can define a great algo from a poor algo and it's important to analyze algos across a breadth of different metrics in order to get a 360-degree view of its previous performance, and hopefully future performance.

Perhaps in a sports analogy, consider the individual sports such as golf or tennis, where most of the time the best #1 ranked player in the world are above average across the breadth of the shots required in the sport, but many other lower ranked competitors are best at any of the 1 single strokes (serving, forehand, backhand, driving off the tee, putting on the green, etc). To be the best in investing is analogous, and in particular building a portfolio, is about selecting the components of your portfolio that are all above average across a breadth of characteristics rather than getting any single metric perfect."

This, I believe gives some insight on why Q insists on using all these metrics, in spite of them being correlated.

And your maths is spot on, of course while using 0.98/sqrt(n) to get the margin of error at 95% confidence with sample size n, the assumption of simple random sampling is crucial. Now its up to you to believe whether every one of those 20 sample points is just as likely as any other in the population, and if you subscribe to the random walk hypothesis :-)

Personally, if you look at how U.S. stocks suffered their worst day in 18 months yesterday, I'd say that confidence level ought to be worse than 22%. But what can you do, just too few data-points.

There are funds which do have Sharpe Ratios of 6.

http://www.globalsigmagroup.com/Past_Performance/Past.html

Hey Sam, certainly a Sharpe Ratio of 6 is impressive, but you also have to factor in survivorship bias. If 100 funds are started all making random trades, then after a year or two there will certainly be a few that have happened to get lucky. These funds are indistinguishable looking at past performance from those that are actually making intelligent trades. Many funds go bankrupt every year, so the funds that survive have a hefty deal of survivorship bias associated with their returns.

If you chase Sharpe Ratios or returns, then you stand a big risk of falling prey to this bias. I would say a first step would be to pick a fund that looks good based on past performance. Then observe it for 3-12 months and check for consistency in performance between the historical numbers and those over the out of sample 3-12 month period.. This is similar to what we have done in the contest and will let you start to get a better sense of whether a fund is actually as good as they say.

You should also try to look at the beta exposures of a fund if possible. It is very easy to get huge returns if you take out leveraged bets during a good market. These funds, of course, are incredibly vulnerable to down markets.