My disqualified algo was mine, and I decided to share it with the community, because I'm not a horder of disqualified algos:) . Link: https://www.quantopian.com/posts/october-prize-number-1-algo-disqualified-worthy-of-the-q-fund
I was a little sad at first, to be true, but I know that I gave my best given the requests for winning the contest, and now I am just a little disappointed. I hope the Q team understands that they began, in the last months, to think like mom and pop investors looking at a stock that performed well in the past, but has reached peak price. Nevertheless, I am thankful for the opportunity given to work in an beautifully designed trading IDE (the best).
Also, I've decided to leave the Quantopian community and work full-time for Misys Financial Systems, my new employer.
Thanks to all, especially to Simon and Grant that trully belived, at first, like me, in this community.

For whom are that Rules if referee do not follow them?

"This algo trades a book of several hundred stocks, presumably chosen using fundamental data. .."

Ok, why not share other pyfolio tearsheets details on the winning algo? Q agrees to keep the source code secret yet have access to a lot of information that algo developers do not have. Is Quantopian going to develop algos inhouse? If not why not publish the winning pyfolio information that you have collected? Do not publish the source code, only information that you have available and we do not... It is in your interest to produce more algos with higher and higher quality....

So the winning algo had 50% cumulative return? I have an algo that has the below numbers backtested back till 2002. My returns are NOT cumulative.

RETURNS: 32.7%
ALPHA: -0.01
BETA: 0.15
SHARPE: 0.07
DRAWDOWN: 20.7%

The drawdown was in 2008 crash. From 2002 to 2008-before-crash it had these numbers (25% return with 5% draw):

RETURNS: 25.8%
ALPHA: 0.00
BETA: 0.16
SHARPE: -0.01
DRAWDOWN: 5.4%

My question is this: Is this considered a good algo? Is it worth trading my money? Is this contest-worthy (just basing on the above parameters)

I would personally not trade a sharpe 0.07 algo, but to each his own!

Interesting. Can someone tell me what would be typical values for the above parameters for
a) Algos that I can use for trading b) Contest-Winners?

Thanks
Sa

Sa,

The premise behind the contest is captured on https://www.quantopian.com/open:

How did you choose this judging criteria? We're building a crowd-sourced hedge fund. These criteria are meant to encourage algorithms that we can use in the hedge fund.

If you then look on https://www.quantopian.com/fund, you'll find an overview of what is wanted (and not wanted) in a strategy for the contest/hedge fund.

Regarding "Algos that I can use for trading" it all depends on what you want to accomplish with your money.

Grant

"The second concern is that the algorithm holds only two securities. That's a lot of concentrated risk. If one of them has a bad event, it's a large risk. We would prefer to invest in algorithms that have more diversified holdings that reduce the exposure to a single stock."

Sorry Quantopian, but this just highlights why you really need to hire some more finance professionals who are as expert at their field as your programming team is at theirs. The two "stocks" were index ETFs composed of 2,100 constituent stocks. Perhaps I missed something, but it appears this algo would be fine with the double secret "market concentration" rule if the writer had foolishly tried to trade all 2,100 stocks individually but when he wisely chose two ETFs that included all those stocks it's too risky? Even though the two portfolios would have exactly the same performance? Actually, no, if he tried to trade the constituents the ridiculous commission and slippage models would eat him alive and he would have a horrible algorithm, but apparently that would be preferable to wisely choosing two of the most liquid ETFs, in fact two of the most liquid stocks in the stock universe, as exact proxies?
You're really losing us with this proliferation of rules that are not only applied ex post facto but don't make sense. At the very least, if you're going to call ETFs composed of thousands of constituents "concentrated risk", you have to know that the vast majority of us are going to think that odd at best. Knowing that, you'd think you would explain up front why you've taken this counterintuitive position, rather than having us all shake our heads and wonder what's going on?
Additionally, you really need to explain if a 2-3 ETF strategy is disqualifying, because there may be lots of us pursuing that with the mistaken idea that an ETF represents a basket of stocks. As Grant pointed out, it's pretty demoralizing if you decide they're not, six months after they've been running an algo. It's Quantopian's game, play as you will, but I would humbly suggest that confusing and demoralizing your top contributors is probably not the best path toward success.
Again, you have hired the best and brightest in the programming world and that shows in the very high quality of the platform. You haven't done that on the finance side, and likewise it shows. You need to hire at least one hedge fund rock star who's been there, done that on the buy side. Otherwise you'll continue flailing about and the "crowd" will lose interest and drift away.

Well said, thanks.

In my view Quantopian are quite correct. If we are talking stock specific risk, then clearly there is none in this case. Well, even that is not strictly speaking true: fraud, mismanagement or other calamity could cause even a physically based ETF to go belly up.

I have serious concerns however with those who fool themselves that the trading of two price series is sufficient. Countless websites push apparently marvellous schemes based on back tests of a very small number of indices and assume they have done their job. They have not.

In a book I published some years ago I gave an example of a very simple system which traded the Dow well for 100 years and then hit a draw down which lasted many years.

My own back testing for any system or concept includes testing on random portfolios and large numbers of securities.

Any back test based on two securities alone is worthless.

Unless it also works on a great many more.

The criticism of Quantopian on these grounds is not, in my opinion, justified.

I have been amused at the criticism of the Contest. You guys have been provided with free data and software which has considerable worth. Zipline itself is a great product with open source code.

It looks to me rather like gifthorses and mouths. I don't subscribe to Quantopians views on how to trade....but who cares. And in any event, what's the big deal about the opportunity to trade $100,000.....fuggedaboudit.......

For a more cogent exposition of my bigoted views see:

http://anthonyfjgarner.net/2014/11/30/dangerous-nonsense-market-timing/

http://anthonyfjgarner.net/2014/09/24/much-quantitative-research-is-misleading-at-best/

Indeed, the platform is a great gift, and getting better all the time. We don't have to subscribe to the contest guidelines, or even enter it, if we don't agree with the direction it is going, and we still benefit from the platform.

For what it's worth, I think the criticism isn't really about 2 securities, or 10 year backtests, or OOS stability, it's that these restrictions have been applied ad hoc and with discretion upon the conclusion of the contest, rather than codified into the ongoing rankings. I understand why, but it really makes the contest feel pretty arbitrary.

I, too, think it's churlish to complain too much about a free service run by genuinely nice folks trying to do the right thing, and about a contest which is offering free capital with basically no strings attached. I think the best approach is to use the tools to make algos that you want and intend to trade with your own money, and enter the contest for additional optionality (contest & fund). If you try to explicitly target the contest or fund, I feel like it's going to be terminally frustrating to chase the moving target and disappointing to have spent so long writing an algo you perhaps can't trade yourself and/or isn't chosen for the fund.

And I'd add to Simon's comment that the free service is spectacular. I have nothing but good things to say about it, and the Q programming team that's putting it together is clearly top notch. And I'd like that platform to continue to be around and continue to be improved for years to come. That only happens if Quantopian is successful in making money. And that only happens if they keep the crowd engaged and excited.
I want success for Quantopian as much as the Quantopian team does. I usually do a good job of caveating this in my critique of Quantopian and apologize that I failed to do so this time. I only offer the critique because I know the Q team is receptive and I want them to succeed.

I must say that ever since joining a few weeks ago, I've been thoroughly impressed with the promptness and feedback from Quantopian staff. I've also observed that the community surrounding this site is genuinely helpful with constructive criticism and suggestions. There are some very intelligent individuals contributing to this community. Regarding the selection process for the October prize winner, I must say that my thoughts have already been expressed quite adequately by previous posters: Kevin Quilliam re the exclusion of ETFs, Anthony FJ Garner re the need for robust backtesting and Simon Thornington re what seems like arbitrary rules added close to the end of a contest. Regardless, I doubt the masses would find much use in an algorithm that has difficulty beating the lowest risk (I hate the use of risk free since no nation can last in perpetuity) alternative. But the beauty is that given the excellent services available, the opportunity exists for anyone to build an algo and invest as they see fit using their own money. Honestly, given the potential, I for one am very grateful.

Hi,

First off: Adrian, I'd like to thank you for your participation in this community, and for sharing your algo. I love that open-sourcing algos from the contest has become a community tradition. Judging the contest is a big responsibility, and I am grateful for the scrutiny and criticism of our decisions.

I agonize over disqualifications because I know the effort required to make a good algo. But, the contest prize is our capital, and we are held accountable for the results. So, I feel compelled to judge as stringently as possible.

Adrian's algo was disqualified because we concluded that its last two years of simulation were overfit. The ten year backtest shows that clearly, and overfitting is a sufficient reason for disqualification. Please note that I don't think Adrian intentionally overfit - I think this type of strategy is very susceptible to inadvertent overfitting.

The small number of positions jumped out at us, because:

1. The small position count makes the algo more susceptible to inadvertent overfitting, due to the attendant drop in observations.
   
2. The biggest systemic problem we see in our population of algorithms is position concentration.

We could have stopped our explanation at overfitting, so why would we point out the position concentration? We are a community, and we need to provide feedback that considers the community as a whole. Disqualifying events, while gut wrenching, are learning moments for the whole community.

We took this opportunity to call attention to the systematic problem: position concentration.

Kevin is right about the use of ETFs. At small capital bases like the contest prize, ETFs are extremely helpful tools for reducing transactions and increasing diversification. We have awarded other winners who also use a small number (read: 2) of ETFs, and we would consider doing so in the future. Position concentration is not a hard rule in the contest, and we would not have disqualified Adrian for that reason alone. But, Anthony is also right that ETFs are not without idiosyncratic risk, and small position counts mean we need much more out of sample data.

This is the last contest that has less than 6 months out of sample. I expect the rules and ranking to stabilize as a result.

Thank you for all your ideas, criticism, and encouragement. Everyone at Q works each day to make the platform, the contest, and the fund better. The community is our motivation and our conscience. We love how seriously you take the mission and the work.

thanks,
fawce

Hi Fawce,

One approach would be to offer Mr. Boca the opportunity to run his disqualified algo for 6 months, after which you would have another look at it. If good, then he would get $100K of capital for 6 months. This should be a safe bet for Q, if your analysis and intuition are correct. The most you could lose would be $5K, since have given yourselves the option of stopping algos at $95K. And you'd look like good sports, too boot! Seems like a win, no?

Grant