Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Contest ranking system: please remove the incentive for bad form
Lionel Ouaknin
edited
Contest Feature Request

Hi fellow Quantopians members and staff,

There something that's been bugging me regarding the contest ranking system for some time.

To illustrate my my point, bear with me this small thought experiment:
Let's say that I submit to the contest two perfectly identical algorithms with one only difference, being that one targets a full leverage using Optimize API MaxGrossExposure of 1.0, while the other targets an exposure of only 0.20, so that the initial capital in the first entry is 10m but only 2m in the second one.
Now, since the two algorithms are identical, one would expect that they'd be ex-aequo, right ?

In fact, there's good cause to suspect that the second algorithm would perform better than the first one. For starters, since Slippage scales with capital, Sharpe ratio and Sortino ratio will be slightly higher with smaller capital. In my tests, this effect is small but not negligible.

Off course the returns of the second algorithm would be 5 times lower (approximately since you can't own share fractions...) with current contest commission policies. But since capital is divided by 5, then volatility, beta, and drawdowns will be also much lower. Stability would be also higher, though I'm not sure to what extent. So out of the 7 statistics that Q computes for algorithm ranking, only one would be lower for the second algorithm, but all the other 6 would be better, slightly for risk adjusted returns, and very highly for pure risk measures.

So even though the two submission are exactly the same algorithm, the second one will beat the fist one by a wide margin either IS or OOS.

Now, say, let's imagine a quant desperate to rank higher in the contest, if he wanted to gain an unfair advantage over some other entry with the same kind of performance/risk statistics, he would simply half his exposure, and see how it goes. And do it again, to satisfaction. I call that half-quantism.

I'm not all knowing, but I have reason to suspect that some of the top contestant use this approach (0.1% vol for 10m equity investment, come on....). This is bad form, because it gives an advantage to contest entries with an unproven ability to scale, over entries with with a proven ability to scale up to 10m. Also, it sets a wrong incentive for people to do the same.

Now, don't mistake me, I'm all in for algorithm that vary active capital with time depending on market dynamics, but i don't think that's what we're dealing with here. Maybe my logic is flawed, in that case please enlighten me. but i stand for it for now.

So I want to suggest this to Q staff:
- Create a qualitative badge named 'Fully invested': It would be granted, if the algorithm is X% invested, Y% of the time.
- Replace beta with jensen alpha: it scales linearly with beta, and it would destroy half-quantism.
- Use the CAPM alpha as a metric.

2 responses
Dan Dunn

Lionel, you've got some very good points in there. We've definitely seen some of the problems you've pointed out. The set of rules isn't perfect, and it lets in some algorithms that just aren't desirable. We've seen some people try to game the system, and we've disqualified a few entries that were the most egregious. The good news is that even with it's flaws, the contest has been "good enough" for it's most important purpose, which is to educate the community about what we're looking for in the algorithm.

We've been doing a lot of thinking over the last few months about what the next iteration of the contest will be. I'm hopeful that we'll be able to share a new proposed framework for the contest in a few months. We will definitely take these ideas into consideration.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Tony Morland

Hi Lionel,
Thanks for your post and in particular for your comment / request regarding scalability and the need to provide incentives that help us all to better understand and achieve the goals or maximize the objective function that is truly important to Quantopian.

Hi Dan,
At the top of the back-test analysis tear sheet there is a table entitled "Performance Statistics". I am familiar with most of these metrics, but there are several that I don't know (specifically Omega ratio, Tail ratio, and Common Sense ratio). I would guess that I'm probably not the only one. Is there a table of definitions of these metrics, and any others that might be of high importance to Q, to help me & others fully understand all the targets that we are or should be aiming for?

PS. Thanks Lionel for introducing me to the concept of "half-quantism" :-))

Cheers, best wishes,
Tony

About Quantopian
Careers
Community
Events
Documentation
Academia
Lectures
Status
Facebook
Twitter
LinkedIn
YouTube
Terms of Use
Privacy Policy

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.