Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
October Prize #1 Algo disqualified - Worthy of the Q fund ?
Adrian Boca
edited Oct 3, 2015
Contest Hedge Fund Market Neutral Risk Management Stat Arb

Hello everybody,

First, let me say congratulations to Andreas R, the winner of the October Prize Quantopian Open. This post is not about his algo.

The algo I am referring to is mine. It was #1 in the leaderboard, before the Q team decided to disqualify it. How I know it was #1? Because they just disqualify entries considered for the prize, in the days prior to announcing a contest winner, and not any other entries. As Dan Dunn said it, ' We reviewed your entry to the Quantopian Open. Unfortunately, we don't think that the algorithm would be a prudent investment, and it has been disqualified. [...] The specific rule reads, in part: "We reserve the right to disqualify any entry at our sole discretion. . . . if we conclude that the algorithm is not suitable or financially prudent to trade with real money." ' Fair enough. Even if, from my point of view, it was far from a risky strategy, but I will go into detail later.

So I decided to share my strategy and code with the community. You will decide on your own if the strategy has potential or not and why.

A few words about the trading strategy and the rationale that lead to it:

• The strategy uses only two securities: QQQ and IWM. As you know, QQQ is an ETF based on a NASDAQ-100 stock index and IWM is an ETF based on Russell 2000 stock index.

• Why only two ETFs and not a basket of stocks? Because the cost of going in and out of stocks is far far far greater than trading only two ETFs that are, at all times, both long and short with approximate the same exposure and need to be rebalanced only from time to time. Also, trading ETFs involves a significantly better probability of the order being filled rapidly and at the price you wanted.

• So it's a stat arb long-short market neutral mean reversion (or gap widening, if the case) momentum strategy? This is exactly what it is.

• Why QQQ versus IWM? The QQQ ETF tracks the performance of the Nasdaq 100 Index that holds large cap U.S. stocks and excludes the financial sector and therefore, tend to be focused on the technology and consumer sector. On the other hand, the IWM ETF tracks the performance of the Russell 2000 Index Fund, that holds mid and small-cap U.S. stocks (its investments are in the smallest 2000 companies from the Russell 3000 Index). It's basically all about crowd psychology - how the crowd sees the large cap U.S. companies (that appear in all the finance news networks) compared to the mid-lower-end broader market (basic economy), the mid and small-cap U.S. companies.

• "Buy tech? Sell tech?" - if QQQvsIWM_recent > 0: vs if QQQvsIWM_recent < 0: ...

• Why this strategy is better than trading only QQQ, without any form of hedge? Because it is considerably reducing intra-day & intra-week "black swans" risk. Yes, it can lose money, but not a considerable amount in a short period, because, even if a flash-crash appears in the market and the QQQ vs IWM develop a wider Gap in price (2-3%, maybe 4% in extreme cases), they will always mean reverse in a matter of minutes or hours - at least, this was the case until now. You can build strategies only by backward-looking and minimizing the risk for the potential "going south" situations.

--------- Dan Dunn, about my algo: "We are concerned with a few features of your algorithm. Firstly, the returns were not good over a long backtest. While your algo has performed well in the last two years, it performed poorly prior to mid-2013. In our experience, that is consistent with an overfit algorithm. The second issue is that your algo only holds 2 positions, which carries a lot of risk. By being invested in only two securities, a strategy becomes more exposed to the movement of either security, which increases risk. We are looking for algos with more diverse portfolios.
Despite the fact that we have disqualified this particular algorithm, we think you're doing good work. We are very interested in the type of strategies you are experimenting with like he one you posted here: https://www.quantopian.com/posts/trading-strategy-worthy-of-q-fund?utm_campaign=trading-strategy-worthy-of-q-fund&utm_medium=email&utm_source=forums. It looks like you have some promising ideas, but that you might need to tinker with them to make them robust to longer-term testing. I suggest that you to keep the long-term goal in mind as you make adjustments and I encourage you to enter the next contest.
" ---------

• Overfitting in the non-machine learning trading systems: every trading model has a certain degree of overfitting. "The models are by necessity backward-looking, based on data about how markets operate in all kinds of conditions."(S. Patterson - Quants[..]) But the market conditions change and, from my point of view, a strategy will work for a limited period of time (a few years). "Overfitting generally occurs when a model is excessively complex, such as having too many parameters relative to the number of observations." (Wikipedia) But if the strategy has a few conditions and parameters and has been consistently profitable in the last few years, the probability is that it will behave well in the near future, until markets conditions change, and this is not overfitting, but just current market conditions.

• Market conditions change: What I am trying to say is that, in my opinion, it is extremely difficult to find a non-machine learning/A.I. system, so basically, a simple trading strategy that was consistently profitable in the last decade and will be the same in the next decade. I think you won't find the high predictive performance holy grail trading strategy, but I hope you prove me wrong. How I see it, you need to see the code of the community algos you are planning to deploy in the hedge fund and make a few machine learning systems that are able to start, pause or stop the simpler trading strategies of the community, as market conditions change.

• My algo has room for improvement? A lot, a lot. It is by far not very elaborate and not perfect, but it has potential. And from my experience, the very basic strategies, with only a few conditions and parameters are the ones that are the most reliable and most adaptable when the market conditions change.

• The most important thing: low short-term risk. My algo, this algo, the "144 - QQQvsIWM Evolution ver.3" (now, the community algo) has it. Yes, very low short-term risk.

Thank you very much! I will leave this community for a period of time, commence Monday. Good luck to all!

Adrian

This is how the algo performed in the paper trading Quantopian Open October Prize Contest (trading from 2015-08-27 to 2015-09-30) :

9 responses
Adrian Boca
Oct 3, 2015

This is the 2 years backtest. I used a leverage of only 1:1. It trades only the money it has and has at all times about $50k long exposure with a $50k short exposure. If it beginins to go south, you can see this little-by-little, in time, with minor capital risk.

Adrian Boca
Oct 3, 2015

And this is the 13 years backtest. Feel free to improve it and share it with the community. It has a logic very easy to understand, I think.

Adrian Boca
Oct 3, 2015

And this is how I could easily trick the backtest, as the Q team didn't had access to the code, but I preferred to stay true to my integrity values.
By adding
if get_datetime().year >= 2013:
... do something
else:
... do exactly the opposite

Adrian Boca
Oct 3, 2015

The process of naming the october prize winner:
https://www.quantopian.com/posts/the-process-of-naming-the-october-prize-winner

Grant Kiehne
Oct 3, 2015

Adrian,

Bummer you didn't get capital. It's clear what Quantopian is aiming for, but on the other hand, might they be throwing the baby out with the bath water? What if your 2-year performance is "real" and persists, at least for awhile? My original understanding was that Q would be grabbing lots of algos for their fund, applying their own "secret sauce" to construct the fund, optimizing performance and managing risk dynamically. A true crowd-sourced approach. Now it seems that they want every algo to be a stand-alone fund that they can pour money into. I have some algos in the contest, but I think I'm gonna ask Q to do a check. If they run for 6 months and I end up winning, and then get disqualified because the strategy is "financially unsound" it'll be frustrating, to say the least.

The other thing is that given your algo uses large ETFs, it should scale easily to $1M-$25M, which is the level Q wants to fund strategies. This doesn't seem to be a consideration at this point in the judging criteria.

Grant

Vladimir
Oct 3, 2015

Adrian,
You are October prize the "Sharpest".

For whom are that Rules if referee do not follow them?
https://www.quantopian.com/posts/for-whom-that-rules-if-referee-do-not-follow-them.

Vladimir

Richard Prokopyshen
Oct 3, 2015

I disagree with Q that "The second issue is that your algo only holds 2 positions..." is sufficient criteria for disqualification.

By going long/short with QQQ and IWM, this algo effectively has positions in thousands of securities creating extreme diversity across multiple sectors at very low cost.

When you look at a backtest you can see the securities being traded. It would seem like a reasonably small step to see which two symbols these are when you have an algo that has beat out hundreds of others and complies with dozens of automated screening rules. To me, this algo meets both the spirit and the letter of the objective "We [Q] are looking for algos with more diverse portfolios".

Grant Kiehne
Oct 3, 2015

It would seem that pairs trading, using only two stocks/ETFs, would be excluded from contest/fund, if the "must have more than 2 positions" rule is invoked. Very confusing.

Viridian Hawk
Jul 7, 2017

It's fun to backtest these "old" algorithms with the out-of-sample timeframe. As frustrating as it must have been for Adrian to come in first and be dqed, it turns out Q were totally right to disqualify it -- it was totally overfitted. The last two years since it was posted it nets a whopping 1.9% return, most of that time spent under water.

About Quantopian
Careers
Community
Events
Documentation
Academia
Lectures
Status
Facebook
Twitter
LinkedIn
YouTube
Terms of Use
Privacy Policy

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.