At Quantopian, we continue to get more experience and deeper insights into how to run our fund and what types of algorithms work best in our investment process. In addition, we have frequent discussions with signed authors, where we learn about their challenges crafting investment algorithms and their questions about the fund. With the benefit of these learnings, we are able to provide actionable insights on what to focus on and what to neglect when developing algorithms.
Our approach has changed quite a bit from before. Initially, we ran your licensed algorithm pretty much exactly as you coded it as its own standalone portfolio. This has several downsides, the most severe being the lofty requirements on the algorithm to appropriately manage trading and risk. That approach also made it very difficult for us to effectively manage risk across multiple strategies.
Our new investment approach is called "factor combination", where we do not view your algorithm as something that emits trades we execute, but rather as a "factor" (i.e. a daily scalar score for every stock in your universe that should be predictive of future returns). While we can't directly observe the underlying factors in your algorithm, we use the end of day (EOD) holdings of your algorithm as an approximation. In this analysis, we completely ignore individual trades and also recompute your returns based on your EOD holdings.
We then select multiple factors from the community, potentially do some transformations on them (e.g., to manage turnover), and then combine them together into a single factor. We then plug this combined factor into our optimizer to form our final portfolio. This approach gives us more flexibility and significantly lowers the requirements on individual strategies, allowing us to license new strategies faster and in greater numbers than before.
Given our shifting investment process, we also want to explain how we are thinking about the contest. We are not immediately making changes to the contest rules or guidelines for the "factor combination" approach. Even with our new approach, the contest continues to be our best resource for finding authors to fund; we look closely at the authors who enter the contest, given the skill and effort it demands. In addition, changing the contest rules is a costly procedure (time and resources for Quantopian engineers and community members) and we'd like to see how the suggestions in this post are received before we make any rule changes.
New Suggestions for Aspiring Fund Authors
With this updated investment process, we have some new suggestions for developing new algorithms / factors:
- Your primary focus should be on coming up with new factors with positive, medium-term alpha (see below on what that means).
- Find alpha in new places: As price-derived factors have been around for ages, your chances of finding alpha that is likely to persist into the future will be maximized if you look at alternative datasets like estimates. You might also consider uploading your own data via self-serve.
- Set your trades to execute 1-2 hours before the close: since we evaluate your algorithm purely based on its EOD holding, it does not matter how you got into your portfolio at the close. Whether you trade at the open or at the close, we won't see a difference, even though it will look different to you in the backtester. By setting your trades to execute near the close they will certainly get filled and your backtester performance will look similar to what our own analysis will show. Finally, when your strategy's performance is influenced by the trade time, it's indicative of short-term alpha, which we are currently not trying to capture (more on that further below). Here is some example code:
schedule_function(rebalance, date_rules.every_day(), time_rules.market_close(hours=1)) - Be mindful of alpha decay: we try to have our final portfolio follow the combined factor as close as possible but there is always going to be some delay due to turnover restrictions. As such, if your portfolio only exhibits alpha on the first day, it might actually be detracting on subsequent days, when it actually makes its way into our portfolio. It's thus critical to know how predictive your factor is when its delayed by a few days. This alpha decay analysis is the central point of our new tearsheet: https://www.quantopian.com/posts/an-updated-method-to-analyze-alpha-factors In fact, this analysis is what we use to evaluate your algorithm and thus should be your primary tool as it shows the alpha decay front and center. Unfortunately the backtest analysis screen will not show you this information: If you have a factor with lots of short-term alpha it will look great in the backtester (which assumes no trading delay) but to us it will not look attractive as we want positive alpha over several days (anywhere from 5 to 10). If you do have short-term alpha that you want to extend, one possible approach is to increase the lookback of your factor (e.g. use 6-month instead of 3-month momentum). Keep in mind that there is no easy trick for finding alpha with slower decay rates; that is where having innovative ideas is most valuable.
- If you need to reduce turn-over, average your factor over multiple days rather than by subsampling (e.g. only trading weekly): Related to the point above, we don’t target 100% turnover per day; we try to build exposure to your portfolios subject to all constraints. We would rather not trade into a stale portfolio which you artificially subsampled. A better method to reduce turnover is to apply a moving average (e.g. 5 days) to your factor scores. Pipeline makes this very easy (
SimpleMovingAverage(inputs=[factor.notnull()], window_length=5)). That way it will reduce turnover while still updating daily. You can use the alpha decay analysis to iterate how much you can slow down the factor to still capture alpha. In general, turnover <20% and alpha decay that doesn't drop below 0.5 in the first 5-10 days is going to be ideal. 20-30% turnover is in the grey area. - Worry less about transaction costs: The impact of one factor in a large mix of other factors where you don't know how it's traded is impossible to estimate. As such, you should worry less about the cost to trade, since our portfolio is now less sensitive to a single fill price. An exception to this rule is an algorithm that only trades small, illiquid names; that portfolio is harder to scale, so we have a harder time funding an algorithm with that universe.
- Use the optimizer with care: while it's tempting to squash all risk by relying on the optimizer, doing so can have very detrimental effects on your alpha. For example, one thing we often observe is that due to certain constraint settings the resulting portfolio ends up being equal weighted. Thus, your original factor that scores stocks where it has a lot of confidence highly and vice-versa is losing all that valuable sensitivity due to the optimizer. Remember that we only see your final EOD holdings, not your actual factor scores. Try to have your final portfolio be the most accurate representation of the original factor. To achieve this, you should use the optimizer as little as possible and not worry too much about exposures, especially if specific returns look good. Code-wise, you should not use
MaximizeAlphaand insteadTargetWeights. This is a good place to start:order_optimal_portfolio(opt.TargetWeights(weights), constraints=[]) - Try to keep your universe as large as possible: Evaluating your factor on a large universe (e.g., the whole QTU) will make it harder for you to overfit and give us more optionality. It's natural for your signal to work better on stocks where factor scores are extreme, but if you keep the sensitivity of your signal (see the previous point) most of your returns will come from those extremes anyway as they will get the largest allocation if you don't squash things to equal-weight with the optimizer.
- Use proper hold-out testing: overfitting is still the biggest risk you can run into and you should be paranoid about it. Something easy to do is to never evaluate your factor on the last 1 to 2 years until you are absolutely happy with it, and then only test it once on the hold-out period. If it fails, you should become very suspicious if you overfit somewhere in your process or if perhaps your factor favors certain market regimes.
We hope that these pointers clarify what we are looking for and help to focus your efforts and to avoid wasting time on the wrong things. It will not be the last time that we learn new things and provide updated advice. Thanks for your eagerness to learn and please let us know if there are any questions. Thanks also to our fund authors for proof reading and providing valuable feedback on this post.
