Hi Enigma,

Here are my thoughts:

Are these numbers real, or is there any indicator that the model
wouldn't work in reality? It trades once per day and uses the standard
commission and slippage model.

If you're trading mostly low liquid small-caps, then yes, the default slippage model may not be accurately reflecting realistic fills.

Is this something worth investing more time into; and if so, in which
direction would I go?

It depends a bit on what your goal is. If you want to do well in the Q Contest, then it might be fine as it is (it might do very well), as long as you can bring down turnover and meet all other requirements. If you're looking to get funded, then I would try to smooth the factor to both bring down turnover, as well as lessen the sudden alpha decay. You could try to take the moving average of the factor, as Thomas' suggested in his 'How to get funded in 2019' post. Or even better, smooth the factor as close to the data source as possible. Personally I would look for lower turnover and longer term alpha though.

Is there a way to test the algorithm out-of-sample, even though it
doesn't meet the contest criteria (without waiting N months)?

Only by holding out data yourself. I don't know of any other way if you can't submit it to the contest.

Is the sudden and horrible alpha decay a general problem which may
potentially make such a strategy infeasible?

That also depends on what you intend to do with it. Personally I think Q would be unlikely to fund it due to the sharp alpha decay, high turnover, high position concentration (not sure what's going on with the position spikes either, but I don't think it's ideal). The long/short percentile holdings graph looks very strange as well.

Looks like there might be mostly intra-day alpha there (especially if you're re-balancing close to the open), and possibly only because the default slippage model is giving you unrealistic fills.

Hi Joakim,

thank you for your feedback! I don't really know what my goal for this one is, yet; I don't think it can be modified easily to pass the contest criteria. As you remarked correcty, this is probably mostly intra-day alpha. Usually this should mean: put it away and start anew. However this one has this incredibly looking track record (well, at least for me those numbers seems somewhat unrealistic: it made over 1000% in 12 years, never had a year below 0% and 2008-2009 was its best year). So it's mostly curiousity if I have merely found a (known?) glitch in the backtester, of if this algorithm provides some interesting insights.

Taking your feedback, I have tested several modified slippage models -- and indeed, after reducing the volume limit and/or increasing the slippage, the profits drop very quickly. Starting a trade a little bit later (i.e. not after market open but after half an hour, as noted in [1]) with the standard slippage model reduces the profits as well, but it still works fine. All in all, I have no real idea what kind of slippage might be realistic for such a scenario.

That also depends on what you intend to do with it. Personally I think
Q would be unlikely to fund it due to the sharp alpha decay, high
turnover, high position concentration (...)

As stated above, this is not necessarily about Q funding my algorithm. It looks waaaay too good to be true -- and I can't simply ignore that.

(...) (not sure what's going on with
the position spikes either, but I don't think it's ideal). The
long/short percentile holdings graph looks very strange as well.

The strategy trades a position when it fulfills certain criteria. If there are too few positions, you see one of those spikes as it uses the remaining Universe to fulfill the dollar neutral requirement.

[1] : https://www.quantopian.com/posts/changes-coming-to-the-default-slippage-model

@Enigma,

I don't want to dampen your spirit but what you've stumbled upon is a known "bug" in the intraday trading portion of the backtester which gives inaccurate performance results. If you can recalibrate your algo following Joakim's suggestions to lower turnover, leverage = 1 and EOD trading then you see what your algo might churn out more accurately.

@James Villa,

What's the known bug? I'm only aware of the Limit Order problem, is that it?

That's part of it. Contact Support to get full details.

Oh yeah, good points. @Enigma, if you’re using limit orders, your returns are unlikely to be real. If you’re not using limit orders, and you’re trading on a reasonably liquid universe, there might be some real (intraday) alpha there. Maybe try limiting your trading universe to Q500US or top 1000 Average Dollar Volume stocks and see if you still get consistently positive returns.

Thank you for your feedback @James Villa.

The leverage is already equal or smaller 1; it is not really doing any intraday trading except that it swaps all positions with a different set of positions at the beginning of each day. Would you mind elaborating a little bit further where the inaccuracity comes from?

The strategy trades a position when it fulfills certain criteria. If
there are too few positions, you see one of those spikes as it uses
the remaining Universe to fulfill the dollar neutral requirement.

This is also a bit of a red flag to me. To go all in (and out) of the full universe only to meet the dollar neutral requirement sounds very expensive to me. It doesn't help with your turnover either. Personally I would limit trading to as little as possible (trading is expensive). Others on here might disagree with me on this, but that's how I see it anyway. :)

@Enigma, I do not have indepth knowledge on the details of the exact problems in intraday trading other than those I read within this forum. The limit order problem as Joakim just describes part of it. There are also problems with getting realistic fills, etc. Bottomline, the backtester is oriented for EOD trading.

@Joakim:

This is also a bit of a red flag to me. To go all in (and out) of the
full universe only to meet the dollar neutral requirement sounds very
expensive to me.

It certainly isn't ideal. This is from back when I hoped I could enter this algorithm into the contest, before I found out about the ridiculously high turnover. Since this algorithm is so far off from meeting the contest constrains anyway, one might reconsider dollar neutrality as well; but if one wants to have it, this is one way to go.

For your liquidity concerns: The algorithm usually uses the QTradableStocksUS. I always thought this meant I was trading a highly liquid Universe already. Isn't that the case? Anyway, I took your advice and tested the algorithm on other Universes (Q500US and Q1500US). In Q500, it barely finds any signal and does almost nothing; it is still positive on Q1500 though.

The QTU is liquid enough to trade yes, but the default slippage model may likely not be aggressive enough still on the low liquid end of the universe (and too aggressive on the liquid end probably). In other words, you’re getting fills in the backtest at prices where you wouldn’t get the same fills in the live market (only market makers would). This is a general problem with the default slippage model (it’s quite difficult to model correctly). And if most of your returns are coming from low liquid stuff well that’s a problem. In the live market only HFT market makers would get filled at those prices and make these type of returns.

In short, with a relatively concentrated portfolio, if you’re mostly trading low liquid stuff, your returns are not likely to be real. If you’re scoring and holding most stocks in the QTU (with relative, not equal, weighting), your turnover is on the low side (less than 20% daily but lower is better), rebalancing near the market close, and your average and median holding period is at least a few days, your returns are much more likely to be real.

Congrats, those are some killer scores.

I've been trading with real money for a little bit now and learned some painful lessons along the way.

Are these numbers real, or is there any indicator that the model wouldn't work in reality? It trades once per day and uses the standard commission and slippage model.

Stocks on the bottom end of the QTU can actually be illiquid enough that it'll be difficult to get a good fill in real life. Quantopian doesn't do a good job of modeling fills on illiquid stocks where spreads can be an issue. Also depending on time-of-day (such as during the first half-hour of trading), spreads can be really bad across all assets. What I've been doing to get around paying spreads is using MOO LOO MOC and LOC orders. I would think for QTU stocks and retail-level portfolio balance, this would be an adequate solution.

If this is a long-short strategy you have to consider availability and borrow fee for stocks on the short side. If you go with a brokerage like IB you'll have access to many stocks for shorting, but you'll be paying commissions in addition to borrow fees. Does your strategy survive IB commissions (which are significantly higher than the default)? With such a high turnover, commissions will add up really quick. If you go with a commission-free brokerage like Alpaca, they only currently support shorting "easy-to-borrow" stocks. Not all of the QTU is easy to borrow. You would need to see if your strategy holds up when limited to ETBs. Because these factors aren't addressed by the backtester, it is easy to find alpha where in reality it's not really there.

Is this something worth investing more time into; and if so, in which direction would I go?

You could figure out a way to get it into the contest, but due to the alpha decay it's highly improbable that Quantopian will take any interest in licensing it for their fund.

As for trading it yourself, that hinges on whether you can get access to the real-time data you need. (I think a FactSet subscription costs more than a new Honda Accord.)

Is the sudden and horrible alpha decay a general problem which may potentially make such a strategy infeasible?

I don't think so. The high turnover is perhaps worrisome, depending on whether you can get frictional costs under control.