Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Training Incomplete - covers the basics well but not the actual signal generation

The training seems to cover some of the basics of time-series econometrics well, but the signals for momentum, mean-reversion, etc, are truly off. The ribbon methods are not on, the physics momentum method is not on. Where did EWMA of returns or differences of EWMAs of prices get left behind. There is an active academic literature on momentum and it is, unfortunately, not being presented.

Some of the methods presented (ribbons? crossing moving averages, etc?) are not really presented well and many trend estimation techniques (a regression in time? really now!) are bound to fail out of sample. Some really just miss the point. Kalman filters are interesting, but why do you include a secular trend? Why not just local levels and local trend models? (i.e., simple MA(1) models are interesting enough and have some good properties).

Financial markets require, more than anything time-series econometrics or signal processing. SDEs don't help much in algo trading, although they produce some good toy models.

Where are the data snooping (p-hacking) tests? Adjusted Sharpe Ratios? Bootstraps....

This is a standard problem of libraries like Zipline and platform like Quantopian that is beautiful from an IT p.o.v., but seem to neglect the underlying statistics. Computer Scientists (including a lot of the ML crowd) and mathematicians (especially those who can do probability and SDEs) don't know that well about much of the modern stats that is needed. It's seriously hard stuff.

13 responses

To be honest Nick, I think that is way off the mark. All of it. You can waste endless hours engaging in math-turbation but you will be no better off than had you stuck to a simple trend following technique.

In my view Q is an excellent product as is Zipline, if you can take the learning curve.

I really can't see what is lacking unless you imagine you can find The Stone.

We are not dealing with physics. You can't run experiments. Backtesting is of limited use in ever changing markets. Most short term trading strategies are just that.

Even those with an "edge" find it disappears - look at HFT recently. Look at the absurd ups and downs of CB arb and volatility arb.

Making big money requires being in the right place at the right time: the Big Short, US IPOs in the 90s, buying big after a crash. HFT a few years back.

Nothing wrong with systematic investing - look at Vanguard and iShares.

But why do people insist in believing that statistics is going to make them a fortune in the stock market?

Beats me. Fees will make you a fortune if you are good at marketing, even if you are charging ETF rather than HF fees.

In any event, what are you trying to achieve? Invest for the long term? Day trade? What are statistics going to do for you? Is it "prediction" you are after?

I honestly think most of what we practitioners do is a complete waste of time. Especially in the Quant investment space.

Endless code, endless obfuscation, and overwhelmingly average results.

Unless you have an edge and exploit it vigorously until it disappears.

Anthony, I think we're talking at cross purposes. The training is ok, but the signals that are presented are far too complex. Ribbons of moving averages? Just a simple EWMA is good enough to get you started. It has lots of interesting properties.

The Quantopian platform is tremendous, don't get me wrong. It's the training materials (I mean the course work taught by the Imperial college prof) which are not so well suited. Did you actually look at Lecture 26?

There is lots of good analysis available for simple strategies, but the training seems to lose direction exactly when it covers momentum strategies. Ribbons of momentum strategies? Distance metrics? Physics-based momentum metrics?

One point I do disagree about is the mathturbation point-- there is a lot of deep analysis that is feasible for simple strategies. That is the beauty of it all. Over-engineering is a huge problem. Two papers try to cover this big problem: Backtesting and Financial Charlatanism. And, as you might say, they do cover to a large extent the absolute futility of what we do! (a lot of deep analysis to show that quants are mostly wrong!).

Consider the world of actuaries and the final salary scheme disaster. Actuaries spend what? 8 years training? Did they spot the mess that would result from guaranteeing final salary pensions schemes? Did sophisticated statistical forecasting and testing enable them to predict companies could not afford these schemes because investment returns would go south?

Like it or not forecasting aka predicting the future is achieved equally well by sticking a finger in the air and seeing which way the wind is blowing.

I see you're a bit of a quant nihilist. This is something you and I won't agree on. I've covered both GRexit, writing a book on the place of uncertainty (not probability, but uncertainty, and robustness) in risk management, and as well do quant investing. Quants work in a (conditional) world where the rules stay the same and probabilities work. If there are rule changes (e.g., Grexit, Cyprus-exit or Brexit even, US Treasury debt ceiling, Scottish referendum, etc), then many of the rules fray and the regular quant based methods often will not work. (although some will....beware RV but simple adaptive strategies did work well).

Oftentimes the compensation for a quant method is merely the fact that you are underwriting the market, being short uncertainty premium.

Does it mean it never works? I don't think so. Quant strategies can and do work and momentum (and mean reversion) both did exceptionally well during the crisis. Carry should not, and did not. It makes decent sense to me.

Nick, many audible chortles, I see I read your post wrong. Yes, I love those papers even if I find it difficult to believe one should not look at optimisation (look, not necessarily practice). But yes, beware Geeks bearing Greeks as the saying goes.

I have been doing this so damn long now I KNOW that I am mostly wrong.

Here are couple of favourite sites - some of the same authors as those papers. Thank goodness some geeks can spot the absurdity of the industry.
.Financial Math Org

Mathematical Investor

I see you're a bit of a quant nihilist

No, I'm more of a closet Boglehead. Who, whatever he might like to think, IS a quant. I believe in following markets through asset allocation but when I see those rare opportunities for huge profits I take them if I can.

Witness the tech IPO boom in the 1990. When anyone who knew the word "flip" could and did make a fortune.

I just don't go with the bullshit and on balance believe that excpet for exceptional circumstances, if you are lucky you will get what the market can give you. And no more. And if you use high leverage you will very likely go bust if you stick at it long enough.

Hey, Nomura lives on in London does it? Most of ECM was chucked out recently plus the derivatives hedging stuff. Glad you survived the cull!

I just don't go with the bullshit and on balance believe that except for exceptional circumstances, if you are lucky you will get what the market can give you. And no more. And if you use high leverage you will very likely go bust if you stick at it long enough.

Anthony, what is your comment then on the seemingly continuous above-the-market performance of the Medallion fund or Berkshire Hathaway?

No disrespect meant her, just sheer curiosity -- I greatly admire your thoughts and your experience.

Anthony, thanks for the posts. I didn't know the second link. Looks really good!

I read your link and realised we were basically saying the same thing. I'm a mathematician and I think mathematicians are often the worst offenders! Statisticians often know the value of parsimony.

Anyone who thinks they'll fit some DNN to the market, good luck. Overfitting for amazon will cause them to lose some clicks, but won't make them go out of business.

Have you seen this paper? Quantifying Backtest Overfitting

In the end, it's not entirely absurd, though. As you put it on your site-robustness is crucial. It is something that has been studied for years and which we can (try to) ensure. Most everyone does not.