All academic finance is built on the idea that the market price is right. Well if the market price is correct then all assets should achieve the same risk adjusted return. Since all academic finance is based on it and has been for years that is what most professionals are taught. We are tested on the concept if we wish to get any certification as well.
Obviously it is a terrible investment practice as we do not actually know what the expected return is. Also we know that Beta isn't set in stone so CAPM falls apart quickly. Why is it still so common then?
I have a few theories. Mostly that financial theories have been around for hundred of years and started with bonds. Bonds yields are much easier to calculate an expected return and these theories fit fairly well. Add equities in and since we have a hammer everything looks like a nail. Now we have hundreds of years of theory and ideas like value investing that have only been around about 70 years. Academia hasn't had time to catch up to be honest and there are no clearer theories as regressions are really the only thing they seem to employ to understand markets. So the best they can do is come up with factor models (Fama-French) that consider not only Beta's but also firm size, assets and etc. It explains markets better and they get a nobel prize.

Great, important topic. Many quants using Quantopian currently have backtested algos (of varying risk levels) that are diversified and suitable for the average investor that have outperformed a standard MPT asset allocation portfolio (of comparable risk) from 2002 - 2015 or 2007-2015. But, MPT lives on stronger than ever...why? The message has not gotten out. The MPTers are very protective of their theory because their management fees and businesses count on MPT. They will lash out at anyone who challenges MPT as a "dangerous market timer" or crazy person who does not understand asset management. MPTers talk about low cost and being fiduciaries, but their laziness is adding risk to investor's portfolios. Somebody finally needs to start publishing some analysis of how algorithmic wealth management is superior to MPT since algos can manage risk better and possibly provide lower volatility and improved long term total returns. MPT is great when everything is going great, but things don't always go great in the markets...and that can be a problem for someone about to retire, or in retirement, or a pension fund counting on returns, or an endowment counting on returns.

@emanuele, @Matthew, @Jeff
To answer yours question: "Why MPT is still in use?"
I search Q for Modern Portfolio Theory, got algo by Wayne Nilsen, cloned it, made some small parameters changes and becktested .
To my mind the results are not as bad as you stated nether for retirement account nor for pension fund.
If you will do the same with Risk Factor Models and get ether lower volatility with same return or grater return with the same volatility please attach backtest and will discuss them.

MPT is built on the assumption that trying to time the market is a futile exercise. Well, it's not if you are a quant and a good one. A quant is always trying to time some behavior - especially mispricing. And good ones get it often right. For instance, buying when there is blood on the street (extreme fear) has proven to be a decent timing tool. It works even for fundamentals driven guys! And yes.

There is a bigger flaw in MPT. Diversification among asset classes would work if the all asset classes involved produce respectable returns but fail temporarily and the failures are uncorrelated! It can't work if one asset class produces 8% and another produces 2% but is inversely correlated with the former! Should I invest in an asset class that offers me a yield of 2% just because it hedges the risk of the part producing 8%. Staying with the asset class that produces 8% and living through the downturn might be better.

As @Emanuele mentioned, the mean and variance are not fully known quantities. There are different regimes in the market. The mean and variance change with the regimes. Real estate in a booming district might offer double digit returns during the boom time, but not forever. That's clearly a regime shift. So all real estate at all times is not equal. Gold does well only when everything else does badly. Emerging market debt offers very close to double digit returns but is exposed to currency risk ........... If I want my portfolio to work well, I still have to churn it and try to time the entry and exit of different parts of it. I need to get rid of real estate in a saturated city (and perhaps get into real estate somewhere else). I need to stay out of the useless yellow metal when the other markets are stable. I need to get out of the equity market when the valuations become frothy, say CAPE of 40......

You can choose to be a bull and you can choose to be a bear if there is quantitative / fundamental reasoning to it. But there is no reason to be a pig. That's unfortunately what MPT recommends.

@Bharath Rao
Are the results of MPT algo I just backtested bad or not bad?
If not bad, then what you wrote bad about MPT has nothing to do with MPT itself but with its subjective implementation.

MPT or not, part of the problem is that folks don't follow the simplest of rules:

1. If there's any chance you'll need the money in 5 years or less, it probably shouldn't be in the market in the first place.
   
2. Put aside 6 months to 1 year of your salary into a cash emergency fund.
   
3. As a percentage, have no more than 100-(your age) in the market. The rest in short-term high-quality bonds, cash, CDs, etc. Something that won't go "Kapoof!"
   
4. Commissions, fees & taxes cut directly into returns, so they are important, no matter what anybody says about the benefits of active management.
   
5. Speculate (gamble) in the market with no more than 5%-10% of your net worth.

If anything, there's probably a psychological benefit to something like MPT. Say, I follow the rules above, and set up a portfolio of something like SPY & BND, and just keep the variance to a minimum (with the #3 age constraint applied). It should minimize the probability of doing something imprudent, like completely pulling out of the market, piling everything into the market, etc. How to manage clients' psychology? Smooth out the rough edges, and give them a measure of confidence that there is a historical theoretical basis for allocations.

According to Markowitz, Using MPT, 'Systematic risk' due to beta, does not diversify away; unsystematic risk does.
In times of financial crisis all assets go down; all correlations go up. Diversification via MPT doesn't help or protect in a crisis.

The opportunity for the quant is to choose when to be in a concentrated portfolio and when to be in a diversified portfolio. It is not either/or but when/if.....
http://www.ifaarchive.com/pdf/does%20portfolio%20theory%20work%20hmm%20mbedits%205-19-09.pdf

@ Vladimir Yevtushenko,

Your results outperform SPY. The point I'm trying to make is - you can't outperform SPY in the passive buy and hold model. As an asset class, SPY beats every other asset class in the passive buy and hold model. However, you can if you are rebalancing your portfolio based on some signals (fundamental, quantitative.....), and if your signals work, you might be able to. That's again an attempt at some sort of timing,

@ mike timmons,

I completely agree with you. Nothing can save a long only, buy and hold investor from a crisis. Being long-short can. But getting it right takes an awful lot of time, expertise and stress. May not be worth it for a retail investor. Being market neutral can. That again takes serious theoretical and practical understanding of stuff like stochastic calculus.

@Bharath Rao

I think my backtest prove that:
MPT is not passive buy and hold model.
MPT is not a long only model.
MPT, if properly used, may outperform SPY, may if not fully protect but substantially help in a crisis.
That is "Why MPT is still in use".

@ Vladimir Yevtushenko,

I'm trying to understand your algorithm by reading the code. I definitely like 'your' version of MPT. My comments were about passive buy and hold (with a bit of rebalancing driven by changes in the risk profile of the investor and redistribution of gains and losses over time.

Let me ask you this. You say that it's not long only. But it is highly correlated with SPY. From the outside, it looks like it is long biased with a high weight given to on SPY (or its constituents). Correct me if I'm wrong.

I am not so sure the Portfolio Selection theory is really used that much in practice, I have been reading some papers and it seems they found-out right after that using and following the efficient portfolio will generate very unstable portfolios.

I am a software engineer and I have doing some post graduation courses in finance, and the feeling I got is that most of their basic assumptions don't hold. Such as the return distribution, or i.i.d but without them I don't think there are many models left. Any ideas and models are built on top of these basic assumptions.

So I guess it is kinda the best we have.

@Bharath Rao
As I mention above it is not my algo.
I just search Quantopian for Modern Portfolio Theory, got algo by Wayne Nilsen, cloned it, made some small parameters changes (replaced one symbol and change re-balancing to 22 days) .
If you want to see what positions are , just clone my algo ,run full backtest, click on daily positions & gains and you will see that mostly it has long and short positions.
Equity has very low correlation to SPY beta 0.2.
It is not the best model I have ever seen but it is not as bad.

@ Vladimir Yevtushenko,
I went through the code. What is your trigger for rebalancing, apart from time?

@Vladimir Yevtushenko
After looking at your answer I will speculate a little. Keeping the facts that the moments of returns are not defined and the fact that your code show that MPT Is not as bad as theory would suggest, this make me think one thing. Seems that the sharpe ratio has some sort of regularization hidden mechanism. Does exist some research in this direction?

MPT on its own can be quite stupid, between allowing short sales, which can have the algorithms go nuts, and also using in very long lookback periods instead of shorter-term periods when momentum holds up.

@Vladimir Yevtushenko One doubt. How your code manage survivorship bias? I am thinking that the "not so bad behaviour" can depend from the survivorship bias.

@emanuele luzio

'xle', 'xlu', 'xlk', 'xlb', 'xlp', 'xly','xli', 'xlv','tlt' There were 9 SPDR Select Sectors ETF in original ago I just replace one of them by TLT.
No survivorship bias and probably will not be.

@Ilya Kipnis

Ilya, let's talk not about MPT as theory but about its implementation in algo which I adopted.
Clone my algo, run full backtest, analize every day position and let me know when it was quite stupid.

@Vladimir Yevtushenko Thanx for reply. Please I am not a professional, so can you explain me a little more why this 9 tickers show that there is no survivorship bias? thanx in advance

@emanuele luzio
Survivorship bias is the tendency for failed companies to be excluded from performance studies because they no longer exist.
All of them existed during backtest period and exist now.

@Vladimir Yevtushenko "All of them existed during backtest period and exist now." Yes, this is why i am wondering for a survivorship bias. I imagine If I select firms from survived ones, I will overestimate the returns. For a correct estimation i have to pick from survived firms and not survived ones random. Isn't it?

@Vladimir, is the algo being used with real money? Is the daily working for live trading?

@Jeff,
You may find the answer in my first post of this thread.