Replying here to comments in the other thread linked above:

@Grant: If interest rates increase we might re-evaluate our decision to set it to 0 (although Dan raises some great points for long/short, but let's ignore that for now). For now, it just doesn't make the cut in terms of prioritization against other things like the risk model, portfolio optimization, the contest, and a million other things we'd like to do. As I said, you can use whatever risk-free rate you want with empyrical, or even better: submit a PR to add it yourself.

Separately, your point on having low correlations against other algorithms already in the portfolio is well taken. In the past we have discussed displaying a correlation score, so that might happen. I also like your idea of including a Quantopian-fund risk factor into the risk model.

@James: These types of fees are very real. Specifically borrow costs and short availability are not handled well in the backtester which can skew results. So at some point we will likely add those. But those are not related.

Moreover, I think we're talking about separate things: I'm not talking about the perspective of an investor into the Q fund who might legitimately compare our expected performance vs fees to alternative investments, including T-Bills. That's a fine discussion but ultimately not related to the contest or to getting an allocation. I'm talking about the quant perspective and how we can create a contest that rewards algorithms that perform well OOS. Subtracting constants from all algorithms when we really care about the rank just doesn't seem like something that is worth focusing on. Does the SR definition include a risk-free rate? Yes, you are 100% correct (although Dan's points highlight it's not so straight-forward in long/short strategies). Does it make a difference for this contest? Provably not.

The algo you keep highlighting is actually an interesting case: It's 99% cash, so let's assume you earned interest on that. As such, whatever you are subtracting as risk-free in the SR, you would need to add back in if you accurately modeled interest in the backtester.

@ Leo - The risk-free rate discussion seems germane, since as I understand, Q is trying to provide the framework for better guidance to users with the new contest on what it will take to get an allocation (if this is not a goal, then I'm really confused). Without subtracting a relevant baseline rate (either the risk-free rate or better yet, higher), users may get the impression their returns are positive, when in fact they are negative relative to an appropriate baseline. A negative excess return result would send the message that more work is required (whereas a positive excess return would mean there is hope, everything else being o.k.). Of course, if they've made up their minds on this point, then any discussion is a waste of time here, or elsewhere.

Regarding publishing a target returns number for authors, I'm figuring Q wouldn't want to do this for a variety of reasons; using 0% is safe. First off, from a regulatory standpoint, I'm guessing it could be problematic. Imposing say, a 5% return to allocate capital to individual algos would imply that the fund would return at least 5%; any sort of implied promise of future returns could be a problem. Also, with prospective customers, say the number is 5%. The number could become a point in negotiations. There are probably other reasons, as well, just to stick with a safe 0% as a baseline return. Of course, the actual number is not 0%, but something higher. But I can see why a specific number would not be published.

The investment industry calculates the Sharpe ratio as: SR = (E[R(m)] - r_f) / σ(R(p)). See Wikipedia if need be: https://en.wikipedia.org/wiki/Sharpe_ratio

By removing the risk-free rate, the ratio is skewed upwards. Meaning that for the very same trading strategy, Q will always present a better Sharpe than anyone else, when in fact, their distorted Sharpe should be reported exactly the same as everyone else, otherwise, it should not be considered as a Sharpe ratio by definition.

They should simply abandon talking about Sharpe, and express their own views as a Quantopian ratio: QR = (E[R(m)] ) / σ(R(p)) which would still be a distorted and scaled SR.

The Sharpe ratio has been a measure for comparing portfolios, comparing funds over the investment industry for ages. And the higher the SR the better. Saying that what you have outperforms the other guy. Not so, in Q's case, they always get: QR > SR, giving a better light to their stuff when in fact, their measure is de facto not comparable.

For those that have not compared QR to SR using historical numbers, QR is overstated by about 50%. Not negligible.

Note that Q is not the only one to distort the Sharpe ratio this way. It gives them the illusion that they are performing better.

@Thomas:

I'm talking about the quant perspective and how we can create a contest that rewards algorithms that perform well OOS. Subtracting constants from all algorithms when we really care about the rank just doesn't seem like something that is worth focusing on.

But what is Q's definition of "algorithms that perform well OOS"? Benchmarked against what? What are the minimum returns you are looking for? If you are scoring algos based on volatility adjusted returns for pure ranking purposes then you are opening yourselves up to the possibility of seeing returns that may be inferior to the risk free rate, most specially when this goes up, considering all the tight risk constraints that the algo is subjected to.

What is the basis of the 2% (a constant) floor on volatilty in the new score function? To me it implies that Q has set the minimum benchmark of returns at 2%. If benchmark is set to 1 in the new score function, by way of mathematical extrapolation, you get 2%. But as per Jamie's answer to me, the 2% floor on volatility is to guard against gaming algos and there is no benchmark so there is no minimum threshold for returns. This just doesn't make sense to me!

The algo you keep highlighting is actually an interesting case: It's 99% cash, so let's assume you earned interest on that. As such, whatever you are subtracting as risk-free in the SR, you would need to add back in if you accurately modeled interest in the backtester.

I don't get you on this. If the Sharpe ratio included the risk free rate adjustment, this algo would not make the cut but instead it remained ranked no.1 for several weeks till I exposed it. This is why I say...risk free rate matters. It exposes the flaw of the current scoring system. Look at the current top 50, they are still several entries with these kind of returns, whether intentional gaming algos or not. If Sharpe and Sortino ratios where calculated as intended by their original authors with the risk free rate adjustment, several of these entries will fall by the wayside. But if you insist on calculating these metrics as you are now, please call them by another name, say, Quantopian's Modified Sharpe and Sortino Ratios. I hope you catch my drift:)

@Karl, I read Jamie's post and I perfectly understand the new scoring system as I expressed here:

you've designed a scoring mechanism that doesn't have a benchmark which includes a constant factor that guards against gaming, purely for ranking purposes, that passes your risk model constraints but without regard to relative returns of the market or other alternative asset class including (apologies for using the bad word) risk free rate.

And concurred by Jamie here:
@James: That sounds like the correct interpretation.

I understand that this is a pure ranking mechanism based on two factors: returns and volatility which is floored at 2%. Basically by way of analogy, it's a fishing expedition, they give the same set of reels and baits (constraints) to all contestants and say anything under 2 inches (volatility floor) you have to throw back but it doesn't say anything about the minimum weight (minimum returns) of the fish. This is what seems lacking to me, the minimum threshold of returns which is by industry standards, the risk free rate. That is why my proposed scoring function is:

score = (algo returns - maximum(risk free rate, market returns)) / maximum(algo volatility, market volatility)

Qualitatively called the Villa Ratio = excess returns / excess risks premium, LOL!

On a serious note, this is a very stringent performance metric. First it captures the true alpha and secondly, it tempers it with a strong risk measurement that relates to the market as opposed to pegging it to a constant number with questionable basis. But having said that, I don't think this is what Q is looking for but something I will use personally..

@ Thomas -

I also like your idea of including a Quantopian-fund risk factor into the risk model.

You might do a head-scratch on that one. My read is that you'd end up publishing the simulated (and real?) returns of the fund under your current framework (via get_factor_returns - see https://www.quantopian.com/posts/introduction-to-the-quantopian-risk-model-in-research). I suggest talking with Fawce and your legal folks about it; the decision may be above your pay grade.

@ Thomas -

An additional perspective is that backtesting should be a pessimistic estimate of potential reality. If it is more of an optimistic back-of-the-envelope model, then it layers on an additional risk. So, by adding the risk-free rate (and other stuff, probably), there will be less risk vis-a-vis trading real money out-of-sample.

@ Thomas -

Another piece of input is that the idea that y'all have no idea what minimum return is required for a $10M long-short equity algo to fly is kinda silly. Just like there's some rough range of baseball batting average it takes to make it in the big league, there must be some sense from your own analysis and the industry as a whole. When Jonathan Larkin was still with you, maybe he provided some guidance? Or is this totally the blind leading the blind? Sorry for the sarcasm, but I'm just finding it hard to believe that the number is zero (in the end, I think your new contest rules will contain the number implicitly, so why not just publish it?). Now, if you have a number, but just don't want to or can't publish it, that's a different story.

Dr. Thomas Wiecki:

The algo you keep highlighting is actually an interesting case: It's 99% cash, so let's assume you earned interest on that. As such, whatever you are subtracting as risk-free in the SR, you would need to add back in if you accurately modeled interest in the backtester.

Your analysis above is absolutely wrong. If interest was accurately modeled in the backtester, this algo which is invested 99% in cash would have earned interest very close to the risk free rate. If Sharpe Ratio was accurately calculated in the backtester as intended by Dr. William F. Sharpe to include the adjustment of risk free rate to algo returns, to reflect the minimum threshold that all other types of asset returns should beat, this said algo will have an SR of something very close to zero. Your "add back" rationale just doesn't make any sense.

That is why when you make statements like..."Trust me, in the industry no one is concerned about the risk-free rate.", guys like me who have been in the industry for 30 years just shake our heads in disgust. The industry actually uses risk free rate as the gold standard of minimum returns to compare to all other forms of assets and we in the industry call these as spreads or premiums. Ask the people from Point72, your investors, if this is the case.

You also keep referring to risk free interest rate as a constant which it is not. It is derived from a financial instrument that is subject to market forces and does changes over time.

The other thing that doesn't seem to sink in to you as this relates to the current contest design and scoring is how it's flawed and could have been easily corrected by including risk free rates in the calculation of Sharpe and Sortino ratios. Look at the current Top 50 entries, there are still some churning out very low returns, low volatilities, low drawdowns that score high because of the miscalculation of Sharpe and Sortino ratios.

Lastly, please don't take my comments as adversarial or personal, I am sincerely trying to help Q improve and achieve it's goals by engaging in healthy debates such as this and pointing out weaknesses as I see them.

@Grant: I thought I gave you a rough range of a SR of around 0.5?

@James: If that algo earned interest the SR formula would look like: (algo_returns + risk_free - risk_free) / algo_vol, where the first risk-free term comes from the interest earned by the strategy, the second one is from Sharpe calculation. Not sure where you see the error.

As I said before, you're completely correct. But I'm not convinced it makes a material difference for the algorithms on the platform. If you have other examples than the contest ones with extremely low leverage I'm happy to hear them. The other low-leverage algorithms would be dequalified if they ranked higher in the contest (or maybe we should do that already now). Moreover, as Dan pointed out, you would earn interest on your shorts. You seem to dismiss that argument by pointing to other fees, but I don't think that's valid as the two are not directly related.

I suspect the risk-free rate is viewed as an academic correction, since to cover hedge fund direct (operational) and indirect (overhead) costs, and to support the 2/20 structure, the returns need to be well in excess of the risk-free rate. In building a solid model, however, it would seem that it should be included, point-in-time. Presumably, Q is in the investment business (and not speculation), and so the thinking should be on investment time horizons (e.g. 10-40 years, perhaps), over which the risk-free rate could rise to a level that it is no longer in the noise. The mechanics of incorporating the rate into calculations would seem to be trivial. The fact that Q is not using the risk-free rate as a baseline might suggest to investors that Q is not thinking long-term, and is not in the investment business. I guess my advice, from this perspective alone, would be for Q to do a head-scratch on the perception of potential investors, since the implication is that Q will be blind to rising interest rates.

An alternative model is that hedge fund profits from trading would scale in proportion to the risk-free rate, but this would say that the market would become less efficient as rates increase. Is there any evidence that this would happen?

@Grant: Thanks for the advice, it's been heard.

@Thomas, with respect to the particular algo we're both referring to and assuming that the backtester accounted for interest income from investment in cash and Sharpe properly accounting for risk free rate, then, your formula is correct. But the fact is Q is not accounting for both right now. Under the current low interest environment, this may seem trivial or immaterial. However, in a high interest scenario such as 1981 when Tbills were at 14%, the banks weren't paying that much interest for savings or CD's and the brokers were charging much higher interest on borrowings for shorts, then these rate disparities becomes material. And this will happen again in the future. The short sightedness of Q's current design is what worries me.

I am not dismissing the argument by pointing to other fees, just pointing out the realities of the hedge fund as a business, as Grant corrrectly points out above. So by design, these costs should be incorporated into the mix because at the end of the day, you're not only trying to satisfy yourselves but also your clients and be competitive in the industry. Even as we strictly talk about just the current algo model, I still believe you need to set a minimum threshold to weed out algos that slip through the cracks.

Thanks, James, I get where you're coming from. We'll keep an eye on it and will adjust accordingly.

Another forward-looking tip would be to understand how you will treat the risk-free rate, SR calculations, etc. in the 1337 Street Fund (a.k.a. Q fund) prospectus and reporting. I would think that you would be in-line with the whatever is used in the industry and by your competition. I'm guessing that institutional investors will see it as relevant, but maybe I'm wrong. In any case, if you report SR values and don't use a conventional approach, they may think that you are trying to pull a fast one. And then you'll need to do a little dance to try to explain why you've left the risk-free rate out. Seems like a safe bet would be to include it across the board.

By the way, I asked for a copy of the prospectus (https://www.quantopian.com/posts/1337-street-fund-aka-q-fund-prospectus) and haven't gotten one yet. Maybe I just need to call or e-mail (see https://www.quantopian.com/investor_relations)? Or maybe I would first have to demonstrate that I'm a qualified (i.e. filthy rich) investor (which I'm not)?

I'm an old storyteller, I know a lot of fairy tales:

https://www.quantopian.com/posts/feedback-requested-improvements-to-quantopians-risk-and-performance-calculations#57b2dae0913f63b329000253

Hi Thomas,
Another disappointment.
Is not that your words?
Thomas Wiecki
Nov 17, 2015
Thanks everyone.
Here's what we'll do:
* Fix the zipline implementation to use risk-adjusted returns in the denominator
* Default zipline to use 1M T-Bills instead of 10Y.
* Have pyfolio use risk-adjust (using 1M T-Bills) when used in research.
Why you changed your mind?
Taking into consideration that risk free rate is not constant, simplification of calculation
does not respect the spirit of what the Sharpe ratio is trying to calculate - namely
the leverage independent quality of a strategy.
The fact that backtester currently do not charge interest for leverage and borrowing cost may not be the reason
for changing formula of Sharpe ratio.

https://www.quantopian.com/posts/feedback-requested-improvements-to-quantopians-risk-and-performance-calculations#57b2e54d4ad7792ff100023a

Tomas,
If I will use my own risk-free rate it will be Vladimir Ratio.
If you will use your own risk-free rate it will be Tomas Ratio and they may differ.
What we need -the industry standard - Sharpe Ratio ( reward-to-variability ratio) named after the winner of the 1990 Nobel Memorial Prize William Forsyth Sharpe.

https://www.quantopian.com/posts/a-new-contest-is-coming-more-winners-and-a-new-scoring-system#5a289e509699ff7e4552ddc9

Jamie,

Me and, I think, James and Tony will be happy if you change:
your algo needs to have positive returns since the start of the 2 year backtest.
to
your algo needs to have positive excess over risk free returns since the start of the 2 year backtest.

https://www.quantopian.com/posts/feedback-requested-improvements-to-quantopians-risk-and-performance-calculations#5a33381e3ca51306de93f5ae

@Tomas,
Don't you think that top ranking of unproductive algo in the contests 22, 24, 34, 35 are results of simplified (without risk free return) calculation of Sharpe and Sortino Ratios?
Quantopian open contest 22 results
The first on finish was Wenkai Zhang with amazing 0.86% return.(Disqualified Quietly)
Wenkai Zhang 1
score_contest_rank 1 score_contest 86.30
Paper Trading
score_pt_rank 2 score_pt 86.30
rankAnnRet_pt 267 annRet_pt 0.86%
rankAnnVol_pt 10 annVol_pt 0.32%
rankSharpeRatio_pt 33 sharpe_pt 2.67
rankMaxDD_pt 7 maxDD_pt -0.07%
rankStability_pt 84 stability_pt 0.79
rankSortino_pt 14 sortino_pt 6.28
rankBetaSPY_pt 15 beta_spy_pt 0.30%

@Vladimir, having been relatively new to the forum, I was not aware that you have brought up these key points a while back. Kudos to you. At least now I know that I'm not the only "crazy" one, hehehe... cheers!

The Sharpe ratio has a strict definition: SR = (E[R(p)] - r_f) / σ(R(p)). Again, see Wikipedia if need be: https://en.wikipedia.org/wiki/Sharpe_ratio

If Q insist on defining a return to risk ratio using: E[R(p)] / σ(R(p)), they can, but then, this is not a Sharpe ratio. And therefore, Q should use another name for it, whatever they want. They could revert back to what it was originally called in the 60's: a "reward-to-variability" ratio as @Vladimir also raised.

But, Q should understand that it is not comparable to what has been defined as a Sharpe ratio for the past 50 years, a very common term used by the whole portfolio management industry.

The historical long-term Sharpe is about 0.40. It goes like this: SR = (0.10 – 0.035)/0.16 = 0.40625. If we eliminate the risk-free rate from the equation we get: 0.625. That is a 50.8% overstatement. And therefore, becomes not that closely comparable to what is considered the industry standard: the Sharpe ratio.

The Sharpe ratio is designed to compare portfolio returns across portfolios. And if we distort its value, we can still compare things, but we would not be comparing portfolios in general on some Sharpe, but some subset following some other definition like: E[R(p)] / σ(R(p)).

For the case where a strategy is 99% in cash, its calculation should go as follows:
E[R(p)] = 0.99∙r_f + 0.01∙β∙(E[R(m)] - r_f). If beta tends to zero, we get: E[R(p)] → 0.99∙r_f

If beta tends to 1.00, we get: E[R(p)] = 0.99∙r_f + 0.01∙(E[R(m)] - r_f) which also tends to E[R(p)] = 0.99∙r_f + 0.01∙(E[R(m)] - r_f) → 0.99∙r_f + 0.000625 → 0.0353

If beta tends to 2.00, we get: E[R(p)] = 0.99∙r_f + 0.01∙2∙(E[R(m)] - r_f) which tends to
E[R(p)] = 0.99∙r_f + 0.01∙2∙(E[R(m)] - r_f) → 0.99∙r_f + 0.0013 → 0.03595

If totally hedged, we get: E[R(p)] → 0.99∙r_f + 0.01∙β∙(E[R(m)] - r_f) – 0.01∙β∙E[R(m)] = 0.99∙r_f +0.01∙β∙r_f. Again, that is not an interesting scenario either.

Am I trying to say that being 99% in cash is not that great a reward for a portfolio?

If you add an execution premium to the equation, meaning that your trading strategy can generate some real alpha, you would get: E[R(p)] = 0.99∙r_f + 0.01∙β∙(E[R(m) + α] - r_f), in fact saying that the alpha would not matter much if 99% in cash. However, just playing the game, you would get a small part of the historical long-term drift.

For a fully invested scenario with market neutral hedging as per the contest you would get:
E[R(p)] = r_f + β∙(E[R(m)] - r_f) – β∙E[R(m)] = r_f – β∙r_f. Again, not a great scenario. Total hedging would give you a beta of zero which would translate to: E[R(p)] = r_f. So, no wonder we see low CAGRs in the contest ranking.

There is only if you succeed in adding some real alpha to the mix that you might outperform, such as in: E[R(p)] = r_f + β∙(E[R(m) + α] - r_f) – β∙E[R(m)] = r_f + β∙ α – β∙r_f. However, as always, real alpha is hard to get. The CAPM sets the alpha at zero. So, you need a lot of skills to put some in.

Quantopian's alpha might be partly an illusion, since a part of it might reside between (E[R(p)] - r_f) / σ(R(p)) and E[R(p)] / σ(R(p)). Note again that E[R(p)] / σ(R(p)) could be overstated by as much as 50% compared to its nemesis.

@ Thomas -

I've been confused for awhile (years, probably) about how Quantopian wants to interact with the crowd (y'all seem a bit confused, as well). This is yet another example. It seems that for this, and numerous other issues, they could be captured on your public Github (https://github.com/quantopian) as bona fide potential problems, and then managed by your engineering staff and management in a prioritized way. Following good engineering practice and traceability, then any accept/reject (along with the rationale) can be captured there.

Or maybe it is not the right process (since there perhaps is no way to put in change requests for the Quantopian API in general on https://github.com/quantopian)? Alternatively, you could open up an internal change request, provide the salient details, and you could report any movement here periodically (simply put a note in the change request, with the link https://www.quantopian.com/posts/risk-free-rate-on-quantopian).

Would this makes sense? Or would it be a waste of time (since the answer has been pre-ordained as 'rejected')?

By the way, you might suggest a focus session for QuantCon on the topic of the relevance of the risk-free rate for Quantopian calculations and the fund industry, as a whole. I'm sure there will be some people there who are experienced industry professionals who could provide some feedback, if you presently don't have the expertise to sort it out. I gather that with the loss of Jonathan Larkin, you are back to zero folks with deep finance experience (although this would seem not to be financial rocket science...subtracting a slowly varying constant point-in-time).

https://personal.vanguard.com/pdf/flgerm.pdf

Vanguard definition of SR:

Sharpe ratio
The Sharpe ratio is a representation of the risk-
adjusted return of a portfolio or security. The
Sharpe ratio measures how much return is being
obtained for each theoretical unit of risk. To calculate
a Sharpe ratio, an asset’s excess return versus a
risk-free asset such as Treasury bills is divided by
the standard deviation of the asset’s returns.

I think Quantopian would be in good company by simply mirroring Vanguard.

Another point of reference:

https://blog.quantopian.com/accurate-slippage-model-comparing-real-simulated-transaction-costs/

At Quantopian, our backtester's primary purpose is to accurately simulate trading strategies. If an algorithm would have performed well historically with real money, we also want our simulation to show positive performance — ideally matching almost exactly with what would have happened in the real world.

In post-processing backtest results, it is very misleading not to include (at a minimum) the risk-free rate as a baseline (benchmark). The simple concept that if the return does not exceed the risk-free rate, the effective return is zero (or negative) should be included. A giant red flag should be displayed.

The other issue, obviously, is that you are assuming your returns will well exceed the risk-free rate (even at its present level). This is not a given. And given that you are basically creating a passive investing vehicle (a bunch of algos running, versus teams of MBAs with green eye shades), in the end, it might be o.k. to just eke out a bit more than the risk-free rate (assuming you get enough capital).

Hi Thomas -

I continue to be mystified by this issue. It would seem that the mechanics of implementing a change to the conventional Sharpe ratio would be trivial, compared to other things your software engineers accomplish. So why is Quantopian using "Sharpe ratio" but not computing it per convention?

@Grant, another way to highlight this shortcoming of not incorporating risk free rate in the conventional computation of Sharpe and Sortino ratios is by way of relating them with regards to current contest metrics/scoring system. Without the adjustment of risk free rates to Sharpe and Sortino ratios, they are not just overstated, but are also being exploited by gamers who churn out returns of below 1% with below 1% volatilities and extremely low drawdowns thus giving them high scores. You see a lot of them in the contest leaderboard, slipping through the cracks which is really frustrating for contestants with real good algo returns. Had the Sharpe and Sortino ratios been strictly calculated and implemented by convention, all these "gaming algos" will fall by the way side. But Q refuses or are still indifferent to my findings. Very frustating for somebody who trying to help them improve their framework.

A little dead horse beating...

Wondering if the launch of Quantopian Enterprise and the SaaS business model, along with the Factset partnership has any bearing on the Quantopian Sharpe Ratio definition? See https://www.quantopian.com/posts/important-news-for-our-community.

I'm thinking that Enterprise users would prefer to use the standard definition, including incorporating the risk-free rate. Also, perhaps Factset would be able to provide the required data (assuming one of the impediments is that Q does not have the data presently, and would have to pay too much for it).

One nitty-gritty point is that on https://www.quantopian.com/allocation, we have:

Consistent Profitability
We are looking for uncorrelated algorithms that show stable profits. If your algorithm makes money while managing risk exposures and avoiding long drawdown periods, it might be a great addition to our portfolio. We are looking for algorithms that consistently have a Sharpe ratio over 1.0.

However, is this the Q SR, or the conventional one, incorporating the risk-free rate?