Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
Sharpe Ratio in Backtest Result differs from than one in Pyfolio
Costantino
posted

The Sharpe Ratio in the Backtest Result is different from that in Pyfolio.
What is even worse: the difference is not consistent and if you compare the same algorithm by Shape ratio, it leads to contradicting results!

For example:
1) Backtest ID 5645e601e7ed821102440b1f, Sharpe Ratio Backtest 2.08, Sharpe Ratio Pyfolio 0.81
2) Backtest ID 5645d32a71785b11076434ac, Sharpe Ratio Backtest 1.89, Sharpe Ratio Pyfolio 0.87
that means, for Pyfolio the second algorithm ist the best, while for the backtest is the opposite!
According to my computation, Pyfolio is right and the Backtest is wrong.

Is it a Bug?

Thanks for the help!

9 responses
Deleted User

Hi Constantino,

Yes, the Sharpe Calculation between pyfolio and the Q Backtester will differ.
The Q Backtester incorporates a risk-free rate, while pyfolio does not by default. (Though you can pass a risk-free rate timeseries to use to the pyfolio Sharpe Ratio calculation as an optional parameter, for those that wish to use one.)

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Vladimir

Justin ,

In definition of Sharpe Ratio ( reward-to-variability ratio) named after the winner of the 1990 Nobel Memorial Prize William Forsyth Sharpe there is no such options:
for those that wish to use risk-free rate...(as in pyfolio) or use yield on 10 year US Treasury bond (as in Zipline used in backtester).
look here

and here

and here

Dont You think that both Quantopian calculations are not The Sharpe Ratio at all?
May be it is the time to bring calculations to industry standard...

Thomas Wiecki

Vladimir,

I think the denominator isn't that clear. I have found both versions and if the risk-free is constant over that time period the std is equivalent.

In pyfolio the risk-free is optional (I think that's OK) and in zipline it's set to use 10Y (maybe should change to 1M or 3M). We could change pyfolio on research to use the same risk-free returns so that the two match up. It's confusing indeed.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Vladimir

Thomas,

I agree that subtracting constant number from Daily Returns do not change standard deviation, but the same time we should take into consideration that risk free rate is not constant and simplification of calculation do not work.
Is that difficult to change dozen lines of code in Zipline Risk Module and pyfolio to make calculations as it should be and identical?
I think Quantopian must do that changes to remove confusion or change the name of the ratio.

Costantino

Yes, I agree with Vladimir, the simplification of the calculation is wrong and lead to contradictory results (two different algorithms, for the backtest the first one has a better Sharpe ratio, while for Pyfolio the opposite is true). Even if the calculation is not identical, it should lead to consistent results.

Simon Thornington

The sharpe ratios in the backtester seem consistently too high. I wonder if there's a bug in the annualization of it.

Thomas Wiecki

Thanks everyone.

Here's what we'll do:
* Fix the zipline implementation to use risk-adjusted returns in the denominator
* Default zipline to use 1M T-Bills instead of 10Y.
* Have pyfolio use risk-adjust (using 1M T-Bills) when used in research.

I think that should address all the issues discussed here-within. Let me know if you disagree. Thanks everyone for helping and sorry if it took a while to get around to this, it's absolutely critical that these values are correct.

You can track progress here: You can track progress here: https://github.com/quantopian/zipline/issues/853

Vladimir

Thanks Thomas,

This is exactly what I recommended a while ago.

Costantino

Thank you, Thomas!
I'm looking forward to testing the new calculation!

About Quantopian
Careers
Community
Events
Documentation
Academia
Lectures
Status
Facebook
Twitter
LinkedIn
YouTube
Terms of Use
Privacy Policy

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian.

In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.