pyfolio
¶There are many ways to evaluate and analyze an algorithm. While we already provide you with some of these measures like a cumulative returns plot in the Quantopian backtester, you may want to dive deeper into what your algorithm is doing. For example, you might want to look at how your portfolio allocation changes over time, or what your exposure to certain risk-factors is.
At Quantopian, we built and open-sourced pyfolio
for exactly that purpose. In this notebook you will learn how you can use this library from within the Quantopian research environment (you can also use this library independently, see the pyfolio website for more information on that).
At the core of pyfolio, we have tear sheets that summarize information about a backtest. Each tear sheet returns a number of plots, as well as other information, about a given topic. There are five main ones:
We have added an interface to the object returned by get_backtest()
to create these various tear sheets. To generate all tear sheets at once, it's as simple as generating a backtest object and calling create_full_tear_sheet
on it:
# Get backtest object
bt = get_backtest('56bb3f8d3ce1db11952648c0')
# Create all tear sheets
bt.create_full_tear_sheet()
There are many metrics being reported in all the tear sheets above. At the top, there are tables that tell you about summary performance statistics like the Sharpe ratio, Sortino ratio, and worst drawdown periods. The following plots are hopefully pretty self-explanatory, but more information can be found on the pyfolio website.
As the name suggests, create_full_tear_sheet()
creates all tear sheets available (except for the Bayesian one, see below). You can also create individual tear sheets. For example, lets create one that only uses the returns of your strategy.
In addition, we will pass in a keyword argument called live_start_date
. The use-case for this feature is that you might have deployed this algorithm and want to see how the out-of-sample period measures up to your backtest. Although it currently is not possible to access returns from live-traded algorithms in research, you could still note the date when you deployed it and run a new backtest over the full time period. This date can be passed with live_start_date
. Lets pretend that we developed and deployed this algorithm on 2014-1-1
. As I had access to 10 years of historical data, I could have easily overfit my algorithm to only work well on that time period. In fact, it is very difficult not to overfit, so comparing in-sample and out-of-sample (OOS) data is a good way to look at that.
This time, we will create just the returns tear sheet on the same backtest object from above:
bt.create_returns_tear_sheet(live_start_date='2014-1-1')
There are a few differences in the returns tear sheet that was created. Note for example that the performance table at the top now has 3 columns: Backtest
, Out_of_Sample
, and All_History
.
The cumulative returns plot also differentiates between in-sample and OOS time periods. In addition, there is a cone that gives you an indiciation of how your algorithm is performing OOS compared to in it's backtest.
At the bottom we also see 3 distribution plots comparing the in-sample and OOS returns distributions. The first one standardizes both distributions to have the same mean and standard deviation of 1. The other two plots relax this standardization.
There are also a few more advanced (and still experimental) analysis methods in pyfolio based on Bayesian statistics.
The main benefit of these methods is uncertainty quantification. All the values you saw above, like the Sharpe ratio, are just single numbers. These estimates are noisy because they have been computed over a limited number of data points. So how much can you trust these numbers? You don't know because there is no sense of uncertainty. That is where Bayesian statistics helps as instead of single values, we are dealing with probability distributions that assign degrees of belief to all possible parameter values.
Lets create the Bayesian tear sheet. Under the hood this is running MCMC sampling in PyMC3
to estimate the posteriors which can take quite a while (that's the reason why we don't generate this by default in create_full_tear_sheet()
).
bt.create_bayesian_tear_sheet(live_start_date='2014-1-1')
Lets go through these row by row:
For more information on Bayesian statistics, check out these resources:
Above, we saw how we can easily create a variety of tear sheets. These are all created using a thin wrapper on top of the pyfolio OSS library. You might also want more fine-grained access over the functionality provided by this library. For this, you can import pyfolio
and use it directly.
import pyfolio as pf
returns = bt.daily_performance.returns
pf.timeseries.cum_returns(returns).plot();
For more information on the usage of the library, check out the pyfolio website or our GitHub repo.
pyfolio is still a very new project — there will be bugs and there are many rough edges. Your help is greatly appreciated.
If you find bugs or have other questions, please report them to our issue tracker. We also appreciate any contributions. For some ideas on where to start, see the 'help wanted' tag.