strong text
With our recent release enabling cloning of notebooks from the Quantopian Research environment, we can now more easily share some of our in-house analytics tools with our community of users. What I’m sharing today is a notebook that you can clone into your own Research environment to analyze and visualize your backtest’s performance across a variety of dimensions.
The notebook contains all of the functions to compute all performance statistics, so it is very self contained, and thus also means a tearsheet can be computed for any timeseries you pass to it. So, you can upload a CSV of the daily returns of your favorite mutual fund and see how it looks, or simply pass in the timeseries of a few stocks. An example of how to accomplish this using a single stock is also included at the end of this notebook.
Performance Statistics
- Similar to what is reported in the Quantopian Backtester but computes a few others.
- Ability to specify an “out-of-sample” or “live trading” date. So, say you’ve started to live papertrade an algo, or started trading it with real money and you’d like to easily perform a side-by-side comparison of the algo’s performance -- this is now as simple as specifying the date which will slice your backtest into 2 pieces, generating performance stats for each.
- If you don’t specify an out-of-sample date, just by default the function will split your backtest into 2 equally sized time periods allowing you to see how consistently the 2nd timeframe matched the 1st timeframe
- Similarity between the distributions of your backtest vs. live trading daily returns. E.g. How closely does the out-of-sample trading of your algo match the expectations set from the backtest.
- If you don’t specify an out-of-sample date, just by default the function will split your backtest into 2 equally sized time periods allowing you to see how consistently the 2nd timeframe matched the 1st timeframe
VaR (Value-At-Risk) Table
- Simple VaR calculation of 1 day and 1 week 2-stdev VaR
- We are working on more extensive building out of VaR metrics
Stress Events Table
- How your algo performed during specific macroeconomic periods of “stress”
- The columns mean/min/max refer to the average daily return of the algo (min), the worst daily return of your algo (min), and best daily return of your algo (max)
Cumulative Returns Plot
- Your algo
- Green: Backtest performance
- Red: Live, out-of-sample, performance
- Green: Backtest performance
- A couple of broad indexes for comparison purposes.
- S&P500 (does not include dividends paid)
- Intermediate Term Bonds ETF (does not include dividends paid)
- These should not necessarily be used as benchmarks for performance so much as they should be used to see when large market/macro shock events occurred and how your algo fared.
- These should not necessarily be used as benchmarks for performance so much as they should be used to see when large market/macro shock events occurred and how your algo fared.
- S&P500 (does not include dividends paid)
- “Cone Charts” (within the Cumulative Returns Plot)
- The cone is meant to serve as a guide post for how we might expect an algorithm to perform in the future based solely upon its backtest.
- RED cone: The live papertrading period (e.g. out-of-sample period). The cone drawn around this live period was based solely on the daily returns data from the backtest period.
- BLUE cone : A blue cone is drawn going out one-year into the future showing the 1.0 stdev expected range of the algo's performance if we were to start trading it today. As well, this blue cone is computed solely on the backtest (GREEN) daily returns/volatility data.
- Math Behind the Cone
- Compute Algo Volatility Expectation from the Backtest.
- Uses only the backtest data to compute the daily volatility of the algo.
- Uses only the backtest data to compute the daily volatility of the algo.
- Compute Algo Profit Expectation from the Backtest
- From the backtest data only, calculate the linear trend of the algo's profit (e.g. the slope of how it goes up-and-to-the-right as it generates profits)
- Use this linear trend established from the backtest daily returns and if the algo performs similarly in live trading, we should expect the live performance to fall within the cone drawn around this linear trend +/- the volatility computed in #1. (The linear trends are shown as dashed lines in the above plot)
- From the backtest data only, calculate the linear trend of the algo's profit (e.g. the slope of how it goes up-and-to-the-right as it generates profits)
- Compute the Cone based on Backtest Returns & Volatility
- The midpoint line of the cone (e.g. average profit expectation) is the linear trend established in #2 above.
- Then draw the cone using the daily volatility computed in #1 above by scaling the daily value based on the number of days that the algo has been live, using the conventional standard rule of volatility scaling. E.g. If I was to scale a daily volatility to an annual volatility I take the daily value and multiply it by the sqrt(252) to annualize it.
- So, for example, an algo with a 1% daily volatility, means it has a ~16% annual volatility: 1% * sqrt(252). This will result in a cone that has a width of +/-16% at a point in time 1-year from the start of the cone. Similarly, the cone width at month 6 would be +/-11 %, since 1% * sqrt(126)=11%. This calculation is simply done every N days from the start of the cone to create the width of the cone at each N-day period from the start of the cone and shapes the cone as time progresses forward.
- The midpoint line of the cone (e.g. average profit expectation) is the linear trend established in #2 above.
- Compute Algo Volatility Expectation from the Backtest.
- Math Behind the Cone
- The cone is meant to serve as a guide post for how we might expect an algorithm to perform in the future based solely upon its backtest.
- This is another area where we are spending more time researching advanced methods for modeling the future returns and variance expectations of an algo based solely on its backtest data. The more accurately we model this will allow us to have greater confidence understanding when an algorithm has stopped working after investing in it, and trigger a warning that the algorithm may have to be turned off.
Rolling Beta Plot
- Your algo’s beta computed over a rolling 6-month window of time
- See how your algo’s exposure to the SP500 market fluctuates over time or if it is steadily market-neutral.
Rolling Sharpe Ratio
- Allows you to see how your algo’s Sharpe Ratio fluctuates over time or if it is very steady and earns profit consistently over time.
Heatmap of Monthly Returns
- Each month’s returns of your algo, heatmapped so you can see the months you made the most or lost the most money.
Annual Returns Bar Chart
- Allows easy visualization of how steadily your algo performs on a yearly basis
Distribution of Monthly Returns
- Easy inspection of whether your algo has a consistent mean profit each month and whether the algo experienced any large “fat-tailed” events
Daily Returns Similarity
- The distribution of your live, out-of-sample, daily returns overlaid upon the distribution of daily returns from your backtest. If they perfectly alighn the result would be 100% similarity.
- This allows you to quickly see if your live out-of-sample performance is aligned with the expectations of your backtest.
- These distributions are determined after scaling the observations underlying each distribution and using a Gaussian kernel density estimator to estimate the probability density function.
- The reason behind scaling the returns observations is to account for the fact that there are most likely many more observations in your backtest than in your live out-of-sample period.
- To see how the scaling was done check out the function in the notebook.
- The reason behind scaling the returns observations is to account for the fact that there are most likely many more observations in your backtest than in your live out-of-sample period.
- We are doing much more work in this area since it is a very important aspect of choosing algorithms for the hedge fund.
- We would love to hear others’ thoughts about ways to improve or revise this current methodology – if you have ideas leave them in the comments!
Box and Whisker Plots
- Show the average, 25th, 75th percentile returns + outliers across three different timeframes: daily, weekly, monthly
“Stress Event” (Grid of Small Plots)
- How well did your algo perform during large stress events.
- Green: Your algo
- Gray: SP500
- Green: Your algo
