First!

All in favor, say aye.

This is fantastic, particularly the "specific returns" and rolling sharpe charts.

Love it! Thank you for your continued efforts to improve the platform!

Would be great to have a few more metrics on the All Backtests page too if possible, ideally all of them, but at least the last 6mo rolling sharpe and beta in my opinion.

The new feature has a good solid feel to it. :)
I can't resist a request: I think it would be so good if the new backtest analysis could provide a concise summary we could copy and store with code like below. Example Notebook.

2017-01-03 to 2017-12-27  $10,000,000  2018-04-16 14:42 US/Eastern  
Returns 25.2 Alpha .16 Beta 0.08 Sharpe 1.67 Drawdown 4.6  
7/9 tests passed.     Score  -0.0069  
Positions concentration    5.04|5.77 FAIL: 98th percentile 5.04 > 5.0  
Leverage   89.47|92.01|102.62|105.62 PASS: 0.92x-1.03x is between 0.8x-1.1x  
Turnover                 30|31|38|39 PASS: 31%-38% is between 5.0%-65.0%  
Net exposure                 5.5|7.8 PASS: 5.5% <= 10.0%  
Beta-to-SPY                0.32|0.33 FAIL: 98th percentile absolute beta of 0.32 > 0.3  
Sector exposure              0.2|0.2 PASS: All between +/-0.20  
Style exposure             0.31|0.31 PASS: All between +/-0.40  
Tradable universe              96|97 PASS: QTradableStocksUS >= 95.0%  
Positive algorithm_returns    25.21% PASS: Positive algorithm_returns  

Pretty sweet!

Hi Blue Seahawk,

I could definitely make use of that. Would you be willing to share the code for it?

Hi Abhijeet -

I gave it a quick run. Some initial comments:

1. The Performance tab provides redundant returns information; the Overview tab already shows the total, specific, and common returns overlaid on a single plot, which would seem to be the best presentation.
   
2. Is the algo code associated with the backtest available on the new page? I can't seem to find it. If not, it should be made available.
   
3. "6-month rolling Sharpe ratio. A measure of risk-adjusted performance, which divides the portfolio's excess return over the risk-free rate by the portfolio's standard deviation." Are you now subtracting the risk-free rate? Previously, this was not the case.
   

As a result of your comment @Joakim I modified a Notebook checker to have some output like that here.

Thanks everyone for the positive feedback! @Grant, algo code is on the new page, under 'Activity > Code'. Performance contains individual graphs for total, specific and common returns largely to handle the case where you might want to drill down into one particular type of return, since we can scale the y-axis accordingly to focus on your return of choice. (There are alternative workflows that we could look at too, like toggling lines on or off). You're right about the Sharpe ratio - the risk-free rate continues to be zero. For those interested, this is a good thread on the rationale behind that choice. We'll update the text to clarify that the RFR used today is zero.

Hi Abhijeet,

Any reason why rolling volatility was not included in the new backtest screen? I think it would be quite useful to have, as well as being able to show multiple metrics in the same graph. For example, how Returns and Volatility (and Sharpe) relate to and affect each other during different market periods.

We've just shipped a couple of important new updates to the backtest screen:

• You can now view your positions and transactions on the new page, via the activity tab.

Positions load automatically, but transactions will load only on-demand, which should be helpful if you're on a metered internet connection and don't want to use more data than you need. To load transactions, just hit the 'Load Transactions' button that you'll see on the transactions tab.

• We've added a metric for 6-month rolling volatility (thanks @Joakim for the suggestion!). You'll find it in the Performance tab, next to Sharpe and max drawdown.

Thanks to the positive feedback, we're going to be taking this page out of beta and making it the default full backtest screen later this week. We'll still provide an option to get to the old page in the short term, to make it a smoother transition for folks who haven't had a chance to use the new interface.

I do not want to offend anyone. But, I do not see the improvement. So, will keep using the old format for a while.

You have rendered the Position and Transaction data completely useless. I admit it is there as it is also in the old format. But now, there is absolutely nothing I can do with it. You can not filter the data on anything. You can not isolate the trades you want to see, or isolate a date you want to expand, or copy any of that data for analysis.

I can understand if it is just an intermediary thing, and at some point, we will regain access to some data analysis, even if it is limited. But, at the moment, I do not see what is provided as some kind of improvement. I view it more as a step back. I will stick to the pre-beta thing, the plain one.

Sorry if this sounds harsh.

Some might view this as not constructive when in fact it could help focusing on what is important to see in those backtests. Just ran a simulation using the “improved” version. Even if it was just for one year, it generated, on average, 5,000 transactions per week. That is a quarter of a million transactions for the year. Should I scroll every line by hand to find something?

Hi @Guy,

Thanks for the feedback - I do agree that the Transactions and Positions tabs need work for cases when the backtest has a high number of transactions. Like you mentioned, it is an intermediate state and the Transactions/Positions tabs are by no means 'done'. We're going to continue to add updates to make them easier to work with, especially for backtests with a high number of transactions. For more advanced analysis, there's also the option of programmatically accessing the positions and transactions data in research via get_backtest('my_backtest_id').positions or get_backtest('my_backtest_id').transactions.

Quick update: after lots of good feedback, the new page is now the default screen when you hit ‘Run Full Backtest’. There are two important things to note:

• You can still access the old page by appending /old to the end of the backtest page URL.
  
• We’re working on improvements to positions/transactions on the new page. Specifically, we’re working on adding in filtering so that it’s easier to manage backtests with a large number of positions/transactions. The old page will continue to be available at /old until the updates to positions/transactions for the new page are in. (Positions and transactions are also programmatically accessible via research using the positions and transactions attributes on the backtest object, for more flexible and powerful analysis).

Thank you to everyone who’s taken the time to provide feedback on the new page - we appreciate it, and it helps us make the platform better.

Happy backtesting,
Abhi

What happend to the performance chart on the first page??

We no longer see the benchmark performance against the total return. I have to now run a backtest notebook to see this !!! can we get this info back?

Thanks

Also what happened to the table performance data? monthly returns and beta etc. Now I only see the chart. If I want to get some return information for April I have to look at the chart and do the math ??!! This is all pretty but less info? Again I want to avoid launching the notebook every time I need to look at the backtest.

Can we get the table data back as well? Thanks

I would also like the Returns vs benchmark performance brought back in some tab even if it is not displayed in the Overview. Others have also mentioned that Relativism vs benchmark is important information and would prefer not to have to run notebook tearsheets for every backtest just to get to that chart.

I would appreciate a lot of the information from the old backtest interface also being displayed on the new interface, especially a benchmark.

I couldn't find the saved algorithm code for the particular backtest in the new version

Oh Yes.. I just noticed that. Where did that feature go? Access to the code for the backtest is super important

@kamran, @Vitaly, algo code is on the new page, under 'Activity > Code'

This is all good feedback and we'll continue to listen and take it into account as we improve the screen.

Thanks
Josh

How do we see the graphs of the custom data in the new tool?

I get the sense that this isn't architecturally general enough. You are talking about adding various other types of trading, universes, etc. as I understand, but this is focused on the present contest rules and need for the 1337 Street Fund. Also, presumably, folks use the site for stuff other than trying to make money via the contest or to get an allocation. So the Quantopian-specific stuff is irrelevant to them. Additionally, it is all kinda unapproachable for a newbie. "Structural constraints? Common returns? Risk constraints? I'm so confused!" Gotta keep it easy-peasy. It should flow from super-simple and conventional, to the more sophisticated, as one delves deeper. I would recommend reverting to a more general interface for the backtester (a generic trading slice and dice, without the Quantopian overlay of requirements). Then, specific analyses can be done in a notebook, to suit the needs of the user. This may sound counter-intuitive, but then you'll have a very generic, approachable trading analysis interface, and can do the specific stuff in notebooks, where it can really be fleshed out to suit the specific case.

By the way, where are the recorded variables? Are they no longer shown? And I note that posted algos aren't really consistent with this new paradigm yet. What is the plan?

+1 for custom data / recorded variables. I need it to display custom measurements that are important to me.

Will this be the backtesting interface also used by Quantopian Enterprise? If so, along the lines of my comments above, it is probably not the kind of thing institutional investors would use, given its primary focus on the contest and Q hedge fund. You might consider how to modularize the interfaces. Presumably, whatever interfaces you will offer to Quantopian Enterprise will also be offered to Quantopian Community for free (on https://www.quantopian.com/posts/important-news-for-our-community, Fawce states "We are committed to maintaining parity between the Quantopian Community and Quantopian Enterprise platform capabilities."). So, one way to set things up would be:

1. Dirt-simple, general interface, for both Community and Enterprise newbies. If someone just wants to get up and going and kick the tires, without a lot of work and cognitive load, this would be the place to start. No Q contest/fund jazz.
   
2. Sophisticated, general professional interface, for both Community and Enterprise users (again, no Q contest/fund stuff). This would be a general backtesting environment, suitable for serious quants, from both the Community and professional Enterprise. This is where I would put your focus. Basically, I think the bulk of your engineering dollars should be focused on developing a professional interface, versus an interface that is narrowly targeted for the Community, for the near-term need to getting a specific type of algo for the 1337 Street Fund. To get started, I'd make a matrix of the functionality of the off-the-shelf and custom tools out there, and compare their features. You could also interview professionals, to understand their various use cases. If you have any Quantopian Enterprise clients lined up, you could invite them to comment here; it would be interesting to get the professional perspective.
   
3. Potentially, develop a set of browser-based modules geared toward specific Quantopian contest and fund objectives; there will be a variety as time progresses. Presently, you would have only one of these: the US equity, long-short, market neutral, etc. (see the contest rules). Eventually, you will have others, and I would suggest separate modules for each. Alternatively, you could not have browser-based modules for specific Quantopian contest and fund objectives, and simply rely on a notebook (or set of notebooks). From a flexibility and maintenance standpoint, I would think that the notebook would be the way to go. And rather than putting effort into devising fancy GUIs that are not general interfaces, I would stick with the kind of concise output that Blue Seahawk shows above and here.

So, in summary, from an architectural standpoint, things would be organized from simple, and general, to the specific (e.g. the Q fund stuff). Also, you would be taking both Community and Enterprise use cases into account.

Some specifics:

1. It would be really nice if any screen/graphic could be shared in a post on the Community forum (e.g. with a simple button or right mouse click). Ideally, something like the Windows Snipping Tool could be used.
   
2. There should be an option to select from a list of notebooks, including user-defined ones. Basically, one should be able to change the default notebook to a different one.
   
3. I note that you added "At present, Quantopian uses a risk-free rate of 0 when calculating Sharpe." It would be nice to be able to select a benchmark for the risk-free rate, over-riding the fixed risk-free rate of 0. I would think that for some Community users and (most?) Enterprise users, this would be very desirable. This option should also extend to analysis in notebooks, as a standard offering.

As a project within Quantopian, how much gas is left? Would you be up to a major re-vamping? Or is the idea to push this out, and then iterate incrementally?

Also, I would recommend, if you want to capture and track changes over the next N months, putting something up in Github or some other public-facing development tool (not the code, but the framework, plans, and proposals, etc.). This way, users (both Community and Enterprise) can see what is proposed, and where you are headed. The Quantopian forum isn't a very good collaborative development tool.

The ability to launch and to then analyze backtests, all within a single notebook would be really nice. Additionally, the ability to programmatically launch a series of backtests in this fashion, run them in parallel, and then post-process the results would be super nice (e.g. to be able to optimize algo hyper-parameters). It would be interesting to hear feedback from both Community and Enterprise users on the potential for this functionality (by the way, will there be a separate forum for Enterprise, or will it be integrated here?).

Hi everyone,

Anyone know how to get back to the old version of backtest analysis from the Beta one? Thanks.

I agree with the above input. The overall issue I see is that the new backtest analysis screens are made purely to show info regarding what Quantopian wants to see in an algo (i.e. specific return data etc.) vs. all the actual details you need to analyze a strategy while building it. Its an overshoot on Qs requirements and ignore the features and stats one actually looks at.

Its almost an academic approach and exercise !

Can we still see the old version of the backtest results/analysis? Also, the get_backtest('algo_id') has not been working for me for a few days now.

Could you please remove the "New Backtest Page Available!" popup dialog from the /old version. Now that the default is the new version and typing /old manually at the end of the url is an explicit request to see the old backtest page there should be no need for a popup reminder of the new backtest page.

Is it possible to put the metrics that is there at the top of the old backtest page "Total Returns, Alpha,Beta ,Sharpe, Sortino,Volatility,Max Drawdown" into the new version or is the rationale for removing them the same as that of removing the benchmark in the chart? I hadn't realized how much I had gotten used to them updating live as the backtest was running.

Taking away information that users have relied in the past for strategy development leads to lesser productivity in my opinion.

Agreed with many here that the default overview page is much too specific and provides no information to actually help develop. The point of these algorithms are A) to make money, and then B) to make money given Quantopian's constraints (only if you care about the contests). The default overview tabs are much too focused on point B, when point A is the main focus and the more difficult part. Anyone can write a random algorithm that satisfies the contest rules that loses money - so we don't need the focus of our backtest results to be these risk constraints.

Variables such as beta, alpha, returns vs benchmark, vol are fundamental characteristics of algos still, and they help you evaluate and create profitable systems. I think they need to be returned to the default page. In addition, custom variables is of great use to me,and I would like that returned as well.

I do not see any positive feedback for the latest improvement (05/25/2018).
But all of them are right.
As many others, I do not want to lose in the full backtester what has crystallized as "good" for many years.

So please let everybody to choose which one will be the default (old or new)until new full backtester screen will be ready
to replace the old one.

Please remove the "New Backtest Page Available!" popup dialog.

P.S. Has anyone tried copy the full code from 'Activity > Code' into the IDE and run it successfully?

I don't think old and new view need to be mutually exclusive and it would be less disruptive to most active users of the platform if a tab (say vs Benchmark) can be provided at the top that provided the charts and metrics of the old backtester.

As has been said many times over in this thread metrics that are important in strategy development phase are vastly different than checking risk constraint before submitting an algo to the contest.

It will be difficult to iteratively develop a viable strategy if the only metrics updating live at the top are "- of 2" and "- of 7" structural/risk constraints met.

Can we for now bring back the old interface and put this back into Beta?! I keep typing /old to do my backtest runs. The new interface is just not very useful for running backtests and developing a strategy. Its more of a final evaluation interface for the contest and Qs requirement check list.

I recommend a back to the drawing board trip ... and this time with some real world traders and strategy developers input.

Thanks

@Leo, @Vladimir: Now that the new screen is the default, we've removed the 'New Backtest Page Available' dialog on the old page.

We're continuing to listen to this feedback and make improvements to the new screen, while the old screen remains accessible by appending /old to the URL.

One thing to note is that you still have access to Sharpe, volatility, max drawdown etc under the performance tab of the new page. The overview tab is focused on contest eligibility, but those metrics themselves are still there under Performance. Currently, you can access recorded variables via the recorded_vars attribute on the notebook. We're looking into adding more support for recorded variables and benchmarks into the new screen, as well as summary point statistics to go along with the rolling performance metrics.

@Abhi, thanks for incorporating user feedback and a creating a roadmap to get back the features that we users relied heavily on (in the old backtest page). For very active users like me this will be of immense help. Having all information (that updated live) in one screen like they do in the old backtest page helps a lot in quickly analyzing effects of algorithm changes.

Abhijeet,

The new backtest screen right now not ready to replace the old one.
Please make on default screen at least " View Old Backtest Page" button.
I support @kamran sokhanvari back to the drawing board trip request.

Hi Abhi -

Not sure if you saw the advice above, so I'll repeat it here:

I would recommend, if you want to capture and track changes over the next N months, putting something up in Github or some other public-facing development tool (not the code, but the framework, plans, and proposals, etc.). This way, users (both Community and Enterprise) can see what is proposed, and where you are headed. The Quantopian forum isn't a very good collaborative development tool.

In this community forum format, it becomes quite impossible to read through user feedback and then sort out your response, and what action might be taken.

Above, you say:

Most importantly, we want your feedback! This page is still evolving, and we want to know what you think so we can make it even better.

I encourage you to sort out how to be more systematic in capturing and responding to user feedback. As a start, perhaps you could edit your original post and list what changes will be made, and what changes are still under consideration. We could then assume that everything else was rejected (which would be o.k.--in the end, it is your engineering investment and your product). Alternatively, you could capture all feedback and your responses in a spreadsheet and provide a link to it.

@Abhi, I'd also recommend a " View Old Backtest Page" button until the changes are incorporated. I run huge number of backtests in a week/month. Not having to type /old in the url everytime is a time saver and convenience, along the same lines as not having to switch multiple tabs in the new backtester for each metric.

Hi Abhi,

One more request: Possible to add that "Bayesian Cone" to display OOS performance from a user-specify cutoff date?

Quite a disconnect to access that useful feature at present - we have to manually do that in the backtest tear sheet Notebook, that is:

bt.create_returns_tear_sheet(live_start_date='2018-01-01')  

ps: Reference to post by Gene Frenkle on the "Bayesian Cone" feature.

+1 for an overhaul of the "Overview" page. Other than the Total Returns, there is nothing. Let's start from scratch. What benchmark(s), rolling stats, custom recorded variables, etc. could be displayed, that would be useful in stimulating thought as the backtest runs, providing feedback if the code should be tweaked and a new backtest run, etc.

One way to think about it is that backtesting is simulated trading. You wouldn't provide a trading interface that just displays the total returns, right? What would a professional trader want to see on a daily basis? I recommend talking to some, and incorporating their feedback into the design (as mentioned above, this should play well with your Quantopian Enterprise clients, too, and as I understand, you actually have a design constraint to maintain parity between Community and Enterprise interfaces, tools, data, etc.).

Providing a top-notch Overview page would not preclude all of the contest/fund post-processing that apparently was a big driver, but it is clear that the paradigm of a generic backtesting interface got lost in the design process. Fortunately, it is software which is relatively easy to fix (compared to physical products, which are a whole different ball game...the design better not be too far off, or you may not have a job).

Cannot copy code from the new backtest interface. Try Ctrl-a in the old and new interface and only the old copies the algorithm code. At another level it will be nice to fork from a backtest and create a new algorithm or just be able to reset the algorithm's current code to a particular backtest. Conveniences..

I want to thank everyone for the feedback and suggestions that have been made regarding the new backtest page. Something that I wanted to revisit was the motivation for the overhaul in the first place: having a better feedback system for our contest and by extension our allocation process. There were too many people who were either unaware about our allocation process or frustrated by not understanding the criteria. Since the change in backtest analysis interface, we have seen many more algorithms passing the contest criteria.

That said, while the new page has helped people learn about our allocation process, it's also a major overhaul to a big part of our user interface and significantly impacted many community members' workflows. Some of the features left out of the new page are important to many in the community. Over the next weeks we're going to make updates to the page that respond to some of the concerns raised in this thread, including:
- Easier access to recorded variables
- Better position/transaction filtering
- Making it easier to view summary stats of a backtest, in addition to the individual charts
- The option to view performance against a benchmark, if needed.
- Miscellaneous interface bugs, like being unable to select code via Ctrl-A, or issues copy-pasting code into the IDE in Firefox.

Thanks again for your feedback,

Abhi

@Ahbijeet, that's great news. Getting tired of always typing /old. But, it is still only a slight inconvenience. Waiting patiently for all the improvements you can bring.

While you are at it with the changes, could you revert back to a white background so that whenever I take a screen snapshot I can print something else than a black sheet of paper?

Thanks @Abhi,

Being able to view live updates of all the key metrics (returns, alpha, beta, sharpe, sortino, max_drawdown, vola, etc.) on the Overview tab, as was the case on the old page, is high on my wish list. Ideally also all the risk metrics like leverage, sector exposure, style exposure, turnover, etc. and perhaps relative (in %) specific and common returns? Maybe even a rolling 63-day Contest Score (somewhat of a Sharpe derivative)?

While I'm wishing, I also think it would be useful to be able to 'superimpose' multiple graphs onto each other. For example, the strategy's volatility and beta to market (left Y-axis) on to Returns (right Y-axis) and see how they relate to each other during different market periods (X-axis).

I have been very curious about how this new improved backtest model calculates the specific (rolling) turnover rate, as well as how a strategy's overall turnover rate (a specific number) on the old version 'Analyze Backtest' is calculated.

In my understanding, there are two ways to calculate daily turnover rate:

One is ticker-based, which means comparing the difference between today and yesterday's portfolio positions based on change of tickers (which can be tricky sometimes especially when only position's direction is changed, let's say, yesterday we have AAPL's long position and today we have AAPL's short position, i.e. ticker doesn't change but extra turnover definitely increased.

And another way to calculate turnover rate is value-based, meaning based on today's value bought/sold divided by previous day's portfolio value. Going back to the example above, if we have $1 mil as initial portfolio capital to trade, yesterday we started by entering $1 mil of AAPL's long positions, today we flipped to $ 1mil of AAPL's short positions. In this case, the turnover rate of today/yesterday I assume is supposed to be 2 mil / 1 mil = 200% (correct me if I am wrong).

So which method (or is it another method I haven't thought of) is what Quantopian has been using here as well as for the old version of backtest results? Appreciate for the answers in advance!

Hey Abhi,

Sorry, maybe I'm nitpicking here, but shouldn't the 'Turnover' graph be on the 'Performance' tab, and the 'Volatility' graph be on the 'Risk' tab? :)

Hi Abhi -

Thanks for the update. Not sure if it was mentioned above, but better version control of algos would be nice, along with a the ability to view a change log. Presently, we have the current version available in the IDE, and a set of versions captured with each full backtest (which are not necessarily new versions, since the backtest parameters could have been changed, but not the code).

@Brad: We use the second model to calculate turnover. Turnover is calculated as the absolute value of the day's transactions, divided by the EOD portfolio value. Assuming no changes in the price of AAPL, your second example of 200% turnover is correct. One thing to note is that the graph you see on the backtest page is actually the trailing 63-day mean turnover. If you'd like to see the daily turnover (not the 63 day mean), it's one of the graphs available when you call create_full_tear_sheet, or directly use pyfolio's get_turnover function.

@Grant: Version control has definitely been on our list, although it's not a near-term priority. While it's not a substitute for full version control, we recently shipped the ability to name your backtest, which can help keep backtests more organized and also serve as an ad hoc changelog. You can also view the diffs between backtests (although you're probably aware of this already!):

@Abhi, a few more issues.

Can't compare across backtests in different pages of the pagination in the all backtests page.

Logs are loading slower in the new backtest than the old backtest interface (compared side by side by switching to logs tab at same time)

Could you add a link to open a backtest in a new tab. Clicking into backtest and then hitting all backtests button and then paginating back to the relevant listing page of backtests seems a very inefficient way of getting back to where one started from.

While editing rename backtest dialog in the new interface, clicking somewhere in the middle of the text to correct something has many times led to cancel button getting clicked automatically. I have had this happen like 20+ times already and is unfavorable user experience when you have to retype again. I am on google chrome windows 10 if that helps.

It used to be I could copy the code from a backtest into the IDE. Now when I try this, I get extra spaces on the left-hand side, and the code won't run. It would be very nice if, in addition to storing the code with a backtest, one could do a "Save As" of the code associated with the backtest, so that it could be edited as a new algo. Needing to copy and paste code is pretty awkward (not to mention trying to sort out how to get rid of the leading spaces!).

I also strongly need the 'copy code' button on the backtest's results page. The old version of interface allows us to do that with the standard copy paste procedure but the new one doesn't.

We've released a fix for the copy bug, and also added the ability to view the benchmark and recorded variables. Take a look at this post for more information.

hi, i am new in this so quick question:
when i click "run full back test" i get the new screen, i would like to see the old one, is it still possible? (a lot of you tube training movies refers to the old screen)

Hi Abhijeet,

On my wish list this holiday season:

To be able to see (and sort by) average Sharpe ratio on the list of backtests page. :)

Simply AMAZING!
Is there a way to plot/access separately the cumulative equity line for long and short positions?
Thanks
JC