Feature request: Provide the ability to schedule multiple backtests with different parameters

Back to Community

posted

When optimizing a trading algorithm, I often run multiple backtests with the same code and only change one parameter at a time. I then copy the results of this backtest into a spreadsheet and analyze the results. I use the correlation function to determine how my parameters are affecting the results, with the goal of fine tuning the parameters with the best results.

After learning more about the Research functionality of Quantopian, I think it would be a great place to automate this process. Let's consider a very simple algorithm, that goes long when the price is above a moving average, and goes short when the price is below that moving average. In this example, the parameter that needs tested is "which moving average provides the best results"?

Algorithm code:

    context.Moving_Average_Days = 20  
    if stock.price > stock.mavg(Moving_Average_Days):  
        order_target_percent(security, 1)  
   else:  
        order_target_percent(security, -1)

Research code:

for Moving_Average_Days in range(2, 200):  
    #  
    # Run_Backtest(algorithm_ID, start_date, end_date, capital, user_variables)  
    #  
    Run_Backtest(142342, 2014-01-01, 2014-12-30, 100000, Moving_Average_Days)

The code above would run 199 backtests, each with a different moving average value. Once these backtests were complete, you could use the Research "get_backtest()" function to analyze the results and find the optimal moving average.

I realize that allowing people to run hundreds of backtests will add a lot of load to the Quantopian servers. I suggest that this can be mitigated by allowing users to submit "jobs" in Research, and then be notified by email when the jobs are finished (which could be days later). This will allow Quantopian to only run Research initiated backtests using excess server capacity, to minimize costs.

What do you guys think?

16 responses

Simon Thornington

You can run multiple zipline backtests within Research already...

Tristan Rhodes

@Simon - Thanks. I'll admit that I have only read the tutorials and docs for Research, but I didn't see how to accomplish that. I know that get_backtest() will display results of a backtest that you have manually ran. Can you share how you accomplished initiating new backtests?

Simon Thornington

Seong has an example of the method here:

https://www.quantopian.com/posts/research-do-you-want-parameter-optimization-click-here-to-get-started-heat-maps-included

Note that this doesn't kick off quantopian backtests, it kicks off zipline backtests, so the usual issues with code shared between zipline and quantopian remain.

Tristan Rhodes

@Simon - Thanks for that link, it looks exactly like what I want to accomplish! I'll have to do a lot more reading and learning so that I can take advantage of it.

BTW, when you say to use Zipline (the open-source software), does that mean (1) I need to run this code on my own server, or do you mean that (2) hosted Research can do all this, but since it uses Zipline I must follow Zipline docs. https://github.com/quantopian/zipline

Simon Thornington

Nope, you can run zipline backtests from within Research. It's just that zipline algos are ever so slightly different from quantopian algos.

André Pajak

Right. I second the motion. We often run the same code with different command-line arguments and options. There is no reason why these should be hardwired into the code.

Alisa Deychman

Andre - check out the link Simon shared above for parameter optimization. This can be done in research, give it a whirl

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Tristan Rhodes

Alisa and Simon - Thanks for your encouragement and direction. I was able to combine Seong's Research example with the Zipline.io basic example (http://www.zipline.io/) to accomplish what I was originally asking for. :)

https://www.quantopian.com/posts/finding-the-best-moving-averages-short-long-for-cross-over-trading-strategy

I was surprised at how easy it was to perform hundreds of backtests and quickly find the optimal values for the strategy. And I was satisfied with the performance: I did 324 backtests, using a single stock over 282 days of daily data, and it only took 14 minutes and 18 seconds. That is 2.6 seconds per backtest! (I expect that minute mode would be around 390 times slower, increasing my testing to 4 days!)

I am going to try to use the power of Research to optimize several of my algorithms, and I hope my example will help others get started with Research.

Grant Kiehne

Hello Tristan,

Note that backtests cannot be run in parallel on the research platform, while they can on the backtest/trading platform. One potential workflow is to randomly sample your parameter space, writing out the parameter values as record variables. So, every time you launch a backtest, a new set of parameter values is selected within the ranges of interest. Once the backtests are run, all of the backtest data can be pulled into the research environment, via get_backtest() (you'll need a list of the backtest ID's).

Regarding,

I realize that allowing people to run hundreds of backtests will add a lot of load to the Quantopian servers. I suggest that this can be mitigated by allowing users to submit "jobs" in Research, and then be notified by email when the jobs are finished (which could be days later). This will allow Quantopian to only run Research initiated backtests using excess server capacity, to minimize costs.

I'm not sure it works this way in the world of cloud computing. If Q owned the servers, and they were paying the electricity bill, capital depreciation, data center lease, etc., then perhaps this model would apply. But I'm not sure that they are sitting on a bunch of dormant servers, just waiting for their 35,000+ users to request backtest jobs.

Blue Seahawk

Someone at Q said we each have our own server. Edit: See below, actually each individual backtest, not each individual, I had read it wrong.

Tristan Rhodes

Grant, you are right about not paying for idle cycles when computing in the cloud, which is what Q runs on:

http://blog.quantopian.com/post-mortem-of-121212-attack-on-quantopian/
"..a front-end built with Ruby on Rails and hosted by Heroku, a pool of algorithm processing servers hosted by Amazon EC2, and several MongoDB databases hosted by MongoHQ."

Grant Kiehne

With the research platform, I think it is true that each user has his/her own server (virtual, no doubt). I've gotten the sense that there are limits, as such (e.g. memory is shared across notebooks running in parallel). This does not appear to be the case for the backtester. There, it seems that each backtest spins up a little stand-alone server/application, grabbing the resources it needs. I don't think there is any limit to the computing resources a given user can command with the backtester (so long as he/she isn't a pig or attempting to do a denial of service attack).

Blue Seahawk

Ok I looked it up and it's even beyond that, in the IDE, each backtest, not each individual:

for backtesting, pretty much every individual backtest gets its own server in the backtest array

Grant Kiehne

@garyha,

Yeah, that's what I gathered. A little virtual server. Just don't get carried away with running too many at once, without first contacting Quantopian. It costs them money and could cause some systemic technical difficulties, as I understand.

André Pajak

@Alisa - I did. Thank you.

Lucas Silva

The way TraderStation does it is super simple. You just define a set of parameter in your strategy then you have a macro view and it will provide you a set of optimization algorithms like GA. I kinda understand why people that have been trading and using existing tools get frustrated here. Existing backtesting tools have a lot of interesting feature still to come to Quantopian.

You've successfully submitted a support ticket.

Our support team will be in touch soon.