Also the performance is inversely proportional to number of securities in algorithm which makes me think that most of the time is lost in I/O.

Performance is key. We want you to be able to quickly test, iterate, and improve your strategies. You can track our performance improvements, and average times, here: https://www.quantopian.com/performance

One of the issue is that the simulation framework computes all the statistics each bar, say each day or each min. This slows down the simulation but gives us the nice live charts we can watch the progress.

Lucas: Yes, that is indeed a lot of unnecessary computation and we have chatted about refactoring this internally. That should give a good boost.

Maybe do those just-in-time.

The statistics is just a chunck of possible optimisation then other part is the history and TA
There is an approach that makes it really efficient it is used to break hashs rainball tables or something like that. Basically most used history / TA are cached in a huge table even pre-computed Terabyte DB
So after the user run a backtest the next runs will be much faster

Here are some relevant links:

See https://www.quantopian.com/posts/zipline-question & https://github.com/florentchandelier/zipline/tree/wip-ImproveBacktestSpeed (and perhaps others).

Seems like as an initial step, zipline could be tweaked, and made available in the research platform, so that backtests could be run using the Quantopian data sets. Or is this what's already done? Surely running a backtest on the research platform doesn't compute intermediate results, since they can't be displayed anyway, when the backtest is run in IPython. Or is it still old-school under the hood?

At a NYC meetup last fall, Quantopian revealed it uses MongoDB as the quotes database. I offered to show them how I would implement a quote server, Q feigned interest, but nothing beyond lip service happened.

Are there any plans to replace MongoDB with something more appropriate?

MongoDB is a fine DB don't think it is the main issue

Hello Thomas,

I just took a quick glance at example backtest code in the research platform. It indeed uses zipline:

import zipline  
from zipline import TradingAlgorithm  
from zipline.api import order_target, record, symbol, history, add_history  

Wouldn't it be feasible to allow the intermediate computations to be toggled off, but leave them on for when zipline is used as the backend for the browser-based backtester? If I or someone else proposed changes to zipline on github, would they propagate all the way to the research platform, if they got implemented?

Grant

André: We have since switched our storage of financial data as mongo was indeed prohibitively slow for this. We are now using bcolz: https://github.com/Blosc/bcolz which is very fast

Grant: We certainly encourage all contributions and if such a change were to get merged it would propagate to research within a week or two. However, we'll probably also want these improvements for the quantopian backtester so we should try to solve this problem once the right way. I know that Eddie is starting to work on this so he might correct me on the plans. I'll ask him to chime in here on what the best way forward is and how we can best collaborate with contributors.

Thanks Thomas,

Understood re: an overall solution. Is it on the roadmap to be able to write an algo in the backtester IDE, pass parameters to it via the research platform, and then get the results back into the research platform (avoiding all of the standard GUI stuff)? If this could be set up, then one could treat the backtest as a function to be optimized, right? I realize that the same thing could be done in the research platform with zipline, but then the workflow is awkward, since the algo would have to be ported over to the IDE. It would be better to find an optimum, run a full backtest in the IDE/browser to verify, and then move on to greener pastures (paper/real-money trading or contest).

One problem is regulating the number of backtests running at any one time by a given user. If you open things up in this way and don't put in some limits, users could fire off a gazillion backtests in parallel and bankrupt y'all.

Grant

Of the 390 minute periods in a day how many are actually used by the majority of strategies? 1? 5? 10 maybe? The evidence that demonstrates that if one puts any code in the handle_data method the backtest performance degrades, should point out an obvious enhancement would be to honor only the schedule_function periods. If handle_data only contains "pass" then short circuit the loop that ships all bars to the strategy and send only those requested by schedule_function. If the history command is dependent upon having all data cached during a backtest this would negate this option. But if history is pulled from a central, continuous database, then the hande_data bypass might work.

Oh, and the pointless continuous shipping of json updates over the life of a test -- make that optional. Do the test, show me the results. It seems that the incremental updates are primarily there to placate the developer through the arduous process of waiting for the test to complete; "And now, for your mild entertainment, a slowly filling P&L chart..."

I think that in the research platform, the data is all there in memory, from start to finish, no? For example, see Research/Tutorials and Documentation/Tutorial (Advanced) - Backtesting with Zipline. I don't think we have schedule_function() there yet? I wonder if the plan is to eventually bring the research platform into full alignment with the backtester?

User choice to update IDE backtest every * minutes, +1

With schedule_function set to, say, 3 minutes of the day, open, midday and close, if a stock doesn't happen to trade in those particular minute bars, a problem, no? Won't scheduled minutes be limited to trading volume during those minutes? Also I'm wondering how daily volume is figured, average or total.

Market Tech, you hit the nail on the head, that's exactly what we are planning to do. Just requires time.

Grant, schedule_function() is in zipline and should hence be in research. Unless we are running 0.7 on research and not the current master.

There are so many software engineers here that if Quantopian made it easier to contribute I am sure many would help.
Maybe add a integration server and a review platform like gerrit or stash so people can propose and send patch to the main branch but only be able to push if a senior developer approves it.

Thanks Thomas,

Sounds like the plan, then, is to be able to do everything in the research platform that can be done in the backtester (although the backtester allows for multiple backtests running in parallel).

I gather that history() would work differently (or not be available at all), since all of the data gets loaded into RAM before the backtest start, right?

Grant

@Grant, history() still works the same in zipline (Research). However you need to add some extra code because the algo only has price information from the date range you specify in get_pricing():
Suppose you call somewhere history(100, '1d', 'price') Then you need:
1. inside initialize() add the following line: add_history(100, '1d', 'price') 2. inside handle_data or other function using data:

def handle_data(context, data):  
     context.tick += 1  
     if context.tick < 100:  
         return  

@Karol,

Thanks.

algo only has price information from the date range you specify in get_pricing()

I gather that this means that the entire data set is in RAM? Is this the same as the custom IDE/GUI backtester, or does it read data from a database on disk on an as-needed basis? Getting everything into RAM sounds like the way to go, if it'll fit. But then RAM costs money, so I'm figuring there are some practical limitations for Quantopian. Looking over http://aws.amazon.com/ec2/instance-types/ (which I think Quantopian uses...can anyone confirm?), 244 GB is the upper limit.

I've noted the potential limitation for the research platform on https://www.quantopian.com/posts/quantopian-research-platform-comments-and-questions. If the entire data set needs to be loaded into RAM, but RAM is limited, then at some point there will be a problem. And since users can't see available RAM and usage, it could be a very unpleasant user experience when scaling up. Has anyone tried loading minute bar data for 5,000 securities over 10 years on the research platform, for example?

@Lucas,

There are so many software engineers here that if Quantopian made it easier to contribute I am sure many would help.

Quantopian tends to be cagey about matters beyond what has been open-sourced on zipline. This is understandable, since if they open up everything, their IP would just get copied. It would be truly revolutionary if they opened up their IP and day-to-day development process to the crowd. My hunch is that they'll continue with their current model of open-sourced zipline and closed-source everything else.

There's also the business side of things, which is a consideration here. They would have to open up about the constraints on various approaches due to cost, which vendors are being considered, etc. And they probably have NDA's, terms, etc. with vendors, and the lawyers wouldn't know what to do with it all ("Huh? You want to 'open-source' the business? No more lawyers? Request denied!"). Again, it would be truly revolutionary to open it all up to the crowd, but my read is that there is a part of the Quantopian operation that won't be open-sourced.

The idea of an integration server / test instance is interesting. I suppose it would require one for the research platform and one for the backtester. Just a matter of dollars and resources. They'll eventually get there, I figure. At the scale of $10B capital, with institutional investors, they'll need a whole QC apparatus. No more testing changes on production systems. It would be nice, though, to be able to pull custom code into the research platform. Then zipline variants could be tried there on actual Quantopian data sets (presumably, Quantopian engineers can do something along these lines already).

@ Market Tech and Thomas,

I don't see how bypassing handle_data() would work with slippage? If all orders are filled within the bar, then perhaps isolated calls to schedule_function() would work. However, what if there is slippage (either due to the slippage model or because of zero volume bars)? Then, you need to have the data streaming in from the data base for all securities, getting updated every call to handle_data(), right? Or maybe under the hood, you add some smarts so that data only get pulled if they are needed (e.g. if open order to be filled, then get required data only).

@GK Number of symbols increased from 200 to 500 however I can unfortunately only run about 40 using symbols() without hitting "Execution timeout" on start after a handful of seconds, even though a 2-yr backtest would run in 20 minutes (limit 2 hrs). Maybe because my code is complex, donno, even though it is a mere 2140 lines w/out comments/blank (characters with spaces ~111,000 all total w/ the limit at 500,000).

@garyha,

Well, it all depends on what you are doing, right? If your code scales O(N) that's one thing, but if it is higher order, then you'll run into problems quickly. Maybe do a plot of run time versus N (the number of securities)?

Also, keep in mind that just like MATLAB, low-level operations will be blindingly fast if you can vectorize them. So, it might be worth reviewing your code in this regard. As a first pass, anywhere you have a loop or nested loop, do a head-scratch.

Are you using any kind of canned iterative optimizer? If so, this might be another place to look. It might not scale well with the number of securities.

Hi,
I can confirm that the Q team is very reactive on this front. Ed did implement quickly something, motivated by the branch I've proposed.
I can also confirm that the simple modification I proposed increased backtest speed by 10 to 20 folds. I simply log the information that matter to me (balance), and compute all information after the fact (DD, peak to peak, CAGR, pain-to-gain ...). I did not yet validated the speed improvements provided by Ed's modification (I would expect it to be even faster).

From my PoV, any stats should be discretionary to the user, but anyways only computed after the fact (after the backtest is completed, making he most of numpy array calculations).

Yet, Zipline being OSS, why aren't any "coders suggesting solutions X, Y, Z" actually contributing it !

Yet, Zipline being OSS, why aren't any "coders suggesting solutions X, Y, Z" actually contributing it !

As a starting point, is there an end-to-end analysis available of where the problems might be, including hardware, I/O, and software? Also, as I mentioned above, economics comes into play here. If Quantopian is paying per backtest and not per unit time of backtest, then what's the motivation to speed things up? Does the overall load on the system matter (e.g. number of users running backtests or code on the research platform)?

any stats should be discretionary to the user

This could be a problem if Quantopian is mining backtest and trading data (the "exhaust" of algorithms). It also presents a problem for shared algos. Only allow sharing if the full complement of stats is included?

@Grant I have the feeling that the current Zipline used by Quantopian has deviated from the open-source quite a bit. The process to apply patches is not that simple. That is why I suggested a simpler process so people can just submit patches and get seniors to review and accept.
https://code.google.com/p/gerrit/

Long time ago I wrote a backtesting framework for a company trading forex, it was TICK based that is 4/6gig of data for 4/5 years and the backtest of 4 years would take 21 seconds. But here we have almost 30min to run a long min backtest. People that used other backtesting system will noticed the problem immediately. With the current framework strategy optimization is just not possible. It will take a day to test 50 parameters.

I tried to use Daily bars, but then with the current slippage model it means even if you try to run your algorithm only once a day a daily bar strategy will not be the same as min.

Lucas: The zipline that's on github is the same we use on Quantopian (with a couple of days delay). There is some code that wraps and extends zipline (things like hooking it up to our internal data sources, backfilling of history, etc). I'm not sure where you see the differences to other projects like gerrit or what is supposedly not simple. You can submit a pull request on github (and many people do), we will review it and merge it if it meets certain requirements. Then once we update Quantopian, those changes will make it to the platform. Here is a good example of an external contribution that made Zipline (and quantopian) much faster: https://github.com/quantopian/zipline/pull/458 If you then check here: https://www.quantopian.com/performance the decrease mid-February was largely caused by this user contribution. Having said that, we do need to be faster in responding to user pull requests.

It's not hard to write a specialized backtester that's faster than zipline. For example, if it only does backtesting over a fixed period you can vectorize most operations. However, zipline has many requirements in how general it needs to be (e.g. it supports paper and live trading). All these make it harder to speed it up.

I appreciate that it might look like we don't care too much about performance from the outside but it's not true. The things that were discussed in this thread are things that we will implement. But as with most things in life, it's harder and takes more time than you thought once you actually get down to it.

Please get in touch with us either on the mailing list, [email protected], or github if you want to help improve the backtester. There's many things to work on where you could have an even bigger impact.

I thought it had deviated because the amount of commits a week was like 2 or 3 so I thought there was a more active branch somewhere. I will indeed make a pull request. I agree if we use zipline for both backtesting and live trading it will impose severe restrictions on optimization, maybe it is worth to split the responsibility.

It's not hard to write a specialized backtester that's faster than zipline. For example, if it only does backtesting over a fixed period you can vectorize most operations. However, zipline has many requirements in how general it needs to be (e.g. it supports paper and live trading). All these make it harder to speed it up.

Wouldn't it make sense to relax the requirements in the research platform. I agree about the final steps of paper & live trading, where one needs all of the bells & whistles. But shouldn't there be a path to much better performance for research & optimization. Sounds like zipline is overburdened, no?

another possible improvement, I think, is pre-calculating benchmark returns. I believe in most cases people only use a small set of benchmarks.

Not only benchmarks but it would probably be helpful to have huge cache on the history and all ta method calls. I would use an object DB something like ZODB since there is a big loss in making the objects/parsing so each request to history or TA would return an object from DB. Of course one cannot pre-compute all since one can call the TA's with different parameters. But I am sure the parameters used are clustered together. The small change the user submitted above took like 2 months to be integrated a huge change like this would probably take ages. It might be worth to fork zipline,