It's an interesting question and it's one that I track pretty closely ;) In practice, what your algorithm actually does matters as much as duration and SIDs.

In the last 24 hours, the average backtest duration was 294 seconds ;)

Hello Dan,

Your point is a good one. But your 294 seconds was daily, minutely or both? What was the average number of SIDs? What was the average backtest duration?

I'm curious about comparing like for like. If someone had a 100 SID 5-year minutely algo completing in 30 minutres maybe I could talk to them about optimising mine‽

P.

Hello Peter,

To profile sections of your code, you can use:

import time

t = time.time()  
# code goes here  
elapsed = (time.time() - t)/1e-3  # milliseconds  

Grant

Hi Dan,

I agree with Peter that minutely backtests run pretty slowly--that darn factor of 390! Seems that your typical daily backtest better execute in the blink of an eye, for a minutely one to seem zippy. Any idea what's slowing things down?

I've attached a backtest for you to profile, if you want. Any way to speed it up?

Grant

Hi Dan,

Does the Quantopian backtester load all of the data needed for a given backtest into working memory, prior to the start of the backtest? Or does it stream the data from disk?

Also, at some point, I gathered that the built-in transforms (e.g. mavg) are considerably faster than the batch transform? Is this still the case? If so, why?

I recognize that some details may be proprietary, but it would be interesting to have some insight into what goes on under the hood.

Based on number-crunching with (unoptimized) MATLAB on similarly sized data sets, it seems that the Quantopian backtester is relatively slow...is this a correct assessment? You may have trouble rolling it into a research/discovery tool, as you have on your product roadmap (I think).

Grant

Hello Grant,

These are the stats on your algo FWIW:

No. Of SIDs = 2
Backtest Duration = 1 Years 8 months
Run Time = 11 minutes 42 seconds
Run Time Per SID Per Year = ((11 + 42/60) / (2 * (20/12))) * 60 = 210.6 seconds
Computations (optional):

I think 'Run Time Per SID Per Year' is quite an interesting metric. But then I would! If Quantopian included this on the 'Backtest Details' page we could start to optimise our algos....

P.

I think if you're optimizing for runtime, you're probably optimizing for the wrong thing. Figure out how to make an algo that makes money. If you can do that, then finding optimization help will be trivial!

That said, when you have an interesting algo that is running too slowly, let us know. We plan on investing more in performance, and it's always good to get guidance on where our real-life bottlenecks are.