I ran some backtests a few hours ago, some failed, some succeeded. Re-ran the failed ones (same code- same time period) some succeeded, a tiny fraction failed again. I am wondering if the failures are dependant on something other than the algorithm : and more of a function of that execution at that time. Database slowness/machine slowness/loading data in custom factors etc. I am seeing failures on long backtests on heavy duty algos that might be taking a good chunk of 5 minutes to complete and that one time maybe hit by server slowness or database slowness? Many times it occurs in the middle like after 55% processed etc. If the algo uses the same pipeline everyday, how can it fail after processing 55% of the days.
Any visibility on how to approach such a problem. Wondering if I need to tune the algo further or if Quantopian can change their backtest code to not exit the first time the before_trading_start takes 5minutes.
Maybe use a more relaxed time for first breach and exit only on repeated delays beyond 5mins/ not exit if the failure was due to internal calls to the database etc?
Or provide us with a premium service that we can pay for to buy more time for the backtests?
