Hello Samuel,

The last few posts on https://www.quantopian.com/posts/sell-vs-short discuss this issue. Dan Dunn's response is "Grant: you are correct that if your algo takes too long (50 seconds) it will get shut down." My understanding is that "shut down" means that a fatal error will result, i.e. a crash.

Later on in the post, Jonathan gives some guidance on how to replicate the behavior on the backtester. I haven't tried it yet.

My sense is that there ought to be some way to catch or to avoid the timeout error, thus avoiding a crash. I'm curious now if a computation could be carried out over multiple calls to handle_data by storing intermediate results in context? I don't recall ever seeing this feat on Quantopian.

Grant

If your algorithm takes longer than a minute to process a call of handle_data during live trading, it will shutdown. You'll receive an error and will need to tweak your code and redeploy your algo to continue trading. This is a conservative measure to prevent your algorithm from getting behind the market and making incorrect trades.

TimeoutException errors happen if your algo is taking longer than 60 seconds to make computations in handle_data. You can improve your algo's performance by optimizing the code. For example, its faster to use history() than batch_transform(). Also, instead of using simple transformations like mavg you can use rolling transformations in conjunction with history to speed up your algo. Hope that helps!

@Grant. I have created an algorithm that does just what you suggest. Dividing complex calculations over several minute calls (handle_data calls). Now it didn't produce the results I wanted so I've abandon the algorithm but if you have something that shows promise it would pretty easy to incorporate I bet.

My general approach was to profile the calculation using the "time" module and then stop doing things after 30 seconds had past and store the current results. I chose 30 seconds arbitrarily to have lots of margin. This is fairly common and simple approach.

Brian,

Glad you gave it a try. Sounds like the trick worked. To the Quantopian staff, this is another thing to add to your long list of potential improvements--a means to carry out a computation over multiple calls to handle_data.

Grant

Yes, such a feature would be very nice. Futures (no pun intended) could be a potentially elegant solution. I've run into this issue in quite a few cases where I attempt to train models on quite a bit of data using third party tools such as scikit-learn. Thus far, I've been able to get things to run in less than a minute without much trouble but I can foresee it being an issue in the future for myself and others. I may certainly be an outlier in my usage, however. I'm not exactly sure what the average algorithm on Quantopian looks like.

Sometimes we are at the mercy of a built-in function. Some of the sklearn functions in particular can be computationally intensive. I have a situation right now where the typical run time of LassoGraphCV is only around 5 seconds, but occasionally it spikes and I crash the backtest. For backtesting, I can skip the problem time period, but it makes me wary of live trading. I would love to have a timing function that will give me an exception to handle if a command takes longer than I specify. That way the default behavior is still to halt the trading, but I have an exception to handle gracefully if I'd like to continue. At home I would use the signal module, but we don't have that available to us within the API.

I'm still getting up the Python learning curve, but I'm wondering if there is a solution to the general problem:

--Call a function
--Wait delta_t seconds for a response
--If no response after delta_t, give up waiting

I found this but don't have time now to try it:

http://code.activestate.com/recipes/576780-timeout-for-nearly-any-callable/

Might it work?

Grant

Hello Ray (and all),

I'd forgotten that the timeout is actually 5 seconds for the build, and 50 seconds otherwise (see the discussion on https://www.quantopian.com/posts/sell-vs-short). So if you are seeing the timeout upon build/validation, you may need to skip the first bar in the algo.

Grant

Yes, Grant, thank you, I was aware that the build only allows 5 seconds. This particular algorithm does nothing until towards the end of the day, so I shouldn't be hitting that. It runs for a couple of years before getting tripped up, so I just skip trading on the days that cause problems. Obviously this would not work for live trading.

Your solution probably would work, but not In Quantopian, since they don't support the threading module.

Ray,

Yeah, I did a few tests, and found that for a variety of approaches published online, the required modules are not available in Quantopian.

One work-around would be to modify the sklearn functions, if possible. For example, if there is a loop in the function, you could check the elapsed time after each execution of the loop, and exit the function if too much time has passed.

Also, I wonder how the Quantopian backtester manages memory? In MATLAB, as I recall, code will keep on chugging, even if it runs out of RAM--it just starts using the hard drive. Might your code be bogging down due to limited memory?

It seems odd that only on a handful of days, your code would time-out. Might there be some input data integrity problem on those days?

Grant

Grant, I agree with your suggested approach. I think instead of using the "CV" version of the algorithm, I would break it up and loop through the alphas myself, checking for time at each iteration. I already split my algo into several frames, but this particular fit function has pretty much the whole frame to itself already. I do realtime (not financial!) programming for a living, so I was kind of hoping not to deal with such things on here, and just use the canned algorithms :)

I don't think it is odd for machine learning algorithms to vary a lot in their execution time... they sometimes get stuck optimizing down a path towards a local maxima/minima and then restart at another point. At least, that's my understanding... I'm not exactly an expert! I do know that the time to fit is usually not an indicator of the quality of the fit. That is, it can solve in 5 seconds or 60 seconds and your fitness won't be any better.

Ray,

Interesting that you have a background in realtime programming. I've tried to imagine how Quantopian works "under the hood" with respect to timing/synchronization with the broker. The way they have it set up (I gather), everything has to happen within ~50 seconds, reserving ~10 seconds for latency/housekeeping. Maybe it wouldn't preclude a background process. One could start the process and then check it every call to handle_data.

Also, I had the thought that maybe the timeout error would be accessible by importing code from zipline. Then, at least, the error could be caught to avoid a crash.

Grant

Hi Grant,

I'm very new to Zipline, having only just installed it locally. I use the trading calendar in it regularly through Quantopian, but otherwise haven't really played with it. I did a cursory search through the code, and it looks like Zipline does no enforcement of execution time at all. They do some time enforcement in the test suite, but they use the external "nose" module to accomplish this, which we do not access to in the API. I'm going to keep playing with Zipline anyway, since it will let me more easily test these machine learning algorithms, if only with daily data.

Ray