Regarding the error of the start date it might be that some pipeline factors has a very long window_length, so long that it requires data before Q database. it happened the same to me sometimes.

The memory error could be due to the universe filter 1500US() . I have the same error everytime I choose a large universe. Try with 500US() .
I know it would be amazing to have no limits, it's frustrating not being able to test what we like, but we have to cope with remote execution of our code. The more Q will succeed the more resources we will have, I guess.

Yeah, Q1500() has to be started on 11/1/2002 or later. There must be some longer windows in there that push the date out. My suggestion to Q support is to follow their paradigm of automatically finding the earliest start date, and jumping to it, since the algo will crash otherwise. Or report the earliest start date possible in the error message. For the code I posted above, it just crashes, and one has to fiddle around to find the earliest start date.

The memory error could be due to the way transactions are stored. It is relatively easy to overload the memory with a long backtest, I've found. It is a mystery to me, since I'd think they could be buffered and written to disk as the backtest runs. Once the backtest ends, they must get stored there, so why not just stick them there as they become available. I guess the entire history is needed in working memory to compute some stats? But then, I think the backtest will also crash if it is run in the background. Perhaps when run in the background, the rolling stats could be suppressed, and then the backtest could be loaded into the research platform (where there is also a limitation that tends to top out for long backtests with portfolios with hundreds of stocks...but if adding memory to the backtester is too expensive or whatever, maybe it could be done more easily in the research platform?).

Thanks for the feedback.

1. I agree, it would be nice if it automatically detected the start date in this case. I created a feature request. You've already discovered the workaround, though, which is to start the backtest at a later date.
   
2. The ML algorithm that Thomas shared does indeed push the limits of what the platform can support right now. His post noted that: "Ideally, we could train on a longer window, like 6 months. But it's very easy to run out-of-memory here. We are acutely aware of this problem and are working hard to resolve it. Until then, we have to make due with being limited in this regard." I'm not sure exactly what triggers this memory problem. Transactions are often a culprit, as are orders.
   

We are regularly making improvements in memory, speed, etc. to support algorithms. For now, as Thomas noted, this one is pushing the boundaries of what we can do. In the future it will be faster and more robust.

Regarding the memory limit Thomas' algo hits, it might be worthwhile just to make sure it is, in fact, the number of transactions/orders (transactions can be gaged by loading the backtest into the research platform), or if there is a new problem, such as a memory leak somewhere in Thomas' code. Maybe running his code without transactions would help sort this out. --Grant

FWIW, I'm pretty sure it's pipeline running out of memory when too big of a window is requested. Although transactions and orders contribute too if you increase the number of traded instruments too much, or use a short rebalance window.

Hi Thomas -

I didn't change the window in your code, and when I load the aborted backtest into the research platform, only 9% of the memory is consumed. So, I don't think it is the overall scale of the algo (transactions, orders, etc.). Not sure what's going on, but as you guys post examples, I'd expect users to try running over the longest time periods possible. Not sure how you did the testing on your end, but if you run all the way back when backtesting, then if some fixes are still required, at least you can add a caveat (which you did, but I'm not clear if it applies here, because apparently there are multiple factors that can cause large scale algos to crash due to memory limitations).

EDIT: Well, I actually had another error:

TimeoutException: Call to before_trading_start timed out
There was a runtime error on line 352.

So, it may be that the memory error I reported was due to transactions. I'll see if I can push past the time-out error...maybe some electrons/holes in your transistors got confused.

Thomas -

Attached, you'll find the code you have posted on https://www.quantopian.com/posts/machine-learning-on-quantopian-part-3-building-an-algorithm (backtest ID: 58517784ee8d8363d0d9790d). It no longer runs to completion, but gives the error:

There was a runtime error.
MemoryError
Algorithm used too much memory. Need to optimize your code for better performance. Learn More

When I load the backtest into the research platform, it only consumes 9% of the memory.

Perhaps you made some changes?

Grant, can you try it again? I just cloned your algo, and it ran to completion.

I'm not sure why it would be intermittent like that, but I'm looking.

Hi Dan,

I cloned a fresh copy of Thomas' code from https://www.quantopian.com/posts/machine-learning-on-quantopian-part-3-building-an-algorithm (Backtest ID: 58517784ee8d8363d0d9790d) and ran it, starting at 8/1/2003. I got:

There was a runtime error.
MemoryError
Algorithm used too much memory. Need to optimize your code for better performance. Learn More

When I load the backtest into the research platform, only 11% of the research platform memory is consumed. So, I've tried it again, and it will not run to completion. Given that so little of the research platform memory is consumed when loading the backtest, I'm skeptical that orders/transactions are the culprit.

Grant, your most recent attached backtest looks like it's running over a longer time frame than Thomas's original. Thomas's run is less than 3 years. It appears that your recent attempt was going for 13 years. It's definitely known and understood that Thomas's ML algo is pushing the limits of what the Quantopian platform will do. It's unfortunate, but I'm not surprised, that it would fail when the backtest parameters are expanded.

What I don't yet understand is why your backtest from yesterday failed. That was only 3 years, and I was able to get it to run. I'm still looking into the cause of that failure.

O.K. No big deal. It just seems odd that my example immediately above (Backtest ID: 585ba441c3d625638d593532) crashes due to a memory limit, but when the backtest is loaded into the research platform, there isn't much being stored. My understanding is that Thomas is doing learning/training on a rolling basis, and that the amount of memory required for that activity should be fixed, not grow with time; but as you suggest, maybe effectively there is a memory leak. If you can look at the memory usage, as or after the backtest runs, then maybe it could be sorted out (perhaps you have some admin-level debug tools?).

@Grant, I bet lots of memory is used by the pipeline and you won't see any trace of that memory usage when loading the backtest into research platform.

@ Luca -

Yes, but it would seem that if things are running on a rolling basis, with finite trailing windows, then the memory usage should be constant versus time (aside from the orders/transactions, which, by definition, will grow). Of course, if expanding windows are used, or ML/pipeline stores a bunch of stuff every time it runs and doesn't "recycle" the memory space, then the memory usage could grow (i.e. a memory leak, if the code could be re-written to provide the same functionality, but not cause the memory usage to grow).

Edit - It is not clear what actually needs to reside in RAM. It would seem that a lot of working memory could be freed up for computing if orders/transactions were written to disk as a backtest runs (unless explicitly stored in context or in some other fashion, as specified in the code).

@Grant

I agree with you, but it could also be that the memory usage is very high since the beginning and as soon as the algorithm stores orders and positions the limit is reached and an error is raised. This could be an explanation that doesn't involve a memory leak.

I remember a reply by Scott that explains the pipeline memory usage. He was talking about the research platform but most of the concepts are valid in backtesting too. Considering the algorithm has quite a lot of pipeline factors, not filtered, I guess the memory consumption is very high.

@Luca -

Yeah, the algo could be "on the hairy edge of disaster" and the orders/transactions push it over the edge. Either way, one is flying blind, without even being able to see the gross amount of memory consumed compared to the amount available (~ 4 GB?). I have confidence that the Q team will eventually fix this; I can't imagine an algo running $10M in capital that could run out of memory unexpectedly. Maybe internally they have a way of flagging which live-trading algos might be at risk?

Another comment that was deemed inappropriate for the thread https://www.quantopian.com/posts/machine-learning-on-quantopian-part-3-building-an-algorithm . I am copying it here, in case anyone is interested:

Hi Thomas,

If I'm reading things correctly, you are applying all factors to the same universe, the Q1500US. There is a more general case, though, of having N factors and M universes, with each factor assigned a universe. That seems straightforward, but then there is a choice of whether the ML is run once, or M+1 times, where the alpha combination step would be run separately on factors grouped by the M universes, and then a final time to combine the results.

The other comment is that it would be nice to be able to be able to combine factors that use minute data (i.e. computed outside pipeline) with pipeline factors. I think you'd said that because the ML is integral to pipeline, this would not be possible, correct? Can the framework be formulated so that the ML runs outside of pipeline (e.g. within before_trading_start as a callable function)?

I'm not sure how important either use case above might end up being, but I figure now's the time to consider them, as you develop the tools.

Hi Grant,

Not sure why we would work with multiple universes?

Re running ML outside of pipline: I think that should be possible. Actually not sure why we moved it into pipeline then in the first place which makes me think that there is probably a reason.

Hello Thomas,

Regarding multiple universes, my thinking is that some factors may work better (or solely) on certain securities. For example, one might have some that work on broad universes of stocks (e.g. Q500US/Q1500US or some such thing), some that work on certain ETFs, some that are IPO-focused, some on stocks that aren't in the Q500US/Q1500US (the subversive/contrarian universe), etc. So, my thinking is that although the workflow flow chart shows a single universe, one could have multiple universes with their associated alpha factors. Then (I think) one would want to combine the alpha factors by universe first, and then do a final combination, prior to the portfolio construction step. This would mean that your architecture would need to support this approach (if it makes sense).

Running ML outside of pipeline would seem to be a lot more flexible. A primary reason would be that for factors that can use pipeline, then by all means, implement them using that API. But if one wants to use minute bars, for example, for some alpha factors, then if your ML module is in pipeline, there will be no way to feed in alpha factors other than from pipeline (I think...unless I'm missing something about pipeline). That is not to say that you would support running the ML more than daily (within before_trading_start, I guess), it would just have the flexibility of being able to take in alpha factors both from pipeline and from the regular functions/classes.

Hi Grant,

Re Multiple Universes: I see, yeah, I think that makes sense. I think one would probably just have an ML classifier piplinefactor for each universe then.

Re ML outside of pipline: I remembered why it's not possible. A pipeline factor can get historical pipeline values. In before_trading_start() you only get the current factor value. I think it would be nice to make this available though.

Regarding ML/alpha combination outside of pipeline, I'd think that the historical pipeline values could be stored in context, right?

The other approach would be to allow feeding of non-pipeline factors into pipeline, so you could still do the combination step within pipeline.

A simple use case would be to use minutely history to compute a daily VWAP of prices, which would then feed into a non-pipeline technical analysis factor that would return daily values (same as a pipeline factor). You'd combine pipeline factors with non-pipeline factors on a daily basis.

My sense is that you'll have a much more universal workflow if you can incorporate non-pipeline factors at the alpha combination step, but apparently there are some fundamental technical limitations.

Another note is that you might do a basic head-scratch on sampling/noise. My intuition is that by using daily OHLCV bars in pipeline, which amounts to very limited sampling of individual trades, with no smoothing, you end up with a lot more noise than necessary. If the idea is to trade daily/weekly (or even monthly?), you'll be in the under-sampled regime, trying to make relatively high-frequency decisions on low-frequency, noisy data. But maybe I'm wrong, and there is good evidence that the kind of equity long-short strategies you are looking to support will work just fine on daily OHLCV bars.

Hi Thomas -

Any progress in devising a way to determine if the ML is doing what it is supposed to? As I suggested, I think you have to be able to input synthetic data/factors that have a predictable/ideal output, to see if the ML combines them properly. Also, one could formulate synthetic factors that are purely noise, to verify that the ML isn't somehow over-fitting.