Wasn't there a memory leak in Q2 that they released last week? I think they are still working on it.

I cloned the algo, and got a memory error, too. I'm at a loss at this point on being able to pinpoint it. Any chance that there is an explicit "accumulator" within the code (e.g. appending lots of data to an array/dataframe every minute/day)?

Was it ever reported that there was a memory problem as part of the Q2 release (e.g. mentioned on the forum, since there is no formal bug tracker)? Maybe it wasn't caught in testdrive, and so Q2 just went forward? Or it wasn't deemed severe enough?

The routine is basically a Fankenstein of a couple of sample algo's mashed together. I certainly did not try to accumulate anything. At line 79 I log the length my two main arrays to verify stuff isn't adding up.

I like the idea of a memory leak as it means I haven't done anything too stupid.

I have been getting the same on that algo I produced....the stocks rebalancing thing.

Anthony,

What do you mean? Are you seeing a memory error, too? What do you mean "the stocks rebalancing thing"?

I'd done some testing in this area, but it was rather extreme corner-case stuff:

https://www.quantopian.com/posts/error-when-ordering-all-stocks

But it did seem strange that a significant memory would be required to process orders (even ~8000 in one shot shouldn't cause a crash).

memory error algorithm used too much memory

"A simple rotation system for stocks". I tried to run it on a monthly reallocation of 50 stocks out of 3000. And eventually gave up!

@Anthony,

Hmm? Did it work under Q1 and not under Q2? Did you send it to Q support for analysis? Doesn't seem like 50 stocks should be a problem.

It did work with your under Q 1. No, I have not taken this up with support. To be honest I like to have control of my own testing environment and have thus been spending time porting my systems to another open source alternative on my local machine.

All credit to Q, but it's too much of a moving target for me and for my simple needs, it is over complex.

I can't find the post now, but I am sure they confirmed that there was a bug in Q2 wherein portfolio positions that were reduced to 0 shares were sticking around in the portfolio indefinitely, leading to a perpetual growth of the portfolio positions dictionary.

Short answer: We're working on a fix for a memory problem.

Memory leaks can be a long chase sometimes. You find a memory leak, you fix it, you test it, you release it. And then you watch the memory keep growing, and you realize you have yet another memory leak that was hidden behind the first one; rinse and repeat.

Our pre-release on testdrive revealed some memory problems. We fixed them before we did the official Q2 launch. As we've been helping people migrate algos to Q2 in the last 10 days, we've gotten a few more reports of algorithms that ran into memory problems. We've got a bug identified and a fix in the works. It appears to specifically affect algorithms that hold a large number of stocks rotating in and out of their portfolio, typically over a longer backtest.

If I were incredibly foolish, or massively optimistic, or both, I'd tell you that we've identified all the memory leaks. But I'm not! I'll post in this thread when we ship the current fix, and then we'll see if there are more problems uncovered.

It's worth mentioning that we do have a broad test suite that looks for this kind of thing before we ship. We found and fixed many issues before we shipped Q2. The unfortunate reality is that bugs sometimes still occur.

Dan, you mean your software isn't perfect like M$ when you release it? tsk, tsk :->

Thanks Dan,

Here's code that reproduces the memory leak error. In this example, the error results after only 6 months of backtesting, so the length of the backtest may not be the critical factor.

By the way, there is another bug (I think). If I comment out if data.can_trade(stock): I get:

Cannot order CFIC on 2002-01-03 15:30:00+00:00 as there is no last price for the security. Try using data.can_trade to check whether an asset is delisted.  
There was a runtime error on line 37.

However, my understanding is that pipeline should only admit stocks after their first trade is encountered. Correct?

This one works, with a 1 stock random rotation, per day.

Hi Dan,

What should the backtester be able to handle? For example, in the code I posted above, how many stocks should it be able to rotate in and out every day? In other words, how will the Q engineers know when they are done? What is their requirement? 100? 500? All ~ 8000?

Grant

Hi Dan,

I like to add my voice to the concern with this big as well. Its been a week or so that I have been getting the memory error while testing a long/short algo that buys and sells 200 positions every week as part of a re-balance routine.

It takes the algo about a years worth of back-test before dying of memory error. At first I thought I was somehow bloating a data structure out of control but after much testing this seems to be related.

Would appreciate feedback on the estimate on when we can get this fixed. Kind of dead in the water with backtesting ...

Thanks

Kamran

Here's another one that fails due to the memory bug. Something to run once it gets fixed, to confirm that a copacetic state of blissful perfection has been achieved.

We shipped a fix at the end of last week that should help with several memory issues. Try running your backtests again!

Thanks. I ran some tests. Even my "science project" algo that buys all 8000+ securities every day runs. --Grant

Hello there,
Need help please as I got the "MemoryError: Algorithm used too much memory" when running my algo for backtest. The algo uses Pipeline to screen over the entire universe. The screening is made of several filters: the builtin filter for tradeability, plus one on market cap, one on price_per_sales_ratio, one on share reduction, one on total revenue absolute change from last year and the last filter is on total revenue absolute change from last 5 years. The thing is that last 3 filters are implemented via custom factor to retrieve past fundamentals values. Also, I have identified and eliminated that the issue is from the call to the last filter (i.e the 5 years back computation = 5*252 trading days). I already tried to optimize by masking the custom factor computation (i.e, using the other filters to consider a smaller universe), but this did not help too much. It's worth to mention that when screening the universe up to the before-last filter, it only counts an average of 45 stocks remaining. So, one should expect an even smaller universe when the latest filter actually works. I have also attempted setting presumably large temporary variables to None, or even delete them (del X), but the error is still there. What I am definitely asking for is whether there is a way for (manually) freeing memory from large variable? I have attempted using Garbage collection but it cannot be imported. So, I wonder if somebody here knows any other way for actively freeing memory while a backtest is running? Please help. Thanks in advance

EDIT One other possible solution would be to query past fundamentals and pick only the value I need at the exact past date, instead of dataframing the past custom factor values over the entire trading days till the concerned date - as this is how it is currently done by setting window_length. I know there are the as_of fields for every morningstar's fundamentals, but I don't know if they actually work and how. Thx