Quantopian's community platform is shutting down. Please read this post for more information and download your code.
Back to Community
What is a run time Error 32 - Broken Pipe?

I noticed that my algo was all of a sudden backtesting far too slowly, then after a few minutes I received the subject error reference I've never seen before referring to a line of code that simply calls a two parameter function that has been working fine all day, no changes, and was working during the sample summary day results printed below of multiple 1 min trades per day.

What is a 'broken pipe'? Is there a current Q system problem?

2013-08-02PRINT('1600', 'Size 0', 'PL 0', 'Cum 0', 'LBT 0', 'SBT 0', 'HLT -4950')
2013-08-05PRINT('1600', 'Size 0', 'PL -3245', 'Cum 0', 'LBT 0', 'SBT 1', 'HLT -4840')
2013-08-06PRINT('1600', 'Size 0', 'PL -952', 'Cum -3245', 'LBT 2', 'SBT 2', 'HLT -4950')
2013-08-07PRINT('1427', 'Size 0', 'PL -5437', 'Cum -9634', 'LBT 1', 'SBT 3', 'HLT -4950', 'MAX HALT')
2013-08-07PRINT('1600', 'Size 0', 'PL 265', 'Cum -9634', 'LBT 1', 'SBT 3', 'HLT -4950')
2013-08-08PRINT('1600', 'Size 0', 'PL 1188', 'Cum -9369', 'LBT 0', 'SBT 0', 'HLT -4950')
2013-08-09PRINT('1600', 'Size 0', 'PL 3469', 'Cum -8181', 'LBT 0', 'SBT 0', 'HLT -5060')
2013-08-12PRINT('1600', 'Size 0', 'PL 3031', 'Cum -4712', 'LBT 0', 'SBT 2', 'HLT -4840')
2013-08-13PRINT('1600', 'Size 0', 'PL 11091', 'Cum -1682', 'LBT 0', 'SBT 2', 'HLT -4620')
2013-08-14PRINT('1600', 'Size 0', 'PL 4689', 'Cum 9410', 'LBT 0', 'SBT 1', 'HLT -4620')
2013-08-15PRINT('1600', 'Size 0', 'PL 3125', 'Cum 14098', 'LBT 1', 'SBT 2', 'HLT -4620')
2013-08-16PRINT('1600', 'Size 0', 'PL -853', 'Cum 17223', 'LBT 0', 'SBT 0', 'HLT -4510')
2013-08-19PRINT('1600', 'Size 0', 'PL 3552', 'Cum 16370', 'LBT 0', 'SBT 1', 'HLT -4510')

The error must have happened sometime during 2013-08-20. I've been running backtests through this time period all day...

Thanks,

Mark

20 responses

I often get the "broken pipe" error when I have a debug breakpoint and exactly on the line where I set the breakpoint.
No breakpoint, no error... it must be a problem in the debugger...

To reproduce the "[Errno 32] Broken pipe" error, build the attached algorithm after having set for example a breakpoint at line 142 and an enough long back-testing period (i.e. from 01/03/2002 to 01/11/2015)

I hope, that the Quantopian Team will prioritize this issue, it's a real bottleneck! I reported it for the first time in Januar.
The Quantopian platform looks very promising, but without a debugger that works properly, it is very hard to do serious development.

You've hit the issue on the head, this error is a bug in the debugger. I've gone ahead to file it internally and we'll take a look to fix it.

I'm not quite sure yet what's causing the message and I'm sorry for the troubles. Once we patch it, I'll follow up in this thread.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

I've been running into several broken pipe error recently, any progress?

Unfortunately the error appears to be random, the same breakpoint will work maybe 85 percent of the time, then it will throw an intermittent error. It does appear to be related to print statements more often than not. I will set a date and time logical test then set the result to print 'OK' after the date time if statement. I set the breakpoint on the print statement to stop processing at the specified date and time. Again, the error appears to be random on the print statement breakpoint. I have been testing all day and have set other breakpoints within the code on 'regular' statements past the 'initial' date and time if statement breakpoint, for example...

if date == 140506 and time == 1320 :
print('OK')

Broken Pipe on stackoverflow

In the neighborhood of buffer, SIGPIPE, keep-alive, seems a process was already abandoned/timed-out and doomed but user doesn't find out until the breakpoint is hit later.

Im my experience this error occurs when the backtesting period is quite long, i.e. from 01/03/2002 to 01/11/2015 and when the code has several if-blocks.

If you servers run on a Linux/Unix System, maybe this could help: http://coding.derkeiler.com/Archive/Python/comp.lang.python/2004-06/3823.html

+1

I get it all the time too

Something new about this problem?
I reported it for the first time 4 months ago.
I think you should prioritize its solving: without a working debugger it's impossible to develop seriously.

The "broken pipe" problem doesn't occur anymore in my algorithm!

I had to many call to get_fundamentals(.) in before_trading_start(.), I limited them introducing a date logic (practically perform get_fundamentals(.) only once in month) and now the debugger works!

Maybe the reason of the "broken pipe" was to many data in memory to be serialized on the hd because of debugging modus.

Workaround:
If it is basically a timeout at some number of minutes (maybe built in to the browser, not sure), then ...
Suppose you have a breakpoint with a condition to break in if the date is near the end of a two year backtest, hitting 'Broken Pipe'.
Set another breakpoint on another line to break at around one half of that time, like str(get_datetime().date()) == '2014-05-01'.
When that first point is hit, it should reset the timer, to be able to make it to the next.

any updates on this?
still running into these at times.

I just ran into this issue as well while backtesting over 9 years. I needed to use the debugger to inspect a variable so I changed my backtest period to the three days I knew would populate the variable I wanted to check. That solved the problem so it does seem to be an issue with the debugger and length of backtest. Even with the backtest at three days though it hung up for almost two minutes before the breakpoint triggered and let me use the debugger which seemed odd since it was just a simple data.history call for 30 days of price data.

Just to bump up this post, the bug isn't fixed yet...

Also having this issue, and it really hurts development time of algorithms. Seems to happen with a test that has a long back-test period, or simply takes a long time.

Nope. Not fixed at all:/

@Danny Zilberg could you elaborate on the issues you are seeing. This thread is quite old and much has changed with the code and the platform. This also is not an issue that get's reported to our support team, so other than the replies above there isn't a lot for our engineering team to go on. Do you have an algo you could attach? Thanks!

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Hi sorry for the late reply. I got ahead in my algorithm and thought it was an internet issue.
Now I've encountered the same problem. If I set the breakpoint at a certain date, way ahead in the backtest, the broken pipe error comes up. If not, the algorithm runs untill it reaches the error that I am trying to find.
For some reason I'm not able to attach that specific algorithm (it only shows 5 of my algorithms)
Sorry and thanks