title: Hadoop streaming job failure: Task process exit with nonzero status of 137
body: <p>I've been banging my head on this one for a few days, and hope that somebody out there will have some insight.</p>
  
  <p>I've written a streaming map reduce job in perl that is prone to having one or two reduce tasks take an extremely long time to execute.  This is due to a natural asymmetry in the data: some of the reduce keys have over a million rows, where most have only a few dozen.</p>
  
  <p>I've had problems with long tasks before, and I've been incrementing counters throughout to ensure that map reduce doesn't time them out.  But now they are failing with an error message I hadn't seen before:  </p>
  
  <pre><code>java.io.IOException: Task process exit with nonzero status of 137.
      at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
  </code></pre>
  
  <p>This is not the standard timeout error message, but the error code 137 = 128+9 suggests that my reducer script received a kill -9 from Hadoop.  The tasktracker log contains the following:</p>
  
  <pre><code>2011-09-05 19:18:31,269 WARN org.mortbay.log: Committed before 410 getMapOutput(attempt_201109051336_0003_m_000029_1,7) failed :
  org.mortbay.jetty.EofException
          at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:787)
          at org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:548)
          at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:569)
          at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:946)
          at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:646)
          at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:577)
          at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2940)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
          at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
          at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
          at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
          at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
          at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
          at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
          at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
          at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
          at org.mortbay.jetty.Server.handle(Server.java:324)
          at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
          at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
          at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
          at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
          at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
          at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
          at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
  Caused by: java.io.IOException: Connection reset by peer
          at sun.nio.ch.FileDispatcher.write0(Native Method)
          at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
          at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:72)
          at sun.nio.ch.IOUtil.write(IOUtil.java:43)
          at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
          at org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:169)
          at org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
          at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:721)
          ... 24 more
  
  2011-09-05 19:18:31,289 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 10.92.8.202:50060, dest: 10.92.8.201:46436, bytes: 7340032, op: MAPRED_SHUFFLE, cliID: attempt_201109051336_0003_m_000029_1
  2011-09-05 19:18:31,292 ERROR org.mortbay.log: /mapOutput
  java.lang.IllegalStateException: Committed
          at org.mortbay.jetty.Response.resetBuffer(Response.java:994)
          at org.mortbay.jetty.Response.sendError(Response.java:240)
          at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2963)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
          at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
          at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
          at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
          at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
          at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
          at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
          at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
          at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
          at org.mortbay.jetty.Server.handle(Server.java:324)
          at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
          at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
          at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
          at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
          at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
          at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
          at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
  </code></pre>
  
  <p>I've been looking around the forums, and it sounds like the Warnings are commonly found in jobs that run without error, and can usually be ignored.  The error makes it look like the reducer lost contact with the map output, but I can't figure out why.  Does anyone have any ideas?</p>
  
  <p>Here's a potentially relevant fact:  These long tasks were making my job take days where it should take minutes.  Since I can live without the output from one or two keys, I tried to implement a ten minute timeout in my reducer as follows:</p>
  
  <pre><code>eval {  
          local $SIG{ALRM} = sub {
              print STDERR "Processing of new merchant names in $prev_merchant_zip timed out...\n";
              print STDERR "reporter:counter:tags,failed_zips,1\n";
              die "timeout";
          };
  
          alarm 600;
  
          #Code that could take a long time to execute
  
          alarm 0;
      };
  </code></pre>
  
  <p>This timeout code works like a charm when I test the script locally, but the strange mapreduce errors started after I introduced it.  However, the failures seem to occur well after the first timeout, so I'm not sure if it's related.</p>
  
  <p>Thanks in advance for any help!</p>