title: Python how to resolve the Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2
body: <p>I am having an issue running a simple python code inside hadoop streaming. 
  I have tried all the suggestions in the previous posts with a similar error and still having the issue. </p>
  
  <ol>
  <li>added the usr/bin/env python</li>
  <li>chmod a+x the mapper and reducer python code</li>
  <li>put "" for the -mapper "python mapper.py -n 1 -r 0.4"</li>
  </ol>
  
  <p>I have run the code outside and it worked well. </p>
  
  <p>UPDATE: I run the code outside of hadoop streaming using the following code:</p>
  
  <pre><code>cat file |python mapper.py -n 5 -r 0.4 |sort|python reducer.py -f 3618 
  </code></pre>
  
  <p>This works fine .. But now I need to run it to HADOOP STREAMING </p>
  
  <pre><code>hadoop jar /usr/lib/hadoop-mapreduce/hadoop-streaming.jar \
  -D mapreduce.job.reduces=5  \
  -files lr \
  -mapper "python lr/mapper.py -n 5 -r 0.4"  \
  -reducer "python lr/reducer.py -f 3618"  \
  -input training \
  -output models 
  </code></pre>
  
  <p>The hadoop streaming is the one failed. I looked at the logs and I did not see anything on it that told me why it is happening?</p>
  
  <p>I have the following <strong>mapper.py</strong>:</p>
  
  <pre><code>#!/usr/bin/env python
  
  import sys
  import random
  
  from optparse import OptionParser
  
  parser = OptionParser()
  parser.add_option("-n", "--model-num", action="store", dest="n_model",
                    help="number of models to train", type="int")
  parser.add_option("-r", "--sample-ratio", action="store", dest="ratio",
                    help="ratio to sample for each ensemble", type="float")
  
  options, args = parser.parse_args(sys.argv)
  
  random.seed(8803)
  r = options.ratio
  for line in sys.stdin:
      # TODO
      # Note: The following lines are only there to help 
      #       you get started (and to have a 'runnable' program). 
      #       You may need to change some or all of the lines below.
      #       Follow the pseudocode given in the PDF.
      key = random.randint(0, options.n_model-1)
      value = line.strip()
      for i in range(1, options.n_model+1):
          m = random.random()
          if m &lt; r:
              print "%d\t%s" % (i, value)
  </code></pre>
  
  <p>and my <strong>reducer.py</strong>:</p>
  
  <pre><code>#!/usr/bin/env python
  import sys
  import pickle
  from optparse import OptionParser
  from lrsgd import LogisticRegressionSGD
  from utils import parse_svm_light_line
  
  parser = OptionParser()
  parser.add_option("-e", "--eta", action="store", dest="eta",
                    default=0.01, help="step size", type="float")
  parser.add_option("-c", "--Regularization-Constant", action="store", dest="C",
                    default=0.0, help="regularization strength", type="float")
  parser.add_option("-f", "--feature-num", action="store", dest="n_feature",
                    help="number of features", type="int")
  options, args = parser.parse_args(sys.argv)
  
  classifier = LogisticRegressionSGD(options.eta, options.C, options.n_feature)
  
  for line in sys.stdin:
      key, value = line.split("\t", 1)
      value = value.strip()
      X, y = parse_svm_light_line(value)
      classifier.fit(X, y)
  
  pickle.dump(classifier, sys.stdout)
  </code></pre>
  
  <p>When I run it outside the code, it runs OK, but when I run it inside hadoop-streaming it gives me the error: </p>
  
  <pre><code>17/02/07 07:44:34 INFO mapreduce.Job: Task Id : attempt_1486438814591_0038_m_000001_2, Status : FAILED
  Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2
      at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
      at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
      at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
  </code></pre>