title: Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1, worked perfectly on local
body: <p>I have googled this error on each and every forum but no luck. I have got the error written below:</p>
  
  <pre><code>18/08/29 00:24:53 INFO mapreduce.Job:  map 0% reduce 0%
  18/08/29 00:24:59 INFO mapreduce.Job: Task Id : attempt_1535105716146_0226_m_000000_0, Status : FAILED
  Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
          at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:325)
          at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:538)
          at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
          at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
          at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
          at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
          at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
          at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAs(Subject.java:422)
          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
          at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
  
  
  18/08/29 00:25:45 INFO mapreduce.Job: Task Id : attempt_1535105716146_0226_r_000000_2, Status : FAILED
  Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
          at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:325)
          at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:538)
          at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:134)
          at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:454)
          at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:393)
          at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAs(Subject.java:422)
          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
          at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
  
  18/08/29 00:25:52 INFO mapreduce.Job:  map 100% reduce 100%
  18/08/29 00:25:53 INFO mapreduce.Job: Job job_1535105716146_0226 failed with state FAILED due to: Task failed task_1535105716146_0226_r_000000
  Job failed as tasks failed. failedMaps:0 failedReduces:1 killedMaps:0 killedReduces: 0
  
  
  18/08/29 00:25:53 ERROR streaming.StreamJob: Job not successful!
  Streaming Command Failed!
  </code></pre>
  
  <p>I have also tried my map-reduce code with the help of python standalone command</p>
  
  <pre><code>cat student1.txt | python mapper.py | python reducer.py
  </code></pre>
  
  <p>The code works perfectly fine. But when I tried it through Hadoop Streaming then it repeatedly throws the above error. My input file size is 3KB. I have tried the running Hadoop-streaming command also after changing the python version but no luck! I have also added <code>#!/usr/bin/python</code> command on the top of script. The directory has nothing inside. I also tried different versions of command:</p>
  
  <p>version 1:</p>
  
  <pre><code>hadoop jar /usr/hdp/3.0.0.0-1634/hadoop-mapreduce/hadoop-streaming-3.1.0.3.0.0.0-1634.jar -Dmapred.reduce.tasks=1 -file /home/mapper.py -mapper mapper.py -file /home/reducer.py -reducer reducer.py -input /data/studentMapReduce/student1.txt -output outputMapReduceFile.txt
  </code></pre>
  
  <p>version 2: python commands with single quotes as well as double quotes</p>
  
  <pre><code>hadoop jar /usr/hdp/3.0.0.0-1634/hadoop-mapreduce/hadoop-streaming-3.1.0.3.0.0.0-1634.jar -Dmapred.reduce.tasks=1 -file /home/mapper.py -mapper "python mapper.py" -file /home/reducer.py -reducer "python reducer.py" -input /data/studentMapReduce/student1.txt -output outputMapReduceFile.txt
  </code></pre>
  
  <p>Simple word-count program is running successfully on the environment, generates correct output also but when I added mysql.connector service in the python script then Hadoop-streaming reports this error. I have also studied the job logs but no such information found.</p>