title: RHadoop: REDUCE capability required is more than the supported max container capability in the cluster
body: <p>Has anybody similar issue with in R (build 1060) on top of sandbox Hadoop (Cloudera5.1/Hortonworks2.1)?
  It seems to be a problem of new R/Hadoop, because on CDH5.0 it works.</p>
  
  <p>Code:</p>
  
  <pre><code>Sys.setenv(HADOOP_CMD="/usr/bin/hadoop")
  Sys.setenv(HADOOP_STREAMING="/usr/lib/hadoop-mapreduce/hadoop-streaming.jar")
  Sys.setenv(JAVA_HOME="/usr/java/jdk1.7.0_55-cloudera")
  library(rhdfs)
  library(rmr2)
  hdfs.init()
  
  ## space and word delimiter
  map &lt;- function(k,lines) {
    words.list &lt;- strsplit(lines, '\\s')
    words &lt;- unlist(words.list)
    return( keyval(words, 1) )
  }
  reduce &lt;- function(word, counts) {
    keyval(word, sum(counts))
  }
  wordcount &lt;- function (input, output=NULL) {
    mapreduce(input=input, output=output, input.format="text", map=map, reduce=reduce)
  }
  
  ## variables
  hdfs.root &lt;- '/user/cloudera'
  hdfs.data &lt;- file.path(hdfs.root, 'scenario_1')
  hdfs.out &lt;- file.path(hdfs.root, 'out')
  
  ## run mapreduce job
  ##out &lt;- wordcount(hdfs.data, hdfs.out)
  system.time(out &lt;- wordcount(hdfs.data, hdfs.out))
  </code></pre>
  
  <p>Error:</p>
  
  <pre><code>&gt; system.time(out &lt;- wordcount(hdfs.data, hdfs.out))
  packageJobJar: [] [/usr/lib/hadoop-mapreduce/hadoop-streaming-2.3.0-cdh5.1.0.jar] /tmp/streamjob8497498354509963133.jar tmpDir=null
  14/09/17 01:49:38 INFO client.RMProxy: Connecting to ResourceManager at quickstart.cloudera/127.0.0.1:8032
  14/09/17 01:49:38 INFO client.RMProxy: Connecting to ResourceManager at quickstart.cloudera/127.0.0.1:8032
  14/09/17 01:49:39 INFO mapred.FileInputFormat: Total input paths to process : 1
  14/09/17 01:49:39 INFO mapreduce.JobSubmitter: number of splits:2
  14/09/17 01:49:39 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1410940439997_0001
  14/09/17 01:49:40 INFO impl.YarnClientImpl: Submitted application application_1410940439997_0001
  14/09/17 01:49:40 INFO mapreduce.Job: The url to track the job: http://quickstart.cloudera:8088/proxy/application_1410940439997_0001/
  14/09/17 01:49:40 INFO mapreduce.Job: Running job: job_1410940439997_0001
  14/09/17 01:49:54 INFO mapreduce.Job: Job job_1410940439997_0001 running in uber mode : false
  14/09/17 01:49:54 INFO mapreduce.Job:  map 100% reduce 100%
  14/09/17 01:49:55 INFO mapreduce.Job: Job job_1410940439997_0001 failed with state KILLED due to: MAP capability required is more than the supported max container capability in the cluster. Killing the Job. mapResourceReqt: 4096 maxContainerCapability:1024
  Job received Kill while in RUNNING state.
  REDUCE capability required is more than the supported max container capability in the cluster. Killing the Job. **reduceResourceReqt: 4096 maxContainerCapability:1024**
  
  14/09/17 01:49:55 INFO mapreduce.Job: Counters: 2
      Job Counters 
          Total time spent by all maps in occupied slots (ms)=0
          Total time spent by all reduces in occupied slots (ms)=0
  14/09/17 01:49:55 ERROR streaming.StreamJob: Job not Successful!
  Streaming Command Failed!
  Error in mr(map = map, reduce = reduce, combine = combine, vectorized.reduce, : hadoop streaming failed with error code 1
  Timing stopped at: 3.681 0.695 20.43 
  </code></pre>
  
  <p>Seems that the issue is in reduceResourceReqt: 4096 maxContainerCapability:1024.
  I have tried to change: yarn-site.xml, but it didn't help. :(</p>
  
  <p>Please, help...</p>