title: Problems using distcp and s3distcp with my EMR job that outputs to HDFS
body: <p>I've run a job on AWS's EMR, and stored the output in the EMR job's HDFS.  I am then trying to copy the result to S3 via distcp or s3distcp, but both are failing as described below. (Note: the reason I'm not just sending my EMR job's output directly to S3 is due to the (currently unresolved) problem I describe in <a href="https://stackoverflow.com/questions/11169708/where-is-my-aws-emr-reducer-output-for-my-completed-job-should-be-on-s3-but-no">Where is my AWS EMR reducer output for my completed job (should be on S3, but nothing there)?</a> </p>
  
  <p>For distcp, I run (following <a href="https://forums.aws.amazon.com/thread.jspa?threadID=34011&amp;tstart=0" rel="nofollow noreferrer">this post's recommendation</a>):</p>
  
  <pre><code>elastic-mapreduce --jobflow &lt;MY-JOB-ID&gt; --jar \
  s3://elasticmapreduce/samples/distcp/distcp.jar \
      --args -overwrite \
      --args hdfs:///output/myJobOutput,s3n://output/myJobOutput \
      --step-name "Distcp output to s3"
  </code></pre>
  
  <p>In error log (/mnt/var/log/hadoop/steps/8), I get:</p>
  
  <pre><code>With failures, global counters are inaccurate; consider running with -i
  Copy failed: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: &lt;SOME-REQUEST-ID&gt;, AWS Error Code: null, AWS Error Message: Forbidden, S3 Extended Request ID: &lt;SOME-EXT-REQUEST-ID&gt;
          at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:548)
          at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:288)
          at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:170)
  ...
  </code></pre>
  
  <p>For s3distcp, I run (following <a href="http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/UsingEMR_s3distcp.html" rel="nofollow noreferrer">the s3distcp documentation</a>):</p>
  
  <pre><code>elastic-mapreduce --jobflow &lt;MY-JOB-ID&gt; --jar \
  s3://us-east-1.elasticmapreduce/libs/s3distcp/1.0.4/s3distcp.jar \
  --args '--src,/output/myJobOutput,--dest,s3n://output/myJobOutput'
  </code></pre>
  
  <p>In the error log (/mnt/var/log/hadoop/steps/9), I get:</p>
  
  <pre><code>java.lang.RuntimeException: Reducer task failed to copy 1 files: hdfs://10.116.203.7:9000/output/myJobOutput/part-00000 etc
          at com.amazon.elasticmapreduce.s3distcp.CopyFilesReducer.close(Unknown Source)
          at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:537)
          at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:428)
          at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAs(Subject.java:396)
          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
          at org.apache.hadoop.mapred.Child.main(Child.java:249)      
  </code></pre>
  
  <p>Any ideas what I'm doing wrong?</p>
  
  <p><strong>Update:</strong> Someone responding on the AWS Forums to a <a href="https://forums.aws.amazon.com/thread.jspa?threadID=93486" rel="nofollow noreferrer">post about a similar distcp error</a> mentions the IAM user user permissions, <strike>but I don't know what this means</strike> (<strong>edit:</strong> I haven't created any IAM users, so it is using the defaults); hopefully it helps pinpoint my problem.</p>
  
  <p><strong>Update 2:</strong> I noticed this error in namenode log file (when re-running s3distcp).. I'm going to look into default EMR permissions to see if it is my problem:</p>
  
  <pre><code>2012-06-24 21:57:21,326 WARN org.apache.hadoop.security.ShellBasedUnixGroupsMapping (IPC Server handler 40 on 9000): got exception trying to get groups for user job_201206242009_0005
  org.apache.hadoop.util.Shell$ExitCodeException: id: job_201206242009_0005: No such user
  
      at org.apache.hadoop.util.Shell.runCommand(Shell.java:255)
      at org.apache.hadoop.util.Shell.run(Shell.java:182)
      at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
      at org.apache.hadoop.util.Shell.execCommand(Shell.java:461)
      at org.apache.hadoop.util.Shell.execCommand(Shell.java:444)
      at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:68)
      at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:45)
      at org.apache.hadoop.security.Groups.getGroups(Groups.java:79)
      at org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:966)
      at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.&lt;init&gt;(FSPermissionChecker.java:50)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5160)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkTraverse(FSNamesystem.java:5143)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:1992)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.getFileInfo(NameNode.java:837)
      ...
  </code></pre>
  
  <p><strong>Update 3:</strong> I contact AWS Support, and they didn't see a problem, so am now waiting to hear back from their engineering team. Will post back as I hear more</p>