title: Use S3DistCp to copy file from S3 to EMR
body: <p>I am struggling to find a way to use S3DistCp in my AWS EMR Cluster.</p>
  
  <p>Some old examples which show how to add s3distcp as an EMR step use <code>elastic-mapreduce</code> command which is not used anymore.</p>
  
  <p>Some other sources suggest to use <code>s3-dist-cp</code> command, which is not found in current EMR clusters. Even official documentation (<a href="http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/UsingEMR_s3distcp.html" rel="nofollow noreferrer">online</a> and EMR developer guide 2016 pdf) present an example like this:</p>
  
  <pre><code>aws emr add-steps --cluster-id j-3GYXXXXXX9IOK --steps Type=CUSTOM_JAR,Name="S3DistCp step",Jar=/home/hadoop/lib/emr-s3distcp-1.0.jar,Args=["--s3Endpoint,s3-eu-west-1.amazonaws.com","--src,s3://mybucket/logs/j-3GYXXXXXX9IOJ/node/","--dest,hdfs:///output","--srcPattern,.*[azA-Z,]+"]
  </code></pre>
  
  <p>But there is no <code>lib</code> folder in the <code>/home/hadoop</code> path. I found some hadoop libraries in this folder: <code>/usr/lib/hadoop/lib</code>, but I cannot find <code>s3distcp</code> from anywhere.</p>
  
  <p>Then I found that there are some libraries available in some S3 buckets. For example, from this <a href="https://stackoverflow.com/questions/14631152/copy-files-from-amazon-s3-to-hdfs-using-s3distcp-fails">question</a>, I found this path: <code>s3://us-east-1.elasticmapreduce/libs/s3distcp/1.latest/s3distcp.jar</code>. This seemed to be a step in the right direction, as adding a new step to a running EMR cluster from the AWS interface with these parameters started the step (which it didn't with previous attempts) but failed after ~15seconds:</p>
  
  <pre><code>JAR location: s3://us-east-1.elasticmapreduce/libs/s3distcp/1.latest/s3distcp.jar
  Main class: None
  Arguments: --s3Endpoint s3-eu-west-1.amazonaws.com --src s3://source-bucket/scripts/ --dest hdfs:///output
  Action on failure: Continue
  </code></pre>
  
  <p>This resulted in the following error:</p>
  
  <pre><code>Exception in thread "main" java.lang.RuntimeException: Unable to retrieve Hadoop configuration for key fs.s3n.awsAccessKeyId
      at com.amazon.external.elasticmapreduce.s3distcp.ConfigurationCredentials.getConfigOrThrow(ConfigurationCredentials.java:29)
      at com.amazon.external.elasticmapreduce.s3distcp.ConfigurationCredentials.&lt;init&gt;(ConfigurationCredentials.java:35)
      at com.amazon.external.elasticmapreduce.s3distcp.S3DistCp.createInputFileListS3(S3DistCp.java:85)
      at com.amazon.external.elasticmapreduce.s3distcp.S3DistCp.createInputFileList(S3DistCp.java:60)
      at com.amazon.external.elasticmapreduce.s3distcp.S3DistCp.run(S3DistCp.java:529)
      at com.amazon.external.elasticmapreduce.s3distcp.S3DistCp.run(S3DistCp.java:216)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
      at com.amazon.external.elasticmapreduce.s3distcp.Main.main(Main.java:12)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
  </code></pre>
  
  <p>I thought this may have been caused by the incompatibility of my S3 location (same as the endpoint) and the location of the s3distcp script, which was from us-east. I replaced it with eu-west-1 and still got the same error about the authentication.
  I have used a similar setup to run my scala scripts (Custom jar type with "command-runner.jar" script with the first argument "spark-submit" to run a spark job and this works, I have not had this problem with the authentication before.</p>
  
  <p><strong>What is the simplest way to copy a file from S3 to an EMR cluster?</strong> Either by adding an additional EMR step with AWS SDK (for Go lang) or somehow inside the Scala spark script? Or from the AWS EMR interface, but not from CLI as I need it to be automated.</p>