title: hadoop copying from hdfs to S3
body: <p>I've successfully completed mahout vectorizing job on Amazon EMR (using <a href="https://cwiki.apache.org/MAHOUT/mahout-on-elastic-mapreduce.html" rel="nofollow noreferrer">Mahout on Elastic MapReduce</a> as reference). Now I want to copy results from HDFS to S3 (to use it in future clustering).</p>
  
  <pre><code>For that I've used hadoop distcp:
  
  den@aws:~$ elastic-mapreduce --jar s3://elasticmapreduce/samples/distcp/distcp.jar \
  &gt; --arg hdfs://my.bucket/prj1/seqfiles \
  &gt; --arg s3n://ACCESS_KEY:SECRET_KEY@my.bucket/prj1/seqfiles \
  &gt; -j $JOBID
  </code></pre>
  
  <p>Failed. Found that suggestion: <a href="https://stackoverflow.com/a/10749610/658346">Use s3distcp</a> Tried it also:</p>
  
  <pre><code>elastic-mapreduce --jobflow $JOBID \
  &gt; --jar --arg s3://eu-west-1.elasticmapreduce/libs/s3distcp/1.latest/s3distcp.jar \
  &gt; --arg --s3Endpoint --arg 's3-eu-west-1.amazonaws.com' \
  &gt; --arg --src --arg 'hdfs://my.bucket/prj1/seqfiles' \
  &gt; --arg --dest --arg 's3://my.bucket/prj1/seqfiles'
  </code></pre>
  
  <p>In both cases I have the same error: <strong>java.net.UnknownHostException: unknown host: my.bucket</strong><br>
  Below the full error output for the 2nd case.  </p>
  
  <pre><code>2012-09-06 13:25:08,209 FATAL com.amazon.external.elasticmapreduce.s3distcp.S3DistCp (main): Failed to get source file system
  java.net.UnknownHostException: unknown host: my.bucket
      at org.apache.hadoop.ipc.Client$Connection.&lt;init&gt;(Client.java:214)
      at org.apache.hadoop.ipc.Client.getConnection(Client.java:1193)
      at org.apache.hadoop.ipc.Client.call(Client.java:1047)
      at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
      at $Proxy1.getProtocolVersion(Unknown Source)
      at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:401)
      at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:384)
      at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:127)
      at org.apache.hadoop.hdfs.DFSClient.&lt;init&gt;(DFSClient.java:249)
      at org.apache.hadoop.hdfs.DFSClient.&lt;init&gt;(DFSClient.java:214)
      at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
      at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1413)
      at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:68)
      at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1431)
      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:256)
      at com.amazon.external.elasticmapreduce.s3distcp.S3DistCp.run(S3DistCp.java:431)
      at com.amazon.external.elasticmapreduce.s3distcp.S3DistCp.run(S3DistCp.java:216)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
      at com.amazon.external.elasticmapreduce.s3distcp.Main.main(Main.java:12)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:187)
  </code></pre>