title: using amazon s3 as input,output and to store intermediate results in EMR map reduce job
body: <p>I am trying to use Amazon s3 storage with EMR. However, when I currently run my code I get multiple errors like </p>
  
  <pre><code>java.lang.IllegalArgumentException: This file system object (hdfs://10.254.37.109:9000)    does not support access to the request path 's3n://energydata/input/centers_200_10k_norm.csv' You possibly called FileSystem.get(conf) when you should have called FileSystem.get(uri, conf) to obtain a file system supporting your path.
  at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:384)
  at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:129)
  at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:154)
  at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:429)
  at edu.stanford.cs246.hw2.KMeans$CentroidMapper.setup(KMeans.java:112)
  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:771)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:375)
  at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:396)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
  at org.apache.hadoop.mapred.Child.main(Child.java:249)
  </code></pre>
  
  <p>In main I set my input and output paths like this and I put s3n://energydata/input/centers_200_10k_norm.csv in configuration CFILE that I retrieve in the mapper and reducer:</p>
  
  <pre><code>FileSystem fs = FileSystem.get(conf);
  conf.set(CFILE, inPath); //inPath in this case is s3n://energydata/input/centers_200_10k_norm.csv
  FileInputFormat.addInputPath(job, new Path(inputDir));
  FileOutputFormat.setOutputPath(job, new Path(outputDir));
  </code></pre>
  
  <p>The specific example where the error above occurs in my mapper and reducer where I try to access CFILE (s3n://energydata/input/centers_200_10k_norm.csv). This is how I try to get the path:</p>
  
  <pre><code>FileSystem fs = FileSystem.get(context.getConfiguration());
  Path cFile = new Path(context.getConfiguration().get(CFILE));
  DataInputStream d = new DataInputStream(fs.open(cFile));  ----&gt;Error
  </code></pre>
  
  <p>s3n://energydata/input/centers_200_10k_norm.csv is one of the input arguments to the program and when I launched my EMR job I specified my input and output directories to be s3n://energydata/input and s3n://energydata/output</p>
  
  <p>I tried doing what was suggested in <a href="https://stackoverflow.com/questions/13577767/file-path-in-hdfs">file path in hdfs</a> but I'm still getting the error. Any help would be appreciated.</p>
  
  <p>thanks!</p>