title: Spark cannot read files stored on AWS S3 in Frankfurt region (Ireland region works fine)
body: <p>I run Apache Spark (2.11, 1.5.2) on a local machine using input files stored in AWS S3. If the files are stored in a bucket in the Ireland region (eu-west-1) it works fine.</p>
  
  <p>But if I try to read files stored in an S3 bucket located in Frankfurt (eu-central-1) it fails with the error message:</p>
  
  <p><strong>The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256</strong></p>
  
  <p>How can I use AWS4-HMAC-SHA256?</p>
  
  <p>The detailed error message is:</p>
  
  <pre><code>Exception in thread "main" org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 GET failed for '/%2myfolder' XML Error Message: &lt;?xml version="1.0" encoding="UTF-8"?&gt;&lt;Error&gt;&lt;Code&gt;InvalidRequest&lt;/Code&gt;&lt;Message&gt;The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.&lt;/Message&gt;&lt;RequestId&gt;ECB53FECECD1C910&lt;/RequestId&gt;&lt;HostId&gt;BmEyVcO/eHZR3IO2Z+8IkEWOn189IBGb2YAgbDxhTu+abuyORCEjHyC14l6nIRVNNnQL2Nyya9I=&lt;/HostId&gt;&lt;/Error&gt;
      at org.apache.hadoop.fs.s3.Jets3tFileSystemStore.get(Jets3tFileSystemStore.java:174)
      at org.apache.hadoop.fs.s3.Jets3tFileSystemStore.retrieveINode(Jets3tFileSystemStore.java:214)
  ...
  Caused by: org.jets3t.service.S3ServiceException: S3 GET failed for '/%2myfolder' XML Error Message: &lt;?xml version="1.0" encoding="UTF-8"?&gt;&lt;Error&gt;&lt;Code&gt;InvalidRequest&lt;/Code&gt;&lt;Message&gt;The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.&lt;/Message&gt;&lt;RequestId&gt;ECB53FECECD1C910&lt;/RequestId&gt;&lt;HostId&gt;BmEyVcO/eHZR3IO2Z+8IkEWOn189IBGb2YAgbDxhTu+abuyORCEjHyC14l6nIRVNNnQL2Nyya9I=&lt;/HostId&gt;&lt;/Error&gt;
      at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:416)
      at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestGet(RestS3Service.java:752)
  </code></pre>
  
  <p>The code is:</p>
  
  <pre><code>import org.apache.spark.api.java.*;
  import org.apache.spark.SparkConf;
  import org.apache.spark.api.java.function.Function;
  
  public class S3Problem {
    public static void main(String[] args) {
      String s3Folder = "s3n://mybucket/myfolder";
  
      SparkConf conf = new SparkConf().setAppName("Simple Application").setMaster("local[*]");
  
      JavaSparkContext sc = new JavaSparkContext(conf);
      JavaRDD&lt;String&gt; myData = sc.textFile(s3Folder).cache();
      long count = myData.count();
  
      System.out.println("Line count: " + count);
    }
  }
  </code></pre>
  
  <p>AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are provided as environment variables.</p>