title: Spark is inventing his own AWS secretKey
body: <p>I'm trying to read a s3 bucket from Spark and up until today Spark always complain that the request return 403</p>
  
  <pre><code>hadoopConf = spark_context._jsc.hadoopConfiguration()
  hadoopConf.set("fs.s3a.access.key", "ACCESSKEY")
  hadoopConf.set("fs.s3a.secret.key", "SECRETKEY")
  hadoopConf.set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
  logs = spark_context.textFile("s3a://mybucket/logs/*)
  </code></pre>
  
  <p>Spark was saying .... Invalid Access key [ACCESSKEY]</p>
  
  <p>However with the same ACCESSKEY and SECRETKEY this was working with aws-cli</p>
  
  <pre><code>aws s3 ls mybucket/logs/
  </code></pre>
  
  <p>and in python boto3 this was working</p>
  
  <pre><code>resource = boto3.resource("s3", region_name="us-east-1")
  resource.Object("mybucket", "logs/text.py") \
              .put(Body=open("text.py", "rb"),ContentType="text/x-py")
  </code></pre>
  
  <p>so my credentials ARE invalid and the problem is definitely something with Spark..</p>
  
  <p>Today I decided to turn on the "DEBUG" log for the entire spark and to my suprise... Spark is NOT using the [SECRETKEY] I have provided but instead... add a random one???</p>
  
  <p>17/03/08 10:40:04 DEBUG request: Sending Request: HEAD <a href="https://mybucket.s3.amazonaws.com" rel="noreferrer">https://mybucket.s3.amazonaws.com</a> / Headers: (Authorization: AWS ACCESSKEY:<strong>[RANDON-SECRET-KEY]</strong>, User-Agent: aws-sdk-java/1.7.4 Mac_OS_X/10.11.6 Java_HotSpot(TM)_64-Bit_Server_VM/25.65-b01/1.8.0_65, Date: Wed, 08 Mar 2017 10:40:04 GMT, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) </p>
  
  <p>This is why it still return 403! Spark is not using the key I provide with fs.s3a.secret.key but instead invent a random one??</p>
  
  <p>For the record I'm running this locally on my machine (OSX) with this command</p>
  
  <pre><code>spark-submit --packages com.amazonaws:aws-java-sdk-pom:1.11.98,org.apache.hadoop:hadoop-aws:2.7.3 test.py
  </code></pre>
  
  <p>Could some one enlighten me on this?</p>