title: Accessing S3 bucket from local pyspark using assume role
body: <p>Background: In order to allow developers to build and unit test code on an easy-to-use environment, we built a local Spark environment with other tools integrated to it. However, we also want to access S3 and Kinesis from Local environment. When we access S3 from Pyspark application from local using assume-role(as per our security standards), it is throwing forbidden error. </p>
  
  <p>FYI - We are below access pattern for accessing resources on AWS account. 
  <a href="https://i.stack.imgur.com/l9JZB.jpg" rel="nofollow noreferrer">assume-role access pattern</a></p>
  
  <p>Code for testing  <em>access-s3-from-pyspark.py</em> :</p>
  
  <pre><code>from pyspark import SparkConf, SparkContext
  
  conf = SparkConf().setAppName("s3a_test").setMaster("local[1]")
  sc = SparkContext(conf=conf)
  
  sc.setSystemProperty("com.amazonaws.services.s3.enableV4", "true")
  
  hadoopConf = {}
  iterator = sc._jsc.hadoopConfiguration().iterator()
  while iterator.hasNext():
      prop = iterator.next()
      hadoopConf[prop.getKey()] = prop.getValue()
  for item in sorted(hadoopConf.items()):
      if "fs.s3" in item[0] :
      print(item)
  
  path="s3a://&lt;your bucket&gt;/test-file.txt"
  
  ## read the file for testing
  lines = sc.textFile(path)
  
  if lines.isEmpty() == False:
      lines.saveAsTextFile("test-file2.text")
  </code></pre>
  
  <p>Property file <em>spark-s3.properties</em></p>
  
  <pre><code>spark.hadoop.fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem
  spark.hadoop.fs.s3a.endpoint s3.eu-central-1.amazonaws.com
  spark.hadoop.fs.s3a.access.key &lt;your access key &gt;
  spark.hadoop.fs.s3a.secret.key &lt;your secret key&gt;
  spark.hadoop.fs.s3a.assumed.role.sts.endpoint sts.eu-central-1.amazonaws.com
  spark.hadoop.fs.s3a.aws.credentials.provider org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider 
  spark.hadoop.fs.s3a.aws.credentials.provider org.apache.hadoop.fs.s3a.AssumedRoleCredentialProvider
  spark.hadoop.fs.s3a.aws.credentials.provider org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider
  spark.hadoop.fs.s3a.assumed.role.session.name testSession1
  spark.haeoop.fs.s3a.assumed.role.session.duration 3600
  spark.hadoop.fs.s3a.assumed.role.arn &lt;role arn&gt;
  spark.hadoop.fs.s3.canned.acl BucketOwnerFullControl
  </code></pre>
  
  <p>How to run the code:</p>
  
  <pre><code>spark-submit --properties-file spark-s3.properties \
          --jars jars/hadoop-aws-2.7.3.jar,jars/aws-java-sdk-1.7.4.jar \
          access-s3-from-pyspark.pyenter code here
  </code></pre>
  
  <p>The above code is returning below error, please note I am able to access S3 via CLI and boto3 using assume-role profile or api.</p>
  
  <pre><code>com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 66FB4D6351898F33, AWS Error Code: null, AWS Error Message: Forbidden, S3 Extended Request ID: J8lZ4qTZ25+a8/R3ZeBTrW5TDHzo98A9iUshbe0/7VcHmiaSXZ5u6fa0TvA3E7ZYvhqXj40tf74=
      at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798)
      at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421)
      at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
      at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
      at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:976)
      at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:956)
      at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:892)
      at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:77)
      at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
      at org.apache.hadoop.fs.Globber.glob(Globber.java:252)
      at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1676)
      at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:259)
      at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)
      at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)
      at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:200)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
      at scala.Option.getOrElse(Option.scala:121)
      at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
      at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
      at scala.Option.getOrElse(Option.scala:121)
      at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
      at org.apache.spark.api.java.JavaRDDLike$class.partitions(JavaRDDLike.scala:61)
      at org.apache.spark.api.java.AbstractJavaRDDLike.partitions(JavaRDDLike.scala:45)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
      at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
      at py4j.Gateway.invoke(Gateway.java:282)
      at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
      at py4j.commands.CallCommand.execute(CallCommand.java:79)
      at py4j.GatewayConnection.run(GatewayConnection.java:238)
      at java.lang.Thread.run(Thread.java:748)
  </code></pre>
  
  <p>Questions:</p>
  
  <p>This is right way of doing?</p>
  
  <p>Is there any other easy way to use AWS resources locally for dev and test (I have also explored localstack package which is working most of the cases but still not fully dependable)</p>
  
  <p>Am I using right jars for this?</p>