title: Scala Spark Read from AWS S3 - com.amazonaws.SdkClientException: Unable to load credentials from service endpoint
body: <p>I'm currently trying to do a simple read from an S3 bucket I've set up, using Spark 3.0.0 (implementation via Scala 2.12.10). However, I am receiving this error when submit the script:</p>
  <pre><code>No AWS Credentials provided by BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: Unable to load credentials from service endpoint: No AWS Credentials provided by BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider : 
  
  com.amazonaws.SdkClientException: Unable to load credentials from service endpoint
  </code></pre>
  <p>I'm implementing the current spark script:</p>
  <pre><code>package org.knd
  
  import scala.util.Properties
  import org.apache.spark.{SparkConf, SparkContext}
  import org.apache.spark.sql.{SparkSession, SQLContext}
  import io.delta.tables._
  import org.apache.spark.sql.functions._
  
  object App {
  
    def main(args: Array[String]) : Unit = {
  
      val spark = SparkSession
        .builder()
        .appName(&quot;covid-delta-lake&quot;)
        .master(&quot;local&quot;)
        .config(&quot;spark.sql.extensions&quot;, &quot;io.delta.sql.DeltaSparkSessionExtension&quot;)
        .config(&quot;spark.sql.catalog.spark_catalog&quot;, &quot;org.apache.spark.sql.delta.catalog.DeltaCatalog&quot;)
        .getOrCreate()
  
      val aws_access_key = scala.util.Properties.envOrElse(&quot;AWS_ACCESS_KEY&quot;, &quot;notAvailable&quot; )
      val aws_secret = scala.util.Properties.envOrElse(&quot;AWS_SECRET_ACCESS_KEY_ID&quot;, &quot;notAvailable&quot; )
  
      spark.sparkContext.hadoopConfiguration.set(&quot;fs.s3a.awsAccessKeyId&quot;, aws_access_key)
      spark.sparkContext.hadoopConfiguration.set(&quot;fs.s3a.awsSecretAccessKey&quot;, aws_secret)
      spark.sparkContext.hadoopConfiguration.set(&quot;fs.s3a.endpoint&quot;, &quot;s3.amazonaws.com&quot;)
  
      print(&quot;\n&quot; + &quot;====================HERE====================&quot; + &quot;\n&quot;)
  
      val data = spark.read.parquet(&quot;s3a://[url-to-my-s3]/*.parquet&quot;)
      data.show(10)
  
  
    }
  
  }
  </code></pre>
  <p>I've double checked my AWS keys and s3 URL, so I'm certain those aren't the issue. I've tried reading from other buckets and am receiving the same error. I've included my POM file below:</p>
  <pre><code>&lt;project xmlns=&quot;http://maven.apache.org/POM/4.0.0&quot; xmlns:xsi=&quot;http://www.w3.org/2001/XMLSchema-instance&quot; xsi:schemaLocation=&quot;http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd&quot;&gt;
    &lt;modelVersion&gt;4.0.0&lt;/modelVersion&gt;
    &lt;groupId&gt;org.knd&lt;/groupId&gt;
    &lt;artifactId&gt;delta-lake-scala&lt;/artifactId&gt;
    &lt;version&gt;1.0-SNAPSHOT&lt;/version&gt;
    &lt;inceptionYear&gt;2020&lt;/inceptionYear&gt;
    &lt;properties&gt;
      &lt;scala.version&gt;2.12.10&lt;/scala.version&gt;
    &lt;/properties&gt;
  
    &lt;repositories&gt;
      &lt;repository&gt;
        &lt;id&gt;scala-tools.org&lt;/id&gt;
        &lt;name&gt;Scala-Tools Maven2 Repository&lt;/name&gt;
        &lt;url&gt;http://scala-tools.org/repo-releases&lt;/url&gt;
      &lt;/repository&gt;
    &lt;/repositories&gt;
    &lt;pluginRepositories&gt;
      &lt;pluginRepository&gt;
        &lt;id&gt;scala-tools.org&lt;/id&gt;
        &lt;name&gt;Scala-Tools Maven2 Repository&lt;/name&gt;
        &lt;url&gt;http://scala-tools.org/repo-releases&lt;/url&gt;
      &lt;/pluginRepository&gt;
    &lt;/pluginRepositories&gt;
  
    &lt;dependencies&gt;
  
      &lt;dependency&gt;
        &lt;groupId&gt;org.scala-lang&lt;/groupId&gt;
        &lt;artifactId&gt;scala-library&lt;/artifactId&gt;
        &lt;version&gt;${scala.version}&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
        &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
        &lt;artifactId&gt;spark-core_2.12&lt;/artifactId&gt;
        &lt;version&gt;3.0.0&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql --&gt;
      &lt;dependency&gt;
        &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
        &lt;artifactId&gt;spark-sql_2.12&lt;/artifactId&gt;
        &lt;version&gt;3.0.0&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
        &lt;groupId&gt;io.delta&lt;/groupId&gt;
        &lt;artifactId&gt;delta-core_2.12&lt;/artifactId&gt;
        &lt;version&gt;0.7.0&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
        &lt;groupId&gt;org.apache.hadoop&lt;/groupId&gt;
        &lt;artifactId&gt;hadoop-common&lt;/artifactId&gt;
        &lt;version&gt;3.0.0&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
        &lt;groupId&gt;org.apache.hadoop&lt;/groupId&gt;
        &lt;artifactId&gt;hadoop-client&lt;/artifactId&gt;
        &lt;version&gt;3.0.0&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
        &lt;groupId&gt;org.apache.hadoop&lt;/groupId&gt;
        &lt;artifactId&gt;hadoop-aws&lt;/artifactId&gt;
        &lt;version&gt;3.0.0&lt;/version&gt;
      &lt;/dependency&gt;
    &lt;/dependencies&gt;
  
    &lt;build&gt;
      &lt;sourceDirectory&gt;src/main/scala&lt;/sourceDirectory&gt;
      &lt;testSourceDirectory&gt;src/test/scala&lt;/testSourceDirectory&gt;
      &lt;plugins&gt;
  
        &lt;plugin&gt;
          &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;
          &lt;artifactId&gt;maven-eclipse-plugin&lt;/artifactId&gt;
          &lt;configuration&gt;
            &lt;downloadSources&gt;true&lt;/downloadSources&gt;
            &lt;buildcommands&gt;
              &lt;buildcommand&gt;ch.epfl.lamp.sdt.core.scalabuilder&lt;/buildcommand&gt;
            &lt;/buildcommands&gt;
            &lt;additionalProjectnatures&gt;
              &lt;projectnature&gt;ch.epfl.lamp.sdt.core.scalanature&lt;/projectnature&gt;
            &lt;/additionalProjectnatures&gt;
            &lt;classpathContainers&gt;
              &lt;classpathContainer&gt;org.eclipse.jdt.launching.JRE_CONTAINER&lt;/classpathContainer&gt;
              &lt;classpathContainer&gt;ch.epfl.lamp.sdt.launching.SCALA_CONTAINER&lt;/classpathContainer&gt;
            &lt;/classpathContainers&gt;
          &lt;/configuration&gt;
        &lt;/plugin&gt;
      &lt;/plugins&gt;
    &lt;/build&gt;
    &lt;reporting&gt;
      &lt;plugins&gt;
  
      &lt;/plugins&gt;
    &lt;/reporting&gt;
  &lt;/project&gt;
  
  
  </code></pre>