title: pyspark read file from AWS S3 not working
body: <p>I installed spark and hadoop using brew:</p>
  
  <pre><code>brew info hadoop #=&gt; hadoop: stable 3.1.2
  brew info apache-spark #=&gt; apache-spark: stable 2.4.4
  </code></pre>
  
  <p>I am now trying to load a csv file hosted on s3, tried many different ways without success (here is one of them):</p>
  
  <pre><code>import pyspark
  
  conf = pyspark.SparkConf()
  
  sc = pyspark.SparkContext('local[4]', conf=conf)
  sc._jsc.hadoopConfiguration().set("fs.s3a.awsAccessKeyId", "key here")
  sc._jsc.hadoopConfiguration().set("fs.s3a.awsSecretAccessKey", "key here")
  
  sql = pyspark.SQLContext(sc)
  df = sql.read.csv('s3a://pilo/fi/data_2014_1.csv')
  </code></pre>
  
  <p>I get this error:</p>
  
  <pre><code>19/09/17 14:34:52 WARN FileStreamSink: Error while looking for metadata directory.
  Traceback (most recent call last):
    File "/Users/cyrusghazanfar/anaconda3/lib/python3.6/runpy.py", line 193, in _run_module_as_main
      "__main__", mod_spec)
    File "/Users/cyrusghazanfar/anaconda3/lib/python3.6/runpy.py", line 85, in _run_code
      exec(code, run_globals)
    File "/Users/cyrusghazanfar/Desktop/startup-studio/pilota_project/pilota_ml/ingestion/clients/aws_s3.py", line 51, in &lt;module&gt;
      df = sql.read.csv('s3a://pilo/fi/data_2014_1.csv')
    File "/Users/cyrusghazanfar/Desktop/startup-studio/pilota_project/pilota_ml/env/lib/python3.6/site-packages/pyspark/sql/readwriter.py", line 476, in csv
      return self._df(self._jreader.csv(self._spark._sc._jvm.PythonUtils.toSeq(path)))
    File "/Users/cyrusghazanfar/Desktop/startup-studio/pilota_project/pilota_ml/env/lib/python3.6/site-packages/py4j/java_gateway.py", line 1257, in __call__
      answer, self.gateway_client, self.target_id, self.name)
    File "/Users/cyrusghazanfar/Desktop/startup-studio/pilota_project/pilota_ml/env/lib/python3.6/site-packages/pyspark/sql/utils.py", line 63, in deco
      return f(*a, **kw)
    File "/Users/cyrusghazanfar/Desktop/startup-studio/pilota_project/pilota_ml/env/lib/python3.6/site-packages/py4j/protocol.py", line 328, in get_return_value
      format(target_id, ".", name), value)
  py4j.protocol.Py4JJavaError: An error occurred while calling o26.csv.
  : java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
          at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)
          at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2654)
          at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
          at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
          at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
          at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
          at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
          at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
          at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary$1.apply(DataSource.scala:547)
          at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary$1.apply(DataSource.scala:545)
          at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
          at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
          at scala.collection.immutable.List.foreach(List.scala:392)
          at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
          at scala.collection.immutable.List.flatMap(List.scala:355)
          at org.apache.spark.sql.execution.datasources.DataSource.org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary(DataSource.scala:545)
          at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:359)
          at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
          at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
          at org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:618)
          at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.base/java.lang.reflect.Method.invoke(Method.java:567)
          at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
          at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
          at py4j.Gateway.invoke(Gateway.java:282)
          at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
          at py4j.commands.CallCommand.execute(CallCommand.java:79)
          at py4j.GatewayConnection.run(GatewayConnection.java:238)
          at java.base/java.lang.Thread.run(Thread.java:835)
  Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
          at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
          at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
          ... 30 more
  </code></pre>
  
  <p>it looks like it has something to do with my aws s3 credentials but not sure how to set that up. (Currently my awe credentiials are in my bash_profile) HELP PLEASE.</p>