title: How do I get Anaconda Pyspark Jupyter to work with S3 under Windows
body: <p>My goal is to have a working jupyter notebook with pyspark and s3/s3a support. I've somehow gotten to the point of having pyspark work but I cannot seem to get it to support s3 or s3a no matter how much I google. Most of the information is for unix/linux, or assumes too much. Please help. The following are the steps I have taken thus far.</p>
  
  <p>Following <a href="https://medium.com/@naomi.fridman/install-pyspark-to-run-on-jupyter-notebook-on-windows-4ec2009de21f" rel="nofollow noreferrer">https://medium.com/@naomi.fridman/install-pyspark-to-run-on-jupyter-notebook-on-windows-4ec2009de21f</a></p>
  
  <ol>
  <li>Installed JAVA 8 (Windowsx64 version)</li>
  <li>Set the JAVA_HOME variable and added PATH variable as per above site (with the newer version paths)</li>
  <li>Downloaded Apache Spark 2.4.4 (Aug 30 2019) PreBuilt for Apache Hadoop 2.7</li>
  <li>Unzipped it to C:\Spark (got spark-2.4.4-bin-hadoop2.7.tar)</li>
  <li>Unzipped that to C:\Spark (got folder spark-2.4.4-bin-hadoop2.7)</li>
  <li>Set more Environment variables and added the bin folder to the PATH variable as per site above</li>
  <li>Downloaded the windows 64-bit winutils.exe file per the site above ran the cmd command it told me to run. I left a copy of it in the C:\Spark\spark-2.4.4-bin-hadoop2.7 folder and its bin folder.</li>
  <li>Ran and tested Anaconda spark - success</li>
  </ol>
  
  <p>Next was to get pyspark working within Jupyter</p>
  
  <ol>
  <li>In Anaconda prompt I installed findspark "conda install -c conda-forge findspark"</li>
  <li>After running</li>
  </ol>
  
  <pre><code>findspark.init()
  findspark.find()
  import pyspark
  findspark.find()
  
  from pyspark import SparkContext, SparkConf
  from pyspark.sql import SparkSession
  conf = pyspark.SparkConf().setAppName('appName').setMaster('local')
  sc = pyspark.SparkContext(conf=conf)
  spark = SparkSession(sc)
  </code></pre>
  
  <p>Hurray, pyspark running under Jupyter, but I need this auto-run as part of a Kernel, of which I only have the option of python3 right now.</p>
  
  <ol start="3">
  <li>So I google things and figure out that I need to create a dir under "C:\ProgramData\Anaconda3\share\jupyter\kernels" so I copy and rename the existing python3 folder and just change the "display_name": to "pyspark", I am supposed to be able to add a line there like "--profile=myCustomProfile" but that kept failing (no module found?) so I gave up on profiles (help here too plz?)</li>
  <li>I then made a 00-pyspark-setup.py file under "C:\Users\Admin.ipython\profile_default\startup" I read it is a sort of autoexec.bat kind of thing so I put the above findspark and sparkContext code in there</li>
  </ol>
  
  <hr>
  
  <p>Here is where I am stuck. The major issue being I cannot access s3/s3a/s3n via a spark.read.load("s3:/xx") command, I also cannot run things like %%configure -f but this is a smaller issue.</p>
  
  <p>I have read solutions via boto3 but does that allow spark.read.load directly from s3? Doesn't sound like it. I could be wrong</p>
  
  <p>The other involves downloading aws-java-sdk-1.7.4.jar and hadoop-aws-2.7.1.jar from maven adding these lines of somewhere (or 2 different places). This should give me the result I want.</p>
  
  <pre><code>spark.executor.extraClassPath      C:\spark\maven\aws-java-sdk-1.7.4.jar;C:\spark\maven\hadoop-aws-2.7.1.jar
  spark.driver.extraClassPath       C:\spark\maven\aws-java-sdk-1.7.4.jar;C:\spark\maven\hadoop-aws-2.7.1.jar
  
  hadoop_conf = spark.sparkContext._jsc.hadoopConfiguration()
  hadoop_conf.set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
  hadoop_conf.set("fs.s3a.access.key", "aKey")
  hadoop_conf.set("fs.s3a.secret.key", "sKey"
  </code></pre>
  
  <p>Can anyone help me? I think I have to create my own "C:\spark\spark-2.4.4-bin-hadoop2.7\conf\spark-defaults.conf" and add the spark.driver codes in there, but it is not working, my error from Jupyter is still and most of the answers online seem like they are referencing linux paths, but here I need to specify the drive do I not?</p>
  
  <p>Py4JJavaError: An error occurred while calling o84.csv.
  : java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
  From
  C:\spark\spark-2.4.4-bin-hadoop2.7\python\lib\py4j-0.10.7-src.zip\py4j\protocol.py in get_return_value(answer, gateway_client, target_id, name)</p>