title: How come PySpark can't find my SPARK_HOME
body: <p>I am trying to run a Jupyter notebook from Archives Unleashed locally on my machine. When the notebooks builds PySpark, it runs into this exception:</p>
  <p><code>Exception: Unable to find py4j, your SPARK_HOME may not be configured correctly</code></p>
  <p>Any idea on how I can configure the SPARK_HOME?</p>
  <p>I have tried to run the notebook in a clean conda environment. This is the full notebook until the error:</p>
  <pre><code>%%capture
  
  !wget &quot;https://github.com/archivesunleashed/aut/releases/download/aut-0.50.0/aut-0.50.0.zip&quot;
  !wget &quot;https://github.com/archivesunleashed/aut/releases/download/aut-0.50.0/aut-0.50.0-fatjar.jar&quot;
  
  !ls
  
  %%capture
  
  !apt-get update
  !apt-get install -y openjdk-8-jdk-headless -qq 
  !apt-get install maven -qq
  
  !curl -L &quot;https://archive.apache.org/dist/spark/spark-2.4.5/spark-2.4.5-bin-hadoop2.7.tgz&quot; &gt; spark-2.4.5-bin-hadoop2.7.tgz
  !tar -xvf spark-2.4.5-bin-hadoop2.7.tgz
  !pip install -q findspark
  
  import os
  os.environ[&quot;JAVA_HOME&quot;] = &quot;/usr/lib/jvm/java-8-openjdk-amd64&quot;
  os.environ[&quot;SPARK_HOME&quot;] = &quot;/content/spark-2.4.5-bin-hadoop2.7&quot;
  os.environ['PYSPARK_SUBMIT_ARGS'] = '--jars aut-0.50.0-fatjar.jar --py-files aut-0.50.0.zip pyspark-shell'
  
  import findspark
  findspark.init()
  import pyspark
  sc = pyspark.SparkContext()
  from pyspark.sql import SQLContext
  sqlContext = SQLContext(sc)
  </code></pre>
  <p>This is what I get returned:</p>
  <pre><code>---------------------------------------------------------------------------
  IndexError                                Traceback (most recent call last)
  ~/opt/miniconda3/envs/arc/lib/python3.8/site-packages/findspark.py in init(spark_home, python_path, edit_rc, edit_profile)
      142     try:
  --&gt; 143         py4j = glob(os.path.join(spark_python, &quot;lib&quot;, &quot;py4j-*.zip&quot;))[0]
      144     except IndexError:
  
  IndexError: list index out of range
  
  During handling of the above exception, another exception occurred:
  
  Exception                                 Traceback (most recent call last)
  &lt;ipython-input-2-03823ebc9ad8&gt; in &lt;module&gt;
        1 import findspark
  ----&gt; 2 findspark.init()
        3 import pyspark
        4 sc = pyspark.SparkContext()
        5 from pyspark.sql import SQLContext
  
  ~/opt/miniconda3/envs/arc/lib/python3.8/site-packages/findspark.py in init(spark_home, python_path, edit_rc, edit_profile)
      143         py4j = glob(os.path.join(spark_python, &quot;lib&quot;, &quot;py4j-*.zip&quot;))[0]
      144     except IndexError:
  --&gt; 145         raise Exception(
      146             &quot;Unable to find py4j, your SPARK_HOME may not be configured correctly&quot;
      147         )
  
  Exception: Unable to find py4j, your SPARK_HOME may not be configured correctly
  </code></pre>