title: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem not found
body: <p>I am new to the world of Spark and Kubernetes. I built a Spark docker image using the official Spark 3.0.1 bundled with Hadoop 3.2 using the docker-image-tool.sh utility.</p>
  <p>I have also created another docker image for Jupyter notebook and am trying to run spark on Kubernetes in client mode. I first run my Jupyter notebook as a pod, do a port forward using kubectl and access the notebook UI from my system at localhost:8888 . All seems to be working fine. I am able to run commands successfully from the notebook.</p>
  <p>Now I am trying to access Azure Data Lake Gen2 from my notebook using <a href="https://hadoop.apache.org/docs/current/hadoop-azure/abfs.html#Default:_Shared_Key" rel="nofollow noreferrer">Hadoop ABFS connector</a>.
  I am setting the Spark context as below.</p>
  <pre><code>from pyspark import SparkContext, SparkConf
  from pyspark.sql import SparkSession
  # Create Spark config for our Kubernetes based cluster manager
  
  
  sparkConf = SparkConf()
  sparkConf.setMaster(&quot;k8s://https://kubernetes.default.svc.cluster.local:443&quot;)
  sparkConf.setAppName(&quot;spark&quot;)
  sparkConf.set(&quot;spark.kubernetes.container.image&quot;, &quot;&lt;&lt;my_repo&gt;&gt;/spark-py:latest&quot;)
  sparkConf.set(&quot;spark.kubernetes.namespace&quot;, &quot;spark&quot;)
  sparkConf.set(&quot;spark.executor.instances&quot;, &quot;3&quot;)
  sparkConf.set(&quot;spark.executor.cores&quot;, &quot;2&quot;)
  sparkConf.set(&quot;spark.driver.memory&quot;, &quot;512m&quot;)
  sparkConf.set(&quot;spark.executor.memory&quot;, &quot;512m&quot;)
  sparkConf.set(&quot;spark.kubernetes.pyspark.pythonVersion&quot;, &quot;3&quot;)
  sparkConf.set(&quot;spark.kubernetes.authenticate.driver.serviceAccountName&quot;, &quot;spark&quot;)
  sparkConf.set(&quot;spark.kubernetes.authenticate.serviceAccountName&quot;, &quot;spark&quot;)
  sparkConf.set(&quot;spark.driver.port&quot;, &quot;29413&quot;)
  sparkConf.set(&quot;spark.driver.host&quot;, &quot;my-notebook-deployment.spark.svc.cluster.local&quot;)
  
  sparkConf.set(&quot;fs.azure.account.auth.type&quot;, &quot;SharedKey&quot;)
  sparkConf.set(&quot;fs.azure.account.key.&lt;&lt;storage_account_name&gt;&gt;.dfs.core.windows.net&quot;,&quot;&lt;&lt;account_key&gt;&gt;&quot;)
  
  spark = SparkSession.builder.config(conf=sparkConf).getOrCreate()
  </code></pre>
  <p>And then I am running the below command to read a csv file present in the ADLS location</p>
  <pre><code>df = spark.read.csv(&quot;abfss://&lt;&lt;container&gt;&gt;@&lt;&lt;storage_account&gt;&gt;.dfs.core.windows.net/&quot;)
  </code></pre>
  <p>On runnining it I am getting the error
  <em><strong>Py4JJavaError: An error occurred while calling o443.csv.
  : java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem not found</strong></em></p>
  <p>After some research, I found that I would have to explicitly include the hadoop-azure jar for the appropriate classes to be available. I downloaded the jar from <a href="https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-azure/3.2.2/hadoop-azure-3.2.2.jar" rel="nofollow noreferrer">here</a> , put it in the /spark-3.0.1-bin-hadoop3.2/jars folder and built the image again.</p>
  <p><strong>Unfortunately I am still getting this error</strong>. I manually verified that the jar file is indeed present in the docker image and contains the class <code>org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem</code></p>
  <p>I looked at the entrypoint.sh present at <code>spark-3.0.1-bin-hadoop3.2\kubernetes\dockerfiles\spark</code> folder which is the entry point of our spark docker image. It adds all the packages present in the <code>spark-3.0.1-bin-hadoop3.2\jar\</code> folder in the class path.</p>
  <pre><code># If HADOOP_HOME is set and SPARK_DIST_CLASSPATH is not set, set it here so Hadoop jars are available to the executor.
  # It does not set SPARK_DIST_CLASSPATH if already set, to avoid overriding customizations of this value from elsewhere e.g. Docker/K8s.
  if [ -n &quot;${HADOOP_HOME}&quot;  ] &amp;&amp; [ -z &quot;${SPARK_DIST_CLASSPATH}&quot;  ]; then
    export SPARK_DIST_CLASSPATH=&quot;$($HADOOP_HOME/bin/hadoop classpath)&quot;
  fi
  
  if ! [ -z ${HADOOP_CONF_DIR+x} ]; then
    SPARK_CLASSPATH=&quot;$HADOOP_CONF_DIR:$SPARK_CLASSPATH&quot;;
  fi
  </code></pre>
  <p>According to my understanding spark should be able to find the class in its classpath with any addition setJar configuration.</p>
  <p>Can someone please guide me how to resolve this? I might be missing something quite basic here.</p>