title: Spark and Zeppelin connecting to WASBS Azure Blob Storage
body: <p>I am trying to run Zeppelin in a Container, alongside Spark, and read files from an Azure Blob Storage.</p>
  
  <p>My Zeppelin Container is configured to send Spark jobs to a Master Server running in a different Container on my Kubernetes Cluster. </p>
  
  <p>When I attempt to read a file from Azure, I get the following error; </p>
  
  <pre><code>java.io.IOException: No FileSystem for scheme: wasbs
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary$1.apply(DataSource.scala:547)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary$1.apply(DataSource.scala:545)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
    at scala.collection.immutable.List.foreach(List.scala:392)
    at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
    at scala.collection.immutable.List.flatMap(List.scala:355)
    at org.apache.spark.sql.execution.datasources.DataSource.org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary(DataSource.scala:545)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:359)
    at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)
    at AccumuloClusterWriter$.main(&lt;console&gt;:62)
  </code></pre>
  
  <p>if I then run a notebook with the following code in it, </p>
  
  <pre><code>sc.hadoopConfiguration.set("fs.wasb.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
  sc.hadoopConfiguration.set("fs.AbstractFileSystem.wasb.impl", "org.apache.hadoop.fs.azure.Wasb")
  
  sc.hadoopConfiguration.set("fs.wasbs.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
  sc.hadoopConfiguration.set("fs.AbstractFileSystem.wasbs.impl", "org.apache.hadoop.fs.azure.Wasbs")
  </code></pre>
  
  <p>I start getting the following error:
  <code>java.lang.NoClassDefFoundError: org/apache/hadoop/fs/StreamCapabilities</code></p>
  
  <p>I am running Zeppelin 0.8.1 alongside Spark 2.4.3</p>
  
  <p>my CLASSPATH is as follows;
  <code>:/jars/hadoop-azure-2.7.0.jar:/jars/azure-storage-3.1.0.jar:</code></p>
  
  <p>the hadoop-azure and azure-storage jars are in my Spark jars directory.</p>
  
  <p>One thing that I am confused about is whether my code is being run ON the Zeppelin Container, or if it is actually being run on one of the Cluster Nodes. I have been trying to correct this issue on the Zeppelin Container, but I wonder if the misconfiguration is actually on the Spark Master Container instead. </p>
  
  <p>Any direction and assistance is greatly appreciated at this point. </p>