title: Spark 3.1.3 with spark cassandra connector 3.1.0 is giving ClassNotFoundException on spark-submit even with jars attached
body: <p>I have used Spark 3.1.3 to connect Astra as well as Cassandra local server, but I am getting <code>java.lang.ClassNotFoundException</code> error on <code>spark-submit</code>. I have confirmed that same is happening with spark above 3.x, this code works fine with spark 2.4.2.</p>
  <p>Here is my <code>Main.scala</code>:</p>
  <pre><code>import com.datastax.spark.connector.{toSparkContextFunctions}
  import org.apache.spark.sql.SparkSession
  import org.apache.spark.{SparkConf, SparkContext}
  
  
  object Main {
  
    def main(args: Array[String]) = {
      val conf = new SparkConf()
        .set(&quot;spark.cassandra.connection.config.cloud.path&quot;,&quot;/path/astradb-secure-connect.zip&quot;)
        .set(&quot;spark.cassandra.auth.username&quot;,&quot;client-id&quot;)
        .set(&quot;spark.cassandra.auth.password&quot;,&quot;client-secret&quot;)
        .setAppName(&quot;SparkTest&quot;)
      val sparkCTX = new SparkContext(conf)
  
      val sparkSess = SparkSession.builder.appName(&quot;MyStream&quot;).getOrCreate()
      sparkCTX.setLogLevel(&quot;Error&quot;)
      println(&quot;\n\n\n\n************\n\n\n\n&quot;)
  
      val Rdd = sparkCTX.cassandraTable(&quot;my_keyspace&quot;, &quot;accounts&quot;) **// Exact Line of Error**
      Rdd.foreach(s =&gt; {
        println(s)
      })
    }
  }
  </code></pre>
  <p>My <code>build.sbt</code> looks like:</p>
  <pre><code>name := &quot;Synchronization&quot;
  
  version := &quot;1.0-SNAPSHOT&quot;
  
  
  scalaVersion := &quot;2.12.15&quot;
  
  idePackagePrefix := Some(&quot;info.myapp.synchronization&quot;)
  
  
  val sparkVersion = &quot;3.1.3&quot;
  
  libraryDependencies ++= Seq(
    &quot;org.apache.spark&quot; %% &quot;spark-core&quot; % sparkVersion,
    &quot;org.apache.spark&quot; %% &quot;spark-sql&quot; % sparkVersion,
    &quot;org.apache.spark&quot; %% &quot;spark-mllib&quot; % sparkVersion,
    &quot;org.apache.spark&quot; %% &quot;spark-streaming&quot; % sparkVersion,
    &quot;io.spray&quot; %%  &quot;spray-json&quot; % &quot;1.3.6&quot;,
    &quot;org.scalaj&quot; %% &quot;scalaj-http&quot; % &quot;2.4.2&quot;
  )
  
  libraryDependencies += &quot;com.datastax.spark&quot; %% &quot;spark-cassandra-connector&quot; % &quot;3.1.0&quot; % &quot;provided&quot;
  
  libraryDependencies += &quot;com.twitter&quot; % &quot;jsr166e&quot; % &quot;1.1.0&quot;
  
  libraryDependencies += &quot;net.liftweb&quot; %% &quot;lift-json&quot; % &quot;3.4.3&quot;
  
  libraryDependencies += &quot;com.sun.mail&quot; % &quot;javax.mail&quot; % &quot;1.6.2&quot;
  
  libraryDependencies += &quot;com.typesafe.akka&quot; %% &quot;akka-stream&quot; % &quot;2.5.22&quot;
  
  libraryDependencies += &quot;com.github.jurajburian&quot; %% &quot;mailer&quot; % &quot;1.2.4&quot;
  </code></pre>
  <p>The code I am using to run this is:</p>
  <pre><code>$ sbt package
  $ spark-submit --class &quot;Main&quot; --jars $(echo localDependencies/*.jar | tr ' ' ',') target/scala-2.12/*jar
  </code></pre>
  <p>My <code>localDependency</code> folder contains jars downloaded from <code>mvnrepository</code> and looks like:</p>
  <p><a href="https://i.stack.imgur.com/4Mihe.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/4Mihe.png" alt="enter image description here" /></a></p>
  <p>My error goes like:</p>
  <pre><code>Exception in thread &quot;main&quot; java.lang.NoClassDefFoundError: com/datastax/spark/connector/CassandraRow
          at Main$.main(Main.scala:20)
          at Main.main(Main.scala)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:498)
          at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
          at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
          at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
          at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
          at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
          at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039)
          at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048)
          at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  Caused by: java.lang.ClassNotFoundException: com.datastax.spark.connector.CassandraRow
          at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
          ... 14 more
  </code></pre>
  <p>Please help me fix the issue, or please provide the working versions combinations</p>