title: spark-submit cannot refer jars specified by "--jars" while the code works in the Spark shell?
body: <p>I created an sbt project using intelliJ. I copied the required jdbc jar <code>sqljdbc42.jar</code> in the <code>lib</code> folder of the project. <code>sbt package</code> finished successfully. I started the spark by <code>spark-shell --driver-class-path C:\sqljdbc_6.0\enu\jre8\sqljdbc42.jar</code> on <strong>Windows</strong>.</p>
  
  <pre><code>import org.apache.spark.SparkContext
  import org.apache.spark.SparkContext._
  import org.apache.spark.SparkConf
  import java.sql._
  
  object ConnTest extends App {
    val conf = new SparkConf()
    val sc = new SparkContext(conf.setAppName("Test").setMaster("local[*]"))
  
    // The following four statements work if running interactively in the Spark shell
    val sqlContext = new org.apache.spark.sql.SQLContext(sc)
    val jdbcSqlConn = "jdbc:sqlserver://...;databaseName=...;user=...;password=...;"
    val jdbcDf = sqlContext.read.format("jdbc").options(Map(
          "url" -&gt; jdbcSqlConn,
          "dbtable" -&gt; "testTable"
        )).load()
    jdbcDf.show(10)
  
    sc.stop()
  }
  </code></pre>
  
  <p>However, the following <code>spark-submit</code> commands got the errors.</p>
  
  <pre><code>spark-submit.cmd --class ConnTest --master local[4] .\target\scala-2.11\test_2.11-1.0.jar
  spark-submit.cmd --class ConnTest --master local[4] .\target\scala-2.11\test_2.11-1.0.jar --jars \sqljdbc_6.0\enu\jre8\sqljdbc42.jar
  </code></pre>
  
  <pre>
  Exception in thread "main" java.sql.SQLException: No suitable driver
          at java.sql.DriverManager.getDriver(Unknown Source)
          at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions$$anonfun$7.apply(JDBCOptions.scala:84)
          at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions$$anonfun$7.apply(JDBCOptions.scala:84)
          at scala.Option.getOrElse(Option.scala:121)
          at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.(JDBCOptions.scala:83)
          at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.(JDBCOptions.scala:34)
          at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:32)
          at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:330)
          at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:152)
          at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:125)
          at ConnTest$.delayedEndpoint$ConnTest$1(main.scala:14)
          at ConnTest$delayedInit$body.apply(main.scala:6)
          at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
          at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
          at scala.App$$anonfun$main$1.apply(App.scala:76)
          at scala.App$$anonfun$main$1.apply(App.scala:76)
          at scala.collection.immutable.List.foreach(List.scala:381)
          at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
          at scala.App$class.main(App.scala:76)
          at ConnTest$.main(main.scala:6)
          at ConnTest.main(main.scala)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
          at java.lang.reflect.Method.invoke(Unknown Source)
          at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:743)
          at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
          at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
          at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
          at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  </pre>
  
  <p><strong>Update:</strong>
  The spark codes work and I can see the table content if I run the statements directly in the Spark shell. </p>
  
  <p><strong>Update 2:</strong>
  It did show the following message when run <code>spark-submit</code></p>
  
  <blockquote>
    <p>17/05/15 16:12:30 INFO SparkContext: Added JAR file:/C:/sqljdbc_6.0/enu/jre8/sqljdbc42.jar at spark://10.8.159.130:7587/jars/sqljdbc42.jar with timestamp 1494879150052</p>
  </blockquote>