title: Spark SQL toDF method fails with java.lang.NoSuchMethodError
body: <h1>Objective</h1>
  
  <p>Understand the cause and the solution to the problem. The problem happens when using spark-submit. Appreciate the help.</p>
  
  <pre><code>spark-submit --class "AuctionDataFrame" --master spark://&lt;hostname&gt;:7077 auction-project_2.11-1.0.jar
  </code></pre>
  
  <p>It does not cause an error when running line by line in a spark-shell.</p>
  
  <pre><code>...
  scala&gt;     val auctionsDF = auctionsRDD.toDF()
  auctionsDF: org.apache.spark.sql.DataFrame = [aucid: string, bid: float, bidtime: float, bidder: string, bidrate: int, openbid: float, price: float, itemtype: string, dtl: int]
  scala&gt; auctionsDF.printSchema()
  root
   |-- aucid: string (nullable = true)
   |-- bid: float (nullable = false)
   |-- bidtime: float (nullable = false)
   |-- bidder: string (nullable = true)
   |-- bidrate: integer (nullable = false)
   |-- openbid: float (nullable = false)
   |-- price: float (nullable = false)
   |-- itemtype: string (nullable = true)
   |-- dtl: integer (nullable = false)
  </code></pre>
  
  <h2>Problem</h2>
  
  <p>Calling toDF method to convert RDD into DataFrame causes the error.</p>
  
  <pre><code>Exception in thread "main" java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaMirrors$JavaMirror;
      at AuctionDataFrame$.main(AuctionDataFrame.scala:52)
      at AuctionDataFrame.main(AuctionDataFrame.scala)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
      at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  </code></pre>
  
  <h2>Code</h2>
  
  <pre><code>case class Auctions(
    aucid: String,
    bid: Float,
    bidtime: Float,
    bidder: String,
    bidrate: Int,
    openbid: Float,
    price: Float,
    itemtype: String,
    dtl: Int)
  
  object AuctionDataFrame {
    val AUCID = 0
    val BID = 1
    val BIDTIME = 2
    val BIDDER = 3
    val BIDRATE = 4
    val OPENBID = 5
    val PRICE = 6
    val ITEMTYPE = 7
    val DTL = 8
  
    def main(args: Array[String]) {
      val conf = new SparkConf().setAppName("AuctionDataFrame")
      val sc = new SparkContext(conf)
      val sqlContext = new org.apache.spark.sql.SQLContext(sc)
      import sqlContext.implicits._
  
      val inputRDD = sc.textFile("/user/wynadmin/auctiondata.csv").map(_.split(","))
      val auctionsRDD = inputRDD.map(a =&gt;
        Auctions(
          a(AUCID),
          a(BID).toFloat,
          a(BIDTIME).toFloat,
          a(BIDDER),
          a(BIDRATE).toInt,
          a(OPENBID).toFloat,
          a(PRICE).toFloat,
          a(ITEMTYPE),
          a(DTL).toInt))
      val auctionsDF = auctionsRDD.toDF()  // &lt;--- line 52 causing the error.
  }
  </code></pre>
  
  <p>build.sbt</p>
  
  <pre><code>name := "Auction Project"
  
  version := "1.0"
  
  scalaVersion := "2.11.8"
  //scalaVersion := "2.10.6"
  
  /* 
  libraryDependencies ++= Seq(
      "org.apache.spark" %% "spark-core" % "1.6.2",
      "org.apache.spark" %% "spark-sql" % "1.6.2",
      "org.apache.spark" %% "spark-mllib" % "1.6.2"
  )
  */
  
  libraryDependencies ++= Seq(
      "org.apache.spark" %% "spark-core" % "1.6.2" % "provided",
      "org.apache.spark" %% "spark-sql" % "1.6.2" % "provided",
      "org.apache.spark" %% "spark-mllib" % "1.6.2" % "provided"
  )
  </code></pre>
  
  <h2>Environment</h2>
  
  <p>Spark on Ubuntu 14.04:</p>
  
  <pre><code>      ____              __
       / __/__  ___ _____/ /__
      _\ \/ _ \/ _ `/ __/  '_/
     /___/ .__/\_,_/_/ /_/\_\   version 1.6.2
        /_/
  
  Using Scala version 2.11.7 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_92)
  </code></pre>
  
  <p>sbt on Windows:</p>
  
  <pre><code>D:\&gt;sbt sbtVersion
  [info] Set current project to root (in build file:/D:/)
  [info] 0.13.12
  </code></pre>
  
  <h2>Research</h2>
  
  <p>Looked into similar issues which suggest Scala version incompatibility that compiled Spark. </p>
  
  <ul>
  <li><a href="http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-4-RDD-to-DF-fails-with-toDF-td23499.html" rel="nofollow noreferrer">Spark 1.4 RDD to DF fails with toDF()</a></li>
  <li><a href="https://stackoverflow.com/questions/32747405/spark-mllib-example-nosuchmethoderror-org-apache-spark-sql-sqlcontext-createda">Spark MLlib example, NoSuchMethodError:</a></li>
  </ul>
  
  <p>Hence changed the Scala version in build.sbt to 2.10 which created 2.10 jar, but the error persisted. Using % provided or not does not change the error. </p>
  
  <pre><code>scalaVersion := "2.10.6"
  </code></pre>