title: Spark/Scala - Error creating DataFrame from Json: java.lang.NoSuchMethodError: org.apache.spark.sql.DataFrameReader.json
body: <p>I am new to Spark and Scala. I am trying to create a Dataframe from a JSONArray. Below is my code:</p>
  
  <pre><code> public class JSONParse{
      public JSONArray actionItems() {
          JSONParser parser = new JSONParser();
          JSONArray results = null;
          try {
              JSONObject obj = (JSONObject) parser.parse(new FileReader("/data/home/actionitems.json"));
              JSONObject obj2 = (JSONObject) obj.get("d");
              results = (JSONArray) obj2.get("results");
              System.out.println(results);
  
          } catch (Exception e) {
              e.printStackTrace();
          }
          return results;
      }
      }
  
  object driver {
    val parse = new JsonParse
    val conf = new SparkConf().setAppName("test")
    val sc = new SparkContext(conf)
    sc.setLogLevel("ERROR")
    val hiveContext = new HiveContext(sc)
    val sqlContext = new SQLContext(sc)
  
    def main(args: Array[String]): Unit = {
      val actionItemsRDD = sc.parallelize(Seq(parse.actionItems.toString))
      val df: DataFrame = hiveContext.read.json(actionItemsRDD)
      df.show
      println("number of records: "+df.count)
      }
  }
  </code></pre>
  
  <p>The Java class <code>JsonParse</code> reads the json from a file and returns the <code>JSONArray</code> to the scala Object <code>driver</code>. In <code>driver</code> , I convert the Json String to an <code>RDD</code> and then create the <code>Dataframe</code> using <code>hiveContext.read.json(actionItemsRDD)</code>. I build using maven and there are no build errors.</p>
  
  <p>However, when I run the jar, I get this error:
  <code>Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.sql.DataFrameReader.json(Lorg/apache/spark/rdd/RDD;)Lorg/apache/spark/sql/Dataset;</code></p>
  
  <p>It throws the exception at the <code>hiveContext.read.json</code> line. I've done this before and had no issues. I am also using the same dependencies from my previous attempt. Below is my pom.xml:</p>
  
  <pre><code>&lt;project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"&gt;
    &lt;modelVersion&gt;4.0.0&lt;/modelVersion&gt;
    &lt;groupId&gt;json&lt;/groupId&gt;
    &lt;artifactId&gt;test&lt;/artifactId&gt;
    &lt;version&gt;1.0-SNAPSHOT&lt;/version&gt;
    &lt;name&gt;${project.artifactId}&lt;/name&gt;
    &lt;build&gt;
      &lt;sourceDirectory&gt;src&lt;/sourceDirectory&gt;
      &lt;plugins&gt;
        &lt;plugin&gt;
          &lt;groupId&gt;net.alchim31.maven&lt;/groupId&gt;
          &lt;artifactId&gt;scala-maven-plugin&lt;/artifactId&gt;
          &lt;version&gt;3.2.2&lt;/version&gt;
          &lt;executions&gt;
            &lt;execution&gt;
              &lt;id&gt;scala-compile-first&lt;/id&gt;
              &lt;phase&gt;process-resources&lt;/phase&gt;
              &lt;goals&gt;
                &lt;goal&gt;compile&lt;/goal&gt;
              &lt;/goals&gt;
            &lt;/execution&gt;
  
            &lt;execution&gt;
              &lt;id&gt;scala-test-compile&lt;/id&gt;
              &lt;phase&gt;process-test-resources&lt;/phase&gt;
              &lt;goals&gt;
                &lt;goal&gt;testCompile&lt;/goal&gt;
              &lt;/goals&gt;
            &lt;/execution&gt;
          &lt;/executions&gt;
        &lt;/plugin&gt;
        &lt;plugin&gt;
          &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;
          &lt;artifactId&gt;maven-shade-plugin&lt;/artifactId&gt;
          &lt;version&gt;2.4.1&lt;/version&gt;
          &lt;executions&gt;
            &lt;execution&gt;
              &lt;phase&gt;package&lt;/phase&gt;
              &lt;goals&gt;
                &lt;goal&gt;shade&lt;/goal&gt;
              &lt;/goals&gt;
              &lt;configuration&gt;
                &lt;relocations&gt;
                  &lt;relocation&gt;
                    &lt;pattern&gt;org.apache.http&lt;/pattern&gt;
                    &lt;shadedPattern&gt;org.shaded.apache.http&lt;/shadedPattern&gt;
                  &lt;/relocation&gt;
                &lt;/relocations&gt;
                &lt;filters&gt;
                  &lt;filter&gt;
                    &lt;artifact&gt;*:*&lt;/artifact&gt;
                      &lt;excludes&gt;
                          &lt;exclude&gt;META-INF/*.SF&lt;/exclude&gt;
                          &lt;exclude&gt;META-INF/*.DSA&lt;/exclude&gt;
                          &lt;exclude&gt;META-INF/*.RSA&lt;/exclude&gt;
                      &lt;/excludes&gt;
                  &lt;/filter&gt;
                &lt;/filters&gt;
                &lt;shadedArtifactAttached&gt;true&lt;/shadedArtifactAttached&gt;
                &lt;shadedClassifierName&gt;shaded&lt;/shadedClassifierName&gt;
              &lt;/configuration&gt;
            &lt;/execution&gt;
          &lt;/executions&gt;
        &lt;/plugin&gt;
      &lt;/plugins&gt;
    &lt;/build&gt;
    &lt;dependencies&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;com.databricks&lt;/groupId&gt;
              &lt;artifactId&gt;spark-csv_2.11&lt;/artifactId&gt;
              &lt;version&gt;1.4.0&lt;/version&gt;
          &lt;/dependency&gt;
  
          &lt;!-- https://mvnrepository.com/artifact/org.apache.spark/spark-hive_2.10 --&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
              &lt;artifactId&gt;spark-hive_2.10&lt;/artifactId&gt;
              &lt;version&gt;1.6.0&lt;/version&gt;                
              &lt;exclusions&gt;
                  &lt;exclusion&gt;
                      &lt;groupId&gt;org.slf4j&lt;/groupId&gt;
                      &lt;artifactId&gt;slf4j-api&lt;/artifactId&gt;
                  &lt;/exclusion&gt;
              &lt;/exclusions&gt;
          &lt;/dependency&gt;
  
          &lt;dependency&gt;
              &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
              &lt;artifactId&gt;spark-sql_2.10&lt;/artifactId&gt;
              &lt;version&gt;1.6.0&lt;/version&gt;             
              &lt;exclusions&gt;
                  &lt;exclusion&gt;
                      &lt;groupId&gt;org.slf4j&lt;/groupId&gt;
                      &lt;artifactId&gt;slf4j-api&lt;/artifactId&gt;
                  &lt;/exclusion&gt;
                  &lt;exclusion&gt;
                      &lt;groupId&gt;org.slf4j&lt;/groupId&gt;
                      &lt;artifactId&gt;jcl-over-slf4j&lt;/artifactId&gt;
                  &lt;/exclusion&gt;
              &lt;/exclusions&gt;
          &lt;/dependency&gt;
  
          &lt;!-- https://mvnrepository.com/artifact/org.scala-lang/scala-library --&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;org.scala-lang&lt;/groupId&gt;
              &lt;artifactId&gt;scala-library&lt;/artifactId&gt;
              &lt;version&gt;2.10.6&lt;/version&gt;
          &lt;/dependency&gt;
  
          &lt;!-- https://mvnrepository.com/artifact/org.jodd/jodd --&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;org.jodd&lt;/groupId&gt;
              &lt;artifactId&gt;jodd&lt;/artifactId&gt;
              &lt;version&gt;3.4.0&lt;/version&gt;
              &lt;type&gt;pom&lt;/type&gt;
          &lt;/dependency&gt;
          &lt;!-- https://mvnrepository.com/artifact/org.json/json --&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;org.json&lt;/groupId&gt;
              &lt;artifactId&gt;json&lt;/artifactId&gt;
              &lt;version&gt;20170516&lt;/version&gt;
          &lt;/dependency&gt;
          &lt;!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient --&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;org.apache.httpcomponents&lt;/groupId&gt;
              &lt;artifactId&gt;httpclient&lt;/artifactId&gt;
              &lt;version&gt;4.5.3&lt;/version&gt;
          &lt;/dependency&gt;
          &lt;!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/httpcore --&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;org.apache.httpcomponents&lt;/groupId&gt;
              &lt;artifactId&gt;httpcore&lt;/artifactId&gt;
              &lt;version&gt;4.4.4&lt;/version&gt;
          &lt;/dependency&gt;
          &lt;!-- https://mvnrepository.com/artifact/com.googlecode.json-simple/json-simple --&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;com.googlecode.json-simple&lt;/groupId&gt;
              &lt;artifactId&gt;json-simple&lt;/artifactId&gt;
              &lt;version&gt;1.1.1&lt;/version&gt;
          &lt;/dependency&gt;
  
          &lt;!-- https://mvnrepository.com/artifact/org.threeten/threetenbp --&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;org.threeten&lt;/groupId&gt;
              &lt;artifactId&gt;threetenbp&lt;/artifactId&gt;
              &lt;version&gt;1.3.3&lt;/version&gt;
          &lt;/dependency&gt;
      &lt;/dependencies&gt;
  &lt;/project&gt;
  </code></pre>
  
  <p>Not sure why this error is showing up and I'm not able to resolve it. Any help would be appreciated. Thank you!</p>