title: Not allowing to set spark.sql.warehouse.dir , it should be set statically for cross-session usages using Java
body: <p>I try To learn Spark but in here i found an Exception
  (Not allowing to set spark.sql.warehouse.dir, it should be set statically for cross-session usages and exception is <em>Exception in thread &quot;main&quot; java.lang.NoClassDefFoundError: org/apache/hadoop/hdfs/DistributedFileSystem</em>). I am working on windows 10 pc.</p>
  <p><strong>Main Class:</strong></p>
  <pre><code>package com.rakib;
  import org.apache.spark.sql.Dataset;
  import org.apache.spark.sql.Row;
  import org.apache.spark.sql.SparkSession;
  
  import java.util.logging.Level;
  import java.util.logging.Logger;
  
  public class App {
      public static void main(String[] args) {
  
          System.setProperty(&quot;hadoop.home.dir&quot;, &quot;c:/hadoop&quot;);
          Logger.getLogger(&quot;org.apache&quot;).setLevel(Level.WARNING);
  
          SparkSession session = SparkSession.builder().appName(&quot;SparkSQL&quot;).master(&quot;local[*]&quot;)
                  .config(&quot;spark.sql.warehouse.dir&quot;, &quot;file:///c:/temp/&quot;)
                  .getOrCreate();
  
          Dataset&lt;Row&gt; dataSet = session.read().option(&quot;header&quot;, true).csv(&quot;src/main/resources/student.csv&quot;);
          dataSet.show();
  
          long numberOfRows = dataSet.count();
          System.out.println(&quot;Total : &quot; + numberOfRows);
  
          session.close();
      }
  }
  </code></pre>
  <p><strong>Exception:</strong></p>
  <pre><code>20/08/17 12:12:27 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:///c:/temp/').
  20/08/17 12:12:27 INFO SharedState: Warehouse path is 'file:///c:/temp/'.
  20/08/17 12:12:27 **WARN SharedState: Not allowing to set spark.sql.warehouse.dir or hive.metastore.warehouse.dir in SparkSession's options, it should be set statically for cross-session usages
  Exception in thread &quot;main&quot; java.lang.NoClassDefFoundError: org/apache/hadoop/hdfs/DistributedFileSystem
      at** org.apache.spark.sql.execution.datasources.InMemoryFileIndex$.listLeafFiles(InMemoryFileIndex.scala:316)
      at org.apache.spark.sql.execution.datasources.InMemoryFileIndex$.$anonfun$bulkListLeafFiles$1(InMemoryFileIndex.scala:195)
      at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
      at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
      at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
      at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
      at scala.collection.TraversableLike.map(TraversableLike.scala:238)
      at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
      at scala.collection.AbstractTraversable.map(Traversable.scala:108)
      at org.apache.spark.sql.execution.datasources.InMemoryFileIndex$.bulkListLeafFiles(InMemoryFileIndex.scala:187)
      at org.apache.spark.sql.execution.datasources.InMemoryFileIndex.listLeafFiles(InMemoryFileIndex.scala:135)
      at org.apache.spark.sql.execution.datasources.InMemoryFileIndex.refresh0(InMemoryFileIndex.scala:98)
      at org.apache.spark.sql.execution.datasources.InMemoryFileIndex.&lt;init&gt;(InMemoryFileIndex.scala:70)
      at org.apache.spark.sql.execution.datasources.DataSource.createInMemoryFileIndex(DataSource.scala:561)
      at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:399)
      at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:279)
      at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:268)
      at scala.Option.getOrElse(Option.scala:189)
      at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:268)
      at org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:705)
      at org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:535)
      at com.rakib.App.main(App.java:21)
  Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.DistributedFileSystem
      at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
      ... 22 more
  20/08/17 12:12:28 INFO SparkContext: Invoking stop() from shutdown hook
  20/08/17 12:12:28 INFO SparkUI: Stopped Spark web UI at http://DESKTOP-3147U79:4040
  20/08/17 12:12:28 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
  20/08/17 12:12:28 INFO MemoryStore: MemoryStore cleared
  20/08/17 12:12:28 INFO BlockManager: BlockManager stopped
  20/08/17 12:12:28 INFO BlockManagerMaster: BlockManagerMaster stopped
  20/08/17 12:12:28 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
  20/08/17 12:12:28 INFO SparkContext: Successfully stopped SparkContext
  20/08/17 12:12:28 INFO ShutdownHookManager: Shutdown hook called
  20/08/17 12:12:28 INFO ShutdownHookManager: Deleting directory C:\Users\itc\AppData\Local\Temp\spark-ab377bad-43d5-48ad-a938-b99234abe546
  </code></pre>
  <p><strong>Pom.XML</strong></p>
  <pre><code>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;
  &lt;project xmlns=&quot;http://maven.apache.org/POM/4.0.0&quot;
           xmlns:xsi=&quot;http://www.w3.org/2001/XMLSchema-instance&quot;
           xsi:schemaLocation=&quot;http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd&quot;&gt;
      &lt;modelVersion&gt;4.0.0&lt;/modelVersion&gt;
  
      &lt;groupId&gt;org.example&lt;/groupId&gt;
      &lt;artifactId&gt;Test_One&lt;/artifactId&gt;
      &lt;version&gt;1.0-SNAPSHOT&lt;/version&gt;
      &lt;build&gt;
          &lt;plugins&gt;
              &lt;plugin&gt;
                  &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;
                  &lt;artifactId&gt;maven-compiler-plugin&lt;/artifactId&gt;
                  &lt;configuration&gt;
                      &lt;source&gt;8&lt;/source&gt;
                      &lt;target&gt;8&lt;/target&gt;
                  &lt;/configuration&gt;
              &lt;/plugin&gt;
          &lt;/plugins&gt;
      &lt;/build&gt;
  
      &lt;dependencies&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
              &lt;artifactId&gt;spark-core_2.12&lt;/artifactId&gt;
              &lt;version&gt;3.0.0&lt;/version&gt;
          &lt;/dependency&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
              &lt;artifactId&gt;spark-sql_2.12&lt;/artifactId&gt;
              &lt;version&gt;3.0.0&lt;/version&gt;
          &lt;/dependency&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;org.apache.hadoop&lt;/groupId&gt;
              &lt;artifactId&gt;hadoop-hdfs&lt;/artifactId&gt;
              &lt;version&gt;3.3.0&lt;/version&gt;
          &lt;/dependency&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;log4j&lt;/groupId&gt;
              &lt;artifactId&gt;log4j&lt;/artifactId&gt;
              &lt;version&gt;1.2.17&lt;/version&gt;
          &lt;/dependency&gt;
  
      &lt;/dependencies&gt;
  &lt;/project&gt;
  </code></pre>