title: Run Spark on Java 10
body: <p><strong>my user case</strong>: We run Spark cluster on Mesos, since our Mesos cluster is also hosting other frameworks such as Storm, Cassandra, we had incidents where  Spark job over-utilizes CPU which caused resource contention with other frameworks.</p>
  
  <p><strong>objective</strong> : run un-modularized spark application (jar is compiled with java 8 compatible sbt compiler) on Java 10 to leverage Java 10 container support. Related link: <a href="https://bugs.openjdk.java.net/browse/JDK-8146115" rel="nofollow noreferrer">https://bugs.openjdk.java.net/browse/JDK-8146115</a></p>
  
  <p>After reading some readings about Java 9, this is my imaginary <em>happy-path</em>:</p>
  
  <ol>
  <li>Point <code>JAVA_HOME</code> to Java 10, </li>
  <li>Run my spark job, resolve <code>ClassNotFoundException</code>, lookup the missing modules in oracle documentation, for example (java.sql <a href="https://docs.oracle.com/javase/9/docs/api/java.sql-summary.html" rel="nofollow noreferrer">https://docs.oracle.com/javase/9/docs/api/java.sql-summary.html</a>) module is missing, add “spark.executor.extraJavaOptions --add-modules java.se.ee” -> conf/spark-defaults.conf</li>
  <li>Repeat step 2 until no more exceptions are thrown </li>
  </ol>
  
  <p>however I found this,</p>
  
  <p>Warning: Local jar ***/java.se.ee does not exist, skipping.</p>
  
  <pre><code>java.lang.ClassNotFoundException: com.**.spark.Main at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:466)
  at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:566)
  at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:499)
  at java.base/java.lang.Class.forName0(Native Method)
  at java.base/java.lang.Class.forName(Class.java:374)
  at org.apache.spark.util.Utils$.classForName(Utils.scala:233)
  at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:732)
  at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
  at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
  at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
  at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  </code></pre>
  
  <p><strong>This is what I have observed after turning verbose mode</strong></p>
  
  <p>...</p>
  
  <p>Using properties file: /tmp/spark-2.2.2-bin-hadoop2.6/conf/spark-defaults.conf</p>
  
  <p>Adding default property: spark.eventLog.enabled=true
  Adding default property: spark.eventLog.dir=hdfs://***
  Adding default property: spark.executor.extraJavaOptions=--add-modules java.se.ee</p>
  
  <p>Parsed arguments:</p>
  
  <pre><code>master                 mesos://localhost:10017
  deployMode             cluster
  executorMemory         16G
  executorCores           2
  totalExecutorCores     50
  propertiesFile         /tmp/spark-2.2.2-bin-hadoop2.6/conf/spark- 
  defaults.conf 
  driverMemory           4G
  driverCores             1
  driverExtraClassPath   null
  driverExtraLibraryPath null
  driverExtraJavaOptions null
  supervise               false
  queue                   null
  numExecutors           null
  files                   null
  pyFiles                 null
  archives               null
  mainClass               com.**.spark.Main
  primaryResource         ***.jar
  name                   ***
  childArgs               [***]
  jars                   null
  packages               null
  packagesExclusions     null
  repositories           null
  verbose                true
  
  Spark properties used, including those specified through
  --conf and those from the properties file /tmp/spark-2.2.2-bin-hadoop2.6/conf/spark-defaults.conf:
  (spark.mesos.uris,hdfs:///***/tmpqIx6x2)
  (spark.driver.memory,4G)
  (spark.eventLog.enabled,true)
  (spark.executor.extraJavaOptions,--add-modules java.se.ee)
  (spark.executor.uri,***/spark-2.2.2-bin-hadoop2.6.tgz)
  (spark.eventLog.dir,hdfs://***)
  </code></pre>
  
  <p>The warning was printed here:</p>
  
  <p><a href="https://github.com/apache/spark/blob/5264164a67df498b73facae207eda12ee133be7d/core/src/main/scala/org/apache/spark/deploy/worker/DriverWrapper.scala#L101" rel="nofollow noreferrer">https://github.com/apache/spark/blob/5264164a67df498b73facae207eda12ee133be7d/core/src/main/scala/org/apache/spark/deploy/worker/DriverWrapper.scala#L101</a></p>
  
  <p><a href="https://github.com/apache/spark/blob/5264164a67df498b73facae207eda12ee133be7d/core/src/main/scala/org/apache/spark/deploy/DependencyUtils.scala#L76" rel="nofollow noreferrer">https://github.com/apache/spark/blob/5264164a67df498b73facae207eda12ee133be7d/core/src/main/scala/org/apache/spark/deploy/DependencyUtils.scala#L76</a></p>
  
  <p>After reading the source code. Seems to me that spark-submit does not understand --add-modules option so it treat java.se.ee as a jar file rather than a module. And I coundn`t make it the way I want it to translate --add-modules when launching executor JVM.</p>
  
  <p>Has anyone done similar experiments running Spark on Java 9/10? </p>
  
  <p>Thanks in advance</p>