title: java.lang.NoSuchMethodError: scala.Predef$.refArrayOps in Spark job with Scala
body: <p>Full error:</p>
  
  <blockquote>
    <p>Exception in thread "main" java.lang.NoSuchMethodError:
    scala.Predef$.refArrayOps([Ljava/lang/Object;)[Ljava/lang/Object;
            at org.spark_module.SparkModule$.main(SparkModule.scala:62)
            at org.spark_module.SparkModule.main(SparkModule.scala)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:498)
            at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
            at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
            at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
            at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
            at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
            at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
            at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
            at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)</p>
  </blockquote>
  
  <p>When I compile and run the code in IntelliJ, it executes fine all the way through. The error shows when I submit the .jar as a spark job (runtime).</p>
  
  <p>Line 62 contains: <code>for ((elem, i) &lt;- args.zipWithIndex)</code>. I commented out the rest of the code to be sure, and the error kept showing on that line.</p>
  
  <p>At first I thought it was <code>zipWithIndex</code>'s fault. Then I changed it for <code>for (elem &lt;- args)</code> and guess what, the error still showed. Is the <code>for</code> causing this?</p>
  
  <p>Google searching always points to Scala versions incompatibility between version used to compile and version used on runtime but I can't figure out a solution. </p>
  
  <p>I tried <a href="https://stackoverflow.com/questions/24363409/what-scala-version-intellij-idea-scala-plugin-uses">this</a> to check Scala version used by IntelliJ and here is everything Scala-related under <em>Modules > Scala</em>:</p>
  
  <p><a href="https://i.stack.imgur.com/H8O4U.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/H8O4U.png" alt="enter image description here"></a></p>
  
  <p>Then I did <a href="https://stackoverflow.com/a/2902449/4770813">this</a> to check the run-time version of Scala and the output is:</p>
  
  <blockquote>
    <p>(file:/C:/Users/me/.gradle/caches/modules-2/files-2.1/org.scala-lang/scala-library/2.12.11/1a0634714a956c1aae9abefc83acaf6d4eabfa7d/scala-library-2.12.11.jar
    )</p>
  </blockquote>
  
  <p>Versions seem to match...</p>
  
  <p>This is my <strong>gradle.build</strong> (includes <code>fatJar</code> task)</p>
  
  <pre><code>group 'org.spark_module'
  version '1.0-SNAPSHOT'
  
  apply plugin: 'scala'
  apply plugin: 'idea'
  apply plugin: 'eclipse'
  
  repositories {
      mavenCentral()
  }
  
  idea {
      project {
          jdkName = '1.8'
          languageLevel = '1.8'
      }
  }
  
  dependencies {
      implementation group: 'org.scala-lang', name: 'scala-library', version: '2.12.11'
      implementation group: 'org.apache.spark', name: 'spark-core_2.12'//, version: '2.4.5'
      implementation group: 'org.apache.spark', name: 'spark-sql_2.12'//, version: '2.4.5'
      implementation group: 'com.datastax.spark', name: 'spark-cassandra-connector_2.12', version: '2.5.0'
      implementation group: 'org.apache.spark', name: 'spark-mllib_2.12', version: '2.4.5'
      implementation group: 'log4j', name: 'log4j', version: '1.2.17'
      implementation group: 'org.scalaj', name: 'scalaj-http_2.12', version: '2.4.2'
  }
  
  task fatJar(type: Jar) {
      zip64 true
      from {
          configurations.runtimeClasspath.collect { it.isDirectory() ? it : zipTree(it) }
      } {
          exclude "META-INF/*.SF"
          exclude "META-INF/*.DSA"
          exclude "META-INF/*.RSA"
      }
  
      manifest {
          attributes 'Main-Class': 'org.spark_module.SparkModule'
      }
  
      with jar
  }
  
  configurations.all {
      resolutionStrategy {
          force 'com.google.guava:guava:12.0.1'
      }
  }
  
  compileScala.targetCompatibility = "1.8"
  compileScala.sourceCompatibility = "1.8"
  
  jar {
      zip64 true
      getArchiveFileName()
      from {
          configurations.compile.collect {
              it.isDirectory() ? it : zipTree(it)
          }
      }
      manifest {
          attributes 'Main-Class': 'org.spark_module.SparkModule'
      }
  
      exclude 'META-INF/*.RSA', 'META-INF/*.SF', 'META-INF/*.DSA'
  
  }
  </code></pre>
  
  <p>To build the (fat) jar:</p>
  
  <pre><code>gradlew fatJar
  </code></pre>
  
  <p>in IntelliJ's terminal.</p>
  
  <p>To run the job:</p>
  
  <pre><code>spark-submit.cmd .\SparkModule-1.0-SNAPSHOT.jar
  </code></pre>
  
  <p>in Windows PowerShell.</p>
  
  <p>Thank you</p>
  
  <p>EDIT:</p>
  
  <p><strong>spark-submit.cmd</strong> and <strong>spark-shell.cmd</strong> both show Scala version <strong>2.11.12</strong>, so yes, they differ from the one I am using in IntelliJ (<strong>2.12.11</strong>). The problem is, in <a href="https://spark.apache.org/downloads.html" rel="nofollow noreferrer">Spark's download page</a>, there is only one <a href="https://www.apache.org/dyn/closer.lua/spark/spark-2.4.5/spark-2.4.5-bin-without-hadoop-scala-2.12.tgz" rel="nofollow noreferrer">Spark distribution for Scala 2.12</a> and it comes without Hadoop; does it mean I have to downgrade from 2.12 to 2.11 in my <strong>gradle.build</strong>?</p>