title: Spark error with google/guava library: java.lang.NoSuchMethodError: com.google.common.cache.CacheBuilder.refreshAfterWrite
body: <p>I have a simple <code>spark</code> project - in which in the <code>pom.xml</code> the dependencies are only the basic <code>scala</code>, <code>scalatest</code>/<code>junit</code>, and <code>spark</code>:</p>
  
  <p></p>
  
  <pre><code>    &lt;dependency&gt;
          &lt;groupId&gt;net.alchim31.maven&lt;/groupId&gt;
          &lt;artifactId&gt;scala-maven-plugin&lt;/artifactId&gt;
          &lt;version&gt;3.2.0&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.scala-lang&lt;/groupId&gt;
          &lt;artifactId&gt;scala-library&lt;/artifactId&gt;
          &lt;version&gt;${scala.version}&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.scala-lang&lt;/groupId&gt;
          &lt;artifactId&gt;scala-compiler&lt;/artifactId&gt;
          &lt;version&gt;${scala.version}&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;junit&lt;/groupId&gt;
          &lt;artifactId&gt;junit&lt;/artifactId&gt;
          &lt;version&gt;4.11&lt;/version&gt;
          &lt;scope&gt;test&lt;/scope&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.scalatest&lt;/groupId&gt;
          &lt;artifactId&gt;scalatest_${scala.binary.version}&lt;/artifactId&gt;
          &lt;version&gt;3.0.1&lt;/version&gt;
          &lt;scope&gt;test&lt;/scope&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
          &lt;artifactId&gt;spark-core_${scala.binary.version}&lt;/artifactId&gt;
          &lt;version&gt;${spark.version}&lt;/version&gt;
          &lt;scope&gt;compile&lt;/scope&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
          &lt;artifactId&gt;spark-mllib_${scala.binary.version}&lt;/artifactId&gt;
          &lt;version&gt;${spark.version}&lt;/version&gt;
          &lt;scope&gt;compile&lt;/scope&gt;
      &lt;/dependency&gt;
  &lt;/dependencies&gt;
  </code></pre>
  
  <p>When attempting to run a basic <code>spark</code> program the <code>SparkSession</code> init fails on this line:</p>
  
  <pre><code> SparkSession.builder.master(master).appName("sparkApp").getOrCreate
  </code></pre>
  
  <p>Here is the output / error:</p>
  
  <pre><code>Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
  18/04/07 18:06:15 INFO SparkContext: Running Spark version 2.2.1
  Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.cache.CacheBuilder
  .refreshAfterWrite(JLjava/util/concurrent/TimeUnit;)
  Lcom/google/common/cache/CacheBuilder;
      at org.apache.hadoop.security.Groups.&lt;init&gt;(Groups.java:96)
      at org.apache.hadoop.security.Groups.&lt;init&gt;(Groups.java:73)
  
  at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:293)
  at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:283)
  at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260)
  at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:789)
  at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:774)
  at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
  at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2424)
  at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2424)
  at scala.Option.getOrElse(Option.scala:121)
  at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2424)
  at org.apache.spark.SparkContext.&lt;init&gt;(SparkContext.scala:295)
  at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2516)
  at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:918)
  at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:910)
  at scala.Option.getOrElse(Option.scala:121)
  at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:910)
  </code></pre>
  
  <p>I have run <code>spark</code> locally many dozens of times on other projects, what might be wrong with this simple one?  Is there a dependency on <code>$HADOOP_HOME</code> environment variable or similar?</p>
  
  <p><strong>Update</strong>  By downgrading the <code>spark</code> version to <code>2.0.1</code> I was able to compile. That does <em>not</em> fix the problem (we need newer) version. But it helps point out the source of the problem</p>
  
  <p><strong>Another update</strong>  In a <em>different</em> project the hack to downgrade to <code>2.0.1</code> does help - i.e. execution proceeds <em>further</em> : but then when writing out to <strong><code>parquet</code></strong> a similar exception does happen.</p>
  
  <pre><code>8/05/07 11:26:11 ERROR Executor: Exception in task 0.0 in stage 2741.0 (TID 2618)
  java.lang.NoSuchMethodError: com.google.common.cache.CacheBuilder.build(Lcom/google/common/cache/CacheLoader;)Lcom/google/common/cache/LoadingCache;
      at org.apache.hadoop.io.compress.CodecPool.createCache(CodecPool.java:62)
      at org.apache.hadoop.io.compress.CodecPool.&lt;clinit&gt;(CodecPool.java:74)
      at org.apache.parquet.hadoop.CodecFactory$BytesCompressor.&lt;init&gt;(CodecFactory.java:92)
      at org.apache.parquet.hadoop.CodecFactory.getCompressor(CodecFactory.java:169)
      at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:303)
      at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:262)
      at org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.&lt;init&gt;(ParquetFileFormat.scala:562)
      at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:139)
      at org.apache.spark.sql.execution.datasources.BaseWriterContainer.newOutputWriter(WriterContainer.scala:131)
      at org.apache.spark.sql.execution.datasources.DefaultWriterContainer.writeRows(WriterContainer.scala:247)
      at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(InsertIntoHadoopFsRelationCommand.scala:143)
      at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(InsertIntoHadoopFsRelationCommand.scala:143)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
      at org.apache.spark.scheduler.Task.run(Task.scala:86)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  </code></pre>