title: How to use gcs-connector and google-cloud-storage alongside in Scala
body: <p>I am trying to list all objects in a bucket, and then read some or all of them as CSV. I have spent two days now, trying to do both, but I can only get one working at a time if I'm using Google's libraries.</p>
  
  <p>I think the problem lies in an incompatibility between Google's own libraries, but I'm not entirely sure. First, I think I should show how I'm doing each thing.</p>
  
  <p>This is how I'm reading a single file. In my version of Scala, you can use a <code>gs://</code> url with <code>spark.read.csv</code>:</p>
  
  <pre class="lang-scala prettyprint-override"><code>val jsonKeyFile = "my-local-keyfile.json"
  ss.sparkContext.hadoopConfiguration.set("google.cloud.auth.service.account.json.keyfile", jsonKeyFile)
  
  spark.read
    .option("header", "true")
    .option("sep", ",")
    .option("inferSchema", "false")
    .option("mode", "FAILFAST")
    .csv(gcsFile)
  </code></pre>
  
  <p>This actually works alone, and I get a working DF from it. Then the problem arises when I try to add Google's Storage library:</p>
  
  <pre class="lang-scala prettyprint-override"><code>libraryDependencies += "com.google.cloud" % "google-cloud-storage" % "1.70.0"
  </code></pre>
  
  <p>If I try to run the same code again, I get this bad boy from the .csv call:</p>
  
  <pre><code>Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
  19/05/14 16:38:00 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  
  An exception or error caused a run to abort: Class com.google.common.base.Suppliers$SupplierOfInstance does not implement the requested interface java.util.function.Supplier 
  java.lang.IncompatibleClassChangeError: Class com.google.common.base.Suppliers$SupplierOfInstance does not implement the requested interface java.util.function.Supplier
      at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.getGcsFs(GoogleHadoopFileSystemBase.java:1488)
      at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.configure(GoogleHadoopFileSystemBase.java:1659)
      at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.initialize(GoogleHadoopFileSystemBase.java:683)
      at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.initialize(GoogleHadoopFileSystemBase.java:646)
      at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3303)
      at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
      ...(lots more trace, probably irrelevant)
  </code></pre>
  
  <p>Then, you might ask, why don't you just not use the library? Well... This is the code that lists the objects in a bucket:</p>
  
  <pre class="lang-scala prettyprint-override"><code>StorageOptions
    .newBuilder()
    .setCredentials(ServiceAccountCredentials.fromStream(
      File(jsonKeyFile).inputStream()))
    .build()
    .getService
    .list(bucket)
    .getValues
    .asScala
    .map(irrelevant)
    .toSeq
    .toDF("irrelevant")
  </code></pre>
  
  <p>And I have not yet found a way to do this easily without the specified library.</p>