title: DataProc Spark Error com.google.api.client.googleapis.json.GoogleJsonResponseException: 410 Gone
body: <p>Having run a spark job on YARN, after about 9 hours the job fails with</p>
  
  <pre><code>org.apache.spark.SparkException: Task failed while writing rows
  at org.apache.spark.sql.execution.datasources.DynamicPartitionWriterContainer.writeRows(WriterContainer.scala:446)
  at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(InsertIntoHadoopFsRelationCommand.scala:143)
  at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(InsertIntoHadoopFsRelationCommand.scala:143)
  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
  at org.apache.spark.scheduler.Task.run(Task.scala:86)
  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:748)
      at java.lang.Thread.run(Thread.java:748)
  Caused by: java.io.IOException: 
  com.google.api.client.googleapis.json.GoogleJsonResponseException: 410 
  Gone
  {
   "code" : 503,
   "errors" : [ {
   "domain" : "global",
   "message" : "Backend Error",
   "reason" : "backendError"
   } ],
   "message" : "Backend Error"
  }
  at  com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel.waitForCompletionAndThrowIfUploadFailed(AbstractGoogleAsyncWriteChannel.java:432)
  at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel.close(AbstractGoogleAsyncWriteChannel.java:287)
  at com.google.cloud.hadoop.gcsio.CacheSupplementedGoogleCloudStorage$WritableByteChannelImpl.close(CacheSupplementedGoogleCloudStorage.java:68)
  at java.nio.channels.Channels$1.close(Channels.java:178)
  at java.io.FilterOutputStream.close(FilterOutputStream.java:159)
  at com.google.cloud.hadoop.fs.gcs.GoogleHadoopOutputStream.close(GoogleHadoopOutputStream.java:126)
  at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
  at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
  at org.apache.parquet.hadoop.ParquetFileWriter.end(ParquetFileWriter.java:400)
  at org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:117)
  at org.apache.parquet.hadoop.ParquetRecordWriter.close(ParquetRecordWriter.java:112)
  at org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.close(ParquetFileFormat.scala:569)
  at org.apache.spark.sql.execution.datasources.DynamicPartitionWriterContainer$$anonfun$writeRows$4.apply$mcV$sp(WriterContainer.scala:422)
  at org.apache.spark.sql.execution.datasources.DynamicPartitionWriterContainer$$anonfun$writeRows$4.apply(WriterContainer.scala:416)
  at org.apache.spark.sql.execution.datasources.DynamicPartitionWriterContainer$$anonfun$writeRows$4.apply(WriterContainer.scala:416)
  at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1348)
  at org.apache.spark.sql.execution.datasources.DynamicPartitionWriterContainer.writeRows(WriterContainer.scala:438)
  ... 8 more
  Suppressed: java.lang.NullPointerException
      at org.apache.parquet.hadoop.InternalParquetRecordWriter.flushRowGroupToStore(InternalParquetRecordWriter.java:147)
      at org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:113)
      at org.apache.parquet.hadoop.ParquetRecordWriter.close(ParquetRecordWriter.java:112)
      at org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.close(ParquetFileFormat.scala:569)
      at org.apache.spark.sql.execution.datasources.DynamicPartitionWriterContainer$$anonfun$writeRows$5.apply$mcV$sp(WriterContainer.scala:440)
      at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1357)
      ... 9 more
  </code></pre>
  
  <p>I have seen solutions here for <a href="https://stackoverflow.com/questions/41215541/dataflow-jobs-fail-after-a-few-410-errors-while-writing-to-gcs">Dataflow jobs fail after a few 410 errors (while writing to GCS)</a> and <a href="https://stackoverflow.com/questions/39165693/how-to-recover-from-cloud-dataflow-job-failed-on-com-google-api-client-googleapi">How to recover from Cloud Dataflow job failed on com.google.api.client.googleapis.json.GoogleJsonResponseException: 410 Gone</a></p>
  
  <p>but these suggest sharding on DataFlow (not DataProc+YARN). I have also seen <a href="https://cloud.google.com/storage/docs/json_api/v1/status-codes#410_Gone" rel="nofollow noreferrer">https://cloud.google.com/storage/docs/json_api/v1/status-codes#410_Gone</a> which suggests a lost re-sumable session which I have no control over.</p>
  
  <p>using </p>
  
  <pre><code>  &lt;dependency&gt;
              &lt;groupId&gt;com.google.cloud&lt;/groupId&gt;
              &lt;artifactId&gt;google-cloud-storage&lt;/artifactId&gt;
          &lt;version&gt;1.1.0&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
          &lt;artifactId&gt;spark-core_${scala.version}&lt;/artifactId&gt;
          &lt;version&gt;2.1.0&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;com.google.cloud.bigdataoss&lt;/groupId&gt;
          &lt;artifactId&gt;gcs-connector&lt;/artifactId&gt;
          &lt;version&gt;1.6.0-hadoop2&lt;/version&gt;
      &lt;/dependency&gt;
  </code></pre>
  
  <p>Are there any sharding, spark, GCP or YARN partition settings that can help avoid/prevent this exception?</p>