title: How to store Spark Streaming Checkpoint Location into S3?
body: <p>I am interested in a Spark Streaming app (Spark v2.3.2) that sources S3 parquet data and writes parquet data to S3. The app's data frame stream makes use of <code>groupByKey()</code> and <code>flatMapGroupsWithState()</code> to make use of the <code>GroupState</code>.</p>
  
  <p>Is it possible to configure this to use an <strong>s3</strong> checkpoint location? For example:</p>
  
  <pre><code>val stream = myDataset.writeStream
      .format("parquet")
      .option("path", s3DataDestination)
      .option("checkpointLocation", s3CheckpointPath)
      .option("truncate", false)
      .option(Trigger.Once)
      .outputMode(OutputMode.Append)
  stream.start().awaitTermination()
  </code></pre>
  
  <p>I confirmed the above is able to successfully write data to the <code>s3DataDestination</code>.</p>
  
  <p>However, an exception is thrown when writing to the s3 checkpoint location:</p>
  
  <pre><code>java.lang.IllegalStateException: Error committing version 1 into HDFSStateStore[id=(op=0, part=9), dir=s3://&lt;my_s3_location&gt;
      at org.apache.spark.sql.execution.streaming.state.HDFSBackedStateStoreProvider$HDFSBackedStateStore.commit(...)
  ...
  Caused by: java.io.IOException: Failed to rename s3://.../checkpoint/state/0/9/temp... to s3://.../checkpoint/state/0/9/1.delta
  </code></pre>
  
  <p>Would this require a custom implementation of a S3 <code>StateStoreProvider</code>? Or, does the checkpoint location need to be written into HDFS?</p>