title: Databricks cannot save stream checkpoint
body: <p>I'm trying to set up the stream to begin processing incoming files. Looks like Databricks is unable to save a checkpoint. I tried location in ADLS Gen2 and DBFS with the same result. Databricks creates needed folder with some scructure but cannot write to it. Are there any special requirements for a checkpoint location?</p>
  <p><a href="https://i.stack.imgur.com/9rq8f.png" rel="nofollow noreferrer">Checkpoint folder</a></p>
  <p>Databricks Community Edition, runtime version: 9.1 LTS (includes Apache Spark 3.1.2, Scala 2.12)</p>
  <pre><code>spark.readStream
  .format(&quot;cloudFiles&quot;)
  .option(&quot;cloudFiles.format&quot;, &quot;parquet&quot;)
  .option(&quot;cloudFiles.partitionColumns&quot;, &quot;year, month, day&quot;)
  .option(&quot;header&quot;, &quot;true&quot;)
  .schema(schema)
  .load(destFolderName)
  .writeStream.format(&quot;delta&quot;)
  .option(&quot;checkpointLocation&quot;, checkpointPath)
  .outputMode(&quot;append&quot;)
  .partitionBy(&quot;year&quot;, &quot;month&quot;, &quot;day&quot;)
  .start(outputPath)
  </code></pre>
  <p>The error:</p>
  <pre><code>java.lang.UnsupportedOperationException: com.databricks.backend.daemon.data.client.DBFSV1.createAtomicIfAbsent(path: Path)
  at com.databricks.tahoe.store.EnhancedDatabricksFileSystemV1.createAtomicIfAbsent(EnhancedFileSystem.scala:324)
  at com.databricks.spark.sql.streaming.AWSCheckpointFileManager.createAtomicIfAbsent(DatabricksCheckpointFileManager.scala:159)
  at com.databricks.spark.sql.streaming.DatabricksCheckpointFileManager.createAtomicIfAbsent(DatabricksCheckpointFileManager.scala:60)
  at com.databricks.sql.streaming.state.RocksDBFileManager.zipToDbfsFile(RocksDBFileManager.scala:497)
  at com.databricks.sql.streaming.state.RocksDBFileManager.saveCheckpointToDbfs(RocksDBFileManager.scala:181)
  at com.databricks.sql.rocksdb.CloudRocksDB.$anonfun$open$5(CloudRocksDB.scala:451)
  at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
  at org.apache.spark.util.Utils$.timeTakenMs(Utils.scala:627)
  at com.databricks.sql.rocksdb.CloudRocksDB.timeTakenMs(CloudRocksDB.scala:527)
  at com.databricks.sql.rocksdb.CloudRocksDB.$anonfun$open$2(CloudRocksDB.scala:439)
  at com.databricks.logging.UsageLogging.$anonfun$recordOperation$1(UsageLogging.scala:395)
  at com.databricks.logging.UsageLogging.executeThunkAndCaptureResultTags$1(UsageLogging.scala:484)
  at com.databricks.logging.UsageLogging.$anonfun$recordOperationWithResultTags$4(UsageLogging.scala:504)
  at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:266)
  at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
  at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:261)
  at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:258)
  at com.databricks.spark.util.PublicDBLogging.withAttributionContext(DatabricksSparkUsageLogger.scala:20)
  </code></pre>