title: exception: org.apache.spark.sql.delta.ConcurrentAppendException: Files were added to the root of the table by a concurrent update
body: <p>I have a simple Spark job that streams data to a Delta table.
  The table is pretty small and is not partitioned.</p>
  <p>A lot of small parquet files are created.</p>
  <p>As recommended in the documentation (<a href="https://docs.delta.io/1.0.0/best-practices.html" rel="nofollow noreferrer">https://docs.delta.io/1.0.0/best-practices.html</a>) I added a compaction job that runs once a day.</p>
  <pre><code>    val path = &quot;...&quot;
      val numFiles = 16
      
      spark.read
       .format(&quot;delta&quot;)
       .load(path)
       .repartition(numFiles)
       .write
       .option(&quot;dataChange&quot;, &quot;false&quot;)
       .format(&quot;delta&quot;)
       .mode(&quot;overwrite&quot;)
       .save(path)
  </code></pre>
  <p>Every time the compaction job runs the streaming job gets the following exception:</p>
  <pre><code>org.apache.spark.sql.delta.ConcurrentAppendException: Files were added to the root of the table by a concurrent update. Please try the operation again.
  </code></pre>
  <p>I tried to add the following config parameters to the streaming job:</p>
  <pre><code>spark.databricks.delta.retryWriteConflict.enabled = true  # would be false by default
  spark.databricks.delta.retryWriteConflict.limit = 3  # optionally limit the maximum amout of retries
  </code></pre>
  <p>It doesn't help.</p>
  <p>Any idea how to solve the problem?</p>