title: Spark FileAlreadyExistsException on Stage Failure
body: <p>I am trying to write a dataframe to s3 location after re-partitioning. But whenever the write stage fails and Spark retry the stage it throws FileAlreadyExistsException.</p>
  
  <p>When I re-submit the job it works fine if spark completes the stage in one try.</p>
  
  <p>Below is my code block</p>
  
  <pre class="lang-py prettyprint-override"><code>df.repartition(&lt;some-value&gt;).write.format("orc").option("compression", "zlib").mode("Overwrite").save(path)
  </code></pre>
  
  <p>I believe Spark should remove files from the failed stage before retry. I understand this will be solved if we set retry to zero but the spark stage is expected to fail and that would not be a proper solution.</p>
  
  <p>Below is the error:</p>
  
  <pre><code>Job aborted due to stage failure: Task 0 in stage 6.1 failed 4 times, most recent failure: Lost task 0.3 in stage 6.1 (TID 740, ip-address, executor 170): org.apache.hadoop.fs.FileAlreadyExistsException: File already exists:s3://&lt;bucket-name&gt;/&lt;path-to-object&gt;/part-00000-c3c40a57-7a50-41da-9ce2-555753cab63a-c000.zlib.orc
      at com.amazon.ws.emr.hadoop.fs.s3.upload.plan.RegularUploadPlanner.checkExistenceIfNotOverwriting(RegularUploadPlanner.java:36)
      at com.amazon.ws.emr.hadoop.fs.s3.upload.plan.RegularUploadPlanner.plan(RegularUploadPlanner.java:30)
      at com.amazon.ws.emr.hadoop.fs.s3.upload.plan.UploadPlannerChain.plan(UploadPlannerChain.java:37)
      at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.create(S3NativeFileSystem.java:601)
      at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:932)
      at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:913)
      at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.create(EmrFileSystem.java:242)
      at org.apache.orc.impl.PhysicalFsWriter.&lt;init&gt;(PhysicalFsWriter.java:95)
      at org.apache.orc.impl.WriterImpl.&lt;init&gt;(WriterImpl.java:170)
      at org.apache.orc.OrcFile.createWriter(OrcFile.java:843)
      at org.apache.orc.mapreduce.OrcOutputFormat.getRecordWriter(OrcOutputFormat.java:50)
      at org.apache.spark.sql.execution.datasources.orc.OrcOutputWriter.&lt;init&gt;(OrcOutputWriter.scala:43)
      at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$$anon$1.newInstance(OrcFileFormat.scala:121)
      at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:120)
      at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.&lt;init&gt;(FileFormatDataWriter.scala:108)
      at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:233)
      at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)
      at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:168)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
      at org.apache.spark.scheduler.Task.run(Task.scala:121)
      at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
      at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
  
  Driver stacktrace:
  </code></pre>
  
  <p>I am using Spark 2.4 with EMR, Please suggest the solution.</p>
  
  <p>Edit 1:
  Please note the issue is not related to overwrite mode, I am already using it. As the question title suggests, the issue is with leftover files in case of stage failure. May be the Spark UI clears it.
  <a href="https://i.stack.imgur.com/PdXF2.png" rel="noreferrer"><img src="https://i.stack.imgur.com/PdXF2.png" alt="enter image description here"></a></p>