title: Spark S3 Eventual Consistency Issues
body: <p>I have several Spark jobs that write data to and read data from S3. Occasionally (about once per week for approximately 3 hours), the Spark jobs will fail with the following exception:</p>
  
  <pre><code>org.apache.spark.sql.AnalysisException: Path does not exist.
  </code></pre>
  
  <p>I've uncovered that this is likely due to the consistency model in S3, where list operations are eventually consistent. <a href="https://hortonworks.com/blog/s3guard-amazon-s3-consistency/" rel="nofollow noreferrer">S3 Guard</a> claims to solve this issue, but I'm in a Spark environment that doesn't support that utility. </p>
  
  <p>Has anyone else run into this issue and figured out a reasonable approach for dealing with it?</p>