title: How to overwrite a parquet file from where DataFrame is being read in Spark
body: <p>This is a microcosm of the problem I am facing, where I am getting an error. Let me try to reproduce it here. </p>
  
  <p>I am saving a <code>DataFrame</code> as a <code>parquet</code>, but when I reload the <code>DataFrame</code> from <code>parquet</code> file and save it once again as <code>parquet</code>, I get an error.</p>
  
  <pre><code>valuesCol = [('Male','2019-09-06'),('Female','2019-09-06'),('Male','2019-09-07')]
  df = spark.createDataFrame(valuesCol,['sex','date'])
  # Save as parquet
  df.repartition(1).write.format('parquet').mode('overwrite').save('.../temp')
  
  # Load it back
  df = spark.read.format('parquet').load('.../temp')
  df = df.where(col('sex')=='Male')
  # Save it back - This produces ERROR   
  df.repartition(1).write.format('parquet').mode('overwrite').save('.../temp')
  </code></pre>
  
  <p><em>Error message</em> -</p>
  
  <blockquote>
    <p>executor 22): java.io.FileNotFoundException: Requested file
    maprfs:///mapr/.../temp/part-00000-f67d5a62-36f2-4dd2-855a-846f422e623f-c000.snappy.parquet
    does not exist. It is possible the underlying files have been updated.
    You can explicitly invalidate the cache in Spark by running 'REFRESH
    TABLE tableName' command in SQL or by recreating the Dataset/DataFrame
    involved.</p>
  </blockquote>
  
  <p>Another SO <a href="https://stackoverflow.com/questions/45809152/how-to-refresh-a-table-and-do-it-concurrently/45809353">question</a> addresses this issue. The proposed solution was to <a href="https://spark.apache.org/docs/2.3.0/sql-programming-guide.html#metadata-refreshing" rel="noreferrer">refresh</a> the table like the code below, but that did not help. The issue is with the refreshing of the metadata. I don't know how to refresh it.</p>
  
  <pre><code>df.createOrReplaceTempView('table_view')
  spark.catalog.refreshTable('table_view')
  df.repartition(1).write.format('parquet').mode('overwrite').save('.../temp')
  </code></pre>
  
  <p><strong>Workaround for this problem:</strong> A non-elegant way to solve this issue is to save the <code>DataFrame</code> as <code>parquet</code> file with a different name, then delete the original <code>parquet</code> file and finally, rename this <code>parquet</code> file to the old name. </p>
  
  <pre><code># Workaround
  import os
  import shutil
  
  # Load it back
  df = spark.read.format('parquet').load('.../temp')
  
  # Save it back as temp1, as opposed to original temp      
  df.repartition(1).write.format('parquet').mode('overwrite').save('.../temp1')
  
  # Delete the original parquet file
  shutil.rmtree('.../temp')
  
  # Renaming the parquet folder.
  os.rename('.../temp1','.../temp')
  </code></pre>
  
  <p>But, the problem is that some DataFrames are quite big and this may not be the best way to deal with it. Not to mention if renaming will cause some problem with the MetaData, that I am not sure of.</p>