title: How to delete a particular month from a parquet file partitioned by month
body: <p>I am having <code>monthly</code> Revenue data for the last 5 years and I am storing the DataFrames for respective months in <a href="http://spark.apache.org/docs/2.4.0/api/python/pyspark.sql.html#pyspark.sql.DataFrameWriter.parquet" rel="nofollow noreferrer"><code>parquet</code></a> formats in <code>append</code> mode, but <a href="http://spark.apache.org/docs/2.4.0/api/python/pyspark.sql.html#pyspark.sql.DataFrameWriter.partitionBy" rel="nofollow noreferrer"><code>partitioned by</code></a> <code>month</code> column. Here is the pseudo-code below - </p>
  
  <pre><code>def Revenue(filename):
      df = spark.read.load(filename)
      .
      .
      df.write.format('parquet').mode('append').partitionBy('month').save('/path/Revenue')
  
  Revenue('Revenue_201501.csv')
  Revenue('Revenue_201502.csv')
  Revenue('Revenue_201503.csv')
  Revenue('Revenue_201504.csv')
  Revenue('Revenue_201505.csv')
  </code></pre>
  
  <p>The <code>df</code> gets stored in <code>parquet</code> format on monthly basis, as can be seen below - </p>
  
  <p><a href="https://i.stack.imgur.com/XbPW8.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/XbPW8.png" alt="enter image description here"></a></p>
  
  <p><strong>Question:</strong> How can I delete the <code>parquet</code> folder corresponding to a particular month?</p>
  
  <p>One way would be to load all these <code>parquet</code> files in a big <code>df</code> and then use <code>.where()</code> clause to filter out that particular month and then save it back into <code>parquet</code> format <code>partitionBy</code> month in <code>overwrite</code> mode, like this -</p>
  
  <pre><code># If we want to remove data from Feb, 2015
  df = spark.read.format('parquet').load('Revenue.parquet')
  df = df.where(col('month') != lit('2015-02-01'))
  df.write.format('parquet').mode('overwrite').partitionBy('month').save('/path/Revenue')
  </code></pre>
  
  <p>But, this approach is quite cumbersome.</p>
  
  <p>Other way is to directly delete the folder of that particular month, but I am not sure if that's a right way to approach things, lest we alter the <code>metadata</code> in an unforseeable way.</p>
  
  <p>What would be the right way to delete the <code>parquet</code> data for a particular month?</p>