title: Pyspark: how to filter by date and read parquet files which is partitioned by date
body: <p>I have a huge dataset of partitioned parquet files stored in AWS s3 in <code>data-store/year=&lt;&gt;/month=&lt;&gt;/day=&lt;&gt;/hour=&lt;&gt;/</code> folder format. e.g <code>data-store/year=2020/month=06/day=01/hour=05</code>.</p>
  <p>I want to read the files only for a specific date range e.g <code>2020/06/01 to 2020/08/30</code> or something like all the records with a date greater than equal to <code>2020/06/01</code>.</p>
  <p>How to do this effectively so that only the required data is loaded to the spark memory.</p>