title: PySpark : Optimize read/load from Delta using selected columns or partitions
body: <p>I am trying to load data from Delta into a pyspark dataframe.</p>
  <pre class="lang-py prettyprint-override"><code>path_to_data = 's3://mybucket/daily_data/'
  df = spark.read.format(&quot;delta&quot;).load(path_to_data)
  </code></pre>
  <p>Now the underlying data is partitioned by date as</p>
  <pre><code>s3://mybucket/daily_data/
      dt=2020-06-12
      dt=2020-06-13
      ...
      dt=2020-06-22
  </code></pre>
  <p>Is there a way to optimize the read as Dataframe, given:</p>
  <ol>
  <li>Only certain date range is needed</li>
  <li>Subset of column is only needed</li>
  </ol>
  <p>Current way, i tried is :</p>
  <pre class="lang-py prettyprint-override"><code>df.registerTempTable(&quot;my_table&quot;)
  new_df = spark.sql(&quot;select col1,col2 from my_table where dt_col &gt; '2020-06-20' &quot;)
  # dt_col is column in dataframe of timestamp dtype.
  </code></pre>
  <p>In the above state, does Spark need to load the whole data, filter the data based on date range and then filter columns needed ? Is there any optimization that can be done in pyspark read, to load data since it is already partitioned ?</p>
  <p>Something on line of :</p>
  <pre class="lang-py prettyprint-override"><code>df = spark.read.format(&quot;delta&quot;).load(path_to_data,cols_to_read=['col1','col2'])
  or 
  df = spark.read.format(&quot;delta&quot;).load(path_to_data,partitions=[...])
  </code></pre>