title: Custom interpretation of Spark dataframe filtering("where") clauses
body: <p>I have a system that stores data in parquet datasets on s3. Each dataset contains data for a single calendar date.</p>
  
  <p>I want to be able to query for a single date, a set of dates, or a range of dates in plain sql terms.
  But instead of needing to fetch all datasets only to discard most of them, I  want to intercept the query interpretation and do the obvious optimisation, based on the date related clauses.</p>
  
  <p>How can one customise spark dataframe/dataest query interpretation? e.g. <code>select * from X where day = '2018-06-16'</code> should only fetch the dataset in <code>/datasets/X/2018-06-16</code>. The same question applies to using the dataframe DSL, not really attached to using SQL.</p>
  
  <p>3rd party connectors (e.g. Cassandra) must be doing the same kind of intercepting to the query AST. If it is too messy, is using UDFs a viable starting point?</p>
  
  <p>I failed to find relevant documentation but could be due to searching for the wrong terminology</p>