title: How to filter a python Spark DataFrame by date between two date format columns
body: <p>I'm using pyspark 2.1 and i have a dataframe with two columns with date format like this: </p>
  
  <pre><code>Column A ,  START_DT       ,  END_DT
  1        ,  2016-01-01     ,  2020-02-04
  16       ,  2017-02-23     ,  2017-12-24
  </code></pre>
  
  <p>I want to filter for a certain date (for example <code>2018-12-31</code>) between the date from <code>START_DT</code> and <code>END_DT</code> (in the example there, the second row would be filtered).</p>
  
  <p>Both <code>START_DT</code> and <code>END_DT</code> columns are already in date format, i was looking for a method like the sql:</p>
  
  <pre><code>SELECT *
  FROM  MYTABLE  WHERE  '2018-12-31' BETWEEN start_dt AND end_dt
  </code></pre>