title: Filtering data between two times in pyspark
body: <p>I have a dataframe imported from CSV with two columns (among others): Date and Time. Date is a date string of format <code>YYYY-MM-DD</code>, and Time is a string of format <code>HH:MM</code>. When importing this into pyspark with <code>inferSchema</code> "Date" is automatically inferred to datetime, which resolves them all to midnight (e.g. <code>2018-05-04 00:00:00.000</code>).</p>
  
  <p>I need to filter the data to only data where the time is between two hours (9am and 5pm). The most basic idea I had was to filter purely on the string, e.g.</p>
  
  <p><code>return dataframe.filter(
          dataframe.Time.like("19%")
      )
  </code></p>
  
  <p>Alternatively, I thought of aggregating Date and Time into a single "Timestamp" column, such as below (probably quite badly, I'm still learning pyspark):</p>
  
  <p><code>data = data.withColumn(
        "Timestamp",
        to_utc_timestamp(concat(date_format(col("Date"), "YYYY-MM-dd "), col("Time")), "GMT")
      # )
  </code></p>
  
  <p>Basically, how can I do this, since I'm not thrilled by either of these methods? If the second method looks reasonable, how would one go about performing the filter? A disadvantage of the second method is that it leaves the dataframe with a new column, which is a potentially unforeseen side effect (I am trying to build this code quite modularly).</p>
  
  <p>Thanks in advance. Apologies for the vagueness of the question, I'm still exploring pyspark and am trying to learn about the different possible ways of accomplishing things.</p>