title: Column filtering in PySpark
body: <p>I have a dataframe <code>df</code> loaded from Hive table and it has a timestamp column, say <code>ts</code>, with string type of format <code>dd-MMM-yy hh.mm.ss.MS a</code> (converted to python datetime library, this is <code>%d-%b-%y %I.%M.%S.%f %p</code>). </p>
  
  <p>Now I want to filter rows from the dataframe that are from the last five minutes:</p>
  
  <pre><code>only_last_5_minutes = df.filter(
      datetime.strptime(df.ts, '%d-%b-%y %I.%M.%S.%f %p') &gt; datetime.now() - timedelta(minutes=5)
  )
  </code></pre>
  
  <p>However, this does not work and I get this message</p>
  
  <pre><code>TypeError: strptime() argument 1 must be string, not Column
  </code></pre>
  
  <p>It looks like I have wrong application of column operation and it seems to me I have to create a lambda function to filter each column that satisfies the desired condition, but being a newbie to Python and lambda expression in particular, I don't know how to create my filter correct. Please advise.</p>
  
  <p>P.S.
  I prefer to express my filters as Python native (or SparkSQL) rather than a filter inside Hive sql query expression 'WHERE'.</p>
  
  <p>preferred: </p>
  
  <pre><code>df = sqlContext.sql("SELECT * FROM my_table")
  df.filter( // filter here)
  </code></pre>
  
  <p>not preferred:</p>
  
  <pre><code>df = sqlContext.sql("SELECT * FROM my_table WHERE...")
  </code></pre>