title: How to subtract a column of days from a column of dates in Pyspark?
body: <p>Given the following PySpark DataFrame</p>
  
  <pre class="lang-py prettyprint-override"><code>df = sqlContext.createDataFrame([('2015-01-15', 10),
                                   ('2015-02-15', 5)],
                                   ('date_col', 'days_col'))
  </code></pre>
  
  <p>How can the days column be subtracted from the date column? In this example, the resulting column should be <code>['2015-01-05', '2015-02-10']</code>.</p>
  
  <p>I looked into <code>pyspark.sql.functions.date_sub()</code>, but it requires a date column and a single day, i.e. <code>date_sub(df['date_col'], 10)</code>. Ideally, I'd prefer to do <code>date_sub(df['date_col'], df['days_col'])</code>.</p>
  
  <p>I also tried creating a UDF:</p>
  
  <pre><code>from datetime import timedelta
  def subtract_date(start_date, days_to_subtract):
      return start_date - timedelta(days_to_subtract)
  
  subtract_date_udf = udf(subtract_date, DateType())
  df.withColumn('subtracted_dates', subtract_date_udf(df['date_col'], df['days_col'])
  </code></pre>
  
  <p>This technically works, but I've read that stepping between Spark and Python can cause performance issues for large datasets. I can stick with this solution for now (no need to prematurely optimize), but my gut says there's just got to be a way to do this simple thing without using a Python UDF.</p>