title: Combine date ranges in Spark dataframe
body: <p>I have a problem similar to <a href="https://stackoverflow.com/questions/44267366/combine-date-ranges-in-pandas-dataframe">this one</a>.</p>
  <p>However, I am dealing with a huge dataset. I was trying to see if I can do the same thing in PySpark instead of pandas. Below is the solution in pandas. Can this be done in PySpark?</p>
  <pre><code>def merge_dates(grp):
      # Find contiguous date groups, and get the first/last start/end date for each group.
      dt_groups = (grp['StartDate'] != grp['EndDate'].shift()).cumsum()
      return grp.groupby(dt_groups).agg({'StartDate': 'first', 'EndDate': 'last'})
  
  # Perform a groupby and apply the merge_dates function, followed by formatting.
  df = df.groupby(['FruitID', 'FruitType']).apply(merge_dates)
  df = df.reset_index().drop('level_2', axis=1) 
  </code></pre>