title: Taking the recency time of client purchase with PySpark
body: <p>The sample of the dataset I am working on:</p>
  <pre><code>data = [(1, &quot;2021-11-08&quot;), (1, &quot;2021-11-06&quot;), (1, &quot;2021-10-08&quot;), (2, &quot;2021-11-01&quot;), (2, &quot;2021-10-20&quot;), 
          (2, &quot;2021-08-05&quot;), (3, &quot;2021-08-02&quot;), (3, &quot;2021-05-08&quot;), (3, &quot;2021-03-01&quot;)]
  
  df = spark.createDataFrame(data=data,schema=columns)
  </code></pre>
  <p><a href="https://i.stack.imgur.com/4e2KA.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/4e2KA.png" alt="enter image description here" /></a></p>
  <p>I'd like to take the customer's most recent purchase (the customer's date.max()) and the previous maximum purchase (the penultimate purchase) and take the difference between the two (I'm assuming the same product on all purchases). I still haven't found something in pyspark that does this. One example of my idea was to do this in a groupby, like a below with the minimum date and maximum date.</p>
  <pre><code>
  
  df1 = df.withColumn('data',to_date(df.data))
  
  dados_agrupados_item = df1.groupBy([&quot;cliente&quot;]).agg(max(&quot;data&quot;), \
           min(&quot;data&quot;), \
       )
  </code></pre>
  <p>The output is:</p>
  <p><a href="https://i.stack.imgur.com/plcL7.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/plcL7.png" alt="enter image description here" /></a></p>
  <p>For my problem, the output would be the maximum date and the penultimate purchase date from that customer. The output should be:</p>
  <p><a href="https://i.stack.imgur.com/pLmyE.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/pLmyE.png" alt="enter image description here" /></a></p>
  <p>Another method could also be directly delivering the difference between these two dates. But I have no idea how to implement it.</p>
  <p>Thank you very much in advance.</p>