title: split row into multiple sessions by time difference in pyspark
body: <p>Here's the pseudo data:</p>
  
  <pre><code>user  ts
  --------
  1     1
  1     3
  1     10
  1     13
  1     21
  1     24
  </code></pre>
  
  <p>It will be split into two sessions if adjacent time difference is >= 6 for each user. So, the above data should be split as below:</p>
  
  <pre><code>user    ts    diff
  -------------------
  1       1     None
  1       3     2
  -------------------
  1       10    7
  1       13    3
  -------------------
  1       21    8
  1       24    3
  </code></pre>
  
  <p>I understand how to generate the diff column in <code>pyspark</code> via Window function illustrated below, but how could I split it into different sessions for each user in <code>pyspark</code> manner? Great thanks!</p>
  
  <pre><code>select
     user,
     ts,
     (lag(ts, 1) over (partition by user order by ts asc)) as diff
  from user_event
  </code></pre>