title: How to create a “sessionId” column using timestamps and userid in PySpark?
body: <p>I have a data set which contains fields such as: userId, event, pageName, and timestamp, while lacking of the sessionId. I want to create a sessionId for each record based on the timestamp and a pre-defined value &quot;finish&quot; (which indicates after how many minutes of inactivity a session ends). Only users with the same UserId can be in the same session.</p>
  <p>If the &quot;finish&quot; value is 30 minutes (1800 difference in timestamp), and a sample DataFrame is:</p>
  <pre><code>from pyspark.sql import functions as F 
  from pyspark.sql.window import Window
  
  df = spark.createDataFrame([
    (&quot;blue&quot;, &quot;view&quot;, 1610494094750, 11),
    (&quot;green&quot;, &quot;add to bag&quot;, 1510593114350, 21),
    (&quot;red&quot;, &quot;close&quot;, 1610493115350, 41),
    (&quot;blue&quot;, &quot;view&quot;, 1610494094350, 11),
    (&quot;blue&quot;, &quot;close&quot;, 1510593114312, 21),
    (&quot;red&quot;, &quot;view&quot;, 1610493114350, 41),
    (&quot;red&quot;, &quot;view&quot;, 1610593114350, 41),
    (&quot;green&quot;, &quot;purchase&quot;, 1610494094350, 31)
  ], [&quot;item&quot;, &quot;event&quot;, &quot;timestamp&quot;, &quot;userId&quot;])
  
  
  +-----+----------+-------------+------+
  | item|     event|    timestamp|userId|
  +-----+----------+-------------+------+
  | blue|      view|1610494094750|    11|
  |green|add to bag|1510593114350|    21|
  |  red|     close|1610493115350|    41|
  | blue|      view|1610494094350|    11|
  | blue|     close|1510593114312|    21|
  |  red|      view|1610493114350|    41|
  |  red|      view|1610593114350|    41|
  |green|  purchase|1610494094350|    31|
  +-----+----------+-------------+------+
  </code></pre>
  <p>The end result should be something like this:</p>
  <pre><code>+--------+----------+-------------+------+---------+
  |    item|     event|    timestamp|userId|sessionId|
  +--------+----------+-------------+------+---------+
  |    blue|     close|1510593114312|    21| session1|
  |   green|add to bag|1510593114350|    21| session1|
  |     red|      view|1610493114350|    41| session2|
  |    blue|      view|1610494094350|    11| session3|
  |     red|     close|1610493115350|    41| session2|
  |   green|  purchase|1610494094350|    31| session4|
  |    blue|      view|1610494094750|    11| session3|
  |     red|      view|1610593114350|    41| session5|
  +--------+----------+-------------+------+---------+
  </code></pre>
  <p>I'm trying to solve this problem using PySpark. Any advice is welcome.</p>
  <p>Edit: I've edited the timestamp</p>