title: pyspark 2.3.2 : dataframe --> shift rows with 1, by a column --> on a column with dates
body: <p>Best</p>
  
  <p>At this moment I'm experimenting with pyspark 2.3.2.<br/>
  And I would like to shift a column based on a certain column (group by).</p>
  
  <p><strong>Input:</strong></p>
  
  <pre><code>    id  timestamp                logintype  start_sessie    timestamp_prev
  0   3   2016-02-09 09:36:57.217     INTERN  True            None
  1   3   2016-02-09 09:51:40.899     INTERN  False           None
  2   3   2016-02-10 10:11:22.131     INTERN  True            None
  3   3   2016-02-10 10:17:16.345     INTERN  False           None
  4   4   2017-08-10 10:18:12.412     INTERN  True            None
  5   4   2017-08-10 10:21:11.788     INTERN  False           None
  6   4   2017-08-11 14:17:33.119     INTERN  True            None
  7   4   2017-08-11 14:11:51.173     INTERN  False           None
  8   4   2017-08-16 11:43:16.609     INTERN  True            None
  9   4   2017-08-16 11:13:35.421     INTERN  False           None
  </code></pre>
  
  <p>Lucky, since pyspark 2.3.x we can use pandas_udf.
  Therefore I had this piece of code into my mind</p>
  
  <p><strong>my code:</strong></p>
  
  <pre><code>from pyspark.sql.functions import pandas_udf, PandasUDFType
  
  @pandas_udf(data_shift_prep.schema, PandasUDFType.GROUPED_MAP)
  def test(pdf):
      pdf["timestamp_prev"] = pdf['timestamp'].shift(1)
      return pdf
  
  data_shift = data_shift_prep.groupby('id').apply(test)
  </code></pre>
  
  <p>with the following expected result:</p>
  
  <p><strong>Expected result:</strong></p>
  
  <pre><code>    id  timestamp                logintype  start_sessie    timestamp_prev
  0   3   2016-02-09 09:36:57.217     INTERN  True            None
  1   3   2016-02-09 09:51:40.899     INTERN  False           2016-02-09 09:36:57.217
  2   3   2016-02-10 10:11:22.131     INTERN  True            2016-02-09 09:51:40.899
  3   3   2016-02-10 10:17:16.345     INTERN  False           2016-02-10 10:11:22.131
  4   4   2017-08-10 10:18:12.412     INTERN  True            None
  5   4   2017-08-10 10:21:11.788     INTERN  False           2017-08-10 10:18:12.412
  6   4   2017-08-11 14:17:33.119     INTERN  True            2017-08-10 10:21:11.788
  7   4   2017-08-11 14:11:51.173     INTERN  False           2017-08-11 14:17:33.119 
  8   4   2017-08-16 11:43:16.609     INTERN  True            2017-08-11 14:11:51.173
  9   4   2017-08-16 11:13:35.421     INTERN  False           2017-08-16 11:43:16.609
  </code></pre>
  
  <p>But unfortunately, I receive always an error.<br/>
  <code>pyarrow.lib.ArrowInvalid: Error converting from Python objects to Int64: Got Python object of type Timestamp but can only handle these types: integer
  </code>
  And i don't know how to handle it. 
  First i thought, it was because of the <strong><em>NaT</em></strong> which are created by the shift function. But i'm not sure about it (I still have a same type of error after replacing the NaT by an <strong>None</strong> value)</p>
  
  <p>Does someone has some more experience with this? (and can you solve this type error)</p>
  
  <p>kind regards</p>
  
  <p>-- Add extra : <strong>schema</strong> --</p>
  
  <pre><code>StructType(List(StructField(id,IntegerType,true),
                  StructField(timestamp,TimestampType,true),
                  StructField(logintype,StringType,true),
                  StructField(start_sessie,BooleanType,true),
                  StructField(timestamp_prev,TimestampType,true)))
  </code></pre>