title: Interpolation in PySpark throws java.lang.IllegalArgumentException
body: <p>I don't know how to interpolate in PySpark when the DataFrame contains many columns. Let me xplain.</p>
  <pre><code>from pyspark.sql.functions import to_timestamp
  
  df = spark.createDataFrame([
      (&quot;John&quot;,  &quot;A&quot;, &quot;2018-02-01 03:00:00&quot;, 60),  
      (&quot;John&quot;,  &quot;A&quot;, &quot;2018-02-01 03:03:00&quot;, 66),  
      (&quot;John&quot;,  &quot;A&quot;, &quot;2018-02-01 03:05:00&quot;, 70),  
      (&quot;John&quot;,  &quot;A&quot;, &quot;2018-02-01 03:08:00&quot;, 76),  
      (&quot;Mo&quot;,    &quot;A&quot;, &quot;2017-06-04 01:05:00&quot;, 10),  
      (&quot;Mo&quot;,    &quot;A&quot;, &quot;2017-06-04 01:07:00&quot;, 20),  
      (&quot;Mo&quot;,    &quot;B&quot;, &quot;2017-06-04 01:10:00&quot;, 35),  
      (&quot;Mo&quot;,    &quot;B&quot;, &quot;2017-06-04 01:11:00&quot;, 40),
  ], (&quot;webID&quot;, &quot;aType&quot;, &quot;timestamp&quot;, &quot;counts&quot;)).withColumn(
    &quot;timestamp&quot;, to_timestamp(&quot;timestamp&quot;)
  )
  </code></pre>
  <p>I need to group by <code>webID</code> and interpolate <code>counts</code> values at 1 minute interval. However, when I apply the below-shown code,</p>
  <pre><code>from operator import attrgetter
  from pyspark.sql.types import StructType
  from pyspark.sql.functions import pandas_udf, PandasUDFType
  
  def resample(schema, freq, timestamp_col = &quot;timestamp&quot;,**kwargs):
      @pandas_udf(
          StructType(sorted(schema, key=attrgetter(&quot;name&quot;))), 
          PandasUDFType.GROUPED_MAP)
      def _(pdf):
          pdf.set_index(timestamp_col, inplace=True)
          pdf = pdf.resample(freq).interpolate()
          pdf.ffill(inplace=True)
          pdf.reset_index(drop=False, inplace=True)
          pdf.sort_index(axis=1, inplace=True)
          return pdf
      return _
  
  
  df.groupBy(&quot;webID&quot;).apply(resample(df.schema, &quot;60S&quot;)).show()
  </code></pre>
  <p>Error:</p>
  <pre><code>py4j.protocol.Py4JJavaError: An error occurred while calling o371.showString.
  : org.apache.spark.SparkException: Job aborted due to stage failure: Task 77 in stage 31.0 failed 4 times, most recent failure: Lost task 77.3 in stage 31.0 (TID 812, 27faa516aadb4c40b7d7586d7493143c0021c825663, executor 2): java.lang.IllegalArgumentException
      at java.nio.ByteBuffer.allocate(ByteBuffer.java:334)
  </code></pre>