title: Spark Time Series - Custom Group By 10 Minute Intervals: Improve Performance
body: <p>We have time series data (timestamp in us since 1970 and integer data value):</p>
  
  <pre><code># load data and cache it
  df_cache = readInData() # read data from several files (paritioned by hour)
  df_cache.persist(pyspark.StorageLevel.MEMORY_AND_DISK)
  df_cache.agg({"data": "max"}).collect()
  
  # now data is cached
  df_cache.show()
  +--------------------+---------+
  |                time|     data| 
  +--------------------+---------+
  |1.448409599861109E15|1551.7468|
  |1.448409599871109E15|1551.7463|
  |1.448409599881109E15|1551.7468|
  </code></pre>
  
  <p>Now we want to calculate some non-trivial things on top of 10 Minute time windows using an external python library. In order to do so, we need to load the data of each time frame in memory, apply the external function and store the result. Therefore a User Defined Aggregate Function (UDAF) is not possible.</p>
  
  <p>Now the problem is, when we apply the GroupBy to the RDD, it is very slow. </p>
  
  <pre><code>df_cache.rdd.groupBy(lambda x: int(x.time / 600e6) ). \ # create 10 minute groups
               map(lambda x: 1). \ # do some calculations, e.g. external library
               collect() # get results
  </code></pre>
  
  <p>This operation takes for 120Mio samples (100Hz data) on two nodes with 6GB Ram around 14 minutes. Spark Details for the groupBy stage:</p>
  
  <pre><code>Total Time Across All Tasks: 1.2 h
  Locality Level Summary: Process local: 8
  Input Size / Records: 1835.0 MB / 12097
  Shuffle Write: 1677.6 MB / 379
  Shuffle Spill (Memory): 79.4 GB
  Shuffle Spill (Disk): 1930.6 MB
  </code></pre>
  
  <p>If I use a simple python script and let it iterate over the input files, it takes way less time to finish.</p>
  
  <p>How can this job be optimized in spark?</p>