title: Cumulative sum in Spark
body: <p>I want to do cumulative sum in Spark. Here is the register table (input):</p>
  
  <pre><code>+---------------+-------------------+----+----+----+
  |     product_id|          date_time| ack|val1|val2|
  +---------------+-------------------+----+----+----+
  |4008607333T.upf|2017-12-13:02:27:01|3-46|  53|  52|
  |4008607333T.upf|2017-12-13:02:27:03|3-47|  53|  52|
  |4008607333T.upf|2017-12-13:02:27:08|3-46|  53|  52|
  |4008607333T.upf|2017-12-13:02:28:01|3-47|  53|  52|
  |4008607333T.upf|2017-12-13:02:28:07|3-46|  15|   1|
  +---------------+-------------------+----+----+----+
  </code></pre>
  
  <p>Hive query:</p>
  
  <pre><code>select *, SUM(val1) over ( Partition by product_id, ack order by date_time rows between unbounded preceding and current row ) val1_sum, SUM(val2) over ( Partition by product_id, ack order by date_time rows between unbounded preceding and current row ) val2_sum from test
  </code></pre>
  
  <p>Output:</p>
  
  <pre><code>+---------------+-------------------+----+----+----+-------+--------+
  |     product_id|          date_time| ack|val1|val2|val_sum|val2_sum|
  +---------------+-------------------+----+----+----+-------+--------+
  |4008607333T.upf|2017-12-13:02:27:01|3-46|  53|  52|     53|      52|
  |4008607333T.upf|2017-12-13:02:27:08|3-46|  53|  52|    106|     104|
  |4008607333T.upf|2017-12-13:02:28:07|3-46|  15|   1|    121|     105|
  |4008607333T.upf|2017-12-13:02:27:03|3-47|  53|  52|     53|      52|
  |4008607333T.upf|2017-12-13:02:28:01|3-47|  53|  52|    106|     104|
  +---------------+-------------------+----+----+----+-------+--------+
  </code></pre>
  
  <p>Using Spark logic, I am getting same above output:</p>
  
  <pre><code>import org.apache.spark.sql.expressions.Window
  val w = Window.partitionBy('product_id, 'ack).orderBy('date_time)
  import org.apache.spark.sql.functions._
  
  val newDf = inputDF.withColumn("val_sum", sum('val1) over w).withColumn("val2_sum", sum('val2) over w)
  newDf.show
  </code></pre>
  
  <p>However, when I try this logic on spark cluster <code>val_sum</code> value will be half of the cumulative sum and something time it is different. I don't know why it is happening on spark cluster. Is it due to partitions?</p>
  
  <p>How I can do cumulative sum of a column on a spark cluster?</p>