title: Avoid performance impact of a single partition mode in Spark window functions
body: <p>My question is triggered by the use case of calculating the differences between consecutive rows in a spark dataframe.</p>
  
  <p>For example, I have:</p>
  
  <pre><code>&gt;&gt;&gt; df.show()
  +-----+----------+
  |index|      col1|
  +-----+----------+
  |  0.0|0.58734024|
  |  1.0|0.67304325|
  |  2.0|0.85154736|
  |  3.0| 0.5449719|
  +-----+----------+
  </code></pre>
  
  <p>If I choose to calculate these using "Window" functions, then I can do that like so:</p>
  
  <pre><code>&gt;&gt;&gt; winSpec = Window.partitionBy(df.index &gt;= 0).orderBy(df.index.asc())
  &gt;&gt;&gt; import pyspark.sql.functions as f
  &gt;&gt;&gt; df.withColumn('diffs_col1', f.lag(df.col1, -1).over(winSpec) - df.col1).show()
  +-----+----------+-----------+
  |index|      col1| diffs_col1|
  +-----+----------+-----------+
  |  0.0|0.58734024|0.085703015|
  |  1.0|0.67304325| 0.17850411|
  |  2.0|0.85154736|-0.30657548|
  |  3.0| 0.5449719|       null|
  +-----+----------+-----------+
  </code></pre>
  
  <p><strong>Question</strong>: I explicitly partitioned the dataframe in a single partition. What is the performance impact of this and, if there is, why is that so and how could I avoid it? Because when I do not specify a partition, I get the following warning:</p>
  
  <pre><code>16/12/24 13:52:27 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
  </code></pre>