title: collect_list by preserving order based on another variable
body: <p>I am trying to create a new column of lists in Pyspark using a groupby aggregation on existing set of columns. An example input data frame is provided below:</p>
  
  <pre><code>------------------------
  id | date        | value
  ------------------------
  1  |2014-01-03   | 10 
  1  |2014-01-04   | 5
  1  |2014-01-05   | 15
  1  |2014-01-06   | 20
  2  |2014-02-10   | 100   
  2  |2014-03-11   | 500
  2  |2014-04-15   | 1500
  </code></pre>
  
  <p>The expected output is:</p>
  
  <pre><code>id | value_list
  ------------------------
  1  | [10, 5, 15, 20]
  2  | [100, 500, 1500]
  </code></pre>
  
  <p>The values within a list are sorted by the date.</p>
  
  <p>I tried using collect_list as follows:</p>
  
  <pre><code>from pyspark.sql import functions as F
  ordered_df = input_df.orderBy(['id','date'],ascending = True)
  grouped_df = ordered_df.groupby("id").agg(F.collect_list("value"))
  </code></pre>
  
  <p>But collect_list doesn't guarantee order even if I sort the input data frame by date before aggregation.</p>
  
  <p>Could someone help on how to do aggregation by preserving the order based on a second (date) variable?</p>