title: Alternative to groupBy in Spark
body: <p>I have a Dataframe df with next info:</p>
  
  <pre><code>id   json_data
  1    {a: "1", b: "2"} 
  1    {a: "1", b: "3"} 
  1    {a: "1", b: "4"}
  2    {a: "1", b: "2"} 
  2    {a: "1", b: "6"}
  </code></pre>
  
  <p>And I need the next final result:</p>
  
  <pre><code>id   json_data
  1    [{a: "1", b: "2"},{a: "1", b: "3"},{a: "1", b: "4"}]
  2    [{a: "1", b: "2"},{a: "1", b: "6"}]
  </code></pre>
  
  <p>I tried two different approaches, using Window functions and groupBy respectively. With both approaches I am getting the desired result.</p>
  
  <p>1º approach: </p>
  
  <pre><code>var user_window = Window.partitionBy("id").orderBy("id")
  val df2 = df.withColumn("json_data",
      collect_list($"json_data").over(user_window))
            .withColumn("rank", row_number().over(user_window))
            .where("rank = 1")
  </code></pre>
  
  <p>2º approach:</p>
  
  <pre><code>val df2 = df.groupBy(df("id")).agg(collect_list($"json_data").as("json_data"))
  </code></pre>
  
  <p>With both approaches I am getting same performance. But reading documentation about Spark, it seems both approaches are not efficient, since rows with the same key will need to travel across the cluster (shuffled) to be together. I am showing a small example, because in production I have a high volume of data. And doing the group or using Window functions it takes a long time. </p>
  
  <p>Any alternatives in order to do it?</p>