title: How to aggregate map columns after groupBy?
body: <p>I need to union two dataframes and combine the columns by keys. The two datafrmaes have the same schema, for example:</p>
  
  <pre><code>root
  |-- id: String (nullable = true)
  |-- cMap: map (nullable = true)
  |    |-- key: string
  |    |-- value: string (valueContainsNull = true)
  </code></pre>
  
  <p>I want to group by "id" and aggregate the "cMap" together to deduplicate.
  I tried the code:</p>
  
  <pre><code>val df = df_a.unionAll(df_b).groupBy("id").agg(collect_list("cMap") as "cMap").
  rdd.map(x =&gt; {
      var map = Map[String,String]()
      x.getAs[Seq[Map[String,String]]]("cMap").foreach( y =&gt; 
          y.foreach( tuple =&gt;
          {
              val key = tuple._1
              val value = tuple._2
              if(!map.contains(key))//deduplicate
                  map += (key -&gt; value)
          }))
  
      Row(x.getAs[String]("id"),map)
      })
  </code></pre>
  
  <p>But it seems collect_list cannnot be used to map structure:</p>
  
  <pre><code>org.apache.spark.sql.AnalysisException: No handler for Hive udf class org.apache.hadoop.hive.ql.udf.generic.GenericUDAFCollectList because: Only primitive type arguments are accepted but map&lt;string,string&gt; was passed as parameter 1..;
  </code></pre>
  
  <p>Is there other solution for the problem?</p>