title: How to flatten columns of type array of structs (as returned by Spark ML API)?
body: <p>Maybe it's just because I'm relatively new to the API, but I feel like Spark ML methods often return DFs that are unnecessarily difficult to work with.</p>
  
  <p>This time, it's the ALS model that's tripping me up. Specifically, the recommendForAllUsers method. Let's reconstruct the type of DF it would return:</p>
  
  <pre class="lang-scala prettyprint-override"><code>scala&gt; val arrayType = ArrayType(new StructType().add("itemId", IntegerType).add("rating", FloatType))
  
  scala&gt; val recs = Seq((1, Array((1, .7), (2, .5))), (2, Array((0, .9), (4, .1)))).
    toDF("userId", "recommendations").
    select($"userId", $"recommendations".cast(arrayType))
  
  scala&gt; recs.show()
  </code></pre>
  
  
  
  <pre><code>+------+------------------+
  |userId|   recommendations|
  +------+------------------+
  |     1|[[1,0.7], [2,0.5]]|
  |     2|[[0,0.9], [4,0.1]]|
  +------+------------------+
  </code></pre>
  
  <pre class="lang-scala prettyprint-override"><code>scala&gt; recs.printSchema
  </code></pre>
  
  
  
  <pre><code>root
   |-- userId: integer (nullable = false)
   |-- recommendations: array (nullable = true)
   |    |-- element: struct (containsNull = true)
   |    |    |-- itemId: integer (nullable = true)
   |    |    |-- rating: float (nullable = true)
  </code></pre>
  
  <p>Now, I only care about the <code>itemId</code> in the <code>recommendations</code> column. After all, the method is <code>recommendForAllUsers</code> not <code>recommendAndScoreForAllUsers</code> (ok ok I'll stop being sassy...)</p>
  
  <p>How do I do this??</p>
  
  <p>I thought I had it when I created a UDF:</p>
  
  <pre class="lang-scala prettyprint-override"><code>scala&gt; val itemIds = udf((arr: Array[(Int, Float)]) =&gt; arr.map(_._1))
  </code></pre>
  
  <p>but that produces an error:</p>
  
  <pre><code>scala&gt; recs.withColumn("items", items($"recommendations"))
  org.apache.spark.sql.AnalysisException: cannot resolve 'UDF(recommendations)' due to data type mismatch: argument 1 requires array&lt;struct&lt;_1:int,_2:float&gt;&gt; type, however, '`recommendations`' is of array&lt;struct&lt;itemId:int,rating:float&gt;&gt; type.;;
  'Project [userId#87, recommendations#92, UDF(recommendations#92) AS items#238]
  +- Project [userId#87, cast(recommendations#88 as array&lt;struct&lt;itemId:int,rating:float&gt;&gt;) AS recommendations#92]
     +- Project [_1#84 AS userId#87, _2#85 AS recommendations#88]
        +- LocalRelation [_1#84, _2#85]
  </code></pre>
  
  <p>Any ideas? thanks!</p>