title: How can I create a Spark DataFrame, where one column has array values?
body: <p>I have the following code, creating recommendations for some users:</p>
  
  
  
  <pre class="lang-python prettyprint-override"><code>als = ALS(rank=4, maxIter=3, regParam=0.01, implicitPrefs=True, alpha=40,
                userCol="userId", itemCol="itemId", ratingCol="rating")
  model = als.fit(train_df)
  </code></pre>
  
  <p>Now I want to take my test DataFrame and expand it to hold the top-k predictions for each user, as an array. For example, if my test DataFrame is:</p>
  
  <pre class="lang-python prettyprint-override"><code>userId    ProductId
  1         2
  1         3
  2         7
  3         4
  </code></pre>
  
  <p>It should transform into:</p>
  
  <pre class="lang-python prettyprint-override"><code>userId    ProductId    Predictions
  1         2            [2, 4, 6, 8 , 3, 9, 10, 3, 50, 14]
  1         3            ...
  2         7            ...
  3         4            ...
  </code></pre>
  
  <p>I am using a <code>udf</code> like this:</p>
  
  <pre class="lang-python prettyprint-override"><code>predictions_udf = udf(user_prediction, ArrayType(DoubleType()))
  predictions_df = test_df.select('*').withColumn('Predictions', predictions_udf())
  </code></pre>
  
  <p>How should I implement this <code>user_prediction</code> method. I have tried everything but could not pass the model, or the DataFrame as a method argument, and cannot return the recommendations as an <code>ArrayType()</code>.</p>