title: How do you add a numpy.array as a new column to a pyspark.SQL DataFrame?
body: <p>Here is the code to create a pyspark.sql DataFrame</p>
  
  <pre><code>import numpy as np
  import pandas as pd
  from pyspark import SparkContext
  from pyspark.sql import SQLContext
  df = pd.DataFrame(np.array([[1,2,3],[4,5,6],[7,8,9],[10,11,12]]), columns=['a','b','c'])
  sparkdf = sqlContext.createDataFrame(df, samplingRatio=0.1)
  </code></pre>
  
  <p>So that sparkdf looks like</p>
  
  <pre><code>a  b  c
  1  2  3
  4  5  6
  7  8  9
  10 11 12
  </code></pre>
  
  <p>Now I would like to add as a new column a numpy array (or even a list)</p>
  
  <pre><code>new_col = np.array([20,20,20,20])
  </code></pre>
  
  <p>But the standard way</p>
  
  <pre><code>sparkdf = sparkdf.withColumn('newcol', new_col)
  </code></pre>
  
  <p>fails.
  Probably an udf is the way to go, but I don't know how to create an udf that assigns one different value per DataFrame row, i.e. that iterates through new_col.
  I have looked at other pyspark and pyspark.sql but couldn't find a solution.
  Also I need to stay within pyspark.sql so not a scala solution. Thanks!</p>