title: Store DenseVector in DataFrame column in PySpark
body: <p>I am trying to store a DenseVector into a DataFrame in a new column.</p>
  
  <p>I tried the following code, but got an <code>AttributeError</code> saying <code>'numpy.ndarray' object has no attribute '_get_object_id'</code>.</p>
  
  <pre><code>from pyspark.sql import functions
  from pyspark.mllib.linalg import Vectors
  
  df = spark.createDataFrame([{'name': 'Alice', 'age': 1},
                              {'name': 'Bob', 'age': 2}])
  
  vec = Vectors.dense([1.0, 3.0, 2.9])
  
  df.withColumn('vector', functions.lit(vec))
  </code></pre>
  
  <p>I'm hoping to store a vector per row for computation purpose. Any help is appreciated.</p>
  
  <p>[Python 3.7.3, Spark version 2.4.3, via Jupyter All-Spark-Notebook]</p>
  
  <p><strong>EDIT</strong></p>
  
  <p>I tried to follow the answer <a href="https://stackoverflow.com/questions/49832877/adding-a-vectors-column-to-a-pyspark-dataframe">here</a> as suggested by Florian, but I could not adapt the udf to take in a custom pre-constructed vector.</p>
  
  <pre><code>conv = functions.udf(lambda x: DenseVector(x), VectorUDT())
  # Same with
  # conv = functions.udf(lambda x: x, VectorUDT())
  
  df.withColumn('vector', conv(vec)).show()
  </code></pre>
  
  <p>I get this error :</p>
  
  <pre><code>TypeError: Invalid argument, not a string or column: [1.0,3.0,2.9] of type &lt;class 'pyspark.mllib.linalg.DenseVector'&gt;. For column literals, use 'lit', 'array', 'struct' or 'create_map' function.
  </code></pre>