title: PCA in Spark 2.3.0 Examples with PySpark
body: <p>I have a Spark dataframe I would like to use to run a simple PCA example. I have looked at <a href="https://spark.apache.org/docs/2.2.0/api/python/pyspark.ml.html#pyspark.ml.feature.PCA" rel="nofollow noreferrer">this example</a> and notice this works because they transpose the features into vectors:</p>
  
  <pre><code>from pyspark.ml.linalg import Vectors
  &gt;&gt;&gt; data = [(Vectors.sparse(5, [(1, 1.0), (3, 7.0)]),),
  ...     (Vectors.dense([2.0, 0.0, 3.0, 4.0, 5.0]),),
  ...     (Vectors.dense([4.0, 0.0, 0.0, 6.0, 7.0]),)]
  &gt;&gt;&gt; df = spark.createDataFrame(data,["features"])
  &gt;&gt;&gt; pca = PCA(k=2, inputCol="features", outputCol="pca_features")
  </code></pre>
  
  <p>I am trying to reproduce the same kind of simple PCA by using a Spark Dataframe I have created my self. How would I transform my Spark DataFrame into a form similar to the above so I could run it with one input column and one output column? </p>
  
  <p>I looked into using RowMatrix <a href="https://spark.apache.org/docs/2.2.0/mllib-dimensionality-reduction.html" rel="nofollow noreferrer">as shown here</a> but I am not understanding if this is the way to go (see error below).</p>
  
  <pre><code>&gt;&gt;&gt;from pyspark.mllib.linalg import Vectors
  &gt;&gt;&gt;from pyspark.mllib.linalg.distributed import RowMatrix
  &gt;&gt;&gt;from pyspark.ml.feature import PCA
  &gt;&gt;&gt;master = pd.read_parquet('master.parquet',engine='fastparquet')
  &gt;&gt;&gt;A = sc.parallelize(master)
  &gt;&gt;&gt;mat = RowMatrix(A)
  &gt;&gt;&gt;pc = mat.computePrincipalComponents(4)
  </code></pre>
  
  <blockquote>
    <p>Py4JJavaError: An error occurred while calling
    o382.computePrincipalComponents. : org.apache.spark.SparkException:
    Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times,
    most recent failure: Lost task 0.0 in stage 1.0 (TID 1, localhost,
    executor driver): org.apache.spark.api.python.PythonException:
    Traceback (most recent call last)</p>
  </blockquote>