title: python spark: narrowing down most relevant features using PCA
body: <p>I am using spark 2.2 with python. I am using PCA from ml.feature module. I am using VectorAssembler to feed my features to PCA. To clarify, let's say I have a table with three columns col1, col2 and col3 then I am doing:</p>
  
  <pre><code>from pyspark.ml.feature import VectorAssembler
  assembler = VectorAssembler(inputCols=table.columns, outputCol="features")
  df = assembler.transform(table).select("features")
  from pyspark.ml.feature import PCA
  pca = PCA(k=2, inputCol="features", outputCol="pcaFeatures")
  model = pca.fit(df)
  </code></pre>
  
  <p>At this time I have run PCA  with 2 components and I can look at its values as:</p>
  
  <pre><code>m = model.pc.values.reshape(3, 2)
  </code></pre>
  
  <p>which corresponds to 3 (= number  of columns in  my original table) rows and 2 (= number of components in my PCA) columns. My question is are the three rows here in the same order in which I had specified my input columns to the vector assembler above? To clarify it further does the above matrix correspond to:</p>
  
  <pre><code>          | PC1 | PC2 |
   ---------|-----|-----|
      col1  |     |     |
   ---------|-----|-----|
      col2  |     |     |
   ---------|-----|-----|
      col3  |     |     |
   ---------+-----+-----+
  </code></pre>
  
  <p>Note that the example here is only for clarity. In my real problem I am dealing with ~1600 columns and bunch of selections. I could not find  any definitive answer to this  in spark documentation. I want to do this to pick best columns / features from my original table to train my model based on the top principal components. Or is there anything else / better in spark ML PCA that I should be looking at to deduce such result?</p>
  
  <p>Or I cannot use PCA for this and have to use other techniques like spearman ranking etc.?</p>