title: Pyspark: Converting RDD to RowMatrix
body: <p>I have an RDD of the form (id1,id2,score). The top(5) rows look like</p>
  
  <pre><code>[(41955624, 42044497, 3.913625989045223e-06),
  (41955624, 42039940, 0.0001018890937469129),
  (41955624, 42037797, 7.901647831291928e-05),
  (41955624, 42011137, -0.00016191403038589588),
  (41955624, 42006663, -0.0005302800991148567)]
  </code></pre>
  
  <p>I want to calculate similarity between id2 members based on scores. I'd like to use RowMatrix.columnSimilarity, but I need to convert it to a RowMatrix first. I want the matrix to be structured id1 x id2 -- i.e., make a row id out of id1 and a column id out of id2.</p>
  
  <p>If my data was smaller I could convert it to a Pyspark dataframe and then use pivot like</p>
  
  <pre><code>rdd_df.groupBy("id1").pivot("id2").sum("score")
  </code></pre>
  
  <p>but that borks with over 10,000 distinct id2, and I have much more than that.</p>
  
  <p>The naive
      rdd_Mat = la.RowMatrix(red)
  brings the data in as a 3-column matrix, which isn't what I want.</p>
  
  <p>Thanks a lot.</p>