title: print CoordinateMatrix after using RowMatrix.columnSimilarities in Apache Spark
body: <p>I am using spark mllib for one of my projects in which I need to calculate document similarities.</p>
  
  <p>I first converted the documents to vectors using tf-idf transform of the mllib, then converted it into RowMatrix and used the columnSimilarities() method.</p>
  
  <p>I referred to <a href="https://spark.apache.org/docs/1.2.1/mllib-feature-extraction.html#tf-idf" rel="nofollow">tf-idf</a> documentation and used the <a href="https://databricks.com/blog/2014/10/20/efficient-similarity-algorithm-now-in-spark-twitter.html" rel="nofollow">DIMSUM</a> implementation for cosine similarities. </p>
  
  <p>in spark-shell this is the scala code is executed:</p>
  
  <pre><code>import org.apache.spark.rdd.RDD
  import org.apache.spark.SparkContext
  import org.apache.spark.mllib.feature.HashingTF
  import org.apache.spark.mllib.linalg.Vector
  import org.apache.spark.mllib.feature.IDF
  import org.apache.spark.mllib.linalg.distributed.RowMatrix
  
  val documents = sc.textFile("test1").map(_.split(" ").toSeq)
  val hashingTF = new HashingTF()
  
  val tf = hashingTF.transform(documents)
  tf.cache()
  
  val idf = new IDF().fit(tf)
  val tfidf = idf.transform(tf)
  
  // now use the RowMatrix to compute cosineSimilarities
  // which implements DIMSUM algorithm
  
  val mat = new RowMatrix(tfidf)
  val sim = mat.columnSimilarities() // returns a CoordinateMatrix
  </code></pre>
  
  <p>Now let's say my <code>input file</code>, <code>test1</code> in this code block is a simple file with 5 short documents (less than 10 terms each), one on each row.</p>
  
  <p><strong>Since I am just testing this code, I would like to see the output of <code>mat.columnSimilarities()</code> which is in object <code>sim</code>.
  I would like to see the similarity of 1st document vector with 2nd, 3rd and so on.</strong></p>
  
  <p>I referred to spark <a href="https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.mllib.linalg.distributed.CoordinateMatrix" rel="nofollow">documentation</a> for <code>CoordinateMatrix</code> which is the type of object returned by <code>columnSimilarities</code> method of <code>RowMatrix</code> class and referred by <code>sim</code>.</p>
  
  <p>By going through more documentation, I figured I could convert the CoordinateMatrix to RowMatrix, then convert the rows of RowMatrix to arrays and then print like this <code>println(sim.toRowMatrix().rows.toArray().mkString("\n"))</code> .</p>
  
  <p>But that gives some output which I couldn't understand.</p>
  
  <p>Can anyone help? Any kind of resource links etc would help a lot!</p>
  
  <p>Thanks!</p>