title: Calculate Cosine Similarity Spark Dataframe
body: <p>I am using Spark Scala to calculate cosine similarity between the Dataframe rows. </p>
  
  <p>Dataframe format is below</p>
  
  <pre class="lang-none prettyprint-override"><code>root
      |-- SKU: double (nullable = true)
      |-- Features: vector (nullable = true)
  </code></pre>
  
  <p>Sample of the dataframe below </p>
  
  <pre class="lang-none prettyprint-override"><code>    +-------+--------------------+
      |    SKU|            Features|
      +-------+--------------------+
      | 9970.0|[4.7143,0.0,5.785...|
      |19676.0|[5.5,0.0,6.4286,4...|
      | 3296.0|[4.7143,1.4286,6....|
      |13658.0|[6.2857,0.7143,4....|
      |    1.0|[4.2308,0.7692,5....|
      |  513.0|[3.0,0.0,4.9091,5...|
      | 3753.0|[5.9231,0.0,4.846...|
      |14967.0|[4.5833,0.8333,5....|
      | 2803.0|[4.2308,0.0,4.846...|
      |11879.0|[3.1429,0.0,4.5,4...|
      +-------+--------------------+
  </code></pre>
  
  <p>I tried to transpose the matrix and check the following mentioned links.<a href="https://stackoverflow.com/questions/43921636/apache-spark-python-cosine-similarity-over-dataframes" title="Apache Spark Python Cosine Similarity over DataFrames">Apache Spark Python Cosine Similarity over DataFrames</a>, <a href="https://stackoverflow.com/questions/32645231/calculating-cosine-similarity-by-featurizing-the-text-into-vector-using-tf-idf" title="calculating-cosine-similarity-by-featurizing-the-text-into-vector-using-tf-idf">calculating-cosine-similarity-by-featurizing-the-text-into-vector-using-tf-idf</a> But I believe there is a better solution</p>
  
  <p>I am tried the below sample code</p>
  
  <pre><code>val irm = new IndexedRowMatrix(inClusters.rdd.map {
    case (v,i:Vector) =&gt; IndexedRow(v, i)
  
  
  }).toCoordinateMatrix.transpose.toRowMatrix.columnSimilarities
  </code></pre>
  
  <p>But I got the below error</p>
  
  <pre><code>Error:(80, 12) constructor cannot be instantiated to expected type;
   found   : (T1, T2)
   required: org.apache.spark.sql.Row
        case (v,i:Vector) =&gt; IndexedRow(v, i)
  </code></pre>
  
  <p>I checked the following Link <a href="https://stackoverflow.com/questions/31567989/apache-spark-how-to-create-a-matrix-from-a-dataframe" title="Apache Spark: How to create a matrix from a DataFrame?">Apache Spark: How to create a matrix from a DataFrame?</a> But can't do it using Scala</p>