title: How to recommend most similar users using Spark ML
body: <p>I have a data about user preferences for specific items in the form:</p>
  
  <pre><code>user, item, preference  
  1,    75,   0.89  
  2,    168,  0.478  
  2,    99,   0.321  
  3,    31,   0.012
  </code></pre>
  
  <p>For each user, I need to make a top N most similar users based on their preferences.The system has thousands of users, and items. Recommendations would be generated in batch process, and served on Elasticsearch.
  In Mahout, simple algorithm that would do it might look like:</p>
  
  <pre><code>  val similarity:UserSimilarity=new PearsonCorrelationSimilarity(dataModel)
    val neighborhood:UserNeighborhood  = new ThresholdUserNeighborhood(0.75, similarity, dataModel)
    val userBasedRecommender:UserBasedRecommender  = new GenericUserBasedRecommender(dataModel, neighborhood, similarity);
    val userid=1
    val howMany=10
    val mostSimilarUserIDs = userBasedRecommender.mostSimilarUserIDs(userid, howMany);
  </code></pre>
  
  <p>I would like to implement that using Spark ML, but could not find any similar example that could help me to move on.</p>
  
  <p>So far, I implemented approach using cosine similarity that compare one user features vector with other users:</p>
  
  <pre><code>def cosineSimilarity(vec1: DoubleMatrix, vec2: DoubleMatrix): Double=
  {
      vec1.dot(vec2)/(vec1.norm2()*vec2.norm2())
  }
  def user2usersimilarity(userid:Integer, recNumber:Integer): Unit ={
      val userFactor=model.userFeatures.lookup(userid).head
      val userVector=new DoubleMatrix(userFactor)
      val s1=cosineSimilarity(userVector,userVector)
      val sims=model.userFeatures.map{case(id,factor)=&gt;
          val factorVector=new DoubleMatrix(factor)
          val sim=cosineSimilarity(factorVector, userVector)
          (id,sim)
      }
      val sortedSims=sims.top(recNumber+1)(Ordering.by[(Int, Double),Double] {case(id, similarity)=&gt;similarity})
      println(sortedSims.slice(1,recNumber+1).mkString("\n"))
   }
  </code></pre>
  
  <p>This approach works fine with the MovieLens dataset in terms of quality of recommendations. However, my concern is related to performance of such algorithm. Since I have to generate recommendations for all users in the system, with this approach I would compare each user with all other users in the system. </p>
  
  <p>I would appreciate if somebody could suggest how to limit comparison of the user to top N neighbors, or some other algorithm that would work better in my use case.</p>