title: Pairwise similarity score in a sparse matrix
body: <p>I am doing a CF item-based problem.</p>
  <p>I have made a TF-idf Matrix (in pandas dataframe) that looks something like this. This is a large and extremely sparse matrix, say, 10^4 x 10^6, and each row has only 5 to 10 non-zero values.</p>
  <pre><code>|  p_Id     | 'coffee' | 'beach' | 'billiards' | ...|
  |-----------|----------|---------|-------------|----|
  |    &quot;A&quot;    |     0    |    0    | 0.3         | ...|
  |    &quot;B&quot;    |     0.1  |    0.   | 0           | ...| 
  </code></pre>
  <p>I want to calculate similarity score pairwise (using the cosine similarity score for example). I have two issues with my code below that I need to overcome:</p>
  <ol>
  <li>Too slow. It takes O(N^2).</li>
  <li>(MOST IMPORTANT) Similarity scores are too small that they are displayed as 0.</li>
  </ol>
  <p>I look forward to your support. Thanks.</p>
  <pre><code>class SimilarityScore: 
    __id_column_name = 'p_id'
    simScore_dict = {}
    # def __pairScore(self, p1, p2, df):
    #   data1 = df[df[self.__id_column_name] == p1]
    #   data2 = df[df[self.__id_column_name] == p2]
    #   return cosine_similarity(data1, data2)
  
    def __pairScore(self, data1, data2):
      return cosine_similarity(data1, data2)[0][0]
    
    def calc_simScores(self,df):
      for i in range(0, len(df)):
        for j in range(i+1, len(df)):
          p1 = int(df.iloc[i][self.__id_column_name])
          p2 = int(df.iloc[j][self.__id_column_name])
          data1 = df.iloc[i][1:].values.reshape(1,-1)
          data2 = df.iloc[j][1:].values.reshape(1,-1)
          # print(data1)
          simScore = self.__pairScore(data1, data2)
          if(p1 not in self.simScore_dict):
            self.simScore_dict[p1] = {}
          self.simScore_dict[p1][p2] = simScore
          if(p2 not in self.simScore_dict):
            self.simScore_dict[p2]={}
          self.simScore_dict[p2][p1] = simScore
  </code></pre>