title: Memory management: building sparse matrix by efficiently iterating through other sparse matrix
body: <p>I'm trying to build a pointwise mutual information matrix. I have a 60k by 60k scipy matrix of word co-occurrences, and I want to transform this into another sparse matrix where entry i,j corresponds to log( p(i,j) / p(i)*p(j) ), for words i and j. I remove positive values to get a PPMI matrix. I am looking for an efficient way to iterate through the first matrix to generate the second, without taking too much memory. </p>
  
  <p>I tried using a copy of the first matrix and iterating on that, and building a new CSR matrix row by row, adding each new row using vstack on 2 sparse matrices. Both processes are killed due to memory errors. What is the best way to build this matrix, and then to save it to reuse it later? </p>
  
  <pre><code>from scipy.sparse import vstack
  from scipy import sparse
  if(inplace):
      for i in range(ctxt_matrix.shape[0]): #row-wise operation    
          #for each row (word vector), reweigh this in 3 steps:
          # 1. get the probability of this context, instead of the raw count (divide by total words)
          # 2. divide this probability by the probability of this row/context occurring together randomly (multiply entry
          #        for word all the other words, do element wise division)
          # 3. take the log of this division, and reassign the row to this.
          row_pmi = np.log(np.divide((ctxt_matrix[i].toarray().T/total_words),(word_probas*word_probas[i]))).T
          if(cutoff_0):
              row_pmi[row_pmi&lt;0] = 0 #0 cutoff
          ctxt_matrix[i, :] = row_pmi
      print('PMI matrix building took:', time.time()-start)
      return ctxt_matrix
  
  else:
      #same as above, but on a new matrix, using vstack.
      pmi_matrix = scipy.sparse.csr_matrix((1, ctxt_matrix.shape[1]))
      for i in range(ctxt_matrix.shape[0]): #row-wise operation
          row_pmi = scipy.sparse.csr_matrix(np.log(np.divide( ((ctxt_matrix[i].toarray().T)/total_words) , word_probas*word_probas[i] )).T)
          if(cutoff_0):
              row_pmi[row_pmi&lt;0] = 0 #0 cutoff            
          pmi_matrix = scipy.vstack((pmi_matrix, row_pmi))
          del row_pmi
      print('PMI matrix building took:', time.time()-start)
      return pmi_matrix
  </code></pre>
  
  <p>TL;DR - I need to do a row-wise operation creating a sparse matrix by iterating over another one. Here's some simplified code for an idea of what i'm doing:</p>
  
  <pre><code>from scipy import sparse
  import time
  start = time.time()
  ctxt_matrix = scipy.sparse.csr_matrix(scipy.sparse.rand(5000, 5000))
  for i in range(ctxt_matrix.shape[0]):   
      row_pmi = np.log(ctxt_matrix[i,:].toarray().T/500) #some row-wise operation on the other matrix
      row_pmi[row_pmi&lt;0] = 0 # don't store negatives in memory
      ctxt_matrix[i,:] = scipy.sparse.csr_matrix(row_pmi).T
      ctxt_matrix[i, :].eliminate_zeros()
  print('PMI matrix building took:', time.time()-start)
  </code></pre>