title: How to efficiently calculate huge matrix multiplication (tfidf features) in Python?
body: <p>I currently want to calculate all-pair document similarity using cosine similarity and Tfidf features in python. My basic approach is the following:</p>
  
  <pre><code>from sklearn.feature_extraction.text import TfidfVectorizer
  #c = [doc1, doc2, ..., docn]
  vec = TfidfVectorizer()
  X = vec.fit_transform(c)
  del vec
  Y = X * X.T
  </code></pre>
  
  <p>Works perfectly fine, but unfortunately, not for my very large datasets. X has a dimension of <code>(350363, 2526183)</code> and hence, the output matrix Y should have <code>(350363, 350363)</code>. X is very sparse due to the tfidf features, and hence, easily fits into memory (around 2GB only). Yet, the multiplication gives me a memory error after running for some time (even though the memory is not full but I suppose that scipy is so clever as to expect the memory usage).</p>
  
  <p>I have already tried to play around with the dtypes without any success. I have also made sure that numpy and scipy have their BLAS libraries linked -- whereas this does not have an effect on the csr_matrix dot functionality as it is implemented in C. I thought of maybe using things like memmap, but I am not sure about that.</p>
  
  <p>Does anyone have an idea of how to best approach this?</p>