title: Memory Error when trying to compute Cosine Similarity Matrix on TFIDF vector
body: <p>I am <strong>trying to build a Movie Plot (content) based recommender function</strong> in python3 to which takes a movie title as an argument and outputs movies with most similar plots.</p>
  
  <p>My wrangled data has <strong>Shape of (45466, 8)</strong> 
  This is what the head of wrangled data looks like:</p>
  
  <p><a href="https://i.stack.imgur.com/W4b7U.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/W4b7U.png" alt="enter image description here"></a></p>
  
  <p>I am using the <code>fit-transform</code> method from <code>sklearn.feature_extraction.text</code>'s <code>TfidVectorizer</code> to build the required TF-IDF matrix on the <strong>overview</strong> feature like so:</p>
  
  <pre><code>tfidf = TfidfVectorizer(stop_words='english')
  
  tfidf_matrix = tfidf.fit_transform(movies['overview'])
  </code></pre>
  
  <p>This results in a matrix of shape (45466, 75827) for the overview of every movie which means--after removing common stop words--<strong>there are 75827 distinct words in the overview soup of all the 45466 movies combined</strong>. </p>
  
  <p>Post this I want to compute the <strong>pairwise cosine similarity score</strong> of every movie based on the tfidf matrix constructed above. This should give me a <strong>45466 x 45466 matrix</strong> where the (i-th, j-th) cell would be the similarity score between movies i &amp; j. I am using <code>sklearn.metrics.pairwise</code>'s <code>linear_kernel</code> method to compute the same:</p>
  
  <p><code>cos_sim = linear_kernel(tfidf_matrix, tfidf_matrix)</code></p>
  
  <p>This is where python3 throws out a Memory Error: </p>
  
  <pre><code>----
  MemoryError                               Traceback (most recent call last)
  &lt;ipython-input-5-d884b8c29067&gt; in &lt;module&gt;
        1 #STEP 2: COMPUTING THE COSINE SIMILARITY MATRIX---------------------------
  ----&gt; 2 cosine_sim = linear_kernel(tv_mat, tv_mat)
  
  ~/.local/lib/python3.6/site-packages/sklearn/metrics/pairwise.py in linear_kernel(X, Y, dense_output)
      990     """
      991     X, Y = check_pairwise_arrays(X, Y)
  --&gt; 992     return safe_sparse_dot(X, Y.T, dense_output=dense_output)
      993 
      994 
  
  ~/.local/lib/python3.6/site-packages/sklearn/utils/extmath.py in safe_sparse_dot(a, b, dense_output)
      153     if (sparse.issparse(a) and sparse.issparse(b)
      154             and dense_output and hasattr(ret, "toarray")):
  --&gt; 155         return ret.toarray()
      156     return ret
      157 
  
  ~/.local/lib/python3.6/site-packages/scipy/sparse/compressed.py in toarray(self, order, out)
     1023         if out is None and order is None:
     1024             order = self._swap('cf')[0]
  -&gt; 1025         out = self._process_toarray_args(order, out)
     1026         if not (out.flags.c_contiguous or out.flags.f_contiguous):
     1027             raise ValueError('Output array must be C or F contiguous')
  
  ~/.local/lib/python3.6/site-packages/scipy/sparse/base.py in _process_toarray_args(self, order, out)
     1187             return out
     1188         else:
  -&gt; 1189             return np.zeros(self.shape, dtype=self.dtype, order=order)
     1190 
     1191 
  
  MemoryError: Unable to allocate 15.4 GiB for an array with shape (45466, 45466) and data type float64
  </code></pre>
  
  <p>I have <strong>8G RAM</strong> and <strong>1G swap partition</strong> on a system running <strong>Ubuntu 18.04</strong>. How do I solve this problem?** Can't upgrade RAM soon enough. </p>
  
  <ul>
  <li>I could perhaps <strong>try this on with a much smaller dataset</strong> to begin with but <strong>that isn't the solution I am looking for</strong>.</li>
  <li>I could perhaps <strong>split</strong> <code>tfidf_matrix</code> <strong>in half and compute the cosine similarity of each half with itself and the other half</strong> and put them back together. Would that work?</li>
  <li>Is there any <strong>simpler solution</strong> that I might be missing?  </li>
  </ul>
  
  <p>TIA!</p>