title: Memory Error: sklearn Knearest Neighbor knn
body: <p>I am working on a Windows 10 64bit 12gb RAM core i5.</p>
  
  <p>right now im testing with amazon dataset around 30k</p>
  
  <p>246621 items in training data, 61656 in test data</p>
  
  <p>i have try with other machine learning in scikit learn works fine but with Knn got problem with memory error.</p>
  
  <p>my code</p>
  
  <pre><code>knn = KNeighborsClassifier(n_neighbors=5).fit(X_train_tfidf, y_train)
  prediction['knn'] = knn.predict(X_test_tfidf)
  accuracy_score(y_test, prediction['knn'])*100
  </code></pre>
  
  <p>my error</p>
  
  <pre><code>MemoryError                               Traceback (most recent call last)
  &lt;ipython-input-13-4d958e7f8f5b&gt; in &lt;module&gt;()
        1 knn = KNeighborsClassifier(n_neighbors=5).fit(X_train_tfidf, y_train)
  ----&gt; 2 prediction['knn'] = knn.predict(X_test_tfidf)
        3 accuracy_score(y_test, prediction['knn'])*100
  
  ~\Anaconda3\lib\site-packages\sklearn\neighbors\classification.py in predict(self, X)
      143         X = check_array(X, accept_sparse='csr')
      144 
  --&gt; 145         neigh_dist, neigh_ind = self.kneighbors(X)
      146 
      147         classes_ = self.classes_
  
  ~\Anaconda3\lib\site-packages\sklearn\neighbors\base.py in kneighbors(self, X, n_neighbors, return_distance)
      355             if self.effective_metric_ == 'euclidean':
      356                 dist = pairwise_distances(X, self._fit_X, 'euclidean',
  --&gt; 357                                           n_jobs=n_jobs, squared=True)
      358             else:
      359                 dist = pairwise_distances(
  
  ~\Anaconda3\lib\site-packages\sklearn\metrics\pairwise.py in pairwise_distances(X, Y, metric, n_jobs, **kwds)
     1245         func = partial(distance.cdist, metric=metric, **kwds)
     1246 
  -&gt; 1247     return _parallel_pairwise(X, Y, func, n_jobs, **kwds)
     1248 
     1249 
  
  ~\Anaconda3\lib\site-packages\sklearn\metrics\pairwise.py in _parallel_pairwise(X, Y, func, n_jobs, **kwds)
     1088     if n_jobs == 1:
     1089         # Special case to avoid picklability checks in delayed
  -&gt; 1090         return func(X, Y, **kwds)
     1091 
     1092     # TODO: in some cases, backend='threading' may be appropriate
  
  ~\Anaconda3\lib\site-packages\sklearn\metrics\pairwise.py in euclidean_distances(X, Y, Y_norm_squared, squared, X_norm_squared)
      244         YY = row_norms(Y, squared=True)[np.newaxis, :]
      245 
  --&gt; 246     distances = safe_sparse_dot(X, Y.T, dense_output=True)
      247     distances *= -2
      248     distances += XX
  
  ~\Anaconda3\lib\site-packages\sklearn\utils\extmath.py in safe_sparse_dot(a, b, dense_output)
      133     """
      134     if issparse(a) or issparse(b):
  --&gt; 135         ret = a * b
      136         if dense_output and hasattr(ret, "toarray"):
      137             ret = ret.toarray()
  
  ~\Anaconda3\lib\site-packages\scipy\sparse\base.py in __mul__(self, other)
      367             if self.shape[1] != other.shape[0]:
      368                 raise ValueError('dimension mismatch')
  --&gt; 369             return self._mul_sparse_matrix(other)
      370 
      371         # If it's a list or whatever, treat it like a matrix
  
  ~\Anaconda3\lib\site-packages\scipy\sparse\compressed.py in _mul_sparse_matrix(self, other)
      538                                     maxval=nnz)
      539         indptr = np.asarray(indptr, dtype=idx_dtype)
  --&gt; 540         indices = np.empty(nnz, dtype=idx_dtype)
      541         data = np.empty(nnz, dtype=upcast(self.dtype, other.dtype))
      542 
  
  MemoryError: 
  </code></pre>