title: How to cluster very big sparse data set using low memory in Python?
body: <p>I have data which forms a sparse matrix in shape of 1000 x 1e9. I want to cluster the 1000 examples into 10 clusters using K-means.</p>
  
  <p>The matrix is very sparse, less than 1/1e6 values.</p>
  
  <p>My laptop got 16 RAM. I tried sparse matrix in scipy. Unfortunately, the matrix makes the clustering process need much more memory than I have. Could anyone suggest a way to do this?</p>
  
  <p>My system crashed when running the following test snippet</p>
  
  <pre><code>import numpy as np
  from scipy.sparse import csr_matrix
  from sklearn.cluster import KMeans
  
  row = np.array([0, 0, 1, 2, 2, 2, 3, 3, 4, 5, 5, 5, 6, 6, 7, 8, 8, 8])
  col = np.array([0, 2, 2, 0, 1, 2] * 3)
  data = np.array([1, 2, 3, 4, 5, 6] * 3)
  X = csr_matrix((data, (row, col)), shape=(9, 1e9))
  
  resC = KMeans(n_clusters=3).fit(X)
  resC.labels_
  </code></pre>
  
  <p>Any helpful suggestion is appreciated.</p>