title: Python: loading a kmeans training dataset and using it to predict a new dataset
body: <p>I have a huge amount of data which I would like to run a kmean classification on. The dataset are so big, that I cannot load the files into memory. </p>
  
  <p>My idea is to run the classifiation on some part of the dataset like a training dataset, and then apply the calssification to the rest of the dataset part by part. </p>
  
  <pre><code>import pandas as pd
  import pickle
  from sklearn.cluster import KMeans
  
  frames = [pd.read_hdf(fin) for fin in ifiles]
  data = pd.concat(frames, ignore_index=True, axis=0)
  data.dropna(inplace=True)
  
  k = 12
  x  = pd.concat(data['A'], data['B'], data['C'], axis=1, keys=['A','B','C'])
  model = KMeans(n_clusters=k, random_state=0, n_jobs = -2)
  model.fit(x)
  
  pickle.dump(model, open(filename, 'wb'))
  </code></pre>
  
  <p>x looks like this: </p>
  
  <pre><code>array([[-2.26732099,  0.24895614,  2.34840191],
     [-2.26732099,  0.22270912,  1.88942378],
     [-1.99246557,  0.04154312,  2.63458941],
     ..., 
     [-4.29596287,  1.97036309, -0.22767511],
     [-4.26055474,  1.72347591, -0.18185197],
     [-4.15980382,  1.73176239, -0.30781225]])
  </code></pre>
  
  <p>The model look like this:</p>
  
  <pre><code>KMeans(algorithm='auto', copy_x=True, init='k-means++', max_iter=300,
  n_clusters=12, n_init=10, n_jobs=-2, precompute_distances='auto',
  random_state=0, tol=0.0001, verbose=0)
  </code></pre>
  
  <p>A plot of two of the model parameters color coded with the model looks like this: 
  <a href="https://i.stack.imgur.com/bQqS1.png" rel="noreferrer"><img src="https://i.stack.imgur.com/bQqS1.png" alt="enter image description here"></a></p>
  
  <p>Now I want to load the model and use it for predicting. As a test example I have loaded the same data (not shown here), and trying to predict the new dataset.</p>
  
  <pre><code>modelnew = pickle.load(open('test.pkl', 'rb'))
  modelnew.predict(x)
  </code></pre>
  
  <p>The result:
  <a href="https://i.stack.imgur.com/gEanC.png" rel="noreferrer"><img src="https://i.stack.imgur.com/gEanC.png" alt="enter image description here"></a></p>
  
  <p>This data does clearly not cluster. What am I missing? Do I need to fix the model parameters in some way?</p>
  
  <p>I have tried to make an example of a test and a train data set. Here it also goes wrong. There is clearly something I am missing:</p>
  
  <pre><code>## Splitting data in a test and train data set
  sample_train, sample_test = train_test_split(x, test_size=0.50)
  
  k = 12 ## Setting number of clusters
  model = KMeans(n_clusters=k, random_state=0, n_jobs = -2) ## Kmeans model
  train = model.fit(sample_train) ## Fitting the training data
  model.predict(sample_test) # Predicting the test data
  
  centroids =  model.cluster_centers_
  labels = model.labels_
  
  ## Figures
  cmap_model = np.array(['red', 'lime', 'black', 'green', 'orange', 'blue', 'gray', 'magenta', 'cyan', 'purple', 'pink', 'lightblue', 'brown', 'yellow'])
  plt.figure()
  plt.scatter(sample_train[:,0], sample_train[:,1], c=cmap_model[train.labels_], s=10, edgecolors='none')
  plt.scatter(centroids[:, 0], centroids[:, 1], c=cmap_model,  marker = "x", s=150, linewidths = 5, zorder = 10)
  
  plt.figure()
  plt.scatter(sample_test[:,0], sample_test[:,1], c=cmap_model[labels], s=10, edgecolors='none')
  plt.scatter(centroids[:, 0], centroids[:, 1], c=cmap_model,  marker = "x", s=150, linewidths = 5, zorder = 10)
  plt.show()
  </code></pre>
  
  <p>Train data:
  <a href="https://i.stack.imgur.com/ImStY.png" rel="noreferrer"><img src="https://i.stack.imgur.com/ImStY.png" alt="Train result"></a></p>
  
  <p>Test data:
  <a href="https://i.stack.imgur.com/hZpKS.png" rel="noreferrer"><img src="https://i.stack.imgur.com/hZpKS.png" alt="Test result"></a></p>