Confused by kmeans results I am using kmeans to cluster some data with 2 features. Not sure I understand why kmeans is producing the clusters I see:



Why would kmeans not cluster these points in a way that matches what we would expect visually looking at the data? Why are seemingly random points in the middle of visual clusters being put into a second cluster like that?

The code I am running:

cols = ['col1', 'col2']
features = map(lambda x: df[x], cols)
input = np.matrix(list(zip(*features)))

scaler = StandardScaler()
scaler.fit(input)
input_scaled = scaler.transform(input)

algo = KMeans(n_clusters=2)
algo.fit(input_scaled)
df['cluster'] = pd.Series(algo.labels_)

sns.lmplot(x=cols[0],y=cols[1],data=df, fit_reg=False, hue='cluster')
Got it. Thanks to everyone who helped. The issue had nothing to do with kmeans.

I did not realize that when you do this:

dataframe[col] = Series, the series gets merged based on the index, as opposed to a simple appending of the column.

My dataframe had been filtered before I generated the lists of features, so the index was not 0,1,2 but rather 0,2,5, etc. I needed to perform reset_index() on the original dataframe before assigning a new column in the dataframe to the labels from the algorithm.