Elbow method on hundreds columns and rows So I have these vectors called matrix_ after I applied TF-IDF (term frequency-inverse document frequency), and I also converted it to dataframe matrixDF_ .

I wanted to cluster these matrix_ vectors with MiniBatchKMeans from sklearn. But since the vectors are large (569 rows and 829 columns), then I need to find the optimum K-Cluster first.

That is why I use KElbowVisualizer from Yellowbrick. At first, it works fine and I got the optimum k-cluster. However, when rerunning it again, it showed me a different number of k-cluster. And that is where I am confused then.

Anyone can help me with this problem?

here is my code:

import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import MiniBatchKMeans
from scipy.spatial.distance import cdist
from sklearn.metrics import silhouette_score
from yellowbrick.cluster import KElbowVisualizer

""" vectorize the words. getting the tf-idf of each words """
vec = TfidfVectorizer(
     lowercase = True,
     stop_words = 'english',
     use_idf=True
)

matrix_ = vec.fit_transform(df)
matrixDF_ = pd.DataFrame(matrix_.toarray(), columns=vec.get_feature_names())

model = MiniBatchKMeans(MiniBatchKMeans(init_size=559, random_state=None))
visualizer = KElbowVisualizer(model,k=(2, 11), metric='distortion', 
timings=True, size=(1080, 720))
visualizer.fit(matrix_)        # Fit the data to the visualizer
visualizer.show()


I also tried fit the dataframe matrixDF_. to the visualizer, but each output after every run has different output number.

Here are some of output examples. First output:


second output:
You are using different random starting points each time you run k-means (random_state=None) This means you may get different clusterings, different clustering metrics each time. That's expected. What you may wish to do is average the results over several different runs to get a more reliable estimate of the loss at each k.