Kmeans using silhouette_score I am using silhouette_score to find the optimal k value. So I am running a for loop with a range of possible k values. I have added my code below. this program takes a very long time to run. Could you suggest some improvements for a more efficient run time?

import numpy as np
import matplotlib.pyplot as plt
import matplotlib.animation as animation
from sklearn.cluster import KMeans
from sklearn import metrics

data=np.load(filename)


coeffs=[]


for i in range(2,8):

    clusters=KMeans(n_clusters=i)
    clusters.fit(data)
    labels = clusters.labels_
    sil_coeff = metrics.silhouette_score(data, labels,metric='euclidean')
    coeffs.append(sil_coeff)


coeffs=np.array(coeffs)    
k=np.argmax(coeffs)+2
Do some measurements to identify your bottlenecks.

Here, I suggest to not use Silhouette. Because it is much slower than k-means. Silhouette need O(n²) distance computations every run!

The obvious was to speed this up would this be to compute and store a distance matrix just once. This won't help k-means, but it will make multiple Silhouette runs a little bit faster (it will still be O(n²), but only array lookups instead of distance computations now). Nevertheless, this will not be scalable to large data.

The better approach is to use a cheaper heuristic to guess the "best" value of k. None such measure is perfect anyway.