Is it possible to increase the number of centroids in KMeans during fitting?
It is not a good idea to do this during optimization, because it changes the optimization procedure substantially. It will essentially reset the whole optimization. There are strategies such as bisecting k-means that try to learn the value of k during clustering, but they are a bit more tricky than increasing k by one - they decide upon one particular cluster to split, and try to choose good initial centroids for this cluster to keep things somewhat stable.
Furthermore, increasing k will not necessarily improve Silhouette. It will trivially improve SSQ, so you cannot use SSQ as a heuristic for choosing k, either.
Last but not least, computing the Silhouette is O(n^2). It is too expensive to run often. If you have large enough amount of data to require MiniBatchKMeans (which really is only for massive data), then you clearly cannot afford to compute Silhouette at all.
I am attempting to use MiniBatchKMeans to stream NLP data in and cluster it, but have no way of determining how many clusters I need. What I would like to do is periodically take the silhouette score and if it drops below a certain threshold, increase the number of centroids. But as far as I can tell, n_clusters is set when you initialize the clusterer and can't be changed without restarting. Am I wrong here? Is there another way to approach this problem that would avoid this issue?