Clustering with k-means for text classification based on similarity
So, the problem is how to cluster texts.
Firstly, an alternative approach to representing document meaning, you can use Doc2Vec and compare similarities between document embeddings (https://medium.com/wisio/a-gentle-introduction-to-doc2vec-db3e8c0cce5e).
Secondly, if you are unsure about the ideal number of clusters, instead of using k-means, you can use agglomerative clustering, which is essentially bottom-up method which clusters individual document embeddings by a distance metric to eventually merge all of the clusters to a &quot;mega-cluster&quot;, containing all documents.
You can then see how the documents cluster together using a dendrogram.
Here is a good resource on agglomerative clustering : https://stackabuse.com/hierarchical-clustering-with-python-and-scikit-learn/