How do I select the "best" unsupervised machine learning algorithm to cluster my specific dataset?
This is a non-trivial question. There is no general law to find the &quot;best&quot; algorithm or the &quot;correct&quot; amount of clusters (assuming you don't know the correct number of clusters).
As you already mentioned, certain algorithms make assumptions on the shape of your clusters or your data in general. One thing I suggest is to look at your data, check the assumptions and rule out algorithms where assumptions are violated. dimensionality reduction methods like PCA, t-SNE, UMAP, etc. are very helpful here if you work with high dimensional data.
Further have a look at the complexity of your clustering algorithm. If your are familiar with the Bias-variance tradeoff and Occam's razor you already know that simpler algorithms are less prone to overfitting and more likely to give you the correct results. BIC and AIC are quite popular measures in this regard.
Generally for unsupervised clustering it is a good idea to look at a number of different algorithms, compare them with different metrics and look where you can find the highes &quot;agreement&quot; among them.