Image clustering by similarity measurement (CW-SSIM)
I would use a regular clustering algorithm and replace the objective function, which is usually the MSE, with a differentiable loss function of your choice. Another way is to learn an embedding that optimizes your similarity metric using a neural network and just cluster that.

If you would rather do similarity-based clustering, here are some papers:


A Similarity-Based Robust Clustering Method
A Discriminative Framework for Clustering via Similarity Functions
Similarity-Based Clustering by Left-Stochastic Matrix Factorization


sklearn implements two similarity clustering methods: Affinity propagation, and spectral clustering.