Clustering algorithms
You can see your affinity matrix as a weighted adjacency matrix of a graph and apply modularity-based community detection algorithms on that. Just note that modularity based algorithms have resolution problem i.e. finding very small communities is difficult in presence of large ones.