Clustering with multiple distance measures
This is just a wild guess, but I was wondering whether your custom distance function is indeed a distance function. 
In particular, your problem might occur because your distance function $d(x, y)$ does not separate observations: This is the case if there exist two observations $x_1$ and $x_2$ that are distinct, $x_1 \neq x_2$, but have zero distance, $d(x_1, d_2) = 0$. Then $x_1$ and $x_2 $ would necessarily be put into the same cluster by your algorithm.
If this happens for many observations in your test set, then you will observe large clusters.

You might be able to check on paper that your function satisfies the four axioms of a distance function (see the Wikipedia article above); if that is not possible, you could at least check it by looping over your test set.