Is there are a metric based on collisions to compare bad hash functions?
There are a few standard quantities related to families of hash functions $H_k\colon \{0,1\}^m \to \{0,1\}^h$ for a uniform random key $k$.  You might call them metrics.  They came to prominence in Carter and Wegman's research program on universal hash families (paywall-free), though their first use in cryptography for one-time authenticators, by Gilbert, MacWilliams, and Sloane (paywall-free), predated the Carter–Wegman program by several years.


An upper bound on $\Pr[H_k(x) = u]$ for any $x \in \{0,1\}^m$ and $u \in \{0,1\}^h$.  If it is $1/2^h$, then we say the family is equidistributed.
An upper bound on $\Pr[H_k(x) = H_k(y)]$ for any distinct $x, y \in \{0,1\}^m$.  For example, if it's at most $1/2^h$, we say the family is universal; if it's at most $\varepsilon = O(1/2^h)$, we say it's $\varepsilon$-almost universal.  The quantity $\Pr[H_k(x) = H_k(y)]$ is sometimes called the collision probability.
An upper bound on $\Pr[H_k(x) \oplus H_k(y) = u]$ for any distinct $x, y, \in \{0,1\}^m$ and any $u \in \{0,1\}^h$.  If it's at most $\varepsilon = O(1/2^h)$, we say it's $\varepsilon$-almost xor-universal.
An upper bound on $\Pr[H_k(x) = u, H_k(y) = v]$ for any distinct $x, y \in \{0,1\}^m$ and any $u, v \in \{0,1\}^h$.  If it's at most $\varepsilon = O(1/2^{2h})$, we say it's $\varepsilon$-almost strongly universal or $\varepsilon$-almost pairwise independent.
An upper bound on $\Pr[H_k(x_1) = u_1, \dots, H_k(x_n) = u_n]$ for any distinct $x_i \in \{0,1\}^m$ and any $u_i \in \{0,1\}^h$.  If it's at most $\varepsilon = O(1/2^{nh})$, we say it's $\varepsilon$-almost $n$-independent or $n$-universal.


There are standard provable examples of functions attaining optimal or near-optimal values of the above bounds.  In cryptography, these bounds are relevant to one-time authenticators such as Poly1305 and GHASH, which are based on equidistributed $\varepsilon$-almost xor-universal hash families for $\varepsilon \approx \lceil m/128\rceil/2^{128}$, as well as to other applications such as hash tables with non-adaptive adversaries.  The quantity $\Pr[H_k(m') = a' \mathrel| H_k(m) = a]$ for $m' \ne m$ is sometimes called the forgery probability.

An $n$-independent hash family has the property that the probability of a collision $H_k(x) = H_k(y)$ for distinct $x, y \in S$ in a subset $S \subseteq \{0,1\}^m$ of $n$ elements is $$1 - \frac{2^h!}{(2^h)^n (2^h - n)!} \approx 1 - e^{-n^2/2^{h + 1}}.$$  However, the key must be enormous to guarantee this.  Further, none of these bounds guarantees anything about the difficulty of deliberately computing a collision knowing the key.

What you are probably getting at is not collision probabilities for unknown keys, but rather the collision resistance of a hash family:


An upper bound on $\Pr[x \ne y, H_k(x) = H_k(y)]$ where $(x, y) = A(k)$ and $A$ is any random algorithm.  If this negligible when the expected cost of $A$ is below $\sqrt{2^h} = 2^{h/2}$, we say the family is collision-resistant.

In particular, if $B_t(f)$ is a random algorithm that applies the birthday paradox to $t$ uniform random inputs to $f\colon \{0,1\}^m \to \{0,1\}^h$ as an oracle and returns a collision if there is one (for example, using van Oorschot and Wiener's parallel collision search machine), then for any equidistributed $f$, $$\Pr[x \ne y, f(x) = f(y)] = 1 - \frac{2^h!}{(2^h)^t (2^h - t)!} \approx 1 - e^{-t^2/2^{h + 1}},$$ where $(x, y) = B_t(f)$.  When $t^2 \lll 2^h$, so that $t \lll \sqrt{2^h} = 2^{h/2}$, this is negligible in $h$.  The best generic attack on $H_k$ with cost $t$ is $A(k) = B_t(H_k)$; if there is no known attack better than this we say that $H_k$ is collision-resistant.


We don't know how to prove this for all random algorithms $A$, however, so this property can only be conjectured of hash families for which we haven't figured out a cheaper collision search algorithm.  For example, we are quite confident that Poly1305 or GHASH is not collision-resistant; it is trivial to compute two-block collisions under any key with one field multiplication and one field addition.  We conjecture that keyed BLAKE2 is collision-resistant because nobody has figured out how to make it collide.

It is of little use to empirically study collision resistance by sampling outputs of a generic hash function.  If you try to use the generic birthday algorithm on Poly1305 or GHASH, it won't do much better than it will on keyed BLAKE2.  But it doesn't take a genius of a cryptanalyst to immediately recognize there is a trivial algorithm to find Poly1305 and GHASH collisions, given the key, much faster than the generic birthday algorithm.  The best we can do is to empirically study it by sampling algorithms for finding collisions—asking a lot of smart cryptanalysts to try to think of them.  So far, empirically, nobody has thought of a way to find collisions in keyed BLAKE2.

Since sampling outputs is not useful for studying collision resistance, when studying simplified versions of hash families, we don't normally study a progression of truncated output sizes. Rather, we structure the hash family to iterate some internal scrambling function for a variable number of rounds, and study a progression of numbers of rounds.  For example, the best known collision attacks on reduced-round versions of BLAKE apply only to 5 rounds, which is why the BLAKE2 designers felt comfortable recommending only 10 rounds.

Loosely, we sometimes also pick a fixed hash function such as SHA3-256, and say that it is collision-resistant if nobody has found a way to compute a collision under that fixed hash function.  Of course, if $h \leq m$, we are guaranteed that a collision exists, so there must exist a trivial random algorithm of negligible cost to compute it: namely an algorithm that simply returns the collision without doing any computation.

So there's no nice way to formalize this, but in practice, it seems to be hard.  Twenty years ago, it was conjectured that SHA-1 was collision-resistant in this loose sense, until someone figured out a way to compute SHA-1 collisions at lower cost than the generic attack in 2004, and then someone published an actual collision last year.  Today, it is still conjectured that SHA-256 is still collision-resistant in this loose sense.