Unique representation of 5-number sets with values 1-13
Here is a solution with 20 bits. Consider the following array:

0x000f2616, 0x000dbcce, 0x000533cf, 0x0007d943, 0x000a180c,
0x000f69f2, 0x0005e214, 0x0009da04, 0x000e101a, 0x000c495c,
0x0009cfaa, 0x000ff60d, 0x0000103d


Each of these numbers $x_1,\ldots,x_{13}$ is 20 bits long. Given a hand $a,b,c,d,e$, compute the 20 least significant bits of $x_a+x_b+x_c+x_d+x_e$. You can easily check that this results in a value that identifies a hand uniquely.

I found these values by randomly generating arrays until one of them worked, so it's possible that a better construction exists (such as one using codes over $\mathbb{F}_5$).



Here is another solution, which results in keys in the range [0,441100]. Let $p = 441101 \approx 2^{18.75}$, which is prime. The key corresponding to a hand $a,b,c,d,e$ is $5^a+5^b+5^c+5^d+5^e \bmod{p}$. This is the minimal prime that works. I calculated it by factoring all differences of $5^a+5^b+5^c+5^d+5^e$ for all pairs of hands.

You can also implement this using the array

1, 5, 25, 125, 625, 3125, 15625, 78125, 390625, 188721, 61403, 307015, 211772


Add the corresponding numbers, and (optionally) reduce modulo 441101.