Understanding Dynamic Range in ADC
Both of those equations are algebraically equivalent.  The first is:
\$20*log(5/0.000076)\$  where 0.000076 comes from \$5/2^{16}= 0.000076V\$
Substitute that back and you get \$20*log(5/(5/2^{16}))\$, which simplifies to
\$20*log(2^{16}/1))\$ which is your second equation.
From that you can see that the denominator has a 1 and not a 1/2 because your definition of dynamic range was the ratio of the maximum value to the step size.

I would think that the quantization error (and thus noise floor due to it) would be at most 1/2 of a step (not 1),

Using the maximum value (which almost never occurs) doesn't make a lot of sense since you'll seldom see an error that large.