How does an op amp compare voltage?
By subtracting one input from the other the reference point (ground) cancels out
$$\underbrace{V_1 + V_0}_{\text{positive input}} - \underbrace{(V_2 + V_0)}_{\text{negative input}} = V_1 - V_2.$$
So the reference voltage does not affect the output.

This is done with the so called differential pair.