What does a shift in time domain do to the phase in frequency domain?
\$z^{-1}\$ is a delay of \$T\$, where \$T\$ is the sampling increment. This is equivalent to the Laplace operator, \$e^{-sT}\$, which transforms to the frequency domain as \$e^{-jwT}\$. The phase contributed by this function at the frequency \$w\$ is \$-wT\$ radians, as \$e^{-jwT} = \cos(wT)-j\sin(wT)\$. So the phase contributon of \$z^{-m}\$ is \$-mwT\$ radians. The gain is not affected since the magnitude of \$e^{-jmwT}\$ is unity.

It may be interesting to note that the sampling operation in, say, a digital controller in a closed-loop system can lead to oscillatory or even unstable behaviour. This is because the additional phase lag reduces the system phase margin (the sampler does not change the gain so there's an additional 'pure' phase lag). Hence the sampling increment must be chosen carefully.