Shouldn't the utility function of two-player zero-sum games be in the range $[-1, 1]$?
it can be either. If you consider the lack of reward as "penalty" then getting 0 reward is bad.

if you use a value estimator through a neural network, the range of rewards will dictate the squashing function you use for the output layer