RL Invertible Value Function approach - why it prevents rewards from exploding?
The answer of your first question can be answered by considering when you do not step forward and $n = 0$. So what happend in your case? $y = h(Q(S_t, a^*))$. However, $y= Q(S_t, a^*)$ could be more reasonable. Hence, we need that $h^{-1}$ to have a more explainable reward in the edge case. 

One answer to the second question could back to the differentiation of the $h$ function. If the $\epsilon x$ is removed, the $\frac{dh(x)}{dx}$ goes to zero when $x$ grows. But, by the presence of $\epsilon$, we will have $\frac{dh(x)}{dx} \to \epsilon$ for big enough values of $x$. Hence, the reward function will not cap for the big values of $x$ and grows increasingly.