RL Invertible Value Function approach - why it prevents rewards from exploding? Authors of "Recurrent Experience Replay in Destributed RL", page 3, use the function $h$ to prevent rewards from exploding:

$$h(x) = \operatorname{sign}(x)(\sqrt{|x|+1} -1) + \epsilon x$$
where $\epsilon$ is a very small number, for example $0.0001$ and $x$ is the score (aka reward). h re-scales the rewards the network receives. 

This is different to a more common "reward-clipping" technique.





Question 1 (most important):

The Q-vaue formula looks like this:

$$y = h \bigg( nstepReward + \gamma^n \cdot h^{-1}(Q(s_{t+n}, a^*)) \bigg)$$

Why do authors pass the Q-scores through the inverse (exponential) function and then through the squashing function $h$? The effect of this is that Q-target-scores remain unchanged, because composition of a function and its inverse is identity. So to me, the $Q$ is still a big source of threat of "exploding rewards", because it's left untouched. Why do we keep it untouched?

It looks like the whole formula merely squashes $nStepReward$, but why not just squash everything, including Q to prevent exploding rewards? In other words, why not to have:

$$y = h \bigg( nstepReward + \gamma^n Q(s_{t+n}, a^*) \bigg)$$



Question 2:

Why to use $\epsilon x$ in the formula for $h$
The answer of your first question can be answered by considering when you do not step forward and $n = 0$. So what happend in your case? $y = h(Q(S_t, a^*))$. However, $y= Q(S_t, a^*)$ could be more reasonable. Hence, we need that $h^{-1}$ to have a more explainable reward in the edge case. 

One answer to the second question could back to the differentiation of the $h$ function. If the $\epsilon x$ is removed, the $\frac{dh(x)}{dx}$ goes to zero when $x$ grows. But, by the presence of $\epsilon$, we will have $\frac{dh(x)}{dx} \to \epsilon$ for big enough values of $x$. Hence, the reward function will not cap for the big values of $x$ and grows increasingly.