What is the optimal value function of the shifted version of the reward function?
Theorem
The optimal state-action value function of $r'(s, a) \triangleq r(s, a) + c$, for $c \in \mathbb{R}$, would be
\begin{align}
q_*(s, a) + c + c\gamma + c \gamma^2 + c \gamma^3 + \dots 
&amp;=q_*(s, a) + c \left( 1 + \gamma + \gamma^2 + \gamma^3 + \dots \right) \\
&amp;= q_*(s, a) + c \left( \sum_{k=0}^{\infty} \gamma^{k} \right) \\
&amp;=q_*(s, a) + c\left(\frac{1}{1 - \gamma}\right) \\
&amp;=q_*(s, a) + \frac{c}{1 - \gamma},
\end{align}
where $\gamma &lt; 1$ is the discount factor (and $\sum_{k=0}^{\infty} \gamma^{k}$ is a geometric series).
Therefore, the optimal policy does not change if we shift the reward function by some constant.
Proof
The proof follows the same reasoning of the proof in this answer, i.e. we show that the Bellman optimality equation also holds in this case.
\begin{align}
q_*(s,a) + \frac{c}{1 - \gamma} 
&amp;= \sum_{s' \in \mathcal{S}, r \in \mathcal{R}}p(s',r \mid s,a)\left((r + c) + \gamma \max_{a' \in\mathcal{A}(s')} \left( q_*(s',a') + \frac{c}{1 - \gamma}  \right) \right) \tag{1}\label{1}
\end{align}
Given that $\frac{c}{1 - \gamma}$ is a constant, it does not affect the max, given that we add this constant to all pairs $(s', a')$, for all $a' \in\mathcal{A}(s')$. This holds even if $c$ is negative. So, we can take that addition out of the max, but note that we sum (rather than multiply, like the proof in the other answer).
\begin{align}
q_*(s,a) + \frac{c}{1 - \gamma} 
&amp;= \sum_{s' \in \mathcal{S}, r \in \mathcal{R}}p(s',r \mid s,a)\left((r + c) + \gamma \left (\frac{c}{1 - \gamma} +  \max_{a'\in\mathcal{A}(s')} q_*(s',a') \right) \right) \\
&amp;= \sum_{s' \in \mathcal{S}, r \in \mathcal{R}}p(s',r \mid s,a)\left((r + c) +  \frac{c \gamma}{1 - \gamma} +  \gamma \max_{a'\in\mathcal{A}(s')} q_*(s',a') \right) \\
&amp;= \sum_{s' \in \mathcal{S}, r \in \mathcal{R}}p(s',r \mid s,a)\left(r + \frac{c(1 - \gamma) + c \gamma}{1 - \gamma} +  \gamma \max_{a'\in\mathcal{A}(s')} q_*(s',a') \right) \\
&amp;= \sum_{s' \in \mathcal{S}, r \in \mathcal{R}}p(s',r \mid s,a)\left(r + \frac{c - c\gamma + c \gamma}{1 - \gamma} +  \gamma \max_{a'\in\mathcal{A}(s')} q_*(s',a') \right) \\
&amp;= \sum_{s' \in \mathcal{S}, r \in \mathcal{R}} \left ( p(s',r \mid s,a)\frac{c}{1 - \gamma} \right) + \\ &amp; \sum_{s' \in \mathcal{S}, r \in \mathcal{R}}  \left( p(s',r \mid s,a) \left(r + \gamma \max_{a'\in\mathcal{A}(s')} q_*(s',a') \right) \right) \tag{2}\label{2} \\
\end{align}
Given that $p(s',r \mid s,a)$ is a probability distribution, then the affine combination of $\frac{c}{1 - \gamma}$, where the affine weights are $p(s',r \mid s,a)$, is $\frac{c}{1 - \gamma}$. To see that this is true, suppose that we have the probability vector $[0.2, 0.5, 0.3]$ and let $c$ be a constant, then $0.2c + 0.5c + 0.3c = c$.
So, equation \ref{2} becomes
\begin{align}
q_*(s,a) + \frac{c}{1 - \gamma} 
&amp;= \frac{c}{1 - \gamma}  + \sum_{s' \in \mathcal{S}, r \in \mathcal{R}} p(s',r \mid s,a) \left(r + \gamma \max_{a'\in\mathcal{A}(s')} q_*(s',a') \right)\\
q_*(s,a)
&amp;=\sum_{s' \in \mathcal{S}, r \in \mathcal{R}} p(s',r \mid s,a) \left(r + \gamma \max_{a'\in\mathcal{A}(s')} q_*(s',a') \right)
\end{align}
which is the Bellman optimality equation (see equation 3.20 of Sutton &amp; Barto book, p. 64).