What happens with policy gradient methods if rewards are differentiable?
That's exactly the point of the Policy Gradient Theorem. Let's go through he proof of this theorem - it relies on our ability to &quot;loop&quot; the reward expression $J(\theta,s)$, expressing it through $J(\theta,s')$ in the next state:
$$J(\theta,s) = \mathbb{E}_{\tau\sim\pi_\theta}\left[ R(\tau) | s_0 = s\right]$$
$$\begin{align}
J(\theta,s) &amp; = \sum_a\pi_\theta(a|s)\sum_{s',r}P(s',r|s,a)(r + J(\theta, s'))\\
&amp; = \sum_aJ(\theta,a,s)
\end{align}
$$
Here, I've introduced the notation $J(\theta,a,s)$ to keep further derivation sane. Taking the gradient:
$$\begin{align}
\nabla_\theta J(\theta,s) = \sum_{a}\left(\vphantom{\sum_a}\right.&amp;\nabla_\theta  \pi_\theta(a|s) \left(\sum_{s',r}P(s',r|s,a)(r + J(s',\theta))\right)  + \\
&amp;+\pi_\theta(a|s) \nabla_\theta \sum_{s',r}P(s',r|s,a)(r + J(s',\theta)\left.\vphantom{\sum_a}\right)\end{align}
$$
With the first term in the sum we do the log trick:
$$
\frac{\nabla_\theta  \pi_\theta(a|s)}{\pi_\theta(a|s)} \left(\pi_\theta(a|s)\sum_{s',r}P(s',r|s,a)(r + J(s',\theta))\right) = J(\theta,a,s)\nabla_\theta\log\pi_\theta(a|s)
$$
And the second term simplifies to:
$$\pi_\theta(a|s) \nabla_\theta \sum_{s',r}P(s',r|s,a)(r + J(s',\theta)) =
\pi_\theta(a|s)  \sum_{s'}P(s'|s,a) \nabla_\theta J(s',\theta) 
$$
The last equality is exactly where we loose the rewards $r$ - the gradient of the constant $r$ is zero, and we can sum over $r$ because of the transition probability normalization $\sum_{r}P(s',r|s,a) = P(s'|s,a)$.
So we've finally get the expression for the gradient:
$$\nabla_\theta J(\theta,s) = \sum_{a}\left(J(\theta,a,s)\nabla_\theta\log\pi_\theta(a|s) + \pi_\theta(a|s)  \sum_{s'}P(s'|s,a) \nabla_\theta J(s',\theta) \right)$$
The Policy Gradient Theorem proof goes on though a couple more steps: unroll the last expression, rewrite through the distribution over states, then convert back from summation to the expectation over trajectories -- going through all this in detail here would too long. While we've already passed the crucial point for your question - we've got rid of the gradient of the rewards.
Edit:
In response to your comment, let me just stress again that this expression is a statement of the Policy Gradient Theorem.
$$\nabla_\theta J(\theta) = \mathbb{E}_{\tau\sim\pi}\left[R(\tau)\sum\nabla_\theta\log \pi_\theta(a_t|s_t)\right]$$
The proof of Policy Gradient Theorem is pretty involved and certainly is not as simple as saying that $\nabla_\theta R(\tau) = 0$. Some authors abuse notation or cut corners when going through the proof of the theorem - for a reasonably strict exposition of the PG theorem I recommend Sutton and Barto, Chapter 13.
Another potential point of confusion is the notation $r(s_t,a_t,s_{t+1})$ that you've been using. In the most general MDP formulation rewards are random variables $r$. The probability of getting a reward $r$ and ending up in state $s_{t+1}$ is encoded in the transition probability $P(s_{t+1},r|s_t,a_t)$. (This covers the case of deterministic rewards by having a deterministic distribution over $r$ ). The derivation above uses this, most general, formulation of the MDP.