Reinforcement Learning - What's the formula for the value function
The formula you have quoted is a bit unwieldy, precisely because the R function as defined needs to "look ahead" to all possible outcomes and their probabilities, but how to do that is not included explicitly.

There are actually a few variants of the Bellman equation that express more  or less detail. A good place to start for a truly generic version is Sutton &amp; Barto (2nd Edition):

$$v^*(s) = \text{max}_{a \in \mathcal{A}}[\sum_{r,s'}p(r,s'|s,a)(r + v^*(s')]$$

Where $\sum_{r,s'}$ is over all possible reward and next state pairs. 

The above equation changes your transition function that only handles next state, to a similar function that handles successor state and reward:

$$p(r,s'|s,a) = Pr\{ R_{t+1} = r, S_{t+1} = s' | S_{t} = s, A_{t} = a \}$$

Usually this does not increase the number of items to sum over, or add much complexity, because reward will most often associated with the transition.

The benefit is that this approach removes the need for a reward function that works with expected reward, and just works with specific rewards. Other variants are possible too, such as an expected reward function based on $(s,a)$ or $(s,a,s')$ - the difference is just a little bit of juggling with the expression so that it remains effectively the same given the subtle differences in definition for $r$.