What is the difference between "expected return" and "expected reward" in the context of RL?
Is the "expected reward" actually $\mathcal{R}^a_{ss'}$ instead of $V^\pi(s)$?


In short, yes. 

Although there is some context associated - $\mathcal{R}^a_{ss'}$ is in the context of specific action and state transition. You will also find $\mathcal{R}^a_{s}$ used for expected reward given only current state and action (which works fine, but moves around some terms in the Bellman equations). 

"Return" may also be called "Utility".

RL suffers a bit from naming differences, however the meaning of reward is not one of them. 

Notation differences also abound, and in Sutton &amp; Barto Reinforcement Learning: An Introduction (2nd edition), you will find:


$R_t$ is a placeholder for reward received at time $t$, a random variable.
$G_t$ is a placeholder for return received after time $t$, and you can express the value equation as $v_{\pi}(s) = \mathbb{E}[G_t|S_t=s] = \mathbb{E}[\sum_{k=0}^{\infty}\gamma^kR_{t+k+1}|S_t=s]$
$r$ is a specific reward value
You won't see "expected reward" used directly in an equation from the book, as the notation in the revised book relies on summing over distribution of reward values.


In some RL contexts, such as control in continuous problems with function approximation, it is more convenient to work with maximising average reward, than maximising expected return. But this is not quite the same as "expected reward", due to differences in context (average reward includes averaging over the expected state distribution when following the policy)