Cannot see what the "notation abuse" is, mentioned by author of book
The mathematical expression is completely legit. The abuse is in the fact that the function $p$, which is defined first time in equation 3.2, which:


  The function $p: S$ x $R$ x $S$ x $A \rightarrow [0,1]$. is an ordinary
  deterministic function of four arguments...


is re-defined slightly differently just two lines after this definition (equation 3.4), as a three-argument function $p: S$ x $S$ x $A \rightarrow [0,1]$.

If they used $p$ to represent the regular probability measure, there would be no abuse. In the authors' notation, $p$ is a deterministic function, while the regular probability function is denoted as $Pr$; and keeping the same name for slightly different functions, is where the "innocent" notation abuse comes from.