Understanding why the expectation is over the new policy $\pi'$ in the proof of the Policy Improvement Theorem
The expectation is over the policy $\pi'$ because the action at the state $S_t = s$ is taken according to $\pi'$, and, for the proof, the book text (2nd edition, paragraph below Equation 4.8) defines $\pi'$ to be a policy that is identical to $\pi$ except that $\pi'(s) = a \neq \pi(s)$, where $s$ is one particular state. 

So, essential, the book text tries to prove that, for such a changed policy $\pi'$, if $q_\pi(s, a) = q_\pi(s, \pi'(s)) &gt; v_\pi(s)$, then the changed policy is better than $\pi$. Note that $\pi'(s) = a$ in this case.

That is why $q_\pi(s, \pi'(s)) = \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1}) \mid S_t = s]$.