Equations in "Intoduction to RL": What is the meaning and difference between E, and E with subscript?
In general, the expectation is taken with respect to some random variable X. Often, when dealing with a single random variable, it can be implicitly inferred over which random variable it is being integrated and thus writing $\mathbb{E}$ suffices. However, when dealing with multiple random variables, this is not the case anymore. Then, a subscript denotes with respect to which random variable the expectation is taken. 

However, the subscript can also denote on which random variable to condition on. This seems to me to be the case on the page you are referring to. Here, $\mathbb{E}_{\pi'}$ means that you are conditioning on actions which are distributed according to $\pi'$. 

(Just as a side note: sometimes you might see something along these lines of: $\mathbb{E}_{a\sim \pi'}$, the subscript here is the actual notation for "$a$ distributed according to $\pi'$") 

For a more technical answer, have a look at this question on Cross Validated.