How can a single sample represent the expectation in gradient temporal difference learning? I was reading the gradient temporal difference learning version 2(GTD2) from rich Sutton's book page-246. At some point, he expressed the whole expectation using a single sample from the environment. But how a single sample can represent the whole expectation. 

I marked this point in this image.
In the, presumably final, printed version the last two equal signs are approximations. This is just because over a large amount of weight updates where you have been sampling the expectation will be approximated by Monte Carlo.