Reinforcement Learning: Policy Gradient derivation question
Actually in the article there is a $\approx$ rather than an $=$. This is because you can approximate expectation values by sampling the respective distribution.

Assume you want to compute

$$ E \left[ f(x) \right] = \int p(x) f(x) dx$$

The integral might not be tractable, you might not even know the distribution $p(x)$. But as long as you can sample from $p(x)$, you can approximate quite well using the Monte Carlo estimator

$$ E \left[ f(x) \right] \approx \frac{1}{N} \sum_{i=1}^N f(x_i)$$

with $x_i \sim p(x)$ for all $i$. This approximation would get better for larger $N$. The distribution $p(x)$ is, in some sense, represented by the samples $x_i$ and their respective frequency.

This is what is going on in the article. You want to compute the expectation value over all possible trajectories, but that is infeasible. Luckily though, you can sample the distribution by running simulations of the environment. The expectation is then approximated using samples of trajectories. The $\pi_\theta(\tau)$ is represented in the sampled $a_{i,t}$ and $s_{i,t}$.

In short, the expectation over all possible paths is approximated by the mean over samples of paths.