objective in policy gradient equation?
Assuming $\tau = (\mathbf{s}_1,\mathbf{a}_1,...,\mathbf{a}_T)$ is a sampled trajectory from $\pi_\theta(\tau)$ with return $r(\tau)=\sum_{t=1}^{T}r(\mathbf{s}_t,\mathbf{a}_t)$ then we define the expected reward (our cost function) as:

$$J(\theta)\equiv E_{\tau\sim \pi_\theta(\tau)}[r(\tau)] = \int \pi_\theta(\tau)r(\tau)d\tau$$

where $\pi_\theta(\tau)$= $\pi_\theta(\mathbf{s}_1,\mathbf{a}_1,...,\mathbf{a}_T)=p(\mathbf{s}_1)\prod_{t=1}^{T}\pi_\theta(\mathbf{a}_t|\mathbf{s}_t)p(\mathbf{s}_{t+1}|\mathbf{s}_t,\mathbf{a}_t)$.

Notice that the p.d.f $\pi_\theta(\tau)$ here  is a joint probability distribution over states and actions. Thus the expectation comes from:

$$E[X]=\int_{-\infty }^{\infty} \int_{-\infty }^{\infty} xf(x,y)dxdy$$

and this is the reason that you need to integrate over the volume of states and actions. $p(\mathbf{s}_1)$ is a distribution over initial states and independent of the integration over actions so it can get out of the 2nd integral. Taking the log of the joint distribution $\pi_\theta(\tau)$, substituting into $J(\theta)$ and differentiating both sides of the first equation,  you get an equation w.r.t the $\theta$ parameters and quantities that you can compute so you can estimate the gradient of $J(\theta)$.

I am not very familiar with your notation (i think you found it in a master thesis) but this is the general approach.