How do the policy gradient's cost function and gradients work?
In policy gradients we are interested in maximizing the expected reward. For this we assume that the expected reward is parametrized by parameters $\theta$ (e.g. a Neural Network. This means that in order to maximize the expected reward we need to find these parameters. In math notation:
$$
\theta^{\star}=\arg \max _{\theta} J(\theta) =\arg \max _{\theta} E_{\tau \sim p_{\theta}(\tau)}\left[\sum_{t} r\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right)\right]
$$

where $\tau$ is a trajectory sampled from policy $p_\theta$. To solve this we need gradient ascent so our parameters are updated: $\theta = \theta + \alpha\nabla J(\theta)$. Thus, if we compute the gradient of the expected reward we will get the proper update of our parameters towards greater expected rewards. You can take a look at Likelihood Ration and REINFORCE which explains analytically the whole optimization process.

To solve this you need the Policy Gradient theorem which will lead you to the equation you have. In other words, by using the gradient form you mentioned we are updating our parameters towards higher expected rewards. Also, PGs are more close to classification (cost-sensitive) rather than regression.

In terms of Neural Networks (and assuming stochastic policies), typically, your inputs are going to be states and your outputs action distribution and/or expected reward, thus mapping states to action probabilities (and/or reward predictions). The additional reward prediction it has been shown that leads to better results than using only the policy loss, as it drives the NN's representations to also predict expected rewards (instead of only action distributions).