Why does REINFORCE work at all?
The key to REINFORCE working is the way the parameters are shifted towards $G \nabla \log \pi(a|s, \theta)$.
Note that $ \nabla \log \pi(a|s, \theta) = \frac{ \nabla \pi(a|s, \theta)}{\pi(a|s, \theta)}$. This makes the update quite intuitive - the numerator shifts the parameters in the direction that gives the highest increase in probability that the action will be repeated, given the state, proportional to the returns - this is easy to see because it is essentially a gradient ascent step. The denominator controls for actions that would have an advantage over other actions because they would be chosen more frequently, by inversely scaling with respect to the probability of the action being taken; imagine if there had been high rewards but the action at time $t$ has low probability of being selected (e.g. 0.1) then this will multiply the returns by 10 leading to a larger update step in the direction that would increase the probability of this action being selected the most (which is what the numerator controls for, as mentioned).
That is for the intuition -- to see why it does work, then think about what we've done. We defined an objective function, $v_\pi(s)$, that we are interested in maximising with respected to our parameters $\theta$. We find the derivative of this objective with respect to our parameters, and then we perform gradient ascent on our parameters to maximise our objective, i.e. to maximise $v_\pi(s)$, thus if we keep performing gradient ascent then our policy parameters will converge (eventually) to values that maximise $v$ and thus our policy would be optimal.