Stability of value function approximation in policy gradients
The short answer is yes. Take a look at Asynchronous Methods for RL. In a similar way that sampling from experience replay helps breaking the correlations, asynchronous methods are based on having multiple agents interacting with multiple instances of the same environment.

The Process:
Each agent (called worker) collects its own experience up to a specified timestep $t_{max}$, which is stored in a batch. Then a master network performs a training update by using this batch. After the update each worker resets to an identical network to the master and they start the task all over again. The update can be synchronous (as I describe here using a batch) or asynchronous by training multiple agents with their own parameters and then update asynchronously the master network.

Why this is helpful? The experience of each agent is independent of the experience of the others and thus more diverse which helps keeping correlation down.

A couple of additional comments on your question:


The goal in Policy Gradients (PG) is to optimize the expected reward w.r.t some parameters which is analogous to the estimation of the gradient of the policy function (parametrized by these parameters).
In RL, and especially in Policy Gradients, you are dealing mainly with 3 things: state sample correlation, bias and variance of your gradient estimator.
The introduction of the baseline is to reduce variance in the gradient estimator without introducing any bias. The baseline is called control variate and is used for reducing variance in Monte Carlo estimators (our gradient estimator here) and has zero mean.