How does being on-policy prevent us from using the replay buffer with the policy gradients? One of the approaches to improving the stability of the Policy
Gradient family of methods is to use multiple environments in
parallel. The reason behind this is the fundamental problem we
discussed in Chapter 6, Deep Q-Network, when we talked about the
correlation between samples, which breaks the independent and
identically distributed (i.i.d) assumption, which is critical for
Stochastic Gradient Descent (SDG) optimization. The negative
consequence of such correlation is very high variance in gradients,
which means that our training batch contains very similar examples,
all of them pushing our network in the same direction. However, this
may be totally the wrong direction in the global sense, as all those
examples could be from one single lucky or unlucky episode. With our
Deep Q-Network (DQN), we solved the issue by storing a large amount of
previous states in the replay buffer and sampling our training batch
from this buffer. If the buffer is large enough, the random sample
from it is much better representation of the states distribution at
large. Unfortunately, this solution won't work for PG methods, at most
of them are on-policy, which means that we have to train on samples
generated by our current policy, so, remembering old transitions is
not possible anymore.

The above excerpt is from Maxim Lapan in the book Deep Reinforcement Learning Hands-on page 284.
How does being on-policy prevent us from using the replay buffer with the policy gradients? Can you explain to me mathematically why we can't use replay buffer with A3C for instance?
Let's say your old policy is $\pi_b$ and your current one is $\pi_a$. If you collected trajectory by using policy $\pi_b$ you would get return $G$ whose expected value is
\begin{align}
E_{\pi_b}[G_t|S_t = s] &amp;= E_{\pi_b}[R_{t+1} + G_{t+1}]\\
&amp;= \sum_a \pi_b(a|s) \sum_{s', r} p(s', r|s, a) [r + E_{\pi_b}[G_{t+1}|S_{t+1} = s']]\\
\end{align}
You can see if you write out this recursively that this expectation depends on $\pi_b(a|s), \pi_b(a'|s'), \ldots$

If you collect trajectory with policy $\pi_a$ you would get expected return that depends on $\pi_a(a|s), \pi_a(a'|s'), \ldots$ Since these are two different policies then $\pi_b(a|s) \neq \pi_a(a|s)$ for some $(s, a)$. That would mean that returns have different expected values and are sampled through different distributions. You cannot then use some return $G$ sampled by following policy $\pi_b$ to update policy $\pi_a$ because it's not sampled according to the proper distribution, if we did, we would be updating policy $\pi_a$ with biased gradient update that does not reflect how policy $\pi_a$ performed.