Learning only using off-policy samples
What you're describing is off-policy learning. A classic example is $Q$-learning, where you follow some policy $\pi$ whilst learning about the greedy policy.
If you're interested in actor-critic methods then a popular off-policy method is the Deep Deterministic Policy Gradient.