Understanding multi-iteration updates of the model in the Proximal Policy Optimization algorithm
Seems this question was asked and answered in openai/baselines github issue. The issue has been closed for a while.
Below is an answer provided by @matthiasplappert which has the most &quot;thumbs up&quot;:

To clarify: PPO is an on-policy algorithm so you are correct that going over the same data multiple times is technically incorrect.
However, we found that PPO is actually quite okay with doing this and we still get stable convergence. This is likely due to the proximal trust region constrained that we enforce, which means that the policy cannot change that much anyway when going over the current set of transitions multiple times, making it still approximately on-policy. You can of course get rid of this but then you'll need more samples.