Do we need multiple parallel environments to train in batches an on-policy algorithm?
We don't need multiple environments.
On-policy algorithms require that new training samples are collected with the newest policy, so we can't use an experience buffer. However we can use the newest policy to collect multiple samples, even over multiple epochs, before updating the weights. This update can be a batch update.