How is parallelism implemented in RL algorithms like PPO? There are multiple ways to implement parallelism in reinforcement learning. One is to use parallel workers running in their own environments to collect data in parallel, instead of using replay memory buffers (this is how A3C works, for example).

However, there are methods, like PPO, that use batch training on purpose. How is parallelism usually implemented for algorithms that still use batch training?

Are gradients accumulated over parallel workers and the combined? Is there another way? What are the benefits of doing parallelism one way over another?
OpenAI have a post on that: https://openai.com/blog/openai-five/

They use a myriad of rollout workers that collect data for 60 seconds and push that data to a GPU cluster where gradients are computed for batches of 4096 observations which are then averaged.

PPO is actually designed to allow this kind of parallelisation as it uses trajectory segments with a fixed size of $T$ to collect data, e.g. 60 seconds for OpenAI Five, where $T$ is supposed to be "much less than the episode length" (p.5 of PPO paper).