How is parallelism implemented in RL algorithms like PPO?
OpenAI have a post on that: https://openai.com/blog/openai-five/

They use a myriad of rollout workers that collect data for 60 seconds and push that data to a GPU cluster where gradients are computed for batches of 4096 observations which are then averaged.

PPO is actually designed to allow this kind of parallelisation as it uses trajectory segments with a fixed size of $T$ to collect data, e.g. 60 seconds for OpenAI Five, where $T$ is supposed to be "much less than the episode length" (p.5 of PPO paper).