What is the advantage of using more than one environment with the advantage actor-critic?
What is the advantage of using more than one environment with a single agent?


There are two main advantages to this approach:


The dataset for training is closer to the independent, identically distributed (i.i.d.) ideal, important for theoretical and practical reasons when training a neural network. Samples taken from a single trajectory are not independent, but instead are correlated due to rules of the environment - so using a single trajectory is furthest from i.i.d. This is a similar motivation to use of experience replay tables for DQN variants of Q-learning. However, experience replay is inherently off-policy, so not a good fit to A2C or A3C that need samples taken when acting under the current policy.
Collecting experience is often a major bottleneck in training RL agents. Being able to do so in parallel in a distributed environment can significantly speed up the training process.