Emergent Coordination Through Competition
  We study the emergence of cooperative behaviors in reinforcement learning
agents by introducing a challenging competitive multi-agent soccer environment
with continuous simulated physics. We demonstrate that decentralized,
population-based training with co-play can lead to a progression in agents'
behaviors: from random, to simple ball chasing, and finally showing evidence of
cooperation. Our study highlights several of the challenges encountered in
large scale multi-agent training in continuous control. In particular, we
demonstrate that the automatic optimization of simple shaping rewards, not
themselves conducive to co-operative behavior, can lead to long-horizon team
behavior. We further apply an evaluation scheme, grounded by game theoretic
principals, that can assess agent performance in the absence of pre-defined
evaluation tasks or human baselines.
