"Other-Play" for Zero-Shot Coordination
  We consider the problem of zero-shot coordination - constructing AI agents
that can coordinate with novel partners they have not seen before (e.g.
humans). Standard Multi-Agent Reinforcement Learning (MARL) methods typically
focus on the self-play (SP) setting where agents construct strategies by
playing the game with themselves repeatedly. Unfortunately, applying SP naively
to the zero-shot coordination problem can produce agents that establish highly
specialized conventions that do not carry over to novel partners they have not
been trained with. We introduce a novel learning algorithm called other-play
(OP), that enhances self-play by looking for more robust strategies, exploiting
the presence of known symmetries in the underlying problem. We characterize OP
theoretically as well as experimentally. We study the cooperative card game
Hanabi and show that OP agents achieve higher scores when paired with
independently trained agents. In preliminary results we also show that our OP
agents obtains higher average scores when paired with human players, compared
to state-of-the-art SP agents.
