Coordinated Proximal Policy Optimization
  We present Coordinated Proximal Policy Optimization (CoPPO), an algorithm
that extends the original Proximal Policy Optimization (PPO) to the multi-agent
setting. The key idea lies in the coordinated adaptation of step size during
the policy update process among multiple agents. We prove the monotonicity of
policy improvement when optimizing a theoretically-grounded joint objective,
and derive a simplified optimization objective based on a set of
approximations. We then interpret that such an objective in CoPPO can achieve
dynamic credit assignment among agents, thereby alleviating the high variance
issue during the concurrent update of agent policies. Finally, we demonstrate
that CoPPO outperforms several strong baselines and is competitive with the
latest multi-agent PPO method (i.e. MAPPO) under typical multi-agent settings,
including cooperative matrix games and the StarCraft II micromanagement tasks.
