Inducing Cooperation via Team Regret Minimization based Multi-Agent Deep
  Reinforcement Learning
  Existing value-factorized based Multi-Agent deep Reinforce-ment Learning
(MARL) approaches are well-performing invarious multi-agent cooperative
environment under thecen-tralized training and decentralized execution(CTDE)
scheme,where all agents are trained together by the centralized valuenetwork
and each agent execute its policy independently. How-ever, an issue remains
open: in the centralized training process,when the environment for the team is
partially observable ornon-stationary, i.e., the observation and action
informationof all the agents cannot represent the global states,
existingmethods perform poorly and sample inefficiently. Regret Min-imization
(RM) can be a promising approach as it performswell in partially observable and
fully competitive settings.However, it tends to model others as opponents and
thus can-not work well under the CTDE scheme. In this work, wepropose a novel
team RM based Bayesian MARL with threekey contributions: (a) we design a novel
RM method to traincooperative agents as a team and obtain a team
regret-basedpolicy for that team; (b) we introduce a novel method to de-compose
the team regret to generate the policy for each agentfor decentralized
execution; (c) to further improve the perfor-mance, we leverage a differential
particle filter (a SequentialMonte Carlo method) network to get an accurate
estimation ofthe state for each agent. Experimental results on two-step ma-trix
games (cooperative game) and battle games (large-scalemixed
cooperative-competitive games) demonstrate that ouralgorithm significantly
outperforms state-of-the-art methods.
