Minimizing Communication while Maximizing Performance in Multi-Agent
  Reinforcement Learning
  Inter-agent communication can significantly increase performance in
multi-agent tasks that require co-ordination to achieve a shared goal. Prior
work has shown that it is possible to learn inter-agent communication protocols
using multi-agent reinforcement learning and message-passing network
architectures. However, these models use an unconstrained broadcast
communication model, in which an agent communicates with all other agents at
every step, even when the task does not require it. In real-world applications,
where communication may be limited by system constraints like bandwidth, power
and network capacity, one might need to reduce the number of messages that are
sent. In this work, we explore a simple method of minimizing communication
while maximizing performance in multi-task learning: simultaneously optimizing
a task-specific objective and a communication penalty. We show that the
objectives can be optimized using Reinforce and the Gumbel-Softmax
reparameterization. We introduce two techniques to stabilize training: 50%
training and message forwarding. Training with the communication penalty on
only 50% of the episodes prevents our models from turning off their outgoing
messages. Second, repeating messages received previously helps models retain
information, and further improves performance. With these techniques, we show
that we can reduce communication by 75% with no loss of performance.
