Title: Lenient Multi-Agent Deep Reinforcement Learning,
Abstract: Much of the success of single agent deep reinforcement learning (DRL) in
recent years can be attributed to the use of experience replay memories (ERM),
which allow Deep Q-Networks (DQNs) to be trained efficiently through sampling
stored state transitions. However, care is required when using ERMs for
multi-agent deep reinforcement learning (MA-DRL), as stored transitions can
become outdated because agents update their policies in parallel [11]. In this
work we apply leniency [23] to MA-DRL. Lenient agents map state-action pairs to
decaying temperature values that control the amount of leniency applied towards
negative policy updates that are sampled from the ERM. This introduces optimism
in the value-function update, and has been shown to facilitate cooperation in
tabular fully-cooperative multi-agent reinforcement learning problems. We
evaluate our Lenient-DQN (LDQN) empirically against the related Hysteretic-DQN
(HDQN) algorithm [22] as well as a modified version we call scheduled-HDQN,
that uses average reward learning near terminal states. Evaluations take place
in extended variations of the Coordinated Multi-Agent Object Transportation
Problem (CMOTP) [8] which include fully-cooperative sub-tasks and stochastic
rewards. We find that LDQN agents are more likely to converge to the optimal
policy in a stochastic reward CMOTP compared to standard and scheduled-HDQN
agents.