Title: Dynamic Safe Interruptibility for Decentralized Multi-Agent Reinforcement Learning,
Abstract: In reinforcement learning, agents learn by performing actions and observing
their outcomes. Sometimes, it is desirable for a human operator to
\textit{interrupt} an agent in order to prevent dangerous situations from
happening. Yet, as part of their learning process, agents may link these
interruptions, that impact their reward, to specific states and deliberately
avoid them. The situation is particularly challenging in a multi-agent context
because agents might not only learn from their own past interruptions, but also
from those of other agents. Orseau and Armstrong defined \emph{safe
interruptibility} for one learner, but their work does not naturally extend to
multi-agent systems. This paper introduces \textit{dynamic safe
interruptibility}, an alternative definition more suited to decentralized
learning problems, and studies this notion in two learning frameworks:
\textit{joint action learners} and \textit{independent learners}. We give
realistic sufficient conditions on the learning algorithm to enable dynamic
safe interruptibility in the case of joint action learners, yet show that these
conditions are not sufficient for independent learners. We show however that if
agents can detect interruptions, it is possible to prune the observations to
ensure dynamic safe interruptibility even for independent learners.