Title: Measuring and avoiding side effects using relative reachability,
Abstract: How can we design reinforcement learning agents that avoid causing
unnecessary disruptions to their environment? We argue that current approaches
to penalizing side effects can introduce bad incentives in tasks that require
irreversible actions, and in environments that contain sources of change other
than the agent. For example, some approaches give the agent an incentive to
prevent any irreversible changes in the environment, including the actions of
other agents. We introduce a general definition of side effects, based on
relative reachability of states compared to a default state, that avoids these
undesirable incentives. Using a set of gridworld experiments illustrating
relevant scenarios, we empirically compare relative reachability to penalties
based on existing definitions and show that it is the only penalty among those
tested that produces the desired behavior in all the scenarios.