Title: Action Robust Reinforcement Learning and Applications in Continuous Control,
Abstract: A policy is said to be robust if it maximizes the reward while considering a
bad, or even adversarial, model. In this work we formalize two new criteria of
robustness to action uncertainty. Specifically, we consider two scenarios in
which the agent attempts to perform an action $\mathbf{a}$, and (i) with
probability $\alpha$, an alternative adversarial action $\bar{\mathbf{a}}$ is
taken, or (ii) an adversary adds a perturbation to the selected action in the
case of continuous action space. We show that our criteria are related to
common forms of uncertainty in robotics domains, such as the occurrence of
abrupt forces, and suggest algorithms in the tabular case. Building on the
suggested algorithms, we generalize our approach to deep reinforcement learning
(DRL) and provide extensive experiments in the various MuJoCo domains. Our
experiments show that not only does our approach produce robust policies, but
it also improves the performance in the absence of perturbations. This
generalization indicates that action-robustness can be thought of as implicit
regularization in RL problems.