Widening the Pipeline in Human-Guided Reinforcement Learning with
  Explanation and Context-Aware Data Augmentation
  Human explanation (e.g., in terms of feature importance) has been recently
used to extend the communication channel between human and agent in interactive
machine learning. Under this setting, human trainers provide not only the
ground truth but also some form of explanation. However, this kind of human
guidance was only investigated in supervised learning tasks, and it remains
unclear how to best incorporate this type of human knowledge into deep
reinforcement learning. In this paper, we present the first study of using
human visual explanations in human-in-the-loop reinforcement learning (HRL). We
focus on the task of learning from feedback, in which the human trainer not
only gives binary evaluative "good" or "bad" feedback for queried state-action
pairs, but also provides a visual explanation by annotating relevant features
in images. We propose EXPAND (EXPlanation AugmeNted feeDback) to encourage the
model to encode task-relevant features through a context-aware data
augmentation that only perturbs irrelevant features in human salient
information. We choose five tasks, namely Pixel-Taxi and four Atari games, to
evaluate the performance and sample efficiency of this approach. We show that
our method significantly outperforms methods leveraging human explanation that
are adapted from supervised learning, and Human-in-the-loop RL baselines that
only utilize evaluative feedback.
