Interactive Learning from Policy-Dependent Human Feedback
  For agents and robots to become more useful, they must be able to quickly
learn from non-technical users. This paper investigates the problem of
interactively learning behaviors communicated by a human teacher using positive
and negative feedback. Much previous work on this problem has made the
assumption that people provide feedback for decisions that is dependent on the
behavior they are teaching and is independent from the learner's current
policy. We present empirical results that show this assumption to be
false---whether human trainers give a positive or negative feedback for a
decision is influenced by the learner's current policy. We argue that
policy-dependent feedback, in addition to being commonplace, enables useful
training strategies from which agents should benefit. Based on this insight, we
introduce Convergent Actor-Critic by Humans (COACH), an algorithm for learning
from policy-dependent feedback that converges to a local optimum. Finally, we
demonstrate that COACH can successfully learn multiple behaviors on a physical
robot, even with noisy image features.
