This paper addresses the problem of learning a task from demonstration. We adopt the framework of inverse reinforcement learning, where tasks are represented in the form of a reward function. Our contribution is a novel active learning algorithm that enables the learning agent to query the expert for more informative demonstrations, thus leading to more sample-efficient learning. For this novel algorithm (Generalized Binary Search for Inverse Reinforcement Learning, or GBS-IRL), we provide a theoretical bound on sample complexity and illustrate its applicability on several different tasks. To our knowledge, GBS-IRL is the first active IRL algorithm with provable sample complexity bounds. We also discuss our method in light of other existing methods in the literature and its general applicability in multi-class classification problems. Finally, motivated by recent work on learning from demonstration in robots, we also discuss how different forms of human feedback can be integrated in a transparent manner in our learning framework.
@cite_27: Ubiquitous robots need the ability to adapt their behaviour to the changing situations and demands they will encounter during their lifetimes. In particular, non-technical users must be able to modify a robot's behaviour to enable it to perform new, previously unknown tasks. Learning from demonstration is a viable means to transfer a desired control policy onto a robot and mixed-initiative control provides a method for smooth transitioning between learning and acting. We present a learning system (dogged learning) that combines learning from demonstration and mixed initiative control to enable lifelong learning for unknown tasks. We have implemented dogged learning on a Sony Aibo and successfully taught it behaviours such as mimicry and ball seeking
@cite_10: We present Confidence-Based Autonomy (CBA), an interactive algorithm for policy learning from demonstration. The CBA algorithm consists of two components which take advantage of the complimentary abilities of humans and computer agents. The first component, Confident Execution, enables the agent to identify states in which demonstration is required, to request a demonstration from the human teacher and to learn a policy based on the acquired data. The algorithm selects demonstrations based on a measure of action selection confidence, and our results show that using Confident Execution the agent requires fewer demonstrations to learn the policy than when demonstrations are selected by a human teacher. The second algorithmic component, Corrective Demonstration, enables the teacher to correct any mistakes made by the agent through additional demonstrations in order to improve the policy and future task performance. CBA and its individual components are compared and evaluated in a complex simulated driving domain. The complete CBA algorithm results in the best overall learning performance, successfully reproducing the behavior of the teacher while balancing the tradeoff between number of demonstrations and number of incorrect actions during learning.
(CBA), proposed by @cite_10 , also enables a robot to learn a task from a human user by building a mapping between situations that the robot has encountered and the adequate actions. This work already incorporates a mechanism that enables the learner to ask the expert for the right action when it encounters a situation in which it is less confident about the correct behavior. The system also allows the human user to provide corrective feedback as the robot executes the learned task. Related ideas are further explored in the architecture of @cite_27 . The querying strategy in CBA can be classified both as and as [see discussions in the survey works of][] settles09tr,dasgupta11tcs . Stream-based, since the learner is presented with a stream of samples (in the case of CBA, samples correspond to possible situations) and only asks for the labels ( correct actions) of those samples it feels uncertain about. Mellow, since it does not seek highly informative samples, but queries any sample that is at all informative.