Title: Inverse Reinforcement Learning from Summary Data,
Abstract: Inverse reinforcement learning (IRL) aims to explain observed strategic
behavior by fitting reinforcement learning models to behavioral data. However,
traditional IRL methods are only applicable when the observations are in the
form of state-action paths. This assumption may not hold in many real-world
modeling settings, where only partial or summarized observations are available.
In general, we may assume that there is a summarizing function $\sigma$, which
acts as a filter between us and the true state-action paths that constitute the
demonstration. Some initial approaches to extending IRL to such situations have
been presented, but with very specific assumptions about the structure of
$\sigma$, such as that only certain state observations are missing. This paper
instead focuses on the most general case of the problem, where no assumptions
are made about the summarizing function, except that it can be evaluated. We
demonstrate that inference is still possible. The paper presents exact and
approximate inference algorithms that allow full posterior inference, which is
particularly important for assessing parameter uncertainty in this challenging
inference situation. Empirical scalability is demonstrated to reasonably sized
problems, and practical applicability is demonstrated by estimating the
posterior for a cognitive science RL model based on an observed user's task
completion time only.