Online Bayesian Goal Inference for Boundedly-Rational Planning Agents
  People routinely infer the goals of others by observing their actions over
time. Remarkably, we can do so even when those actions lead to failure,
enabling us to assist others when we detect that they might not achieve their
goals. How might we endow machines with similar capabilities? Here we present
an architecture capable of inferring an agent's goals online from both optimal
and non-optimal sequences of actions. Our architecture models agents as
boundedly-rational planners that interleave search with execution by
replanning, thereby accounting for sub-optimal behavior. These models are
specified as probabilistic programs, allowing us to represent and perform
efficient Bayesian inference over an agent's goals and internal planning
processes. To perform such inference, we develop Sequential Inverse Plan Search
(SIPS), a sequential Monte Carlo algorithm that exploits the online replanning
assumption of these models, limiting computation by incrementally extending
inferred plans as new actions are observed. We present experiments showing that
this modeling and inference architecture outperforms Bayesian inverse
reinforcement learning baselines, accurately inferring goals from both optimal
and non-optimal trajectories involving failure and back-tracking, while
generalizing across domains with compositional structure and sparse rewards.
