SPOTTER: Extending Symbolic Planning Operators through Targeted
  Reinforcement Learning
  Symbolic planning models allow decision-making agents to sequence actions in
arbitrary ways to achieve a variety of goals in dynamic domains. However, they
are typically handcrafted and tend to require precise formulations that are not
robust to human error. Reinforcement learning (RL) approaches do not require
such models, and instead learn domain dynamics by exploring the environment and
collecting rewards. However, RL approaches tend to require millions of episodes
of experience and often learn policies that are not easily transferable to
other tasks. In this paper, we address one aspect of the open problem of
integrating these approaches: how can decision-making agents resolve
discrepancies in their symbolic planning models while attempting to accomplish
goals? We propose an integrated framework named SPOTTER that uses RL to augment
and support ("spot") a planning agent by discovering new operators needed by
the agent to accomplish goals that are initially unreachable for the agent.
SPOTTER outperforms pure-RL approaches while also discovering transferable
symbolic knowledge and does not require supervision, successful plan traces or
any a priori knowledge about the missing planning operator.
