Human-Level Reinforcement Learning through Theory-Based Modeling,
  Exploration, and Planning
  Reinforcement learning (RL) studies how an agent comes to achieve reward in
an environment through interactions over time. Recent advances in machine RL
have surpassed human expertise at the world's oldest board games and many
classic video games, but they require vast quantities of experience to learn
successfully -- none of today's algorithms account for the human ability to
learn so many different tasks, so quickly. Here we propose a new approach to
this challenge based on a particularly strong form of model-based RL which we
call Theory-Based Reinforcement Learning, because it uses human-like intuitive
theories -- rich, abstract, causal models of physical objects, intentional
agents, and their interactions -- to explore and model an environment, and plan
effectively to achieve task goals. We instantiate the approach in a video game
playing agent called EMPA (the Exploring, Modeling, and Planning Agent), which
performs Bayesian inference to learn probabilistic generative models expressed
as programs for a game-engine simulator, and runs internal simulations over
these models to support efficient object-based, relational exploration and
heuristic planning. EMPA closely matches human learning efficiency on a suite
of 90 challenging Atari-style video games, learning new games in just minutes
of game play and generalizing robustly to new game situations and new levels.
The model also captures fine-grained structure in people's exploration
trajectories and learning dynamics. Its design and behavior suggest a way
forward for building more general human-like AI systems.
