Title: SOLAR: Deep Structured Latent Representations for Model-Based Reinforcement Learning,
Abstract: Model-based reinforcement learning (RL) methods can be broadly categorized as
global model methods, which depend on learning models that provide sensible
predictions in a wide range of states, or local model methods, which
iteratively refit simple models that are used for policy improvement. While
predicting future states that will result from the current actions is
difficult, local model methods only attempt to understand system dynamics in
the neighborhood of the current policy, making it possible to produce local
improvements without ever learning to predict accurately far into the future.
The main idea in this paper is that we can learn representations that make it
easy to retrospectively infer simple dynamics given the data from the current
policy, thus enabling local models to be used for policy learning in complex
systems. To that end, we focus on learning representations with probabilistic
graphical model (PGM) structure, which allows us to devise an efficient local
model method that infers dynamics from real-world rollouts with the PGM as a
global prior. We compare our method to other model-based and model-free RL
methods on a suite of robotics tasks, including manipulation tasks on a real
Sawyer robotic arm directly from camera images. Videos of our results are
available at this https URL