Organizing Experience: A Deeper Look at Replay Mechanisms for
  Sample-based Planning in Continuous State Domains
  Model-based strategies for control are critical to obtain sample efficient
learning. Dyna is a planning paradigm that naturally interleaves learning and
planning, by simulating one-step experience to update the action-value
function. This elegant planning strategy has been mostly explored in the
tabular setting. The aim of this paper is to revisit sample-based planning, in
stochastic and continuous domains with learned models. We first highlight the
flexibility afforded by a model over Experience Replay (ER). Replay-based
methods can be seen as stochastic planning methods that repeatedly sample from
a buffer of recent agent-environment interactions and perform updates to
improve data efficiency. We show that a model, as opposed to a replay buffer,
is particularly useful for specifying which states to sample from during
planning, such as predecessor states that propagate information in reverse from
a state more quickly. We introduce a semi-parametric model learning approach,
called Reweighted Experience Models (REMs), that makes it simple to sample next
states or predecessors. We demonstrate that REM-Dyna exhibits similar
advantages over replay-based methods in learning in continuous state problems,
and that the performance gap grows when moving to stochastic domains, of
increasing size.
