Hierarchy through Composition with Linearly Solvable Markov Decision
  Processes
  Hierarchical architectures are critical to the scalability of reinforcement
learning methods. Current hierarchical frameworks execute actions serially,
with macro-actions comprising sequences of primitive actions. We propose a
novel alternative to these control hierarchies based on concurrent execution of
many actions in parallel. Our scheme uses the concurrent compositionality
provided by the linearly solvable Markov decision process (LMDP) framework,
which naturally enables a learning agent to draw on several macro-actions
simultaneously to solve new tasks. We introduce the Multitask LMDP module,
which maintains a parallel distributed representation of tasks and may be
stacked to form deep hierarchies abstracted in space and time.
