Automatic formation of the structure of abstract machines in
  hierarchical reinforcement learning with state clustering
  We introduce a new approach to hierarchy formation and task decomposition in
hierarchical reinforcement learning. Our method is based on the Hierarchy Of
Abstract Machines (HAM) framework because HAM approach is able to design
efficient controllers that will realize specific behaviors in real robots. The
key to our algorithm is the introduction of the internal or "mental"
environment in which the state represents the structure of the HAM hierarchy.
The internal action in this environment leads to changes the hierarchy of HAMs.
We propose the classical Q-learning procedure in the internal environment which
allows the agent to obtain an optimal hierarchy. We extends the HAM framework
by adding on-model approach to select the appropriate sub-machine to execute
action sequences for certain class of external environment states. Preliminary
experiments demonstrated the prospects of the method.
