Continuous Homeostatic Reinforcement Learning for Self-Regulated
  Autonomous Agents
  Homeostasis is a prevalent process by which living beings maintain their
internal milieu around optimal levels. Multiple lines of evidence suggest that
living beings learn to act to predicatively ensure homeostasis (allostasis). A
classical theory for such regulation is drive reduction, where a function of
the difference between the current and the optimal internal state. The recently
introduced homeostatic regulated reinforcement learning theory (HRRL), by
defining within the framework of reinforcement learning a reward function based
on the internal state of the agent, makes the link between the theories of
drive reduction and reinforcement learning. The HRRL makes it possible to
explain multiple eating disorders. However, the lack of continuous change in
the internal state of the agent with the discrete-time modeling has been so far
a key shortcoming of the HRRL theory. Here, we propose an extension of the
homeostatic reinforcement learning theory to a continuous environment in space
and time, while maintaining the validity of the theoretical results and the
behaviors explained by the model in discrete time. Inspired by the
self-regulating mechanisms abundantly present in biology, we also introduce a
model for the dynamics of the agent internal state, requiring the agent to
continuously take actions to maintain homeostasis. Based on the
Hamilton-Jacobi-Bellman equation and function approximation with neural
networks, we derive a numerical scheme allowing the agent to learn directly how
its internal mechanism works, and to choose appropriate action policies via
reinforcement learning and an appropriate exploration of the environment. Our
numerical experiments show that the agent does indeed learn to behave in a way
that is beneficial to its survival in the environment, making our framework
promising for modeling animal dynamics and decision-making.
