Hierarchical Reinforcement Learning as a Model of Human Task
  Interleaving
  How do people decide how long to continue in a task, when to switch, and to
which other task? Understanding the mechanisms that underpin task interleaving
is a long-standing goal in the cognitive sciences. Prior work suggests greedy
heuristics and a policy maximizing the marginal rate of return. However, it is
unclear how such a strategy would allow for adaptation to everyday environments
that offer multiple tasks with complex switch costs and delayed rewards. Here
we develop a hierarchical model of supervisory control driven by reinforcement
learning (RL). The supervisory level learns to switch using task-specific
approximate utility estimates, which are computed on the lower level. A
hierarchically optimal value function decomposition can be learned from
experience, even in conditions with multiple tasks and arbitrary and uncertain
reward and cost structures. The model reproduces known empirical effects of
task interleaving. It yields better predictions of individual-level data than a
myopic baseline in a six-task problem (N=211). The results support hierarchical
RL as a plausible model of task interleaving.
