Many real world tasks exhibit rich structure that is repeated across different parts of the state space or in time. In this work we study the possibility of leveraging such repeated structure to speed up and regularize learning. We start from the KL regularized expected reward objective which introduces an additional component, a default policy. Instead of relying on a fixed default policy, we learn it from data. But crucially, we restrict the amount of information the default policy receives, forcing it to learn reusable behaviors that help the policy learn faster. We formalize this strategy and discuss connections to information bottleneck approaches and to the variational EM algorithm. We present empirical results in both discrete and continuous action domains and demonstrate that, for certain tasks, learning a default policy alongside the policy can significantly speed up and improve learning.
@cite_2: A growing interest in decision making in psychology is evidenced by the recent publication of Edwards’ review article in the Psychological Bulletin (1) and the Santa Monica Conference volume, Decision Processes (7). In this work, much attention has been focused on the characterization of rational choice, and because the latter topic has been a central concern in economics, the theory of decision making has become a natural meeting ground for psychological and economic theory. A comparative examination of the models of adaptive behavior employed in psychology (e.g., learning theories), and of the models of rational behavior employed in economics, shows that in almost all respects the latter postulate a much greater complexity in the choice mechanisms, and a much larger capacity in the organism for obtaining information and performing computations, than do the former. Moreover, in the limited range of situations where the predictions of the two theories have been compared (see [7, Ch. 9, 10, 18]), the learning theories appear to account for the observed behavior rather better than do the theories of rational behavior. Both from these scanty data and from an examination of the postulates of the economic models it appears probable that, however adaptive the behavior of organisms in learning and choice situations, this adaptiveness falls far short of the ideal of “maximizing” postulated in economic theory. Evidently, organisms adapt well enough to “satisfice”; they do not, in general, “optimize.” If this is the case, a great deal can be learned about rational decision making by taking into account, at the outset, the limitations upon the capacities and complexity of the organism, and by taking account of the fact that the environments to which it must adapt possess properties that permit further simplication of its choice mechanisms. It may be useful, therefore, to ask: How simple a set of choice mechanisms can we postulate and still obtain the gross features of observed adaptive choice behavior? In a previous paper (6) I have put forth some suggestions as to the kinds of “approximate” rationality that might be employed by an organism possessing limited information and limited computational facilities. The suggestions were “hypothetical” in that, lacking definitive knowledge of the human decisional processes, we can only conjecture on the basis of our everyday experiences, our introspection, and a very limited body of psychological literature what these
The DISTRAL algorithm as well as the present paper can be seen as taking an intermediate position: unlike in the class of RL-as-inference algorithms the default policy is not fixed but learned, but unlike in the classical EM policy search the final result of the optimization remains regularized since the default policy is constrained relative to the policy. As explained above this can be seen as analogous to the relative roles of learned model and observation specific posterior in fitting a generative model. Similar to DISTRAL, Divide and Conquer learns an ensemble of policies, each specializing to a particular context, which are regularized towards one another via a symmetric KL penalty, with the behavior of the ensemble distilled to a single fixed policy. In concurrent work @cite_2 propose an information bottleneck architecture for policies with latent variables that leads to a KL-regularized formulation similar to the one described in Appendix . The information bottleneck is implemented in latent space and the default policy is obtained by marginalization with a goal-agnostic prior.