Question: Represent the article for finding a single-sentence TLDR

Question: Recent work has shown that deep reinforcement-learning agents can learn to follow language-like instructions from infrequent environment rewards. However, this places on environment designers the onus of designing language-conditional reward functions which may not be easily or tractably implemented as the complexity of the environment and the language scales. To overcome this limitation, we present a framework within which instruction-conditional RL agents are trained using rewards obtained not from the environment, but from reward models which are jointly trained from expert examples. As reward models improve, they learn to accurately reward agents for completing tasks for environment configurations---and for instructions---not present amongst the expert data. This framework effectively separates the representation of what instructions require from how they can be executed. In a simple grid world, it enables an agent to learn a range of commands requiring interaction with blocks and understanding of spatial relations and underspecified abstract arrangements. We further show the method allows our agent to adapt to changes in the environment without requiring new expert examples. Figure 1: Different valid goal states for the instruction "build an L-like shape from red blocks".Developing agents that can learn to follow user instructions pertaining to an environment is a longstanding goal of AI research BID33. Recent work has shown deep reinforcement learning (RL) to be a promising paradigm for learning to follow language-like instructions in both 2D and 3D worlds (e.g. BID11 ; BID5, see Section 4 for a review). In each of these cases, being able to reward an agent for successfully completing a task specified by an instruction requires the implementation of a full interpreter of the instruction language. This interpreter must be able to evaluate the instruction against environment states to determine when reward must be granted to the agent, and in doing so requires full knowledge (on the part of the designer) of the semantics of the instruction language relative to the environment. Consider, for example, 4 arrangements of blocks presented in Figure 1. Each of them can be interpreted as a of successfully executing the instruction "build an L-like shape from red blocks", despite the fact that these arrangements differ in the location and the orientation of the target shape, as well as in the positioning of the irrelevant blue blocks. At best (e.g. for instructions such as the * Work done during an internship at DeepMind.† Now at Facebook AI Research.aforementioned one), implementing such an interpreter is feasible, although typically onerous in terms of engineering efforts to ensure reward can be given-for any admissible instruction in the language-in potentially complex or large environments. At worst, if we wish to scale to the full complexity of natural language, with all its ambiguity and underspecification, this requires solving fundamental problems of natural language understanding. If instruction-conditional reward functions cannot conveniently or tractably be implemented, can we somehow learn them in order to then train instruction-conditional policies? When there is a single implicit task, Inverse Reinforcement Learning (IRL; BID20 BID36 methods in general, and Generative Adversarial Imitation Learning BID12 in particular, have yielded some success in jointly learning reward functions from expert data and training policies from learned reward models. In this paper, we wish to investigate whether such mechanisms can be adapted to the more general case of jointly learning to understand language which specifies task objectives (e.g. instructions, goal specifications, directives), and use such understanding to reward language-conditional policies which are trained to complete such tasks. For simplicity, we explore a facet of this general problem in this paper by focussing on the case of declarative commands that specify sets of possible goal-states (e.g. "arrange the red blocks in a circle. "), and where expert examples need only be goal states rather than full trajectories or demonstrations, leaving such extensions for further work. We introduce a framework-Adversarial Goal-Induced Learning from Examples (AGILE)-for jointly training an instruction-conditional reward model using expert examples of completed instructions alongside a policy which will learn to complete instructions by maximising the thus-modelled reward. In this respect, AGILE relies on familiar RL objectives, with free choice of model architecture or training mechanisms, the only difference being that the reward comes from a learned reward model rather than from the environment. We first verify that our method works in settings where a comparison between AGILE-trained policies with policies trained from environment reward is possible, to which end we implement instructionconditional reward functions. In this setting, we show that the learning speed and performance of A3C agents trained with AGILE reward models is superior to A3C agents trained against environment reward, and comparable to that of true-reward A3C agents supplemented by auxiliary unsupervised reward prediction objectives. To simulate an instruction-learning setting in which implementing a reward function would be problematic, we construct a dataset of instructions and goal-states for the task of building colored orientation-invariant arrangements of blocks. On this task, without us ever having to implement the reward function, the agent trained within AGILE learns to construct arrangements as instructed. Finally, we study how well AGILE's reward model generalises beyond the examples on which it was trained. Our experiments show it can be reused to allow the policy to adapt to changes in the environment. Here, we introduce AGILE ("Adversarial Goal-Induced Learning from Examples", in homage to the adversarial learning mechanisms that inspire it), a framework for jointly learning to model reward for instructions, and learn a policy from such a reward model. Specifically, we learn an instructionconditional policy π θ with parameters θ, from a data stream G π θ obtained from interaction with the environment, by adjusting θ to maximise the expected total reward R π (θ) based on stepwise reward r t given to the policy, exactly as done in any normal Reinforcement Learning setup. The difference lies in the source of the reward: we introduce an additional discriminator network D φ, the reward model, whose purpose is to define a meaningful reward function for training π θ. We jointly learn this reward model alongside the policy by training it to predict whether a given state s is a goal state for a given instruction c or not. Rather than obtain positive and negative examples of instruction, state pairs from a purely static dataset, we sample them from a policy-dependent data stream. This stream is defined as follows: positive examples are drawn from a fixed dataset D of instructions c i paired with goal states s i; negative examples are drawn from a constantly-changing buffer of states obtained from the policy acting on the environment, paired with the instruction given to the policy. Formally, the policy is trained to maximize a return R π (θ) and the reward model is trained to minimize a cross-entropy loss L D (φ), the equations for which are: DISPLAYFORM0 DISPLAYFORM1 wherer t = [D φ (c, s t) > 0.5] In the equations above, the Iverson Bracket [. . .] maps truth to 1 and falsehood to 0, e.g. [x > 0] = 1 iff x > 0 and 0 otherwise. γ is the discount factor. With (c, s 1:∞) ∼ G π θ, we denote a state trajectory that was obtained by sampling (c, s 0) ∼ G and running π θ conditioned on c starting from s 0. B denotes a replay buffer to which (c, s) pairs from T -step episodes are added; i.e. it is the undiscounted occupancy measure over the first T steps. D φ (c, s) is the probability of (c, s) having a positive label according to the reward model, and thus [D φ (c, s t) > 0.5] indicates that a given state s t is more likely to be a goal state for instruction c than not, according to D. H(π θ) is the policy's entropy, and α is a hyperparameter. The approach is illustrated in Fig 2. Pseudocode is available in Appendix A. We note that Equation 1 differs from a traditional RL objective only in that the modelled rewardr t is used instead of the ground-truth reward r t. Indeed, in Section 3, we will compare policies trained with AGILE to policies trained with traditional RL, simply by varying the reward source from the reward model to the environment. Figure 2: Information flow during AGILE training. The policy acts conditioned on the instruction and is trained using the reward from the reward model (Figure 2a). The reward model is trained, as a discriminator, to distinguish between "A", the instruction, goal-state pairs from the dataset (Figure 2b), and "B", the instruction, state pairs from the agent's experience. Dealing with False Negatives Let us call Γ(c) the objective set of goal states which satisfy instruction c (which is typically unknown to us). Compared to the ideal case where all (c, s) would be deemed positive if-and-only-if s ∈ Γ(c), the labelling of examples implied by Equation 2 has a fundamental limitation when the policy performs well. As the policy improves, by definition, a increasing share of (c, s) ∈ B are objective goal-states from Γ(c). However, as they are treated as negative examples in Equation 2, the discriminator accuracy drops, causing the policy to get worse. We therefore propose the following simple heuristic to rectify this fundamental limitation by approximately identifying the false negatives. We rank (c, s) examples in B according to the reward model's output D φ (c, s) and discard the top 1 − ρ percent as potential false negatives. Only the other ρ percent are used as negative examples of the reward model. Formally speaking, the first term in Equation 2 becomes E(c,s)∼B D φ,ρ − log(1 − D φ (c, s)), where B D φ,ρ stands for the ρ percent of B selected, using D φ, as described above. We will henceforth refer to ρ as the anticipated negative rate. Setting ρ to 100% means using B D φ,100 = B like in Equation 2, but our preliminary experiments have shown clearly that this inhibits the reward model's capability to correctly learn a reward function. Using too small a value for ρ on the other hand may deprive the reward model of the most informative negative examples. We thus recommend to tune ρ as a hyperparameter on a task-specific basis. Reusability of the Reward Model An appealing advantage of AGILE is the fact that the reward model D φ and the policy π θ learn two related but distinct aspects of an instruction: the reward model focuses on recognizing the goal-states (what should be done), whereas the policy learns what to do in order to get to a goal-state (how it should be done). The intuition motivating this design is that the knowledge about how instructions define goals should generalize more strongly than the knowledge about which behavior is needed to execute instructions. Following this intuition, we propose to reuse a reward model trained in AGILE as a reward function for training or fine-tuning policies. Relation to GAIL AGILE is strongly inspired by-and retains close relations to-Generative Adversarial Imitation Learning (GAIL; BID12, which likewise trains both a reward function and a policy. The former is trained to distinguish between the expert's and the policy's trajectories, while the latter is trained to maximize the modelled reward. GAIL differs from AGILE in a number of important respects. First, AGILE is conditioned on instructions c so a single AGILE agent can learn combinatorially many skills rather than just one. Second, in AGILE the reward model observes only states s i (either goal states from an expert, or states from the agent acting on the environment) rather than state-action traces (s 1, a 1), (s 2, a 2),..., learning to reward the agent based on "what" needs to be done rather than according to "how" it must be done. Finally, in AGILE the policy's reward is the thresholded probability [D φ (c, s t)] as opposed to the log-probability log D φ (s t, a t) used in GAIL. Our reasoning for this change is that, when adapted to the setting with goal-specifications, a GAIL-style reward log D φ (c, s t) could take arbitrarily low values for intermediate states visited by the agent, as the reward model D φ becomes confident that those are not goal states. Empirically, we found that dropping the logarithm from GAIL-style rewards is indeed crucial for AGILE's performance, and that using the probability D φ (c, s t) as the rewardr t in a performance level similar to that of the discretized AGILE rewardr t = [D φ (c, s t)]. We experiment with AGILE in a grid world environment that we call GridLU, short for Grid Language Understanding and after the famous SHRDLU world BID33. GridLU is a fully observable grid world in which the agent can walk around the grid (moving up, down left or right), pick blocks up and drop them at new locations (see FIG2 for an illustration and Appendix C for a detailed description of the environment). All our models receive the world state as a 56x56 RGB image. With regard to processing the instruction, we will experiment with two kinds of models: Neural Module Networks (NMN) that treat the instruction as a structured expression, and a generic model that takes an unstructured instruction representation and encodes it with an LSTM.Because the language of our instructions is generated from a simple grammar, we perform most of our experiments using policy and reward model networks that are constructed using the NMN BID2 paradigm. NMN is an elegant architecture for grounded language processing in which a tree of neural modules is constructed based on the language input. The visual input is then fed to the leaf modules, which send their outputs to their parent modules, which process is repeated until the root of the tree. We mimick the structure of the instructions when constructing the tree of modules; for example, the NMN corresponding to the instruction DISPLAYFORM0, where m x denotes the module corresponding to the token x, and h s is a representation of state s. Each module m x performs a convolution (weights shared by all modules) followed by a token-specific Feature-Wise Linear Modulation (FiLM): DISPLAYFORM1 where h l and h r are module inputs, γ x is a vector of FiLM multipliers, β x are FiLM biases, and ⊕ are element-wise multiplication and addition with broadcasting, * denotes convolution. The representation h s is produced by a convnet. The NMN's output h N M N undergoes max-pooling and is fed through a 1-layer MLP to produce action probabilities or the reward model's output. Note, that while structure-wise our policy and reward model are mostly similar, they do not share parameters. NMN is an excellent model when the language structure is known, but this may not be the case for natural language. To showcase AGILE's generality we also experiment with a very basic structure-agnostic architecture. We use FiLM to condition a standard convnet on an instruction representation h LST M produced by an LSTM. The k-th layer of the convnet performs a computation DISPLAYFORM2 The same procedure as described above for h N M N is used to produce the network outputs using the output h 5 of the 5 th layer of the convnet. In the rest of the paper we will refer to the architectures described above as FiLM-NMN and FiLM-LSTM respectively. FiLM-NMN will be the default model in all experiments unless explicitly specified otherwise. Detailed information about network architectures can be found in Appendix G. For the purpose of training the policy networks both within AGILE, and for our baseline trained from ground-truth reward r t instead of the modelled rewardr t, we used the Asynchronous Advantage Actor-Critic (A3C; . Any alternative training mechanism which uses reward could be used-since the only difference in AGILE is the source of the reward signal, and for any such alternative the appropriate baseline for fair comparison would be that same algorithm applied to train a policy from ground-truth reward. We will refer to the policy trained within AGILE as AGILE-A3C. The A3C's hyperparameters γ and λ were set to 0.99 and 0 respectively, i.e. we did not use without temporal difference learning for the baseline network. The length of an episode was 30, but we trained the agent on advantage estimation rollouts of length 15. Every experiment was repeated 5 times. We considered an episode to be a success if the final state was a goal state as judged by a task-specific success criterion, which we describe for the individual tasks below. We use the success rate (i.e. the percentage of successful episodes) as our main performance metric for the agents. Unless otherwise specified we use the NMN-based policy and reward model in our experiments. Full experimental details can be found in Appendix D. Green triangle west of a red circle Our first task, GridLU-Relations, is an adaptation of the SHAPES visual question answering dataset BID2 in which the blocks can be moved around freely. GridLU-Relations requires the agent to induce the meaning of spatial relations such as above or right of, and to manipulate the world in order to instantiate these relationships. Named GridLU-Relations, the task involves five spatial relationships (NorthFrom, SouthFrom, EastFrom, WestFrom, SameLocation), whose arguments can be either the blocks, which are referred to by their shapes and colors, or the agent itself. To generate the full set of possible instructions spanned by these relations and our grid objects, we define a formal grammar that generates strings such as: DISPLAYFORM0 This string carries the meaning'put a red circle north from (above) a blue square'. In general, when a block is the argument to a relation, it can be referred to by specifying both the shape and the color, like in the example above, or by specifying just one of these attributes. In addition, the AGENT constant can be an argument to all relations, in which case the agent itself must move into a particular spatial relation with an object. FIG2 shows two examples of GridLU-Relations instructions and their respective goal states. There are 990 possible instructions in the GridLU-Relations task, and the number of distinct training instances can be loosely lower-bounded by 1.8 · 10 7 (see Appendix E for details).Notice that, even for the highly concrete spatial relationships in the GridLU-Relations language, the instructions are underspecified and somewhat ambiguous-is a block in the top-right corner of the grid above a block in the bottom left corner? We therefore decided (arbitrarily) to consider all relations to refer to immediate adjacency (so that Instruction equation 3 is satisfied if and only if there is a red circle in the location immediately above a blue square). Notice that the commands are still underspecified in this case (since they refer to the relationship between two entities, not their absolute positions), even if the degree of ambiguity in their meaning is less than in many real-world cases. The policy and reward model trained within AGILE then have to infer this specific sense of what these spatial relations mean from goal-state examples, while the baseline agent is allowed to access our programmed ground-truth reward. The binary ground-truth reward (true if the state is a goal state) is also used as the success criterion for evaluating AGILE.Having formally defined the semantics of the relationships and programmed a reward function, we compared the performance of an AGILE-A3C agent against a priviliged baseline A3C agent trained using ground-truth reward. Interestingly, we found that AGILE-A3C learned the task more easily than standard A3C (see the respective curves in FIG3). We hypothesize this is because the modeled rewards are easy to learn at first and become more sparse as the reward model slowly improves. This naturally emerging curriculum expedites learning in the AGILE-A3C when compared to the A3C-trained policy that only receives signal upon reaching a perfect goal state. We did observe, however, that the A3C algorithm could be improved significantly by applying the auxiliary task of reward prediction (RP; BID13, which was applied to language learning tasks by BID11 (see the A3C and A3C-RP curves in FIG3). This objective reinforces the association between instructions and states by having the agent replay the states immediately prior to a non-zero reward and predict whether or not it the reward was positive (i.e. the states match the instruction) or not. This mechanism made a significant difference to the A3C performance, increasing performance to 99.9%. AGILE-A3C also achieved nearly perfect performance (99.5%). We found this to be a very promising , since within AGILE, we induce the reward function from a limited set of examples. The best with AGILE-A3C were obtained using the anticipated negative rate ρ = 25%. When we used larger values of ρ AGILE-A3C training started quicker but after 100-200 million steps the performance started to deteriorate (see AGILE curves in FIG3), while it remained stable with ρ = 25%.Data efficiency These suggest that the AGILE reward model was able to induce a near perfect reward function from a limited set of instruction, goal-state pairs. We therefore explored how small this training set of examples could be to achieve reasonable performance. We found that with a training set of only 8000 examples, the AGILE-A3C agent could reach a performance of 60% (massively above chance). However, the optimal performance was achieved with more than 100,000 examples. The full are available in Appendix D. In the experiments we have reported so far the AGILE agent was trained on all 990 possible GridLU-Relation instructions. In order to test generalization to unseen instructions we held out 10% of the instructions as the test set and used the rest 90% as the training set. Specifically, we restricted the training instances and instruction, goal-state pairs to only contain instructions from the training set. The performance of the trained model on the test instructions was the same as on the training set, showing that AGILE did not just memorise the training instructions but learnt a general interpretation of GridLU-Relations instructions. AGILE with Structure-Agnostic Models We report the for AGILE with a structureagnostic FILM-LSTM model in FIG3 (middle). AGILE with ρ = 25% achieves a high 97.5% success rate, and notably it trains almost as fast as an RL-RP agent with the same architecture. Analyzing the reward model We compare the binary reward provided by the reward model with the ground-truth from the environment during training on the GridLU-Relation task. With ρ = 25% the accuracy of the reward model peaks at 99.5%. As shown in FIG3 (right) the reward model learns faster in the beginning with larger values of ρ but then deteriorates, which confirms our intuition about why ρ is an important hyperparameter and is aligned with the success rate learning curves in FIG3 (left). We also observe during training that the false negative rate is always kept reasonably low (<3% of rewards) whereas the reward model will initially be more generous with false positives (20-50% depending on ρ during the first 20M steps of training) and will produce an increasing number of false positives for insufficiently small values of ρ (see plots in Appendix E). We hypothesize that early false positives may facilitate the policy's training by providing it with a sort of curriculum, possibly explaining the improvement over agents trained from ground-truth reward, as shown above. The reward model as general reward function An instruction-following agent should be able to carry-out known instructions in a range of different contexts, not just settings that match identically the specific setting in which those skills were learned. To test whether the AGILE framework is robust to (semantically-unimportant) changes to the environment dynamics, we first trained the policy and reward model as normal and then modified the effective physics of the world by making all red square objects immovable. In this case, following instructions correctly is still possible in almost all cases, but not all solutions available during training are available at test time. As expected, this change impaired the policy and the agent's success rate on the instructions referring to a red square dropped from 98% to 52%. However, after fine-tuning the policy (additional training of the policy on the test episodes using the reward from the previously-trained-then-frozen reward model), the success rate went up to 69.3% (FIG4). This experiment suggests that the AGILE reward model learns useful and generalisable linguistic knowledge. The knowledge can be applied to help policies adapt in scenarios where the high-level meaning of commands is familiar but the low-level physical dynamics is not. The experiments thus far demonstrate that even without directly using the reward function AGILE-A3C performs comparably to its pure A3C counter-part. However, the principal motivation for the AGILE framework is to avoid programming the reward function. To model this setting more explicitly, we developed the task GridLU-Arrangements, in which each instruction is associated with multiple viable goal-states that share some (more abstract) common form. The complete set of instructions and forms is illustrated in FIG2. To get training data, we built a generator to produce random instantiations (i.e. any translation, rotation, reflection or color mapping of the illustrated forms) of these goal-state classes, as positive examples for the reward model. In the real world, this process of generating goal-states could be replaced by finding, or having humans annotate, labelled images. In total, there are 36 possible instructions in GridLU-Arrangements, which together refer to a total of 390 million correct goal-states (see Appendix F for details). Despite this enormous space of potentially correct goal-states, we found that for good performance it was necessary to train AGILE on only 100,000 (less than 0.3%) of these goal-states, sampled from the same distribution as observed in the episodes. To replicate the conditions of a potential AGILE application as close as possible, we did not write a reward function for GridLU-Arrangements (even though it would have been theoretically possible), and instead carried out all evaluation manually. Half of the episodes began with four square blocks (all of the same color), and the agent, in random unique positions, and an instruction sampled uniformly from the list of possible arrangement words. In the other half of the episodes, four square blocks of one color and four square blocks of a different color were initially each positioned randomly. The instruction in these episodes specified one of the two colors together with an arrangement word. We trained policies and reward models using AGILE with 10 different seeds for each level, and selected the best pair based on how well the policy maximised modelled reward. We then manually assessed the final state of each of 200 evaluation episodes, using human judgement that the correct shape has been produced as success criterion to evaluate AGILE. We found that the agent made the correct arrangement in 58% of the episodes. The failure cases were almost always in the episodes involving eight blocks 1. In these cases, the AGILE agent tended towards building the correct arrangement, but was impeded by the randomly positioned non-target-color blocks and could not recover. Nonetheless, these scores, and the compelling behaviour observed in the video (https://www.youtube.com/watch?v=07S-x3MkEoQ), demonstrate the potential of AGILE for teaching agents to execute semantically vague or underspecified instructions. Learning to follow language instructions has been approached in many different ways, for example by reinforcement learning using a reward function programmed by a system designer. consider instruction-following in 2D or 3D environments and reward the agent for arriving at the correct location or object. BID14 and BID18 train RL agents to produce goal-states given instructions. As discussed, these approaches are constrained by the difficulty of programming language-related reward functions, a task that requires an programming expert, detailed access to the state of the environment and hard choices above how language should map to the world. Agents can be trained to follow instructions using complete demonstrations, that is sequences of correct actions describing instruction execution for given initial states.; BID3 train semantic parsers to produce a formal representation of the query that when fed to a predefined execution model matches exactly the sequence of actions from the demonstration. BID1; BID17 sidestep the intermediate formal representation and train a Conditional Random Field (CRF) and a sequenceto-sequence neural model respectively to directly predict the actions from the demonstrations. A underlying assumption behind all these approaches is that the agent and the demonstrator share the same actuation model, which might not always be the case. In the case of navigational instructions the trajectories of the agent and the demonstrators can sometimes be compared without relying on the actions, like e.g. BID28, but for other types of instructions such a hard-coded comparison may be infeasible. train a log-linear model to map instruction constituents into their groundings, which can be objects, places, state sequences, etc. Their approach requires access to a structured representation of the world environment as well as intermediate supervision for grounding the constituents. Our work can be categorized as apprenticeship (imitation) learning, which studies learning to perform tasks from demonstrations and feedback. Many approaches to apprenticeship learning are variants of inverse reinforcement learning (IRL), which aims to recover a reward function from expert demonstrations BID0 BID36. As stated at the end of Section 2, the method most closely related to AGILE is the GAIL algorithm from the IRL family BID12. There have been earlier attempts to use IRL-style methods for instruction following BID16 BID30, but unlike AGILE, they relied on the availability of a formal reward specification language. To our knowledge, ours and the concurrent work by BID9 are the first works to showcase learning reward models for instructions from pixels directly. Besides IRL-style approaches, other apprenticeship learning methods involve training a policy BID15 BID29 or a reward function BID31 BID7 directly from human feedback. Several recent imitation learning works consider using goal-states directly for defining the task BID10 BID23. AGILE differs from these approaches in that goal-states are only used to train the reward module, which we show generalises to new environment configurations or instructions, relative to those seen in the expert data. We have proposed AGILE, a framework for training instruction-conditional RL agents using rewards from learned reward models, which are jointly trained from data provided by both experts and the agent being trained, rather than reward provided by an instruction interpreter within the environment. This opens up new possibilities for training language-aware agents: in the real world, and even in rich simulated environments BID4 BID34, acquiring such data via human annotation would often be much more viable than defining and implementing reward functions programmatically. Indeed, programming rewards to teach robust and general instruction-following may ultimately be as challenging as writing a program to interpret language directly, an endeavour that is notoriously laborious BID32, and some say, ultimately futile BID33.As well as a means to learn from a potentially more prevalent form of data, our experiments demonstrate that policies trained in the AGILE framework perform comparably with and can learn as fast as those trained against ground-truth reward and additional auxiliary tasks. Our analysis of the reward model's classifications gives a sense of how this is possible; the false positive decisions that it makes early in the training help the policy to start learning. The fact that AGILEs objective attenuates learning issues due to the sparsity of reward states within episodes in a manner similar to reward prediction suggests that the reward model within AGILE learns some form of shaped reward BID21, and could serve not only in the cases where a reward function need to be learned in the absence of true reward, but also in cases where environment reward is defined but sparse. As these cases are not the focus of this study, we note this here, but leave such investigation for future work. As the policy improves, false negatives can cause the reward model accuracy to deteriorate. We determined a simple method to mitigate this, however, leading to robust training that is comparable to RL with reward prediction and unlimited access to a perfect reward function. Another attractive aspect of AGILE is that learning "what should be done" and "how it should be done" is performed by two different model components. Our experiments confirm that the "what" kind of knowledge generalizes better to new environments. When the dynamics of the environment changed at test time, fine-tuning using frozen reward model allowed to the policy recover some of its original capability in the new setting. While there is a large gap to be closed between the sort of tasks and language experimented with in this paper and those which might be presented in "real world" situations or more complex environments, our provide an encouraging first step in this direction. Indeed, it is interesting to consider how AGILE could be applied to more realistic learning settings, for instance involving first-person vision of 3D environments. Two issues would need to be dealt with, namely training the agent to factor out the difference in perspective between the expert data and the agent's observations, and training the agent to ignore its own body parts if they are visible in the observations. Future work could focus on applying third-person imitation learning methods recently proposed by BID26 learn the aforementioned invariances. Most of our experiments were conducted with a formal language with a known structure, however AGILE also performed very well when we used a structure-agnostic FiLM-LSTM model which processed the instruction as a plain sequence of tokens. This suggest that in future work AGILE could be used with natural language instructions. Require: The policy network π θ, the discriminator network D φ, the anticipated negative rate ρ, a dataset D, a replay buffer B, the batch size BS, a stream of training instances G, the episode length T, the rollout length R. 1: while Not Converged do 2:Sample a training instance (c, s0) ∈ G. 3:t ← 0 4: DISPLAYFORM0 Act with π θ (c, s) and produce a rollout (c, st...t+R). Add (c, s) pairs from (c, st...t+R) to the replay buffer B. Remove old pairs from B if it is overflowing. ComputeLD DISPLAYFORM0 Compute the gradient DISPLAYFORM0 and use it to update φ. Synchronise θ and φ with other workers. 13: DISPLAYFORM0 end while 15: end while Require: The policy network π θ, the discriminator network D φ, a dataset D, a replay buffer B, a stream of training instances G, the episode length T. 1: while Not Converged do 2:Sample a training instance (c, s0) ∈ G. t ← 0 4: DISPLAYFORM0 Act with π θ (c, s) and produce a rollout (c, st...t+R). Use the discriminator D φ to compute the rewards rτ = [D φ (c, sτ) > 0.5]. Perform an RL update for θ using the rewards rτ. 8:Synchronise θ and φ with other workers. 9: t ← t + R 10: end while 11: end while We trained the policy π θ and the discriminator D φ concurrently using RMSProp as the optimizer and Asynchronous Advantage Actor-Critic (A3C) as the RL method. A baseline predictor (see Appendix G for details) was trained to predict the discounted return by minimizing the mean square error. The RMSProp hyperparameters were different for π θ and D φ, see TAB0. A designated worker was used to train the discriminator (see Algorithm 1). Other workers trained only the policy (see Algorithm 2). We tried having all workers write to the replay buffer B that was used for the discriminator training and found that this gave the same performance as using (c, s) pairs produced by the discriminator worker only. We found it crucial to regularize the discriminator by clipping columns of all weights matrices to have the L2 norm of at most 1. In particular, we multiply incoming weights w u of each unit u by min(1, 1/||w u || 2) after each gradient update as proposed by BID25. We linearly rescaled the policy's rewards to the [0; 0.1] interval for both RL and AGILE. When using RL with reward prediction we fetch a batch from the replay buffer and compute the extra gradient for every rollout. For the exact values of hyperparameters for the GridLU-Relations task we refer the reader to TAB0. The hyperparameters for GridLU-Arrangements were mostly the same, with the exception of the episode length and the rollout length, which were 45 and 30 respectively. For training the RL baseline for GridLU-Relations we used the same hyperparameter settings as for the AGILE policy. The GridLU world is a 5 × 5 gridworld surrounded by walls. The cells of the grid can be occupied by blocks of 3 possible shapes (circle, triangle, and square) and 3 possible colors (red, blue, and green). The grid also contains an agent sprite. The agent may carry a block; when it does so, the agent sprite changes color 2. When the agent is free, i.e. when it does not carry anything, it is able to enter cells with blocks. A free agent can pick a block in the cell where both are situated. An agent that carries a block cannot enter non-empty cells, but it can instead drop the block that it carries in any non-empty cell. Both picking up and dropping are realized by the INTERACT action. Other available actions are LEFT, RIGHT, UP and DOWN and NOOP. The GridLU agent can be seen as a cursor (and this is also how it is rendered) that can be moved to select a block or a position where the block should be released. FIG7 illustrates the GridLU world and its dynamics. We render the state of the world as a color image by displaying each cell as an 8 × 8 patch 3 and stitching these patches in a 56 × 56 image 4. All neural networks take this image as an input. Every experiment was repeated 5 times and the average is reported. RL vs. AGILE All agents were trained for 5 · 10 8 steps. Data Efficiency We trained AGILE policies with datasets D of different sizes for 5 · 10 8 steps. For each policy we report the maximum success rate that it showed in the course of training. GridLU-Arrangements We trained the agent for 100M time steps, saving checkpoints periodically, and selected the checkpoint that best fooled the discriminator according to the agent's internal reward. Data Efficiency We measure how many examples of instructions and goal-states are required by AGILE in order to understand the semantics of the GridLU-Relations instruction language. The are reported in FIG8. The AGILE-trained agent succeeds in more than 50% of cases starting from 8000 examples, but as many as 130000 is required for the best performance. All GridLU instructions can be generated from <instruction> using the following Backus-Naur form, with one exception: The first expansion of <obj> must not be identical to the second expansion of <obj> in <bring to instruction>. There are 15 unique possibilities to expand the nonterminal <obj>, so there are 150 unique possibilities to expand <go to instruction> and 840 unique possibilities to expand <bring to instruction> (not counting the exceptions mentioned above). Hence there are 990 unique instructions in total. However, several syntactically different instructions can be semantically equivalent, such as EastFrom(AGENT, Shape(rect, SCENE)) and WestFrom(Shape(rect, SCENE), AGENT).Every instruction partially specifies what kind of objects need to be available in the environment. For go-to-instructions we generate one object and for bring-to-instructions we generate two objects according to this partial specification (unspecified shapes or colors are picked uniformly at random). Additionally, we generate one "distractor object". This distractor object is drawn uniformly at random from the 9 possible objects. All of these objects and the agent are each placed uniformly at random into one of 25 cells in the 5x5 grid. The instance generator does not sample an instruction uniformly at random from a list of all possible instructions. Instead, it generates the environment at the same time as the instruction according to the procedure above. Afterwards we impose two'sanity checks': are any two objects in the same location or are they all identical? If any of these two checks fail, the instance is discarded and we start over with a new instance. Because of this rejection sampling technique, go-to-instructions are ultimately generated with approximately 25% probability even though they only represent ≈ 15% of all possible instructions. The number of different initial arrangements of three objects can be lower-bounded by 9 3 = 2300 if we disregard their permutation. Hence every bring-to-instruction has at least K = 2300 · 9 ≈ 2 · 10 4 associated initial arrangements. Therefore the total number of task instances can be lower-bounded with 840 · K ≈ 1.7 · 10 7, disregarding the initial position of the agent. During the training on GridLU-Relations we compared the predictions of the discriminator with those of the ground-truth reward checker. This allowed us to monitor several performance indicators of the discriminator, see FIG10. Instruction Syntax We used two types of instructions in the GridLU-Arrangements task, those referring only to the arrangement and others that also specified the color of the blocks. Examples Connected(AGENT, SCENE) and Snake(AGENT, Color('yellow', SCENE)) illustrate the syntax that we used for both instruction types. Number of Distinct Goal-States TAB1 presents our computation of the number of distinct goal-states in the GridLU-Arrangements Task. In this section we explain in detail the neural architectures that we used in our experiments. We will use * to denote convolution,, ⊕ to denote element-wise addition of a vector to a 3D tensor with broadcasting (i.e. same vector will be added/multiplied at each location of the feature map). We used ReLU as the nonlinearity in all layers with the exception of LSTM.FiLM-NMN We will first describe the FiLM-NMN discriminator D φ. The discriminator takes a 56x56 RGB image s as the representation of the state. The image s is fed through a stem convnet that consisted of an 8x8 convolution with 16 kernels and a 3x3 convolution with 64 kernels. The ing tensor h stem had a 5x5x64 shape. As a Neural Module Metwork BID2, the FiLM-NMN is constructed of modules. The module m x corresponding to a token x takes a left-hand side input h l and a right-hand side input h r and performs the following computation with them: DISPLAYFORM0 where γ x and β x are FiLM coefficients corresponding to the token x, W m is a weight tensor for a 3x3 convolution with 128 input features and 64 output features. Zero-padding is used to ensure that the output of m x has the same shape as h l and h r. The equation above describes a binary module that takes two operands. For the unary modules that received only one input (e.g. m red, m square) we present the input as h l and zeroed out h r. This way we are able to use the same set of weights W m for all modules. We have 12 modules in total, 3 for color words, 3 for shape words, 5 for relations words and one m AGEN T module used in go-to instructions. The modules are selected and connected based on the instructions, and the output of the root module is used for further processing. For example, the following computation would be performed for the instruction c 1 =NorthFrom(Color('red', Shape('circle', SCENE)), Color('blue', Shape('square', SCENE))): h nmn = m N orthF rom (m red (m circle (h stem)), m blue (m square (h stem))),and the following one for c 2 =NorthFrom(AGENT, Shape('triangle', SCENE)): h nmn = m N orthF rom (m AGEN T (h stem), m triangle (h stem)).Finally, the output of the discriminator is computed by max-pooling the output of the FiLM-NMN across spatial dimensions and feeding it to an MLP with a hidden layer of 100 units: DISPLAYFORM1 where w, W and b are weights and biases, σ(x) = e x /(1 + e x) is the sigmoid function. The policy network π φ is similar to the discriminator network D θ. The only difference is that it outputs softmax probabilites for 5 actions instead of one real number we use an additional convolutional layer to combine the output of FiLM-NMN and h stem: h merge = ReLU (W merge * [h nmn ; h stem] + b merge ), π(c, s) = softmax(W 2 ReLU (W 1 maxpool (h merge) + b 1 ) + b 2 ), the output h merge of which is further used in the policy network instead of h nmn. FIG11 illustrates our FiLM-NMN policy and discriminator networks. FiLM-LSTM For our structure-agnostic models we use an LSTM of 100 hidden units to predict FiLM biases and multipliers for a 5 layer convnet. More specifically, let h LST M be the final state of the LSTM after it consumes the instruction c. We compute the FiLM coefficients for the layer k ∈ [1; 5] as follows: DISPLAYFORM2 DISPLAYFORM3 and use them as described by the equation below: DISPLAYFORM4 where W k are the convolutional weights, h 0 is set to the pixel-level representation of the world state s. The characteristics of the 5 layers were the following: (8x8, 16, VALID), (3x3, 32, VALID), (3x3, 64, SAME), (3x3, 64, SAME), (3x3, 64, SAME), where (mxm, n out, p) stands for a convolutional layer with mxm filters, n out output features, and p ∈ {SAME, VALID} padding strategy. Layers with p = VALID do not use padding, whereas in those with p = SAME zero padding is added in order to produce an output with the same shape as the input. The layer 5 is also connected to layer 3 by a residual connection. Similarly to FiLM-NMN, the output h 5 of the convnet is max-pooled and fed into an MLP with 100 hidden units to produce the outputs: DISPLAYFORM5 Baseline prediction In all policy networks the baseline predictor is a linear layer that took the same input as the softmax layer. The gradients of the baseline predictor are allowed to propagate through the rest of the network. Reward prediction We use the h maxpool of the max-pooling operation (which was a part of all models that we considered) as the input to the reward prediction pathway of our model. h maxpool is fed through a linear layer and softmax to produce probabilities of the reward being positive or zero (the reward is never negative in AGILE)., where f an in is the product of kernel width, kernel height and the number of input features.

Represent the short summary for finding the corresponding scientific article (long)

We propose AGILE, a framework for training agents to perform instructions from examples of respective goal-states.

Represent the TLDR for finding the corresponding scientific article.

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.


Question: Represent the scientific article to find a short TLDR

Question: We present Multitask Soft Option Learning (MSOL), a hierarchical multi-task framework based on Planning-as-Inference. MSOL extends the concept of Options, using separate variational posteriors for each task, regularized by a shared prior. The learned soft-options are temporally extended, allowing a higher-level master policy to train faster on new tasks by making decisions with lower frequency. Additionally, MSOL allows fine-tuning of soft-options for new tasks without unlearning previously useful behavior, and avoids problems with local minima in multitask training. We demonstrate empirically that MSOL significantly outperforms both hierarchical and flat transfer-learning baselines in challenging multi-task environments. A key challenge in Reinforcement Learning (RL) is to scale current approaches to higher complexity tasks without requiring a prohibitive number of environmental interactions. However, for many tasks, it is possible to construct or learn efficient exploration priors that allow to focus on more relevant parts of the state-action space, reducing the number of required interactions. These include, for example, reward shaping , curriculum learning , some meta-learning algorithms (; ; ; ;), and transfer learning (; ; ; ; ;). One promising way to capture prior knowledge is to decompose policies into a hierarchy of subpolicies (or skills) that can be reused and combined in novel ways to solve new tasks (; ; ; ;). The idea of Hierarchical RL (HRL) is also supported by findings that humans appear to employ a hierarchical mental structure when solving tasks . In such a hierarchical RL policy, lower-level, temporally extended skills yield directed behavior over multiple time steps. This has two advantages: i) it allows efficient exploration, as the target states of skills can be reached without having to explore much of the state space in between, and ii) directed behavior also reduces the variance of the future reward, which accelerates convergence of estimates thereof. On the other hand, while a hierarchical approach can therefore significantly speed up exploration and training, it can also severely limit the expressiveness of the final policy and lead to suboptimal performance when the temporally extended skills are not able to express the required policy for the task at hand . Many methods exist for constructing and/or learning skills for particular tasks (; ; ; ; Ş imşek & ; ; ; ; a). Training on multiple tasks simultaneously is one promising approach to learn skills that are both relevant and generalise across tasks (; ; ; ;). Ideally, the entire hierarchy can be trained end-to-end on the obtained return, obviating the need to specify proxy rewards for skills . However, learning hierarchical policies end-to-end in a multitask setting poses two major challenges: i) because skills optimize environmental rewards directly, correctly updating them relies on already (nearly) converged master policies that use them similarly across all tasks, requiring complex training schedules , and ii) the end-to-end optimization is prone to local minima in which multiple skills have learned similar behavior. This second points is explained in more detail in Appendix A. In this paper, we propose Multitask Soft Option Learning (MSOL), a novel approach to learning hierarchical policies in a multi-task setting that extends Options , a common definition for skills, and casts the concept into the Planning as Inference (PAI) framework (see, e.g., , for a review). MSOL brings multiple advantages: i) it stabilizes end-to-end multitask training, removing the need for complex training schedules like in , ii) it gives rise to coordination between master policies, avoiding local minima of the type described in Appendix A, iii) it allows fine-tuning of options, i.e. adapting them to new tasks at test-time without the risk of unlearning previously acquired useful behavior, thereby avoiding suboptimal performance due to restricted expressiveness, iv) and lastly, we show how the soft option framework gives rise to a natural solution to the challenging task of learning option-termination policies. MSOL differentiates between a prior policy for each option, shared across all tasks, and a flexible task-specific posterior policy. The option prior can be fixed once it is fully trained, preventing unlearning of useful behavior even when the posteriors are updated. On new tasks, the option posteriors are initialized to the priors and regularized towards them, but are still adaptable to the specific task. This allows the same accelerated training as with'hard' options, but can solve more tasks due to the adjustable posteriors. Furthermore, during option learning, we can train prior and posterior policies simultaneously , all without the need for complex training schedules : training is stabilized because only the priors are shared across tasks. Our experiments demonstrate that MSOL outperforms previous hierarchical and transfer learning algorithms during transfer tasks in a multitask setting. MSOL only modifies the regularized reward and loss function, but does not require any specialized architecture. In particular, it also does not require artificial restrictions on the expressiveness of either the higher-level or intra-option policies. 2 PRELIMINARIES An agent's task is formalized as a MDP (S, A, ρ, P, r), consisting of the state space S, the action space A, the initial state distribution ρ, the transition probability P (s t+1 |s t, a t) of reaching state s t+1 by executing action a t in state s t, and the reward r(s t, a t) an agent receives for this transition. Planning as inference (PAI) (; ;) frames RL as a probabilistic-inference problem . The agent learns a distribution q φ (a|s) over actions a given states s, i.e., a policy, parameterized by φ, which induces a distribution over trajectories τ of length T, i.e., τ = (s 1, a 1, s 2, . . ., a T, s T +1): This can be seen as a structured variational approximation of the optimal trajectory distribution. Note that the true initial state probability ρ(s 1) and transition probability P (s t+1 |s t, a t) are used in the variational posterior, as we can only control the policy, not the environment. An advantage of this formulation is that we can incorporate information both from prior knowledge, in the form of a prior policy distribution, and the task at hand through a likelihood function that is defined in terms of the achieved reward. The prior policy p(a t |s t) can be specified by hand or, as in our case, learned (see Section 3). To incorporate the reward, we introduce a binary optimality variable O t , whose likelihood is highest along the optimal trajectory that maximizes return: p(O t = 1|s t, a t) = exp r(s t, a t)/β. The constraint r ∈ (−∞, 0] can be relaxed without changing the inference procedure . For brevity, we denote If a given prior policy p(a t |s t) explores the state-action space sufficiently, then p(τ, O 1:T) is the distribution of desirable trajectories. PAI aims to find a policy such that the variational posterior in approximates this distribution by minimizing the Kullback-Leibler (KL) divergence 2.2 MULTI-TASK LEARNING In a multi-task setting, we have a set of different tasks i ∈ T, drawn from a task distribution with probability ξ(i). All tasks share state space S and action space A, but each task has its own initialstate distribution ρ i, transition probability P i (s t+1 |s t, a t), and reward function r i. Our goal is to learn n tasks concurrently, distilling common information that can be leveraged to learn faster on new tasks from T. In this setting, the prior policy p θ (a t |s t) can be learned jointly with the taskspecific posterior policies q φi (a t |s t) . To do so, we simply extend to where is the regularised reward. Minimizing the loss in is equivalent to maximizing the regularized reward R reg i,t. Moreover, minimizing the term E τ ∼q ln implicitly minimizes the expected KL-divergence. In practise (see Appendix C.1) we will also make use of a discount factor γ ∈. For details on how γ arises in the PAI framework we refer to. Options are skills that generalize primitive actions and consist of three components: i) an intra-option policy p(a t |s t, z t) selecting primitive actions according to the currently active option z t, ii) a probability p(b t |s t, z t−1) of terminating the previously active option z t−1, and iii) an initiation set I ⊆ S, which we simply assume to be S. Note that by construction, the higherlevel (or master-) policy p(z t |z t−1, s t, b t) can only select a new option z t if the previous option z t−1 has terminated. 3 METHOD We aim to learn a reusable set of options that allow for faster training on new tasks from a given distribution. We learn both intra-option and termination policies, while preventing multiple options from learning the same behavior. To differentiate ourselves from classical'hard' options, which, once learned, do not change during new tasks, we call our novel approach soft-options (this is further discussed in Appendix B). Each soft-option consists of an option prior, which is shared across all tasks, and a taskspecific option posterior. The priors of both the intraoption policy and the termination policy capture how an option typically behaves and remain fixed once they are fully learned. At the beginning of training on a new task, they are used to initialize the task-specific posterior distribution. During training, the posterior is then regularized against the prior to prevent inadvertent unlearning. However, if maximizing the reward on certain tasks is not achievable with the prior policy, the posterior is free to deviate from it. We can thus speed up training using options, while remaining flexible enough to solve any task. Additionally, this soft option framework also allows for learning good priors in a multitask setting while avoiding local minima in which several options learn the same behavior. See Figure 1 for an overview over the hierarchical prior-posterior architecture that we explain further below. To express options in the PAI framework, we introduce two additional variables at each time step t: option selections z t, representing the currently selected option, and decisions b t to terminate them and allow the higher-level (master) policy to choose a new option. The agent's behavior depends on the currently selected option z t, by drawing actions a t from the intra-option posterior policy q L φi (a t |s t, z t). The selection z t itself is drawn from a master policy q, which conditions on b t ∈ {0, 1}, drawn by the termination posterior policy q T φi (b t |s t, z t−1). The master policy either continues with the previous z t−1 or draws a new option, where we set b 1 = 1 at the beginning of each episode. We slightly abuse notation by referring by δ(z t − z t−1) to the Kronecker delta δ zt,zt−1 for discrete and the Dirac delta distribution for continuous z t. The joint posterior policy is While z t can be a continuous variable, we consider only z t ∈ {1 . . . m}, where m is the number of available options. The induced distribution q φi (τ) over trajectories of task i, τ = (Our framework transfers knowledge between tasks by a shared prior p θ (a t, z t, b t |s t, z t−1) over all joint policies: H, and p L θ correctly, we can learn useful temporally extended options. The parameterized priors p T θ (b t |s t, z t−1) and p L θ (a t |s t, z t) are structurally equivalent to the posterior policies q T φi and q L φi so that they can be used as initialization for the latter. Optimizing the regularized return (see next section) w.r.t. θ distills the common behavior into the prior policy and softly enforces similarity across posterior distributions of each option amongst all tasks i. m selects the previous option z t−1 if b t = 0, and otherwise draws options uniformly to ensure exploration. Because the posterior master policy is different on each task, there is no need to distill common behavior into a joint prior. We extend the multitask objective in by substituting p θ (τ, O 1:T) and p φi (τ) with those induced by our hierarchical posterior policy in and the corresponding prior. The ing objective has the same form but with a new regularized reward that is maximized: As we maximize E q [R reg i,t], this corresponds to maximizing the expectation over, along the on-policy trajectories drawn from q φi (τ). Term 1 of the regularization encourages exploration in the space of options. It can also be seen as a form of deliberation cost as it is only nonzero whenever we terminate an option and the master policy needs to select another to execute. Term 2 softly enforces similarity between option posteriors across tasks and updates the prior towards the'average' posterior. It also encourages the master policy to pick the most specialized option whose posteriors across all tasks are most similar. In other words, the master policy q H φi is encouraged to pick option z t which maximizes r i, but minimizes term 2 by picking the option z t for which prior p L θ and posterior q L φi are the most similar. Because the prior is the average posterior, this rewards the master policy to pick the most specialized option (that still achieves high reward). As discussed in more detail in Appendix A, this allows us to escape the local optimization minima that hard options face in multitask learning, while still having fully specialized options after training. Lastly, we can use 3 to also encourage temporal abstraction of options. To do so, during option learning, we fix the termination prior p Choosing a large α encourages prolonged execution of one option, but allows switching whenever necessary. This is also similar to deliberation costs but with a more flexible cost model. Additionally, we can still distill a termination prior p T θ which can be used on future tasks. Instead of learning p T θ by minimizing the KL against the posterior termination policies, we can get more decisive terminations by minimizing andq φi (b = 1|s t, z t−1) = zt =zt−1 q H φi (z t |s t, z t−1, b t = 1) i.e., the learned termination prior distills the probability that the tasks' master policies would change the active option if they had the opportunity. Details on how we optimized the MSOL objective are given in Appendix C. Most hierarchical approaches rely on proxy rewards to train the lower level components and their terminations. Some of them aim to reach pre-specified subgoals , which are often found by analyzing the structure of the MDP (; ; Ş imşek et al., 2005; Ş imşek &), previously learned policies or predictability . Those methods typically require knowledge, or a sufficient approximation, of the transition model, both of which are often infeasible. Recently, several authors have proposed unsupervised training objectives for learning diverse skills based on their distinctiveness (; ; ;). However, those approaches don't learn termination functions and cannot guarantee that the required behavior on the downstream task is included in the set of learned skills. also incorporate reward information, but do not learn termination policies and are therefore restricted to learning multiple solutions to the provided task instead of learning a decomposition of the task solutions which can be re-composed to solve new tasks. A third usage of proxy rewards is by training lower level policies to move towards goals defined by the higher levels. When those goals are set in the original state space (a), this approach has difficulty scaling to high dimensional state spaces like images. Setting the goals in a learned embedding space (; ; b) can be difficult to train, though. In both cases, the temporal extension of the learned skills are set manually. On the other hand, also learn a hierarchical agent, but not to transfer skills, but to find decisions states based on how much information is encoded in the latent layer. also take an inference motivated approach to learning options. In particular propose a similarly structured hierarchical policy, albeit in a single task setting. However, they do not utilize learned prior and posterior distributions, but instead use expectation maximization to iteratively infer a hierarchical policy to explain the current rewardweighted trajectory distribution. Several previous works try to overcome the restrictive nature of options that can lead to sub-optimal solutions by allowing the higher-level actions to modulate the behavior of the lower-level policies;;. However, this significantly increases the required complexity of the higher-level policy and therefore the learning time. The multitask-and transfer-learning setup used in this work is inspired by and who suggest extracting options by using commonalities between solutions to multiple tasks. Prior multitask approaches often rely on additional human supervision like policy sketches or desirable sub-goals (; ;) in order to learn skills which transfer well between tasks. In contrast, our work aims at finding good termination states without such supervision. investigate the use of different priors for the higher-level policy while we are focussing on learning transferrable option priors. Closest to our work is Meta Learning of Shared Hierarchies (MLSH) which, however, shares the lower-level policies across all tasks without distinguishing between prior and posterior and does not learn termination policies. As discussed, this leads to local minima and insufficient diversity in the learned options. Similarly to us, differentiate between prior and posterior policies on multiple tasks and utilize a KLdivergence between them for training. However, they do not consider termination probabilities and instead only choose one option per task. Instead of transferring option policies between tasks, aim to share behavior through a latent embedding. Another interesting approach to multitask learning is which learns decision regions that are linear in the state instead of learning nonlinear master-and termination policies. Our approach is closely related to DISTRAL with which we share the multitask learning of prior and posterior policies. However, DISTRAL has no hierarchical structure and applies the same prior distribution over primitive actions, independent of the task. As a necessary hierarchical heuristic, the authors propose to also condition on the last primitive action taken. This works well when the last action is indicative of future behavior; however, in Section 5 we show several failure cases where a learned hierarchy is needed. We conduct a series of experiments to show: i) when learning hierarchies in a multitask setting, MSOL successfully overcomes the local minimum of insufficient option diversity, as described in Appendix A; ii) MSOL can learn useful termination policies; iii) MSOL is equally applicable to discrete as well as continuous domains; and iv) using soft options yields fast transfer learning while still reaching optimal performance, even on new, out-of-distribution tasks. All architectural details and hyper-parameters can be found in the appendix. For all experiments, we first train the exploration priors and options on n tasks from the available task distribution T (training phase is plotted in Appendix E). Subsequently, we test how quickly we can learn new tasks from T (or another distribution T). We compare the following algorithms: MSOL is our proposed method that utilizes soft options both during option learning and transfer. MSOL(frozen) uses the soft options framework during learning to find more diverse skills, but does not allow fine-tuning the posterior sub-policies after transfer. DISTRAL is a strong non-hierarchical transfer learning algorithm that also utilizes prior and posterior distributions. DISTRAL(+action) utilizes the last action as option-heuristic which works well in some tasks but fails when the last action is not sufficiently informative. MLSH is a multitask option learning algorithm like MSOL, but utilizes'hard' options for both learning and transfer, i.e., sub-policies that are shared exactly across tasks. It also relies on fixed option durations and requires a complex training schedule between master and intra-option policies to stabilize training. We use the author's MLSH implementation. Lastly, we compare against Option Critic (OC) , which takes the task-id as additional input in order to apply it to a multitask setting. We start with the 2D Moving Bandits environment proposed and implemented by , which is similar to the example in Appendix A. In each episode, the agent receives a reward of 1 for each time step it is sufficiently close to one of two randomly sampled, distinguishable, marked positions in the environment. The agent can take actions that move it in one of the four cardinal directions. Which position is not signaled in the observation. Each episode lasts 50 time steps. We compare against MLSH and DISTRAL to highlight challenges that arise in multitask training. We allow MLSH and MSOL to learn two options. During transfer, optimal performance can only be achieved when both options successfully learned to reach different marked locations, i.e., when they are diverse. In Figure 2 (a) we can see that MSOL is able to do so but the options learned by MLSH are not sufficiently diverse, for the reason explain in Appendix A. DISTRAL, even with the last action provided as additional input, is not able to quickly utilize the prior knowledge. The last action only conveys meaningful information when taking the goal locations into account: DISTRAL agents need to infer the intention based on the last action and the relative goal positions. While this is possible, in practice the agent was not able to do so, even with a much larger network. However, longer training allows DISTRAL to perform as well as MSOL, since its posterior is flexible, denoted by "DISTRAL(+action) limit". Lastly, MSOL(frozen) also outperforms DISTRAL(+action) and MLSH, but performs worse that MSOL. This highlights the utility of making options soft, i.e. adaptable. Next, we use a slightly modified version of the original Taxi domain to show learning of termination functions as well as transfer-and generalization capabilities. To solve the task, the agent must pick up a passenger on one of four possible locations by moving to their location and executing a special'pickup/drop-off' action. Then, the passenger must be dropped off at one of the other three locations, again using the same action executed at the corresponding location. The domain has a discrete state space with 30 locations arranged on a grid and a flag indicating whether the passenger was already picked up. The observation is a one-hot encoding of the discrete state, excluding passenger-and goal location. This introduces an information-asymmetry between the task-specific master policy, and the shared options, allowing them to generalize well . Walls (see Figure 3) limit the movement of the agent and invalid actions. We investigate two versions of Taxi. In the original (, just called Taxi), the action space consists of one no-op, one'pickup/drop-off' action and four actions to move in all cardinal directions. In Directional Taxi, we extend this setup: the agent faces in one of the cardinal directions and the available movements are to move forward or rotate either clockwise or counter-clockwise. In both environments the set of tasks T are the 12 different combinations of pickup/drop-off locations. Episodes last at most 50 steps and there is a reward of 2 for delivering the passenger to its goal and a penalty of -0.1 for each time step. During training, the agent is initialized to any valid state. During testing, the agent is always initialized without the passenger on board. We allow four learnable options in MLSH and MSOL. This necessitates the options to be diverse, i.e., one option to reach each of the four pickup/drop-off locations. Importantly, it also requires the options to learn to terminate when a passenger is picked up. As one can see in Figure 2 (b), MLSH struggles both with option-diversity and due to its fixed option duration which is not flexible enough for this environment. DISTRAL(+action) performs well in the original Taxi environment, as seen in Figure 2 (b), since here the last action is a good indicator for the agent's intention. However, in the directional case shown in Figure 2 (c), the actions are less informative and make it much harder for DISTRAL to use prior knowledge. By contrast, MSOL performs well in both taxi environments. Comparing its performance with MSOL(frozen) shows the utility of adaptable soft options during transfer. Figure 3, which visualizes the options learned by MSOL, shows that it successfully learns useful movement primitives and termination functions. The same soft option represents different behavior depending on whether it already picked up the passenger. This is expected as this behavior does not need to terminate the current option on three of the 12 tasks. Next we show how learning information-asymmetric soft options can help with transfer to unseen tasks. In Figure 2 (d) we show learning on four tasks from T using options that were trained on the remaining eight, comparing against A2C and OC. Note that in OC, there is no informationasymmetry: We share the same networks across all tasks and provide the task-id as additional input, including to the option-policies. This prevents them from generalizing well to unseen tasks. On the other hand, withholding the task-information from them would be similar to MLSH, which we already showed to struggle with local minima. The strong performance of MSOL on this task shows that we need soft options to be able to train information-asymmetric options that generalize well. We also investigate the utility of flexible soft options: In Figure 2 (e) we show learning performance on twelve changed tasks in which the pickup/dropoff locations where moved by one cell while the options were trained with the original locations. As expected, hard options are not able to solve this tasks. Even with additional access to primitive actions, exploration is inefficient . On the other hand, MSOL is able to quickly learn this new task by adapting the previously learned options, outperforming hard options and flat policies learned from scratch. Lastly, we show that MSOL can also be applied to continuous multitask domains. In particular, we investigate the MuJoCo environment'Swimmer' . Instead of rewarding forward movement as in the original implementation, now the rewarded movement direction depends on the task from T = {up, down, lef t, right}. We also include a small amount of additive action noise (details in the Appendix). We show that MSOL performs competitive even in the absence of known failure cases of DISTRAL (see Figure 2 (f)). Multitask Soft Option Learning (MSOL) proposes reformulating options using the perspective of prior and posterior distributions. This offers several key advantages. First, during transfer, it allows us to distinguish between fixed, and therefore knowledge-preserving option priors, and flexible option posteriors that can adjust to the reward structure of the task at hand. This effects a similar speed-up in learning as the original options framework, while avoiding sub-optimal performance when the available options are not perfectly aligned to the task. Second, utilizing this'soft' version of options in a multitask learning setup increases optimization stability and removes the need for complex training schedules between master and lower level policies. Furthermore, this framework naturally allows master policies to coordinate across tasks and avoid local minima of insufficient option diversity. It also allows for autonomously learning option-termination policies, a very challenging task which is often avoided by fixing option durations manually. Lastly, using this formulation also allows inclusion of prior information in a principled manner without imposing too rigid a structure on the ing hierarchy. We utilize this advantage to explicitly incorporate the bias that good options should be temporally extended. In future research, other types of information can be explored. As an example, one could investigate sets of tasks which would benefit from a learned master prior, like walking on different types of terrain. Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in neural information processing systems, pp. 3675-3683, 2016. where all posteriors of z 1 are moving to A, but for z 2, the posteriors on different tasks a and b move to different targets. Crucially, to maximize the regularized reward, the KL-divergences between priors and posteriors should be minimized along they trajectories, i.e. weighted by how likely they are to occur. Consequently, the higher D KL [z 1 is always used to reach A and z 2 (b) is always used to reach B, allowing bothz 1 andz 2 to also lead to A and B respectively. Assume we are faced with a new task and are given some prior knowledge in the form of a set of skills that we can use. Using those skills and their termination probabilities as prior policies p T and p L in the soft option framework, we can see β as a temperature parameter determining how closely we are restricted to following them. For β → ∞ we recover the classical'hard' option case and our posterior option policies are restricted to the prior. 1 For β = 0 the priors only initialize the otherwise unconstrained policy, quickly unlearning behavior that may be useful down the line. Lastly, for 0 < β < ∞ we use the prior information to guide exploration but are only softly restricted to the given skills and can also explore and use policies'close' to them. Even though R reg i depends on φ i, its gradient w.r.t. φ i vanishes. 2 Consequently, we can treat the regularized reward as a classical RL reward and use any RL algorithm to find the optimal hierarchical policy parameters φ i. In the following, we explain how to adapt A2C to soft options. The extension to PPO is straightforward. The joint posterior policy in depends on the current state s t and the previously selected option z t−1. The expected sum of regularized future rewards of task i, the value function V i, must therefore also condition on this pair: As V i (s t, z t−1) cannot be directly observed, we approximate it with a parametrized model V φi (s t, z t−1). The k-step advantage estimation at time t of trajectory τ is given by where the superscript'−' indicates treating the term as a constant. The approximate value function V φi can be optimized towards its bootstrapped k-step target by minimizing L V (φ i, τ 1: As per A2C, k ∈ [1 . . . n s] depending on the state . The corresponding policy gradient loss is The gradient w.r.t. the prior parameters θ is whereb t = δ zt−1 (z t) and z t ∼ q H (z t |s t, z t−1, b t = 1). To encourage exploration in all policies of the hierarchy, we also include an entropy maximization loss: Note that term 1 in already encourages maximizing L H (φ i, τ) for the master policy, since we chose a uniform prior p H (z t |b t = 1). As both terms serve the same purpose, we are free to drop either one of them. In our experiments, we chose to drop the term for q H in R reg t, which proved slightly more stable to optimize that the alternative. We can optimize all parameters jointly with a combined loss over all tasks i, based on sampled trajectories τ i:= τ i 1:T ∼ q φi and corresponding sampled values ofb i:=b C.2 TRAINING SCHEDULE For faster training, it is important to prevent the master policies q H from converging too quickly to allow sufficient updating of all options. On the other hand, a lower exploration rate leads to more clearly defined options. We consequently anneal the exploration bonus λ H with a linear schedule during training. Similarly, a high value of β leads to better options but can prevent finding the extrinsic reward r i (s t, a t) early on in training. Consequently, we increase β over the course of training, also using a linear schedule. All policies and value functions share the same encoder network with two fully connected hidden layers of size 64 for the Moving Bandits environment and three hidden layers of sizes 512, 256, and 512 for the Taxi environments. Distral was tested with both model sizes on the Moving Bandits task to make sure that limited capacity is not the problem. Both models ed in similar performance, the shown in the paper are for the larger model. On swimmer the encoder model size is 1024× 256 × 512. Master-policies, as well as all prior-and posterior policies and value functions consist of only one layer which takes the latent embedding produced by the encoder as input. Furthermore, the encoder is shared across tasks, allowing for much faster training since observations can be batched together. Options are specified as an additional one-hot encoded input to the corresponding network that is passed through a single 128 dimensional fully connected layer and concatenated to the state embedding before the last hidden layer. We implement the single-column architecture of Distral as a hierarchical policy with just one option and with a modified loss function that does not include terms for the master and termination policies. Our implementation builds on the A2C/PPO implementation by , and we use the implementation for MLSH that is provided by the authors (https://github.com/openai/mlsh). We use 2λ V = λ A = λ P = 1 in all experiments. Furthermore, we train on all tasks from the task distribution, regularly resetting individual tasks by resetting the corresponding master and re-initializing the posterior policies. Optimizing β for MSOL and Distral was done over {0.01, 0.02, 0.04, 0.1, 0.2, 0.4}. We use γ = 0.95 for Moving Bandits and Taxi and γ = 0.995 for Swimmer. For MLSH, we use the original hyper-parameters . The duration of each option is fixed to 10. The required warm-up duration is set to 9 and the training duration set to 1. We also use 30 parallel environments split between 10 tasks. This and the training duration are the main differences to the original paper. Originally, MLSH was trained on 120 parallel environments which we were unable to do due to hardware constraints. Training is done over 6 million frames per task. For MSOL and Distral we use the same number of 10 tasks and 30 processes. The duration of options are learned and we do not require a warm-up period. We set the learning rate to 0.01 and β = 0.2, α = 0.95, λ H = 0.05. Training is done over 0.6 million frames per task. For Distral we use β = 0.04, λ H = 0.05 and also 0.6 million frames per task. For MSOL we anneal β from 0.02 to 0.1 and λ H from 0.1 to 0.05. For Distral we use β = 0.04. We use 3 processes per task to collect experience for a batch size of 15 per task. Training is done over 1.4 million frames per task for Taxi and 4 million frames per task for Directional Taxi. MLSH was trained on 0.6 million frames for Taxi as due to it's long runtime of several days, using more frames was infeasible. Training was already converged. As shown in the main experiments, soft options can still be useful, even when the task distribution changes. It is unsurprising that hard options are not able to solve the task. However, interestingly, providing hard options and primitive actions to the master policy performs much worse than just learning from scratch. This phenomenon was investigated previously by and further justifies using soft options for transfer to out-of-distribution tasks. Whether training from scratch or re-using misspecified options that were trained on a different set of tasks is learning faster mainly depends on i) how strongly the options are misspecified and ii) how difficult the exploration problem is in the environment. This tradeoff is shown in Figure 6: On a smaller 8x8 grid (left), learning from scratch performs competitively because exploration is sufficiently simple. On the other hand, on a 10x10 grid (right, from the main paper), exploration is harder and soft options allow for significantly faster learning because they can guide the exploration in a helpful way. For training Distral and MSOL we use PPO instead of A2C as it generally achieves better performance on continuous tasks. We have λ H = 0.0004 for both MSOL and Distral for primitive actions and λ H = 0.1 for the master-and termination policies in MSOL. We use a learning rate of 0.0002, GAE with τ = 0.98. We collect 2000 steps in parallel on 6 processes per task, ing in a batchsize of 12, 000 per task. Training is done over 6 million frames with a linearly scheduled increase of β from 0 to 0.04 for MSOL and 0.01 for Distral. We set α = 0.98. MSOL and MSOL(frozen) share the same training as they only differ during testing. Further, note that the highest achievable performance for Taxi and Directional Taxi is higher during training as they can be initialized closer to the final goal (i.e. with the passenger on board). learning from scratch performs competitively. For larger grid sizes, soft options can accelerate training through faster exploration, even if they are misspecified because they were trained on a different set of tasks.

In Hierarchical RL, we introduce the notion of a 'soft', i.e. adaptable, option and show that this helps learning in multitask settings.