Question: represent the scientific article to find a short tldr

Question: Grasping an object and precisely stacking it on another is a difficult task for traditional robotic control or hand-engineered approaches. Here we examine the problem in simulation and provide techniques aimed at solving it via deep reinforcement learning. We introduce two straightforward extensions to the Deep Deterministic Policy Gradient algorithm (DDPG), which make it significantly more data-efficient and scalable. Our show that by making extensive use of off-policy data and replay, it is possible to find high-performance control policies. Further, our hint that it may soon be feasible to train successful stacking policies by collecting interactions on real robots. Dexterous manipulation is a fundamental challenge in robotics. Researchers have long sought a way to enable robots to robustly and flexibly interact with fixed and free objects of different shapes, materials, and surface properties in the context of a broad range of tasks and environmental conditions. Such flexibility is very difficult to achieve with manually designed controllers. The recent resurgence of neural networks and "deep learning" has inspired hope that these methods will be as effective in the control domain as they are for perception. Indeed, recent work has used neural networks to learn solutions to a variety of control problems BID31 BID6 BID30 BID10 BID16.While the flexibility and generality of learning approaches is promising for robotics, these methods typically require a large amount of data that grows with the complexity of the task. What is feasible on a simulated system, where hundreds of millions of control steps are possible BID22 BID31, does not necessarily transfer to real robot applications due to unrealistic learning times. One solution to this problem is to restrict the generality of the controller by incorporating task specific knowledge, e.g. in the form of dynamic movement primitives BID29, or in the form of strong teaching signals, e.g. kinesthetic teaching of trajectories BID23. Recent works have had success learning flexible neural network policies directly on real robots (e.g. BID4 BID38), but tasks as complex as precise grasping-and-stacking remain daunting. In this paper we investigate in simulation the possibility of learning precise manipulation skills endto-end with a general purpose model-free deep reinforcement learning algorithm. We assess the feasibility of performing analogous experiments on real robotics hardware and provide guidance with respect to the choice of learning algorithm, experimental setup, and the performance that we can hope to achieve. We consider the task of picking up a Lego brick from the table and stacking it onto a second nearby brick using a robotic arm and gripper. This task involves contact-rich interactions between the robotic arm and two freely moving objects. It also requires mastering several sub-skills (reaching, grasping, lifting, and stacking). Each of these sub-skills is challenging in its own right as they require both precision (for instance, successful stacking requires accurate alignment of the two bricks) and as well as robust generalization over a large state space (e.g. different initial positions of the bricks and the initial configuration of the arm). Finally, there exist non-trivial and long-ranging dependencies between the solutions for different sub-tasks: for instance, the ability to successfully stack the brick depends critically on having picked up the brick in a sensible way beforehand. This paper makes several contributions: 1. We build on the Deep Deterministic Policy Gradient (DDPG;), a general purpose model-free reinforcement learning algorithm for continuous actions, and extend it in two ways: firstly, we improve the data efficiency of the algorithm by scheduling updates of the network parameters independently of interactions with the environment. Secondly, we overcome the computational and experimental bottlenecks of single-machine single-robot learning by introducing a distributed version of DDPG which allows data collection and network training to be spread out over multiple computers and robots. 2. We show how to use these straightforward algorithmic developments to solve a complex, multi-stage manipulation problem. We further propose two broadly applicable strategies that allow us to reliably find solutions to complex tasks and further reduce the amount of environmental interaction. The first of these strategies is a recipe for designing effective shaping rewards for compositional tasks, while the second biases the distribution of initial states to achieve an effect akin a form of apprenticeship learning. In combination these contributions allow us to reliably learn robust policies for the full stacking task from scratch in less than 10 million environment transitions. This corresponds to less than 10 hours of interaction time on 16 robots. In addition, we show that when states from demonstration trajectories are used as the start states for learning trials the full task can be learned with 1 million transitions (i.e. less than 1 hour of interaction on 16 robots). To our knowledge our provide the first demonstration of end-to-end learning for a complex manipulation problem involving multiple freely moving objects. They are also suggest that it may be possible to learn such non-trivial manipulation skills directly on real robots. Reinforcement learning (RL) approaches solve tasks through repeated interactions with the environment guided by a reward signal of success or failure BID33. A distinction is often made between value-based and policy search methods. The latter have been routinely applied in robotics, in part because they straightforwardly handle continuous and high-dimensional action spaces BID2, and applications include manipulation BID25 BID36 BID4 BID38 BID7, locomotion e.g. BID15 BID20, and a range of other challenges such as helicopter flight BID0 ). However, policy search methods can scale poorly with the number of parameters that need to be estimated, requiring the need for restricted policy classes, that in turn might not be powerful enough for solving complex tasks. One exception are guided policy search methods (GPS) BID38. These employ a teacher algorithm to locally optimize trajectories which are then summarized by a neural network policy. They gain data-efficiency by employing aggressive local policy updates and extensive training of their neural network policy. The teacher can use model-based or model-free BID38 trajectory optimization. The former can struggle with strong discontinuities in the dynamics, and both rely on access to a well defined and fully observed state space. Alternatively, model-free value function approaches enable effective reuse of data and do not require full access to the state space or to a model of the environment. The use of rich function approximators such as neural networks in value function methods dates back many years, e.g. BID37 BID34 BID11 BID9, and recent success with deep learning has driven the development of new end-to-end training methods for challenging control problems BID21 BID5 c;. Closely related to the ideas followed in this paper, BID4 demonstrates that value-based methods using neural network approximators can be used for relatively simple robotic manipulation tasks in the real world BID6 ). This work also followed a recent trend towards the use of experimental rigs that allow parallelized data collection, e.g. BID26, via the use of multiple robots from which experience is gathered simultaneously BID4 BID38.Finally, the use of demonstration data has played an important role in robot learning, both as a means to obtain suitable cost functions BID1 BID13 BID3 BID7 but also to bootstrap and thus speed up learning. For the latter, kinesthetic teaching is widely used BID25 BID38, though the need for a human operator to be able to guide the robot through the full movement can be limiting. In this section we explain the learning problem and summarize the DDPG algorithm. We explain its relationship to other Q-function based RL algorithms in the Appendix. The RL problem consists of an agent interacting with an environment in a sequential manner to maximize the expected sum of rewards. At time t the agent observes the state x t of the system and produces a control u t = π(x t ; θ) according to policy π with parameters θ. This leads the environment to transition to a new state x t+1 according to the dynamics x t+1 ∼ p(·|x t, u t), and the agent receives a reward r t = r(x t, u t). The goal is to maximize the expected sum of discounted rewards J(θ) = E τ ∼ρ θ t γ t−1 r(x t, u t), where ρ θ is the distribution over trajectories τ = (x 0, u 0, x 1, u 1, . . .) induced by the current policy: DISPLAYFORM0 DPG BID32 ) is a policy gradient algorithm for continuous action spaces that improves the deterministic policy function π via backpropagation of the action-value gradient from a learned approximation to the Q-function. Specifically, DPG maintains a parametric approximation Q(x t, u t ; φ) to the action value function Q π (x t, u t) associated with π and φ is chosen to minimize DISPLAYFORM1 where y t = r(x t, u t) + γQ(x t+1, π(x t+1)).ρ is usually close to the marginal transition distribution induced by π but often not identical. For instance, during learning u t may be chosen to be a noisy version of π(x t ; θ), e.g. u t = π(x t ; θ) + where ∼ N (0, σ 2) andρ is then the transition distribution induced by this noisy policy. The policy parameters θ are then updated according to DISPLAYFORM2 DDPG incorporates experience replay and target networks to the original DPG algorithm: Experience is collected into a buffer and updates to θ and φ (eqs. 1, 2) are computed using mini-batch updates with samples from this buffer. A second set of "target-networks" is maintained with parameters θ and φ. These are used to compute y t in eqn. and their parameters are slowly updated towards the current parameters θ, φ. Both measures significantly improve the stability of DDPG.The use of a Q-function facilitates off-policy learning. This decouples the collection of experience data from the updates of the policy and value networks which allows us to make many parameter update steps per step in the environment, ensuring that the networks are well fit to the data that is currently available. The full task that we consider in this paper is to use the arm to pick up one Lego brick from the table and stack it onto the remaining brick. This "composite" task can be decomposed into several subtasks, including grasping and stacking. We consider the full task as well as the two sub-tasks in isolation:Starting state Reward Grasp Both bricks on In every episode the arm starts in a random configuration with an appropriate positioning of gripper and brick. We implement the experiments in a physically plausible simulation in MuJoCo BID35 with the simulated arm being closely matched to a real-world Jaco arm setup in our lab. Episodes are terminated after 150 steps of 50ms of physical simulation time. The agent thus has 7.5 seconds to perform the task. Unless otherwise noted we give a reward of one upon successful completion of the task and zero otherwise. The observation contains information about the angles and angular velocities of the 6 joints of the arm and 3 fingers of the gripper, as well as the position and orientation of the two bricks and relative distances of the two bricks to the pinch position of the gripper (roughly the position where the fingertips would meet if the fingers are closed). The 9-dimensional continuous action directly sets the velocities of the arm and finger joints. In experiments not reported in this paper we have tried using observations with only the raw state of the brick and the arm configuration (i.e. without the vector between the end-effector and brick) This increased the number of environment interactions needed roughly by a factor of two to three. For each experimental condition we optimize the learning rate and train and measure the performance of 10 agents with different random initial network parameters. After every 30 training episodes the agent is evaluated for 10 episodes. We used the mean performance at each evaluation phase as the performance measure presented in all plots. In the plots the line shows the mean performance across agents and the shaded regions correspond to the range between the worst and best performing one In all plots the x-axis represents the number of environment transitions seen so far at an evaluation point (in millions) and the y-axis represent episode return. A video of the full setup and examples of policies solving the component and full tasks can be found here: https://www.youtube.com/watch?v=7vmXOGwLq24. In this section we study two methods for extending the DDPG algorithm and find that they can have significant effect on data and computation efficiency, in some cases making the difference between finding a solution to a task or not. Multiple mini-batch replay steps Deep neural networks can require many steps of gradient descent to converge. In a supervised learning setting this affects purely computation time. In reinforcement learning, however, neural network training is interleaved with the acquisition of interaction experience giving rise to a complex interaction. To gain a better understanding of this effect we modified the original DDPG algorithm as described in to perform a fixed but configurable number of mini-batch updates per step in the environment. In one update was performed after each new interaction step. We refer to DDPG with a configurable number of update steps as DPG-R and tested the impact of this modification on the two primitive tasks Grasp and StackInHand. The are shown in FIG1. The number of update steps has a dramatic effect on the amount of experience data required. After one million interactions the original version of DDPG with a single update step (blue traces) appears to have made no progress towards a successful policy for stacking, and only a small number of controllers have learned to grasp. Increasing the number of updates per interaction to 5 greatly improves the (green traces), and with 40 updates (purple) the first successful policies for stacking and grasping are obtained after 200,000 and 300,000 interactions respectively (corresponding to 1,300 and 2,000 episodes). Notably, for both tasks we continue to see a reduction in total environment interaction up to 40 update steps, the maximum used in the experiment. One possible explanation for this effect is the interaction alluded to above: insufficient training may lead to a form of underfitting of the policy. Since the policy is then used for exploration this affects the quality of the data collected in the next iteration which in turn has an effect on training in future iterations leading to overall slow learning. We have observed in various experiments (not shown) that other aspects of the network architecture (layer sizes, non-linearities) can similarly affect learning speed. Finally, it is important to note that one cannot replicate the effect of multiple replay steps simply by increasing the learning rate. In practice we find that attempts to do so make training unstable. Asynchronous DPG Increasing the number of update steps relative to the number of environment interactions greatly improves the data efficiency but also dramatically increases compute time. When the overall run time is dominated by the network updates it may scale linearly with the number of replay steps. In this setting experiments can quickly become impractical and parallelizing computation can provide a solution. Similarly, in a robotics setup the overall run time is typically dominated by the collection of interactions. In this case it is desirable to be able to collect experience from multiple robots simultaneously (e.g. as in BID38 BID4).We therefore develop an asynchronous version of DPG that allows parallelization of training and environment interaction by combining multiple instances of an DPG-R actor and critic that each share their network parameters and can be configured to either share or have independent experience replay buffers. This is inspired by the A3C algorithm proposed in BID22, and also analogous to BID4 BID38: We employ asynchronous updates whereby each worker has its own copy of the parameters and uses it for computing gradients which are then applied to a shared parameter instance without any synchronization. We use the Adam optimizer BID14 with local non-shared first-order statistics and a single shared instance of second-order statistics. The pseudo code of the asynchronous DPG-R is shown in algorithm box 1. Initialize global shared critic and actor network parameters: θ Q and θ µ Pseudo code for each learner thread: Initialize critic network Q(s, a|θ Q) and policy network µ(s|θ µ) with weights θ Q and θ µ.Initialize target network Q and µ with weights: DISPLAYFORM0 Initialize replay buffer R for episode = 1, M do Receive initial observation state s1 for t = 1, T do Select action at = µ(st|θ µ) + Nt according to the current policy and exploration noise Perform action at, observe reward rt and new state st+1 Store transition (st, at, rt, st+1) in R for update = 1, R do Sample a random minibatch of N transitions (si, ai, ri, si+1) from R Set yi = ri + γQ (si+1, µ (si+1|θ µ)|θ Q ) Perform asynchronous update of the shared critic parameters by minimizing the loss: DISPLAYFORM1 2 ) Perform asynchronous update of the shared policy parameters using the sampled gradient: DISPLAYFORM2 Copy the shared parameters to the local ones: DISPLAYFORM3 Every S update steps, update the target networks: Figure 2 (right) compares the performance of ADPG-R for different number of update steps and 16 workers (all workers performing both data collection and computing updates). Similar to FIG1 (left) we find that increasing the ratio of update steps per environment steps improves data efficiency, although the effect appears to be somewhat less pronounced than for DPG-R. FIG2 (left) directly compares the single-worker and asynchronous version of DPG-R. In both cases we choose the best performing number of replay steps and learning rate. As we can see, the use of multiple workers does not affect overall data efficiency for StackInHand but it reduced roughly in half for Grasp, with the note that the single worker still hasn't quite converged. DISPLAYFORM4 for end for end forFigure 3 (right) plots the same data but as a function of environment steps per worker. This measure corresponds to the optimal wall clock efficiency that we can achieve, under the assumption that communication time between workers is negligible compared to environment interaction and gradient computation (this usually holds up to a certain degree of parallelization). The theoretical wall clock time for 16 workers is about 16x lower for StackInHand and roughly 8x lower for Grasp. Overall these show that distributing neural network training and data collection across multiple computers and robots can be an extremely effective way of reducing the overall run time of experiments and thus making it feasible to run more challenging experiments. We make extensive use of asynchronous DPG for remaining the experiments. The reward function in the previous section was "sparse" or "pure" reward where a reward of 1 was given for states that correspond to successful task completion (brick lifted above 3cm for grasp; for stack) and 0 otherwise. For this reward to be useful it is necessary that the agent enters the goal region at least some of the time. While possible for each of the two subtasks in isolation, this is highly unlikely for the full task: without further guidance naïve random exploration is very unlikely to lead to a successful grasp-and -stack as we experimentally verify in FIG3.One solution are informative shaping rewards that provide a learning signal even for simple exploration strategies, e.g. by embedding information about the value function in the reward function. This is a convenient way of embedding prior knowledge about the solution and is a widely and successfully used approach for simple problems. For complex sequential or compositional tasks such as the one we are interested in here, however, a suitable reward function is often non-obvious and may require considerable effort and experimentation. In this section we propose and analyze several reward functions for the full Stack task, and provide a general recipe that can be applied to other tasks with compositional structure. Shaping rewards are often defined using a distance from or progress towards a goal state. Analogously our composite (shaping) reward functions return an increasing reward as the agent completes components of the full task. They are either piece-wise constant or smoothly varying across different regions of the state space that correspond to completed subtasks. In the case of Stack we use the following reward components (see the Appendix): These reward components can be combined in different ways. We consider three different composite rewards in additional to the original sparse task reward: Grasp shaping: Grasp brick 1 and Stack brick 1, i.e. the agent receives a reward of 0.25 when brick 1 has been grasped and a reward of 1.0 after completion of the full task. Reach and grasp shaping: Reach brick 1, Grasp brick 1 and Stack brick 1, i.e. the agent receives a reward of 0.125 when close to brick 1, a reward of 0.25 when brick 1 has been grasped, and a reward of 1.0 after completion of the full task. Full composite shaping: the sparse reward components as before in combination with the distancebased smoothly varying components A full description of the reward functions is provided in the Appendix. The actual reward functions given above are specific to the stacking task. But the general principle, a piecewise-constant sequence of rewards that increases as components of the tasks are completed, augmented with simple smoothly varying rewards that guide towards completion of individual subtasks should be widely applicable. It is important to note that the above reward functions do not describe all aspects of the task solution: we do not tell the agent how to grasp or stack but merely to bring the arm into a position where grasping (stacking) can be discovered from exploration and the sparse reward component. This eases the burden on the designer and is less likely to change the optimal solution in unwanted ways. In the previous section we described a strategy for designing effective compositional reward functions that alleviate the burden of exploration. However, designing such rewards can still be error prone and we did indeed encounter several unexpected failure cases as shown in the supplemental video (https://www.youtube.com/watch?v=7vmXOGwLq24) and detailed in the Appendix. Furthermore, suitable rewards may rely on privileged information not easily available in a real robotics setup. In this section we describe a second, complementary strategy for embedding prior knowledge into the training process and improving exploration. Specifically we propose to let the distribution of states at which the learning agent is initialized at the beginning of an episode reflect the compositional nature of the task: e.g., instead of initializing the agent at the beginning of the full task with both bricks on the table, we can initialize the agent occasionally with the brick already in its hand and thus prepared for stacking in the same way as when learning the subtask StackInHand in section 5.More generally, we can initialize episodes with states taken from anywhere along or close to successful trajectories. Suitable states can be either manually defined (as in section 5), or they can be obtained from a human demonstrator or a previously trained agent that can partially solve the task. This can be seen as a form of apprenticeship learning in which we provide teacher information by influencing the state visitation distribution. Unlike many other forms of imitation or apprenticeship learning, however, this approach requires neither complete trajectories nor demonstrator actions.. On all plots, x-axis is millions of transitions of total experience and y-axis is mean episode return. Policies with mean return over 100 robustly perform the full Stack from different starting states. Without reward shaping and basic start states only (a, blue) there is no learning progress. Instructive start states allow learning even with very uninformative sparse rewards indicating only overall task success (a,red).We perform experiments with two methods for generating the starting states. The first one uses the manually defined initial states from section 5 (both bricks located on the table or in states where the first brick is already in the gripper as if the agent just performed a successful grasp). The second method initializes the learning agent at start states sampled randomly from successful demonstration trajectories (derived from agents previously trained end-to-end on the compositional reward).The of these experiments are shown in FIG3. Green traces show for the four reward functions from section 6 in combination with the manually defined start states (from section 5). While there is still no learning for the sparse reward case, obtained with all other reward functions are improved. In particular, even for the second simplest reward function (Grasp shaping) we obtain some controllers that can solve the full task. Learning with the full composite shaping reward is faster and more robust than without the use of instructive states. The leftmost plot of FIG3 (red trace) shows for the case where the episode is initialized anywhere along trajectories from a pre-trained controller (which was obtained using full composite shaping; rightmost blue curve). We use this start state distribution in combination with the basic sparse reward for the overall case (Stack without shaping). Episodes were configured to be 50 steps, which we found to be better suited to this setup with assisted exploration. During testing we still used episodes with 150 steps as before (so that the traces are comparable). We can see a large improvement in performance in comparison to the two-state method variant even in the absence of any shaping rewards. We can learn a robust policy for all seeds within a total of 1 million environment transitions -less than 1 hour of interaction time on 16 simulated robots. These suggest that an appropriate start state distribution not only speeds up learning, it also allows simpler reward functions to be used. In our final experiment we found that the simplest reward function (i.e. only indicating overall experimental success) was sufficient to solve the task. In this case the robustness of trained policies to starting state variation is also encouraging. Over 1000 test trials we obtain 99.2% success for Grasp, 98.2% for StackInHand, and 95.5% for the full Stack task. We have introduced two extensions to the DDPG algorithm which make it a practical method for learning robust policies for complex continuous control tasks. We have shown that by decoupling the frequency of network updates from the environment interaction we can dramatically improve data-efficiency. Parallelizing data acquisition and learning substantially reduces wall clock time. In addition, we presented two methods that help to guide the learning process towards good solutions and thus reduce the pressure on exploration strategies and speed up learning. In combination these contributions allow us to solve a challenging manipulation problem end-to-end, suggesting that many hard control problems lie within the reach of modern learning methods. It is of course challenging to judge the transfer of in simulation to the real world. We have taken care to design a physically realistic simulation, and in initial experiments, which we have performed both in simulation and on the physical robot, we generally find a good correspondence of performance and learning speed between simulation and real world. This makes us optimistic that performance numbers may also hold when going to the real world. A second limitation of our simulated setup is that it currently uses information about the state of the environment would require additional instrumentation of the experimental setup, e.g. to determine the position of the two bricks in the work space. These are issues that need to be addressed with care as experiments move to robotics hardware in the lab. Nevertheless, the algorithms and techniques presented here offer important guidance for the application of deep reinforcement learning methods to dexterous manipulation on a real robot. 9 DDPG AND OTHER ALGORITHMS DDPG bears a relation to several other recent model free RL algorithms: The NAF algorithm BID6 which has recently been applied to a real-world robotics problem BID4 can be viewed as a DDPG variant where the Q-function is quadratic in the action so that the optimal action can be easily recovered directly from the Q-function, making a separate representation of the policy unnecessary. DDPG and especially NAF are the continuous action counterparts of DQN BID21, a Q-learning algorithm that recently re-popularized the use of experience replay and target networks to stabilize learning with powerful function approximators such as neural networks. DDPG, NAF, and DQN all interleave mini-batch updates of the Q-function (and the policy for DDPG) with data collection via interaction with the environment. These mini-batch based updates set DDPG and DQN apart from the otherwise closely related NFQ and NFQCA algorithms for discrete and continuous actions respectively. NFQ BID28 and NFQCA BID8 employ the same basic update as DDPG and DQN, however, they are batch algorithms that perform updates less frequently and fully re-fit the Q-function and the policy network after every episode with several hundred iterations of gradient descent with Rprop BID27 and using full-batch updates with the entire replay buffer. The aggressive training makes NFQCA data efficient, but the full batch updates can become impractical with large networks, large observation spaces, or when the number of training episodes is large. Finally, DPG can be seen as the deterministic limit of a particular instance of the stochastic value gradients (SVG) family BID10, which also computes policy gradient via back-propagation of value gradients, but optimizes stochastic policies. Target networks DQN DDPG, NAF Full-batch learning with Rprop Parameter resetting NFQ NFQCA In this section we provide further details regarding the composite reward functions described in the main text. For our experiments we derived these from the state vector of the simulation, but they could also be obtained through instrumentation in hardware. The reward functions are defined in terms of the following quantities:• b z: height of brick 1 above table • s B1 {x,y,z}: x,y,z positions of site located roughly in the center of brick 1 • s B2 {x,y,z}: x,y,z positions of site located just above brick 2, at the position where s B1 will be located when brick 1 is stacked on top of brick 2.• s P {x,y,z}: x,y,z positions of the pinch site of the hand -roughly the position where the fingertips would meet if the fingers are closed.. Using the above we can define the following conditions for the successful completion of subtasks:Reach Brick 1 The pinch site of the fingers is within a virtual box around the first brick position. DISPLAYFORM0 where ∆ reach {x,y,z} denote the half-lengths of the sides of the virtual box for reaching. Grasp Brick 1 Brick 1 is located above the table surface by a threshold, θ, that is possible only if the arm is the brick has been lifted. grasp =b z > θ Stack Brick 1 is stacked on brick 2. This is expressed as a box constraint on the displacement between brick 1 and brick 2 measured in the coordinate system of brick 2. stack =(|C DISPLAYFORM1 where ∆ stack {x,y,z} denote the half-lengths of the sides of the virtual box for stacking, and C is the rotation matrix that projects a vector into the coordinate system of brick 2. This projection into the coordinate system of brick 2 is necessary since brick 2 is allowed to move freely. It ensures that the box constraint is considered relative to the pose of brick 2. While this criterion for a successful stack is quite complicated to express in terms of sites, it could be easily implemented in hardware e.g. via a contact sensor attached to brick 2. The full composite reward also includes two distance based shaping components that guide the hand to the brick 1 and then brick 1 to brick 2. These could be approximate and would be relatively simple to implement with a hardware visual system that can only roughly identify the centroid of an object. The shaping components of the reward are given as follows:Reaching to brick 1: DISPLAYFORM0 Reaching to brick 2 for stacking r S2 (s B1, s B2) = 1 − tanh 2 (w 2 s B1 − s 2). Using the above components the reward functions we implement the composite reward functions described in the main text: Stack, Grasp shaping, Reach and grasp shaping, and Full composite shaping can be expressed as in equations below. These make use of the predicates above to determine whether which subtasks have been completed and return a reward accordingly.

Represent the short summary for finding the corresponding scientific article (long)

Data-efficient deep reinforcement learning can be used to learning precise stacking policies.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

The applicability of inverse reinforcement learning is often hampered by the expense of collecting expert demonstrations; this paper seeks to broaden its applicability by incorporating prior task information through meta-learning.


Question:
represent the article for finding a single-sentence tldr
Examples:
Provided: Deep reinforcement learning has demonstrated increasing capabilities for continuous control problems, including agents that can move with skill and agility through their environment. An open problem in this setting is that of developing good strategies for integrating or merging policies for multiple skills, where each individual skill is a specialist in a specific skill and its associated state distribution. We extend policy distillation methods to the continuous action setting and leverage this technique to combine \expert policies, as evaluated in the domain of simulated bipedal locomotion across different classes of terrain. We also introduce an input injection method for augmenting an existing policy network to exploit new input features. Lastly, our method uses transfer learning to assist in the efficient acquisition of new skills. The combination of these methods allows a policy to be incrementally augmented with new skills. We compare our progressive learning and integration via distillation (PLAID) method against three alternative baselines. As they gain experience, humans develop rich repertoires of motion skills that are useful in different contexts and environments. Recent advances in reinforcement learning provide an opportunity to understand how motion repertoires can best be learned, recalled, and augmented. Inspired by studies on the development and recall of movement patterns useful for different locomotion contexts BID17, we develop and evaluate an approach for learning multi-skilled movement repertoires. In what follows, we refer to the proposed method as PLAID: Progressive Learning and Integration via Distillation. For long lived applications of complex control tasks a learning system may need to acquire and integrate additional skills. Accordingly, our problem is defined by the sequential acquisition and integration of new skills. Given an existing controller that is capable of one-or-more skills, we wish to: (a) efficiently learn a new skill or movement pattern in a way that is informed by the existing control policy, and (b) to reintegrate that into a single controller that is capable of the full motion repertoire. This process can then be repeated as necessary. We view PLAID as a continual learning method, in that we consider a context where all tasks are not known in advance and we wish to learn any new task in an efficient manner. However, it is also proves surprisingly effective as a multitask solution, given the three specific benchmarks that we compare against. In the process of acquiring a new skill, we also allow for a control policy to be augmented with additional inputs, without adversely impacting its performance. This is a process we refer to as input injection. Understanding the time course of sensorimotor learning in human motor control is an open research problem BID31 ) that exists concurrently with recent advances in deep reinforcement learning. Issues of generalization, context-dependent recall, transfer or "savings" in fast learning, forgetting, and scalability are all in play for both human motor control models and the learning curricula proposed in reinforcement learning. While the development of hierarchical models for skills offers one particular solution that supports scalability and that avoids problems related to forgetting, we eschew this approach in this work and instead investigate a progressive approach to integration into a control policy defined by a single deep network. Distillation refers to the problem of combining the policies of one or more experts in order to create one single controller that can perform the tasks of a set of experts. It can be cast as a supervised regression problem where the objective is to learn a model that matches the output distributions of all expert policies BID13 BID28 BID19. However, given a new task for which an expert is not given, it is less clear how to learn the new task while successfully integrating this new skill in the pre-existing repertoire of the control policy for an agent. One wellknown technique in machine learning to significantly improve sample efficiency across similar tasks is to use Transfer Learning (TL) BID12, which seeks to reuse knowledge learned from solving a previous task to efficiently learn a new task. However, transferring knowledge from previous tasks to new tasks may not be straightforward; there can be negative transfer wherein a previously-trained model can take longer to learn a new task via fine-tuning than would a randomlyinitialized model BID16. Additionally, while learning a new skill, the control policy should not forget how to perform old skills. The core contribution of this paper is a method Progressive Learning and Integration via Distillation (PLAiD) to repeatedly expand and integrate a motion control repertoire. The main building blocks consist of policy transfer and multi-task policy distillation, and the method is evaluated in the context of a continuous motor control problem, that of robust locomotion over distinct classes of terrain. We evaluate the method against three alternative baselines. We also introduce input injection, a convenient mechanism for adding inputs to control policies in support of new skills, while preserving existing capabilities. Transfer learning and distillation are of broad interest in machine learning and RL BID12 BID27 BID28. Here we outline some of the most relevant work in the area of Deep Reinforcement Learning (DRL) for continuous control environments. Distillation Recent works have explored the problem of combining multiple expert policies in the reinforcement learning setting. A popular approach uses supervised learning to combine each policy by regression over the action distribution. This approach yields model compression BID19 as well as a viable method for multi-task policy transfer BID13 on discrete action domains including the Arcade Learning Environment BID0. We adopt these techniques and extend them for the case of complex continuous action space tasks and make use of them as building block. Transfer Learning Transfer learning exploits the structure learned from a previous task in learning a new task. Our focus here is on transfer learning in environments consisting of continuous control tasks. The concept of appending additional network structure while keeping the previous structure to reduce catastrophic forgetting has worked well on Atari games BID19 BID13 BID20 BID2 Other methods reproduce data from all tasks to reduce the possibility of forgetting how to perform previously learned skills e.g, BID23 BID9. Recent work seeks to mitigate this issue using selective learning rates for specific network parameters. A different approach to combining policies is to use a hierarchical structure BID29. In this setting, previously-learned policies are available as options to execute for a policy trained on a new task. However, this approach assumes that the new tasks will be at least a partial composition of previous tasks, and there is no reintegration of newly learned tasks. A recent promising approach has been to apply meta-learning to achieve control policies that can quickly adapt their behaviour according to current rewards BID4. This work is demonstrated on parameterized task domains. The Powerplay method provides a general framework for training an increasingly general problem solver BID21 BID25. It is based on iteratively: inventing a new task using play or invention; solving this task; and, lastly, demonstrating the ability to solve all the previous tasks. The last two stages are broadly similar to our PLAID approach, although to the best of our knowledge, there are no experiments on motor control tasks of comparable complexity to the ones we tackle. In our work, we develop a specific progressive learning-and-distillation methodology for motor skills, and provide a detailed evaluation as compared to three other plausible baselines. We are specifically interested in understanding issues that arise from the interplay between transfer from related tasks and the forgetting that may occur. Hierarchical RL further uses modularity to achieve transfer learning for robotic tasks BID29 This allows for the substitution of network modules for different robot types over a similar tasks BID3. Other methods use Hierarchical Reinforcement Learning (HRL) as a method for simplifying a complex motor control problem, defining a decomposition of the overall task into smaller tasks BID7 BID5 BID15 While these methods examine knowledge transfer, they do not examine the reintegration of policies for related tasks and the associated problems such as catastrophic forgetting. Recent work examines learned motions that can be shaped by prior mocap clips BID11, and that these can then be integrated in a hierarchical controller. In this section we outline the details of the Reinforcement Learning (RL) framework. We also give an introduction to the concepts of TL and distillation. Leveraging the framework of reinforcement learning, we frame the problem as a Markov Decision Processes (MDP): at each time step t, the world (including the agent) is in a state s t ∈ S, wherein the agent is able to perform actions a t ∈ A, sampled from a policy π(s t, a t) = p(a t |s t) and ing in state s t+1 ∈ S according to transition probabilities T (s t, a t, s t+1). Performing action a t from state s t produces a reward r t; the expected cumulative reward earned from following some policy π may then be written as: DISPLAYFORM0 where T is the time horizon, and γ is the discount factor, defining the planning horizon length. The agent's goal is to learn an optimal policy, π *, maximizing J(π). If the policy has parameters θ π, then the goal may be reformulated to identify the optimal parameters θ * π: DISPLAYFORM1 Our policy models a Gaussian distribution with a mean state dependent mean, µ θt (s t). Thus, our stochastic policy may be formulated as follows: DISPLAYFORM2 where Σ is a diagonal covariance matrix with entries σ 2 i on the diagonal, similar to BID15.To optimize our policy, we use stochastic policy gradient methods, which are well-established family of techniques for reinforcement learning BID26. The gradient of the expected reward with respect to the policy parameters, ∇ θπ J(π(·|θ π)), is given by: DISPLAYFORM3 where DISPLAYFORM4 represents the initial state distribution, and p 0 (s 0)(s 0 → s | t, π 0) models the likelihood of reaching state s by starting at state s 0 and following the policy π(a, s|θ π) for T steps BID24. A π (s, a) represents an advantage function BID22. In this work, we use the Positive Temporal Difference (PTD) update proposed by BID30 for A π (s, a): DISPLAYFORM5 DISPLAYFORM6 where V π (s) = E T t=0 γ t r t | s 0 = s is the value function, which gives the expected discounted cumulative reward from following policy π starting in state s. PTD has the benefit of being insensitive to the advantage function scale. Furthermore, limiting policy updates in this way to be only in the direction of actions that have a positive advantage has been found to increase the stability of learning BID30. Because the true value function is unknown, an approximation V π (· | θ v) with parameters θ v is learned, which is formulated as the regression problem: DISPLAYFORM7 3.2 POLICY DISTILLATION Given a set of expert agents that have solved/mastered different tasks we may want to combine the skills of these different experts into a single multi-skilled agent. This process is referred to as distillation. Distillation does not necessarily produce an optimal mix of the given experts but instead tries to produce an expert that best matches the action distributions produced by all experts. This method functions independent of the reward functions used to train each expert. Distillation also scales well with respect to the number of tasks or experts that are being combined. Given an expert that has solved/mastered a task we want to reuse that expert knowledge in order to learn a new task efficiently. This problem falls in the area of Transfer Learning BID12. Considering the state distribution expert is skilled at solving, (D ωi the source distribution) it can be advantageous to start learning a new, target task ω i+1 with target distribution D ωi+1 using assistance from the expert. The agent learning how to solve the target task with domain D ωi+1 is referred to as the student. When the expert is used to assist the student in learning the target task it can be referred to as the teacher. The success of these methods are dependent on overlap between the D ωi and D ωi+1 state distributions. Although we focus on the problem of being presented with tasks sequentially, there exist other methods for learning a multi-skilled character. We considered 4 overall integration methods for learning multiple skills, the first being a controller that learns multiple tasks at the same time (MultiTasker), where a number of skills are learned at the same time. It has been shown that learning many tasks together can be faster than learning each task separately BID13. The curriculum for using this method is shown in Figure 1a were during a single RL simulation all tasks are learned together. It is also possible to randomly initialize controllers and train in parallel (Parallel) and then combine the ing policies Figure 1b. We found that learning many skills from scratch was challenging, we were only able to get fair for the flat task. Also, when a new task is to be learned with the Parallel model it would occur outside of the original parallel learning, leading to a more sequential method. A TL-Only method that uses TL while learning tasks in a sequence Figure 1c, possibly ending with a distillation step to combine the learned policies to decrease forgetting. For more details see Appendix: 8.4. The last version (PLAiD) learns each task sequentially using TL from the previous, most skilled policy, in the end ing in a policy capable of solving all tasks Figure 1d. This method works well for both combining learned skills and learning new skills. DISPLAYFORM0 Figure 1: Different curriculum learning process. The red box with a D in it denotes a distillation step that combines policies. Each gray box denotes one iteration of learning a new policy. The larger red boxes with an L terrain−type denotes a learning step where a new skill is learned. In this section, we detail our proposed learning framework for continual policy transfer and distillation (PLAiD). In the acquisition (TL) step, we are interested in learning a new task ω i+1. Here transfer can be beneficial if the task structure is somewhat similar to previous tasks ω i. We adopt the TL strategy of using an existing policy network and fine-tuning it to a new task. Since we are not concerned with retaining previous skills in this step, we can update this policy without concern for forgetting. As the agent learns it will develop more skills and the addition of every new skill can increase the probability of transferring knowledge to assist the learning of the next skill. In the integration (distillation) step, we are interested in combining all past skills (π 0, . . ., π i) with the newly acquired skill π i+1. Traditional approaches have used policy regression where data is generated by collecting trajectories of the expert policy on a task. Training the student on these trajectories does not always in robust behaviour. This poor behaviour is caused by the student experiencing a different distribution of trajectories than the expert during evaluation. To compensate for this distribution difference, portions of the trajectories should be generated by the student. This allows the expert to suggest behaviour that will pull the state distribution of the student closer to the expert's. This is a common problem in learning a model to reproduce a given distribution of trajectories BID18 BID1 BID10 BID8. We use a method similar to the DAGGER algorithm BID18 which is useful for distilling policies BID13. See Appendix: 8.2.1 for more details. As our RL algorithm is an actor-critic method, we also perform regression on the critic by fitting both in the same step. The presented in this work cover a range of tasks that share a similar action space and state space. Our focus is to demonstrate continual learning between related tasks. In addition, the conceptual framework allows for extensions that would permit differing state spaces, described later in Section: 5.2. In this experiment, our set of tasks consists of 5 different terrains that a 2D humanoid walker (pdbiped) learns to traverse. The humanoid walker is trained to navigate multiple types of terrain including flat in FIG5 ), incline FIG5 ), steps FIG5 ), slopes FIG5 ), gaps FIG5 and a combination of all terrains mixed FIG5 ) on which agents are trained. The goal in these tasks is to maintain a consistent forward velocity traversing various terrains, while also matching a motion capture clip of a natural human walking gait on flat ground, similar to BID14. The pd-biped receives as input both a character and (eventually) a terrain state representation, consisting of the terrains heights of 50 equally-spaced points in front of the character. The action space is 11-dimensional, corresponding to the joints. Reasonable torque limits are applied, which helps produce more natural motions and makes the control problem more difficult. A detailed description of the experimental setup is included in Section: 8.5. The tasks are presented to the agent sequentially and the goal is to progressively learn to traverse all terrain types. We evaluate our approach against three baselines. First, we compare the above learning curriculum from learning new tasks in PLAiD with learning new tasks in Parallel. This will demonstrate that knowledge from previous tasks can be effectively transferred after distillation steps. Second, we compare to the MultiTasker to demonstrate that iterated distillation is effective for the retention of learned skills. The MultiTasker is also used as a baseline for comparing learning speed. Last, a method that performs TL between tasks and concludes with a distillation step is evaluated to illustrate the of different TL and distillation schedules. The of the PLAiD controller are displayed in the accompanying Video 1 5.1 TRANSFER LEARNING First, the pd-biped is trained to produce a walking motion on flat ground (flat). In FIG0 PLAiD is compared to the three baselines for training on incline. The TL-Only method learns fast as it is given significant information about how to perform similar skills. The Parallel method is given no prior information leading to a less skilled policy. The first MultiTasker for the incline task is initialized from a terrain injected controller that was trained to walk on flat ground. Any subsequent MultiTasker is initialized from the final MultiTasker model of the preceding task. This controller has to learn multiple tasks together, which can complicate the learning process, as simulation for each task is split across the training and the overall RL task can be challenging. This is in contrast to using PLAiD, that is also initialized with the same policy trained on flat, that will integrate skills together after each new skill is learned. In FIG0 the MultiTasker is learning the new task (steps) with similar speed to PLAiD. However, after adding more tasks the MultiTasker is beginning to struggle in FIG0 and starts to forget in FIG0, with the number of tasks it must learn at the same time. While PLAiD learns the new tasks faster and is able to integrate the new skill required to solve the task robustly. TL-Only is also able to learn the new tasks very efficiently. An appealing property of using distillation in PLAiD is that the combined policy model need not resemble that of the individual expert controllers. For example, two different experts lacking state features and trained without a local map of the terrain can be combined into a single policy that has new state features for the terrain. These new terrain features can assist the agent in the task domain in which it operates. We introduce the idea of input injection for this purpose. We augment a policy with additional input features while allowing it to retain its original functional behaviour similar to BID2. This is achieved by adding additional inputs to the neural network and initializing the connecting layer weights and biases to 0. By only setting the weights and biases in the layer connecting the new features to the original network to 0, the gradient can still propagate to any lower layers which are initialized random without changing the functional behaviour. This is performed when distilling the flat and incline experts. Further details can be found in Appendix: 8.3. Training over multiple tasks at the same time may help the agent learn skills quicker, but this may not scale with respect to the number of tasks. When training the MultiTasker over two or even three tasks (FIG1) the method displays good , however when learning a fourth or more tasks the method struggles, as shown in FIG1 and 3b. Part of the reason for this struggle is when new Table 1: These values are relative percentage changes in the average reward, where a value of 0 is no forgetting and a value of −1 corresponds to completely forgetting how to perform the task. A value > 0 corresponds to the agent learning how to better perform a task after training on other tasks. Here, the final policy after training on gaps compared to the original polices produced at the end of training for the task noted in the column heading. The TL-Only baseline forgets more than PLAiD. The MultiTasker forgets less than PLAiD but has a lower average reward over the tasks.tasks are added the MultiTasker has to make trade-offs between more tasks to maximizes. As more tasks are added, this trade-off becomes increasingly complex ing in the MultiTasker favouring easier tasks. Using PLAiD to combine the skills of many policies appears to scale better with respect to the number of skills being integrated. This is likely because distillation is a semi-supervised method which is more stable than the un-supervised RL solution. This can be seen in FIG1, 3e and especially in 3f where PLAiD combines the skills faster and can find higher value policies in practice. PLAiD also presents zero-shot training on tasks which it has never been trained on. In Figure 7 this generalization is shown as the agent navigates across the mixed environment. This is also reflected in Table 1, that shows the final average reward when comparing methods before and after distillation. The TL-Only is able to achieve high performance but much is lost when learning new tasks. A final distillation step helps mitigate this issue but does not work as well as PLAiD. It is possible performing a large final distillation step can lead to over-fitting. There are some indications that distillation is hindering training during the initial few iterations. We are initializing the network used in distillation with the most recently learning policy after TL. The large change in the initial state distribution from the previous seen distribution during TL could be causing larger gradients to appear, disrupting some of the structure learned during the TL step, shown in FIG1 and 3e. There also might not exist a smooth transition in policy space between the newly learned policy and the previous policy distribution. MultiTasker vs PLAiD: The MultiTasker may be able to produce a policy that has higher overall average reward, but in practise constraints can keep the method from combining skills gracefully. If the reward functions are different between tasks, the MultiTasker can favour a task with higher rewards, as these tasks may receive higher advantage. It is also a non-trivial task to normalize the reward functions for each task in order to combine them. The MultiTasker may also favour tasks that are easier than other tasks in general. We have shown that the PLAiD scales better with respect to the number of tasks than the MultiTasker. We expect PLAiD would further outperform the MultiTasker if the tasks were more difficult and the reward functions dissimilar. In our evaluation we compare the number of iterations PLAiD uses to the number the MultiTasker uses on only the new task, which is not necessarily fair. The MultiTasker gains its benefits from training on the other tasks together. If the idea is to reduce the number of simulation samples that are needed to learn new tasks then the MultiTasker would fall far behind. Distillation is also very efficient with respect to the number of simulation steps needed. Data could be collected from the simulator in groups and learned from in many batches before more data is needed as is common for behavioural cloning. We expect another reason distillation benefits learning multiple tasks is that the integration process assists in pulling policies out of the local minima RL is prone to. Transfer Learning: Because we are using an actor-critic learning method, we also studied the possibility of using the value functions for TL. We did not discover any empirical evidence that this assisted the learning process. When transferring to a new task, the state distribution has changed and the reward function may be completely different. This makes it unlikely that the value function will be accurate on this new task. In addition, value functions are in general easier and faster to learn than policies, implying that value function reuse is less important to transfer. We also find that helpfulness of TL depends on not only the task difficulty but the reward function as well. Two tasks may overlap in state space but the area they overlap could be easily reachable. In this case TL may not give significant benefit because the overall RL problem is easy. The greatest benefit is gained from TL when the state space that overlaps for two tasks is difficult to reach and in that difficult to reach area is where the highest rewards are achieved. Once integrated, the skills for our locomotion tasks are self-selecting based on their context, i.e., the knowledge of the upcoming terrain. It may be that other augmentation and distillation strategies are better for situations where either the reward functions are different or a one-hot vector is used to select the currently active expert. In our transfer learning we could be over fitting the initial expert for the particular task it was learning. Making it more challenging for the policy to learn a new task, ing in negative transfer. After learning many new tasks the previous tasks may not receive a large enough potion of the distillation training process to preserve the experts skill well enough. How best to chose which data should be trained on next to best preserve the behaviour of experts is a general problem with multi-task learning. Distillation treats all tasks equally independent of their reward. This can in very low value tasks, receiving potentially more distribution than desired and high value tasks receiving not enough. We have not needed the use a one-hot vector to indicate what task the agent is performing. We want the agent to be able to recognize which task it is given but we do realize that some tasks could be too similar to differentiate, such as, walking vs jogging on flat ground. It would be interesting to develop a method to prioritize tasks during the distillation step. This could assist the agent with forgetting issues or help with relearning tasks. While we currently use the Mean Squared Error (MSE) to pull the distributions of student policies in line with expert polices for distillation, better distance metrics would likely be helpful. Previous methods have used KL Divergence in the discrete action space domain where the state-action value function encodes the policy, e.g., as with Deep Q-Network (DQN). In this work we do not focus on producing the best policy from a mixture of experts, but instead we match the distributions from a number of experts. The difference is subtle but in practice it can be more challengine to balance many experts with respect to their reward functions. It could also be beneficial to use a KL penalty while performing distillation, i.e., something similar to the work in BID28 in order to keep the policy from changing too rapidly during training. We have proposed and evaluated a method for the progressive learning and integration (via distillation) of motion skills. The method exploits transfer learning to speed learning of new skills, along with input injection where needed, as well as continuous-action distillation, using DAGGER-style learning. This compares favorably to baselines consisting of learning all skills together, or learning all the skills individually before integration. We believe that there remains much to learned about the best training and integration methods for movement skill repertoires, as is also reflected in the human motor learning literature. We augment the blind network design by adding features for terrain to create an agent with sight. This network with terrain features has a single convolution layer with 8 filters of width 3. This constitutional layer is followed by a dense layer of 32 units. The dense layer is then concatenated twice, once along each of the original two hidden layers in the blind version of the policy. The policy network models a Gaussian distribution by outputting a state dependant mean. We use a state independent standard deviation that normalized with respect to the action space and multiplied by 0.1. We also use a version of epsilon greedy exploration where with probability an exploration action is generated. For all of our experiments we linearly anneal from 0.2 to 0.1 in 100, 000 iterations and leave it from that point on. Each training simulation takes approximately 5 hours across 8 threads. For network training we use Stochastic Gradient Decent (SGD) with momentum. During the distillation step we use gradually anneal the probability of selecting an expert action from 1 to 0 over 10, 000 iterations. For the evaluation of each model on a particular task we use the average reward achieved by the agent over at most 100 seconds of simulation time. We average this over running the agent over a number of randomly generated simulation runs. For each of the distillation steps we initialize the policy from the most recently trained policy. This policy has seen all of the tasks thus far but may have overfit the most recent tasks. We us a version of the DAGGER algorithm for the distillation process BID18. We anneal from selecting actions from the expert polices to selecting actions from the student policy The probability of selecting an action from the expert is annealed to near zero after 10, 000 training updates. We still add exploration noise to the policies when generating actions to take in the simulation. This is also annealed along with the probability of selecting from the expert policy. The actions used for training always come from the expert policy. Although some actions are applied in the simulation from the student, during a training update those actions will be replaced with ones from the proper expert. The expert used to generate actions for tasks 0 − i is π i and the expert used to generate action for task i + 1 is π i+1. We keep around at most 2 policies at any time. In order to add additional input features to the policy network we construct a new network. This new network has a portion of it that is the same design as the previous network plus additional parameters. First we initialize the new network with random parameters. Then we copy over the values from the previous network into the new one for the portion of the network design that matches the old. Then the weight for the layers that connect the old portion of the network to the new are set to 0. This will allow the network to preserve the previous distribution it modeled. Having the parameters from the old network will also help generate gradients to train the new 0 valued network parameters. We use feature injection to assist the learning method with differentiating between different states. For example, it could be challenging to discover the difference between the flat and incline tasks using only the character features. Therefore, we add new terrain features to allow the controller to better differentiate between these two different tasks. We also evaluate a baseline where we TL for all tasks. In this baseline TL is performed for a number of tasks and then distillation is used to combined these many learned skills. This method can be considered a version of PLAiD where tasks are learned in groups and after some number of tasks, a collection of policies/skills are distilled together. In FIG3 the learning curves for the TL-Only baseline are given. The TL-Only method learns new tasks well. We do not show the incline tasks as the two methods are the same up to starting the steps tasks. In Table 1 the amount of forgetting is compared between methods. To compare the amount of forgetting between TL-Only and PLAiD we show the relative loss in average reward between the original policy trained for the tasks steps and slopes and the final polices for each method on gaps. The TL-Only method shows a larger drop in policy performance corresponding to a large amount of forgeting compared to PLAiD, in particular for the more complex tasks steps and slopes. Interestingly, the final distllation step for TL-Only appears to reduce the performance of the policy. We believe this is related to the final distillation step being more challenging than performing a simpler distillation after each new task. Note that we only compare these two steps because the process for the first two tasks for PLAiD and TL-Only are the same. A comparison of the average rewards for the final policies are given in TAB1. The agent used in the simulation models the dimensions and masses of the average adult. The size of the character state is 50 parameters that include the relative position and velocity of the links in the agent FIG2 ). The action space consists of 11 parameters that indicate target joint positions for the agent. The target joint positions (pd-targets) are turned into joint torques via proportional derivative controllers at each joint. The reward function for the agent consists of 3 primary terms. The first is a velocity term the rewards the agent for going at velocity of 1 m/s The second term is the difference between the pose of the agent and the current pose of a kinematic character controlled via a motion capture clip. The difference between the agent and the clip consists of the rotational difference between each corresponding joint and the difference in angular velocity. The angular velocity for the clip is approximated via finite differences between the current pose of the clip and it's last pose. The last term is an L2 penalty on the torques generated by the agent to help reduce spastic motions. We also impose torque limits on the joints to reduce unrealistic behaviour, limits: Hips 150, knees 125, ankles 100, shoulders 100, elbows 75 and neck 50 N/m. Terrain Types All terrain types are randomly generated per episode, except for the flat terrain. The incline terrain is slanted and the slant of the terrain is randomly sampled between 20 and 25 degrees. The steps terrain consists of flat segments with widths randomly sampled from 1.0 m to 1.5 m followed by sharp steps that have randomly generated heights between 5 cm and 15 cm. The slopes terrain is randomly generated by updating the slope of the previous point in the ground with a value sampled from −20 and 20 degrees to generate a new portion of the ground every 10 cm. The gaps terrain generate gaps of width 25 -30 cm separated by flat segments of widths sampled from 2.0 m to 2.5 m. The mixed terrain is a combination of the above terrains where a portion is randomly chosen from the above terrain types. Figure 7: Still frame shots of the pd-biped traversing the mixed environment. In certain cases the MultiTasker can learn new task faster than PLAiD. In Figure 8a we present the MultiTasker and compare it to PLAiD. In this case the MultiTasker splits its training time across multiple tasks, here we compare the two methods with respect to the time spent learning on the single new task. This is a good baseline to compare our method against but in some ways this is not fair. If the real measure of how efficient a learning method is the number of simulation samples that are needed to learn would fall far behind as the MultiTasker needs to train across all tasks to gain the benefits of improving a single task without forgetting the old tasks.(a) Figure 8: (a) Shows that the MultiTasker can learn faster on steps, flat and incline than PLAiD (expert) learning the single task steps with TL. Match: A continual learning method that uses distillation to combine expert policies and transfer learning to accelerate learning new skills. Hard Negative: We propose a novel Intrinsically Motivated Goal Exploration architecture with unsupervised learning of goal space representations, and evaluate how various implementations enable the discovery of a diversity of policies.

Question: In recent years deep reinforcement learning has been shown to be adept at solving sequential decision processes with high-dimensional state spaces such as in the Atari games. Many reinforcement learning problems, however, involve high-dimensional discrete action spaces as well as high-dimensional state spaces. In this paper, we develop a novel policy gradient methodology for the case of large multidimensional discrete action spaces. We propose two approaches for creating parameterized policies: LSTM parameterization and a Modified MDP (MMDP) giving rise to Feed-Forward Network (FFN) parameterization. Both of these approaches provide expressive models to which backpropagation can be applied for training. We then consider entropy bonus, which is typically added to the reward function to enhance exploration. In the case of high-dimensional action spaces, calculating the entropy and the gradient of the entropy requires enumerating all the actions in the action space and running forward and backpropagation for each action, which may be computationally infeasible. We develop several novel unbiased estimators for the entropy bonus and its gradient. Finally, we test our algorithms on two environments: a multi-hunter multi-rabbit grid game and a multi-agent multi-arm bandit problem. In recent years deep reinforcement learning has been shown to be adept at solving sequential decision processes with high-dimensional state spaces such as in the Go game ) and Atari games BID6, BID7, BID9, BID11, BID12, BID22, BID1 ). In all of these success stories, the size of the action space was relatively small. Many reinforcement learning problems, however, involve high-dimensional action spaces as well as high-dimensional state spaces. Examples include StarCraft BID21, BID4 ), where there are many agents each of which can take a finite number of actions; and coordinating self-driving cars at an intersection, where each car can take a finite set of actions BID17 ).In this paper, we develop a novel policy gradient methodology for the case of large multidimensional action spaces. There are two major challenges in developing such a methodology:• For large multidimensional action spaces, how can we design expressive and differentiable parameterized policies which can be efficiently sampled?• In policy gradient, in order to encourage sufficient exploration, an entropy bonus term is typically added to the objective function. However, in the case of high-dimensional action spaces, calculating the entropy and its gradient requires enumerating all the actions in the action space and running forward and backpropagation for each action, which may be computationally infeasible. How can we efficiently approximate the entropy and its gradient while maintaining desirable exploration?In this paper, we first propose two approaches for parameterizing the policy: a LSTM model and a Modified MDP (MMDP) giving rise to Feed-Forward Network (FFN) model. For both of these parameterizations, actions can be efficiently sampled from the policy distribution, and backpropagation can be employed for training. We then develop several novel unbiased estimators for the entropy bonus and its gradient. These estimators can be combined with stochastic gradient descent giving a new a class of policy gradient algorithms with desirable exploration. Finally, we test our algorithms on two environments: a multi-agent multi-arm bandit problem and a multi-agent hunter-rabbit grid game. Consider an MDP with a d-dimensional action space A = A 1 × A 2 × · · · × A d. Denote a = (a 1, . . ., a d) for an action in A. A policy π(·|s) specifies for each state s a distribution over the action space A. In the standard RL setting, an agent interacts with an environment over a number of discrete timesteps BID19 BID15 ). At timestep t, the agent is in state s t and samples an action a t from the policy distribution π(·|s t). The agent then receives a scalar reward r t and the environment enters the next state s t+1. The agent then samples a t+1 from π(·|s t+1) and so on. The process continues until the end of the episode, denoted by T. The return R t = T −t k=0 γ k r t+k is the discounted accumulated return from time step t until the end of the episode. In the policy gradient formulation, we consider a set of parameterized policies π θ (·|s), θ ∈ Θ, and attempt to find a good θ within a parameter set Θ. Typically the policy π θ (·|s) is generated by a neural network with θ denoting the weights and biases in the network. The parameters θ are updated by performing stochastic gradient ascent on the expected reward. One example of such an algorithm is REINFORCE, proposed by BID23, where in a given episode at timestep t the parameters θ are updated as follows: DISPLAYFORM0 where b t (s t) is a baseline. It is well known that the policy gradient algorithm often converges to a local optimum. To discourage convergence to a highly suboptimal policy, the policy entropy is typically added to the update rule: DISPLAYFORM1 where DISPLAYFORM2 This approach is often referred to as adding entropy bonus or entropy regularization BID23 ) and is widely used in different applications of neural networks, such as optimal control in Atari games ), multi-agent games BID5 ) and optimizer search for supervised machine learning with RL BID0 ). β is referred to as the entropy weight. In applying policy gradient to MDP with large multidimensional action spaces, there are two challenges. First, how do we design an expressive and differentiable parameterized policy which can be efficiently sampled? Second, for the case of large multidimensional action spaces, calculating the entropy and its gradient requires enumerating all the actions in the action space, which may be infeasible. How do we then enhance exploration in a principled way? To abbreviate the notation, we write p θ (a) for π θ (a|s t), with the conditioning on s t being implicit. We consider schemes whereby the sample components a i, i = 1,..., d, are sequentially generated. In particular, after obtaining a 1, a 2,..., a i−1, we will generate a i ∈ A i from some parameterized distribution p θ (·|a 1, a 2, . . ., a i−1) defined over the one-dimensional set A i. After generating the distribution p θ (·|a 1, a 2, . . ., a i−1), i = 1,..., d and the action components a 1,..., a d sequentially, we can then define DISPLAYFORM0.., a i−1 ). We now propose two methods for creating the parameterized distributions p θ (a|a 1, a 2, . . ., a i−1), a ∈ A i. To our knowledge, these models are novel and have not been studied in multidimensional action space literature. We assume that the size of the one-dimensional action sets are equal, that is, DISPLAYFORM1 To handle action sets of different sizes, we include inconsequential actions if needed. The policy p θ (a) can be learned with a recurrent neural network (RNN). Long Short-Term Memory (LSTM), a special flavor of RNN, has recently been used with great success to represent conditional probabilities in language translation tasks BID18 ). Here, as shown in FIG0 (a), we use an LSTM to generate a parameterized multidimensional distribution p θ (·) and to sample a = (a 1, . . ., a d) from that distribution. Specifically, p θ (a|a 1, a 2, . . ., a i−1), a ∈ A i is given by the output of the LSTM. To generate a i, we run a forward pass through the LSTM with the input being a i−1 and the current state s t (and implicitly on a 1, . . ., a i−1 which influences h i−1). This produces a hidden state h i, which is then passed through a linear layer, producing a K dimensional vector. The softmax of this vector is taken to produce the one-dimensional conditional distribution p θ (a|a 1, a 2, ..., a i−1), a ∈ A i. Finally, a i is sampled from this one-dimensional distribution, and is then fed into the next stage of the LSTM to produce a i+1.After generating the action a = (a 1, . . ., a d), and the conditional probabilities p θ (·|a 1, a 2, . . ., a i−1), i = 1,..., d, we can evaluate p θ (a) as the product of the conditional probabilities. During training, we can also use backpropagation to efficiently calculate the first term on the RHS of the update rule in. As an alternative to using a LSTM to create parameterized multidimensional policies, we can modify the underlying MDP to create an equivalent MDP for which the action space is one dimensional at each time step. We refer to this MDP as the Modified MDP (MMDP). In the original MDP, we have state space S and action space A = A 1 × A 2 × · · · × A d where A i = {1, 2, . . ., K}. In MMDP, the state is modified to encapsulate the original state and all the action dimensions selected for state s so far, i.e., (s, a 1, a 2, . . ., a i, 0, . . ., 0) with a 1,..., a i being selected values for action dimensions 1 to i, and 0 being the placeholder for d − i − 1 dimensions. The new action space is A = {0, 1, . . ., K} and the new state space is S × {0, 1, . . ., K} d−1. The state transition probabilities for the MMDP are given by DISPLAYFORM0 where P (s |s, a 1, . . ., a d) is the transition probabiliy of the original MDP. The reward is only generated after all d component actions are taken. It is easily seen that the MMDP is equivalent to the original MDP.Since the MMDP has an one-dimensional action space, we can use a feed-forward network (FFN) to generate each action component as shown in FIG0 ). Note that the FFN input layer size is always |S| + K − 1 and the output layer size is K. As shown in, an entropy bonus is typically included to enhance exploration. However, for large multidimensional action spaces, calculating the entropy and the gradient of the entropy requires enumerating all the actions in the action space and running forward and backpropagation for each action. In this section we develop computationally efficient unbiased estimates for the entropy and its gradient. Let A = (A 1, . . ., A d) denote a random variable with distribution p θ (·). Let H θ denote the exact entropy of the distribution p θ (a): DISPLAYFORM0 (a) The RNN architecture. To generate ai, we input st and ai−1 into the RNN and then pass the ing hidden state hi through a linear layer and a softmax to generate a distribution, from which we sample ai. DISPLAYFORM1... DISPLAYFORM2 The MMDP architecture. To generate ai, we input st and a1, a2,..., ai−1 into a FFN. The output is passed through a softmax layer, providing a distribution from which we sample ai. Since the input size of the FFN is fixed, when generating ai, constants 0 serve as placeholders for ai+1,..., a d−1 in the input to the FFN. During training within an episode, for each state s t, the policy (using, for example, LSTM or MMDP) generates an action a = (a 1, a 2, . . ., a d). A crude approximation of the entropy bonus is: DISPLAYFORM0 This approximation is an unbiased estimate of H θ but its variance is likely to be large. To reduce the variance, we can generate M action samples a, a,..., a (M) when in s t and average the log action probabilities over the samples. However, generating a large number of samples is costly, especially when each sample is generated from a neural network, since each sample requires one additional forward pass. In this section, we develop an alternative unbiased estimator for entropy which only requires the one episodic sample. In the course of an episode, an action a = (a 1, a 2, . . ., a d) is generated for each s t. The alternative estimator accounts for the entropy along each dimension of the action space. DISPLAYFORM0 where DISPLAYFORM1 which is the entropy of A i conditioned on a i−1. This approximation of entropy bonus is computationally efficient since for each dimension i, we need to obtain p θ (·|a i−1), its log and gradient anyway during training. We refer to this approximation as the smoothed entropy. The smoothed entropy H θ (A) has several appealing properties. The proofs of Theorem 1 and Theorem 3 are straightforward and omitted. DISPLAYFORM2 is an unbiased estimator of the exact entropy H θ.Theorem 2. If p θ (a) has a multivariable normal distribution with mean and variance depending on θ, then: DISPLAYFORM3 Thus, the smoothed entropy equals the exact entropy for a multi-variate normal parameterization of the policy (Proof in Appendix B). Theorem 3. (i) If there is a sequence of weights θ 1, θ 2,... such that p θn (·) converges to the uniform distribution over A, then sup DISPLAYFORM4 Thus, the smoothed entropy H θ (a) mimics the exact entropy in that it has the same supremum and infinum values as the exact entropy. The above theorems indicate that H θ (a) may serve as a good proxy for H θ: it is an unbiased estimator for H θ, it has the same minimum and maximum values when varying θ; and in the special case when p θ (a) has a multivariate normal distribution, it is actually equal to H θ for all a ∈ A. Our numerical experiments have shown that the smoothed estimator H θ (a) typically has lower variance than the crude estimator H crude θ (a). However, it is not generally true that the smoothed estimator always has lower variance as counterexamples can be found. So far we have been looking at estimates of entropy. But the policy gradient algorithm uses the gradient of the entropy rather than just simply the entropy. As it turns out, the gradient of estimators H crude θ (a) and H θ (a) are not unbiased estimates of the gradient of the entropy. In this subsection, we provide unbiased estimators for the gradient of the entropy. For simplicity, in this section, we assume an one-step decision setting, such as in a multi-armed bandit problem. A straightforward calculation shows: DISPLAYFORM0 Suppose a is one sample from p θ (·). A crude unbiased estimator for the gradient of the entropy therefore is: − log p θ (a)∇ θ log p θ (a) = log p θ (a)∇ θ H crude θ (a). Note that this estimator is equal to the gradient of the crude estimator multiplied by a correction factor. Analogous to the smoothed estimator for entropy, we can also derive a smoothed estimator for the gradient of the entropy. Theorem 4. If a is a sample from p θ (·), then DISPLAYFORM1 is an unbiased estimator for the gradient of the entropy (Proof in Appendix C).Note that this estimate for the gradient of the entropy is equal to the gradient of the smoothed estimate H θ (a) plus a correction term. We refer to this estimate of the entropy gradient as the unbiased gradient estimate. We designed experiments to compare the LSTM and MMDP models, and to also compare how the different entropy approximations perform for both. For each entropy approximation, the entropy weight as described in was tuned to give the highest episode reward. For MMDP, the number of hidden layers was also tuned from 1 to 7. The rest of the hyperparameters are listed in Appendix A. In this environment, there is a n × n grid. At the beginning of each episode d hunters and d rabbits are randomly placed in the grid. The rabbits remain fixed in the episode, and each hunter can move to a neighboring square (including diagonal neighbors) or stay at the current square. So each hunter has nine possible actions, and altogether there are |A| = 9 d actions at each time step. When a hunter enters a square with a rabbit, the hunter captures the rabbit and remains there until the end of the game. In each episode, the goal is for the hunters to capture the rabbits as quickly as possible. Each episode is allowed to run for at most 10,000 time steps. To provide a dense reward signal, we formalize the goal with the following modification: capturing a rabbit gives a reward of 1, which is discounted by the number of time steps taken since the beginning of the episode. The discount factor is 0.8. The goal is to maximize the episode's total discounted reward. After a hunter captures a rabbit, they both become inactive. The representation of an active hunter or rabbit is (1, y position, x position). The representation of an inactive hunter or rabbit is (0, -1, -1). TAB0 shows the performance of the LSTM and MMDP models with different entropy estimates. (smoothed mode entropy is explained in Appendix D). The evaluation was performed in a square grid of 5 by 5 with 5 hunters and 5 rabbits. Training was run for 1 million episodes for each of the seeds. All evaluations are averaged over 1,000 episodes per seed for a total of 5,000 episodes. First, we observe that the LSTM model always does better than the MMDP model, particularly for the episode length. Second, we note that policies obtained with the entropy approximations all perform better than policies obtained without entropy or with crude entropy. For the LSTM model, the best performing approximation is smoothed entropy, reducing the mean episode length by 45% and increasing the mean episode reward by 10% compared to without entropy. We also note that there is not a significant difference in performance between the smoothed entropy estimate, smoothed mode estimate, and the unbiased gradient estimate. As shown in TAB1, smoothed entropy is also more robust to the initial seed than without entropy. For example, for the LSTM model, in the case of without entropy, seed 0 leads to significantly worse than the seeds 1-4. This does not happen to smoothed entropy. We now consider how policies trained with entropy approximations compare with polices trained with exact entropy. In order to calculate exact entropy in an acceptable amount of time, we reduced the number of hunters and rabbits to 4 hunters and 4 rabbits. Training was run for 50,000 episodes. TAB2 shows the performance differences between policies trained with entropy approximations and exact entropy. We see that the best entropy approximations perform only slightly worse than exact entropy for both LSTM and MMDP. Once again we see that the LSTM model performs better than the MMDP model. We examine a multi-agent version of the standard multi-armed bandit problem, where there are d agents each pulling one of K arms, with d ≤ K. The k th arm generates a reward r k. The total reward in a round is generated as follows. In each round, each agent chooses an arm. All of the chosen arms are then pulled, with each pulled arm generating a reward. Note that the total number of arms chosen, c, may be less than d since some agents may choose the same arm. The total reward is the sum of rewards from the c chosen arms. The optimal policy is for the d agents to collectively pull the d arms with the highest rewards. Additionally, among all the optimal assignments of d agents to the d arms that yield the highest reward, we add a bonus reward with probability p * if one particular agent-to-arms assignment is chosen. We performed experiments with 4 agents and 10 arms, with the k th arm providing a reward of k. The exceptional assignment gets a bonus of 200 with probability 0.01, and no bonus with probability 0.99. Thus the maximum expected reward is 36. Training was run for 100,000 rounds for each of the seeds. TAB3 shows average for the last 500 of the 100,000 rounds. The for the multi-agent bandit problem are consistent with those for the hunter-rabbit problem. Policies obtained with the entropy approximations all perform better than policies obtained without entropy or with crude entropy, particularly for the percentage of optimal arms pulled. We again note that using the unbiased gradient estimate does not perform significantly better than using the smoothed entropy estimate. There has been limited attention in the RL literature with regards to large discrete action spaces. BID13 proposes generalized value functions in the form of H-value functions, and also propose approximate linear programming as a solution technique. Their methodology is not suited for deep RL, and approximate linear programming may lead to highly sub-optimal solutions. Dulac-Arnold et al. FORMULA1 embeds discrete actions in a continuous space, picks actions in the continuous space and map these actions back into the discrete space. However, their algorithm introduces a new hyper-parameter that requires tuning for every new task. Our approach involves no new hyper-parameter other than those normally used in deep learning. In BID17, each action dimension is treated as an agent and backpropagation is used to learn coordination between the agents. The approach is particularly adept for problems where agents leave and enter the system. However, the approach requires homogenous agents, and has not been shown to solve large-scale problems. Furthermore, the decentralized approach will potentially lead to highly suboptimal polices even though communication is optimized among the agents. To our knowledge, we are the first to propose using LSTMs and a modified MDP to create policies for RL problems with large multidimensional action spaces. Although this leads to algorithms that are straightforward, the approaches are natural and well-suited to multidimensional action spaces. We also propose novel estimators for the entropy regularization term that is often used in policy gradient. To the best of our knowledge, no prior work has dealt with approximating the policy entropy for MDP with large multidimensional discrete action space. On the other hand, there has been many attempts to devise methods to encourage beneficial exploration for policy gradient. BID10 modifies the entropy term by adding weights to the log action probabilities, leading to a new optimization objective termed under-appreciated reward exploration. While entropy regularization has been mostly used in algorithms that explicitly parameterize the policies, BID14 applies entropy regularization to Q-learning methods. They make an important observation about the equivalence between policy gradient and entropy regularized Q-learning, which they term soft Q-learning. In this paper, we developed a novel policy gradient methodology for the case of large multidimensional discrete action spaces. We proposed two approaches for creating parameterized policies: LSTM parameterization and a Modified MDP (MMDP) giving rise to Feed-Forward Network (FFN) parameterization. Both of these approaches provide expressive models to which backpropagation can be applied for training. We then developed several novel unbiased estimators for entropy bonus and its gradient. We did experimental work for two environments with large multidimensional action space. For these environments, we found that both the LSTM and MMDP approach could successfully solve large multidimensional action space problems, with the LSTM approach generally performing better. We also found that the smoothed estimates of the entropy and the unbiased gradient estimate of the entropy gradient can help reduce computational cost while not sacrificing significant loss in performance. Hyperparameters for hunter-rabbit gameThe LSTM policy has 128 hidden nodes. For the MMDP policy, the number of hidden layers for smoothed entropy, smoothed mode entropy, unbiased gradient estimate, crude entropy and without entropy are 5, 3, 3, 4 3 and 3 respectively. Each MMDP layer has 128 nodes. We parameterize the baseline in with a feed forward neural network with one hidden layer of size 64. This network was trained using first visit Monte Carlo return to minimize the L1 loss between actual and predicted values of states visited during the epidode. Both the policies and baseline are optimized after each episode with RMSprop BID20 ). The RHS of FORMULA1 To obtain the in TAB2, the entropy weights for LSTM smoothed entropy, LSTM exact entropy, MMDP unbiased gradient estimate and MMDP exact entropy are 0.03, 0.01, 0.03 and 0.01 respectively. The MMDP networks have three layers with 128 nodes in each layer. Experimental are averaged over five seeds. The experiments were run with 4 agents and 10 arms. For the 10 arms, their rewards are i for i = 1,..., 10. The LSTM policy has 32 hidden nodes. The baseline in is a truncated average of the reward of the last 100 rounds. The entropy weight for crude entropy, smoothed entropy and unbiased gradient estimate are 0.005, 0.001 and 0.003 respectively. The learning rates for without entropy, crude entropy, smoothed entropy and unbiased gradient estimate are 0.006, 0.008, 0.002 and 0.005 respectively. Experimental are averaged over ten seeds. We first note that for DISPLAYFORM0 where X 1 and X 2 are random vectors, we have X 2 | X 1 = x 1 ∼ N (μ,Σ) wherē DISPLAYFORM1 Observe that the covariance matrix of the conditional distribution does not depend on the value of x 1 .Also note that for X ∼ N (µ, Σ), the entropy of X takes the form DISPLAYFORM2 where k is the dimension of X and | · | denotes the determinant. Therefore, the entropy of a multivariate normal random variable depends only on the variance and not on the mean. Because A is multivariate normal, the distribution of A i given A 1 = a 1,..., A i−1 = a i−1 has a normal distribution with a variance σ 2 i that does not depend on a 1,..., a i−1. Therefore DISPLAYFORM3 does not depend on a 1,..., a i−1 and hence H θ (a) does not depend on a. Combining this with the fact that H θ (a) is an unbiased estimator for H θ gives H θ (a) = H θ for all a ∈ A. From, we have: DISPLAYFORM0 We will now use conditional expectation to calculate the terms in the double sum. For i < j: DISPLAYFORM1 For i > j: Combining these three conditional expectations with, we obtain: DISPLAYFORM2 DISPLAYFORM3 Depending on the episodic action a at a given time step in the episode, the smoothed entropy H θ (a) may give unsatisfactory . For example, suppose for a particular episodic action a, H θ (a) H θ. In this case, the policy gradient may ignore the entropy bonus term, thinking that the policy already has enough entropy when it perhaps does not. We therefore consider alternative approximations which may improve performance at modest additional computational cost. First consider Thus in this case, instead of calculating the entropy over a sample action a, we calculate it over the most likely action a *. The problem here is that it is not easy to find a * when the given conditional probabilities p θ (a|a 1, . . ., a i−1) are not in closed form but only available algorithmically as outputs of neural networks. DISPLAYFORM0 A more computationally efficient approach would be to choose the action greedily: a 1 = argmax The actionâ is an approximation for the mode of the distribution p θ (·). As often done in NLP, we can use beam search to determine an action a that is a better approximation, that is, p θ (a) ≥ p θ (â). Indeed, the above H θ definition is beam search with beam size equal to 1. We refer to H θ as smoothed mode entropy. H θ with an appropriate beam size may be a better approximation for the entropy H θ than H θ (a). However, calculating H θ and its gradient comes with some computational cost. For example, with a beam size equal to one, we would have to make two passes through the neural network at each time step: one to obtain the episodic sample a and the other to obtain the greedy actionâ. For beam size n we would need to make n + 1 passes. We note that H θ is a biased estimator for H θ but with no variance. Thus there is a bias-variance tradeoff between H θ (a) and H θ. Note that H θ also satisfies Theorems 2 and 3 in subsection 4.2.

Represent the TLDR for finding the corresponding scientific article

policy parameterizations and unbiased policy entropy estimators for MDP with large multidimensional discrete action space

Neural network regression should use Dirichlet output distribution when targets are probabilities in order to quantify uncertainty of predictions.