Question: Represent the article for finding a single-sentence TLDR

Question: This work adopts the very successful distributional perspective on reinforcement learning and adapts it to the continuous control setting. We combine this within a distributed framework for off-policy learning in order to develop what we call the Distributed Distributional Deep Deterministic Policy Gradient algorithm, D4PG. We also combine this technique with a number of additional, simple improvements such as the use of N-step returns and prioritized experience replay. Experimentally we examine the contribution of each of these individual components, and show how they interact, as well as their combined contributions. Our show that across a wide variety of simple control tasks, difficult manipulation tasks, and a set of hard obstacle-based locomotion tasks the D4PG algorithm achieves state of the art performance. The ability to solve complex control tasks with high-dimensional input and action spaces is a key milestone in developing real-world artificial intelligence. The use of reinforcement learning to solve these types of tasks has exploded following the work of the Deep Q Network (DQN) algorithm BID11, capable of human-level performance on many Atari games. Similarly, ground breaking achievements have been made in classical games such as Go. However, these algorithms are restricted to problems with a finite number of discrete actions. In control tasks, commonly seen in the robotics domain, continuous action spaces are the norm. For algorithms such as DQN the policy is only implicitly defined in terms of its value function, with actions selected by maximizing this function. In the continuous control domain this would require either a costly optimization step or discretization of the action space. While discretization is perhaps the most straightforward solution, this can prove a particularly poor approximation in highdimensional settings or those that require finer grained control. Instead, a more principled approach is to parameterize the policy explicitly and directly optimize the long term value of following this policy. In this work we consider a number of modifications to the Deep Deterministic Policy Gradient (DDPG) algorithm BID9. This algorithm has several properties that make it ideal for the enhancements we consider, which is at its core an off-policy actor-critic method. In particular, the policy gradient used to update the actor network depends only on a learned critic. This means that any improvements to the critic learning procedure will directly improve the quality of the actor updates. In this work we utilize a distributional BID0 version of the critic update which provides a better, more stable learning signal. Such distributions model the randomness due to intrinsic factors, among these is the inherent uncertainty imposed by function approximation in a continuous environment. We will see that using this distributional update directly in better gradients and hence improves the performance of the learning algorithm. Due to the fact that DDPG is capable of learning off-policy it is also possible to modify the way in which experience is gathered. In this work we utilize this fact to run many actors in parallel, all feeding into a single replay table. This allows us to seamlessly distribute the task of gathering Authors contributed equally. The Deterministic Policy Gradient (DPG) algorithm BID19 upon which this work is based starts from a different set of ideas, namely the policy gradient theorem of BID22. The deterministic policy gradient theorem builds upon this earlier approach, but replaces the stochastic policy with one that includes no randomness. This approach is particularly important because it had previously been believed that the deterministic policy gradient did not exist in a model-free setting. The form of this gradient is also interesting in that it does not require one to integrate over the action space, and hence may require less samples to learn. DPG was later built upon by BID9 who extended this algorithm and made use of a deep neural network as the function approximator, primarily as a mechanism for extending these to work with vision-based inputs. Further, this entire endeavor lends itself very readily to an off-policy actorcritic architecture such that the actor's gradients depend only on derivatives through the learned critic. This means that by improving estimation of the critic one is directly able to improve the actor gradients. Most interestingly, there have also been recent attempts to distribute updates for the DDPG algorithm, (e.g. BID15 and more generally in this work we build on work of BID5 for implementing distributed actors. Recently, BID0 showed that the distribution over returns, whose expectation is the value function, obeys a distributional Bellman equation. Although the idea of estimating a distribution over returns has been revisited before BID21 BID13, Bellemare et al. demonstrated that this estimation alone was enough to achieve state-of-the-art on the Atari 2600 benchmarks. Crucially, this technique achieves these gains by directly improving updates for the critic. In this work we consider a standard reinforcement learning setting wherein an agent interacts with an environment in discrete time. At each timestep t the agent makes observations x t P X, takes actions a t P A, and receives rewards rpx t, a t q P R. Although we will in general make no assumptions about the inputs X, we will assume that the environments considered in this work have real-valued actions A " R d .In this standard setup, the agent's behavior is controlled by a policy π : X Ñ A which maps each observation to an action. The state-action value function, which describes the expected return conditioned on first taking action a P A from state x P X and subsequently acting according to π, is defined as DISPLAYFORM0 and is commonly used to evaluate the quality of a policy. While it is possible to derive an updated policy directly from Q π, such an approach typically requires maximizing this function with respect to a and is made complicated by the continuous action space. Instead we will consider a parameterized policy π θ and maximize the expected value of this policy by optimizing Jpθq " ErQ π θ px, π θ pxqqs. By making use of the deterministic policy gradient theorem BID19 one can write the gradient of this objective as DISPLAYFORM1 where ρ is the state-visitation distribution associated with some behavior policy. Note that by letting the behavior policy differ from π we are able to empirically evaluate this gradient using data gathered off-policy. While the exact gradient given by assumes access to the true value function of the current policy, we can instead approximate this quantity with a parameterized critic Q w px, aq. By introducing the Bellman operator pT π Qqpx, aq " rpx, aq`γE " DISPLAYFORM2 whose expectation is taken with respect to the next state x 1, we can minimize the temporal difference (TD) error, i.e. the difference between the value function before and after applying the Bellman update. Typically the TD error will be evaluated under separate target policy and value networks, i.e. networks with separate parameters pθ 1, w 1 q, in order to stabilize learning. By taking the twonorm of this error we can write the ing loss as DISPLAYFORM3 In practice we will periodically replace the target networks with copies of the current network weights. Finally, by training a neural network policy using the deterministic policy gradient in and training a deep neural to minimize the TD error in we obtain the Deep Deterministic Policy Gradient (DDPG) algorithm. Here a sample-based approximation to these gradients is employed by using data gathered in some replay table. The approach taken in this work starts from the DDPG algorithm and includes a number of enhancements. These extensions, which we will detail in this section, include a distributional critic update, the use of distributed parallel actors, N -step returns, and prioritization of the experience replay. First, and perhaps most crucially, we consider the inclusion of a distributional critic as introduced in BID0. In order to introduce the distributional update we first revisit in terms of the return as a random variable Z π, such that Q π px, aq " E Z π px, aq. The distributional Bellman operator can be defined as DISPLAYFORM0 where equality is with respect to the probability law of the random variables; note that this expectation is taken with respect to distribution of Z as well as the transition dynamics. While the definition of this operator looks very similar to the canonical Bellman operator defined in, it differs in the types of functions it acts on. The distributional variant takes functions which map from state-action pairs to distributions, and returns a function of the same form. In order to use this function within the context of the actor-critic architecture introduced above, we must parameterize this distribution and define a loss similar to that of Equation 4. We will write the loss as DISPLAYFORM1 for some metric d that measures the distance between two distributions. Two components that can have a significant impact on the performance of this algorithm are the specific parameterization used for Z w and the metric d used to measure the distributional TD error. In both cases we will give further details in Appendix A; in the experiments that follow we will use the Categorical distribution detailed in that section. We can complete this distributional policy gradient algorithm by including the action-value distribution inside the actor update from Equation 2. This is done by taking the expectation with respect to the action-value distribution, i.e. DISPLAYFORM2 Algorithm 1 D4PG Input: batch size M, trajectory length N, number of actors K, replay size R, exploration constant, initial learning rates α 0 and β 0 1: Initialize network weights pθ, wq at random 2: Initialize target weights pθ 1, w 1 q Ð pθ, wq 3: Launch K actors and replicate network weights pθ, wq to each actor 4: for t " 1,..., T do Sample M transitions px i:i`N, a i:i`N´1, r i:i`N´1 q of length N from replay with priority p i 6: DISPLAYFORM0 Compute the actor and critic updates DISPLAYFORM1 Update network parameters θ Ð θ`α t δ θ, w Ð w`β t δ w If t " 0 mod t target, update the target networks pθ 1, w 1 q Ð pθ, wq If t " 0 mod t actors, replicate network weights to the actors 11: end for 12: return policy parameters θ DISPLAYFORM0 Sample action a " π θ pxq` N p0, 1q DISPLAYFORM1 Execute action a, observe reward r and state x 1 4:Store px, a, r, x 1 q in replay 5: until learner finishes As before, this update can be empirically evaluated by replacing the outer expectation with a samplebased approximation. Next, we consider a modification to the DDPG update which utilizes N -step returns when estimating the TD error. This can be seen as replacing the Bellman operator with an N -step variant DISPLAYFORM2 where the expectation is with respect to the N -step transition dynamics. Although not used by, N -step returns are widely used in the context of many policy gradient algorithms (e.g. BID12 as well as Q-learning variants BID4 . This modification can be applied analogously to the distributional Bellman operator in order to make use of it when updating the distributional critic. Finally, we also modify the standard training procedure in order to distribute the process of gathering experience. Note from Equations that the actor and critic updates rely entirely on sampling from some state-visitation distribution ρ. We can parallelize this process by using K independent actors, each writing to the same replay table. A learner process can then sample from some replay table of size R and perform the necessary network updates using this data. Additionally sampling can be implemented using non-uniform priorities p i as in BID16. Note that this requires the use of importance sampling, implemented by weighting the critic update by a factor of 1{Rp i . We implement this procedure using the ApeX framework BID5 and refer the reader there for more details. Algorithm pseudocode for the D4PG algorithm which includes all the above-mentioned modifications can be found in Algorithm 1. Here the actor and critic parameters are updated using stochastic gradient descent with learning rates, α t and β t respectively, which are adjusted online using ADAM BID7 . While this pseudocode focuses on the learning process, also shown is pseudocode for actor processes which in parallel fill the replay table with data. The left-most set illustrates the actor network and critic torso used for the standard control and manipulation domains. The full critic architecture is completed by feeding the output of the critic torso into a relevant distribution, e.g. the categorical distribution, as defined in Section A. The right half of the figure similarly illustrates the architecture used by the parkour domains. In this section we describe the performance of the D4PG algorithm across a variety of continuous control tasks. To do so, in each environment we run our learning procedure and periodically snapshot the policy in order to test it without exploration noise. We will primarily be interested in the performance as a function of wall clock time, however we will also examine the data efficiency. Most interestingly, from a scientific perspective, we also perform a number of ablations which individually remove components of the D4PG algorithm in order to determine their specific contributions. First, we experiment with and without distributional updates. In this setting we focus on use of a categorical distribution as we found in preliminary experiments that the use of a mixture of Gaussians performed worse and was less stable with respect to hyperparameter values across different tasks; a selection of these runs can be found in Appendix C. Across all tasks-except for one which we will introduce later-we use 51 atoms for the categorical distribution. In what follows we will refer to non-distributional variants of this algorithm as Distributed DDPG (D3PG).Next, we consider prioritized and non-prioritized versions of these algorithm variants. For the nonprioritized variants, transitions are sampled from replay uniformly. For prioritized variants we use the absolute TD-error to sample from replay in the case of D3PG, and for D4PG we use the absolute distributional TD-error as described in Section A. We also vary the trajectory length N P t1, 5u. In all experiments we use a replay table of size R " 1ˆ10 6 and only consider behavior policies which add fixed Gaussian noise N p0, 1q to the current online policy; in all experiments we use a value of " 0.3. We experimented with correlated noise drawn from an Ornstein-Uhlenbeck process, as suggested by ), however we found this was unnecessary and did not add to performance. For all algorithms we initialize the learning rates for both actor and critic updates to the same value. In the next section we will present a suite of simple control problems for which this value corresponds to α 0 " β 0 " 1ˆ10´4; for the following, harder problems we set this to a smaller value of α 0 " β 0 " 5ˆ10´5. Similarly for the control suite we utilize a batch size of M " 256 and for all subsequent problems we will increase this to M " 512. We first consider evaluating performance on a number of simple, physical control tasks by utilizing a suite of benchmark tasks BID23 developed in the MuJoCo physics simulator BID24. Each task is run for exactly 1000 steps and provides either an immediate dense reward r t P r0, 1s or sparse reward r t P t0, 1u depending on the particular task. For each domain, the inputs presented to the agent consist of reasonably low-dimensional observations, many consisting of phys- ical state, joint angles, etc. These observations range between 6 and 60 dimensions, however note that the difficulty of the task is not immediately associated with its dimensionality. For example the acrobot is one of the lowest dimensional tasks in this suite which, due to its level of controllability, can prove much more difficult to learn than other, higher dimensional tasks. For an illustration of these domains see Figure 9; see Appendix D for more details. For algorithms in these experiments we consider actor and critic architectures of the form given in FIG0 and for each experiment we use K " 32 actors. FIG1 shows the performance of D4PG and its various ablations across the entire suite of control tasks. This set of plots is quite busy, however it serves as a broad set of tasks with which we can obtain a general idea of the algorithms performance. Later experiments on harder domains look more closely at the difference between algorithms. Here we also compare against the canonical (non-distributed) DDPG algorithm as a baseline, shown as a dotted black line. This removes all the enhancements proposed in this paper, and we can see that except on the simplest domain, Cartpole (Swingup), it performs worse than all other methods. This performance disparity worsens as we increase the difficulty of tasks, and hence for further experiments we will drop this line from the plot. Next, across all tasks we see that the best performance is obtained by the full D4PG algorithm (shown in purple and bold). Here we see that the longer unroll length of N " 5 is uniformly better (we show these as solid lines), and in particular we sometimes see for both D3PG and D4PG that an unroll length of N " 1 (shown as dashed lines) can occasionally in instability. This is especially apparent in the Cheetah (Walk) and Cartpole (Swingup Sparse) tasks. The next biggest gain is arguably due to the inclusion of the distributional critic update, where it is particularly helpful on the hardest tasks e.g. Humanoid (Run) and Acrobot. The manipulator is also quite difficult among this suite of tasks, and here we see that the inclusion of the distributional update does not help as much as in other tasks, although note that here the D3PG and D4PG variants obtain approximately the same performance. As far as the use of prioritization is concerned, it does not appear to contribute significantly to the performance of D4PG. This is not the case for D3PG, however, which on many tasks is helped significantly by the inclusion of prioritization. Next, we consider a set of tasks designed to highlight the ability of the D4PG agent to learn dexterous manipulation. Tasks of this form can prove difficult for many reasons, most notably the higher dimensionality of the control task, intermittent contact dynamics, and potential under-actuation of the manipulator. Here we use a simulated hand model implemented within MuJoCo, consisting of 13 actuators which control 22 degrees of freedom. For these experiments the wrist site is attached to a fixed location in space, about which it is allowed to rotate axially. In particular this allows the hand to pick up objects, rotate into a palm-up position, and manipulate them. We first consider a task in which a cylinder is dropped onto the hand from a random height, and the goal of the task is to catch the falling cylinder. The next task requires the agent to pick up an object from the tabletop and then maneuver it to a target position and orientation. The final task is one wherein a broad cylinder must be rotated inhand in order to match a target orientation. See Appendix E for further details regarding both the model and the tasks. For these tasks we use the same network architectures as in the previous section as well as K " 64 actors. In FIG3 we again compare the D4PG algorithm against ablations of its constituent components. Here we split the algorithms between N " 1 in the top row and N " 5 in the bottom row, and in particular we can see that across all algorithms N " 5 is uniformly better. For all tasks, the full D4PG algorithm performs either at the same level or better than other ablations; this is particularly apparent in the N " 5 case. Overall the use of priorization never seems to harm D4PG, however it does appear to be of limited additional value. Interestingly this is not necessarily the case with the D3PG variant (i.e. without distributional updates). Here we can see that prioritization sometimes harms the performance of D3PG, and this is very readily seen in the N " 1 case where the algorithm can either become unstable, or in the case of the Pickup and Orient task it completely fails to learn. Finally, we consider the parkour domain introduced by. In this setting the agent controls a simplified robotic walker which is rewarded for forward movement, but is impeded by a number of randomly sampled obstacles; see Figure 4 for a visualization and refer to the earlier work for further details. The first of our experiments considers a two-dimensional walker, i.e. a domain in which the walker is allowed to move horizontally and vertically, but is constrained to a fixed depth position. In this domain the obstacles presented to the agent include gaps in the floor surface, barriers it must jump over, and platforms that it can either run over or underneath. The agent is presented with proprioceptive observations x proprio P R 19 corresponding to the angles of its limbs and other functions of these quantities. It is also given access to observations x terrain P R Figure 4: Example frames taken from trained agents running in the two parkour domains.which includes features such as a depth map of the upcoming terrain, etc. In order to accommodate these inputs we utilize a network architecture as specified in FIG0. In particular we make use of a stack of feed-forward layers which process the terrain information to reduce it to a smaller number of hidden units before concatenating with the proporioceptive information for further processing. The actions in this domain take the form of torque controls a P R 6.In order to examine the performance of the D4PG algorithm in this setting we consider the ablations of the previous sections and we have further introduced a PPO baseline as utilized in the earlier paper of. For all algorithms, including PPO, we use K " 64 actors. These are shown in Figure 5 in the top row. As before we examine the performance separately for N " 1 and N " 5, and again we see that the higher unroll length in better performance. Note that we show the PPO baseline on both plots for consistency, but in both plots this is the same algorithm, with settings proposed in the earlier paper and unrolls of length 50.Here we again see a clear delineation and clear gains for each of the other algorithm components. The biggest gain comes from the inclusion of the distributional update, which we can see by comparing the non-prioritized D3PG/D4PG variants. We see marginal benefit to using prioritization for D3PG, but this gain disappears when we consider the distributional update. Finally, we can see when comparing to the PPO baseline that this algorithm compares favorably to D3PG in the case of N " 1, however is outperformed by D4PG; when N " 5 all algorithms outperform PPO.Next, in the plots shown in Figure 5 on the bottom row we also consider the performance not just in terms of training time, but also in terms of the sample complexity. In order to do so we plot the performance of each algorithm versus the number of actor steps, i.e. the quantity of transitions collected. This is perhaps more favorable to PPO, as the parallel actors considered in this work are not necessarily tuned for sample efficiency. Here we see that PPO is able to out-perform the non-prioritized version of D3PG, and early on in training is favorable compared to the prioritized version, although this trails off. However, we still see significant performance gains by utilizing the distributional updates, both in a prioritized and non-prioritized setting. Interestingly we see that the use of prioritization does not gain much, if any over the non-prioritized D4PG version. Early in the trajectory for N " 5, in fact, we see that the non-prioritized D4PG exhibits better performance, however later these performance curves level out. With respect to wall-clock time these small differences may be due to small latencies in the scheduling of different runs, as we see that this difference is less for the plot with respect to actor steps. Finally we consider a humanoid walker which is able to move in all three dimensions. The obstacles in this domain consist of gaps in the floor, barriers that must be jumped over, and walls with gaps that allow the agent to run through. For this experiment we utilize the same network architecture as in the previous experiment, except now the observations are of size x proprio P R 79 and x terrain P R 461. Again actions are torque controls, but in 21 dimensions. In this task we also increased the number of atoms for the categorical distribution from 51 to 101. This change increases the level of resolution for the distribution in order to keep the resolution roughly consistent with other tasks. This is a much higher dimensional problem than the previous parkour task with a significantly more difficult control task: the walker is more unstable and there are many more ways for the agent to fail than in the previous experiment. The for this particular domain are displayed in Figure 6, and here we concentrate on performance as a function of wall-clock time, restricted to the previously best performing roll-out length of N " 5. In this setting we see a clear delineation between first the PPO which are the poorest performing, the D3PG where the prioritized version has a slight edge, and finally the D4PG . Interestingly for D4PG we again see as in the twodimensional walker case, the use of prioritization seems to have no benefit, with both versions have almost identical performance curves; in fact the performance here is perhaps even closer than that of the previous set of experiments. In this work we introduced the D4PG, or Distributed Distributional DDPG, algorithm. Our main contributions include the inclusion of a distributional updates to the DDPG algorithm, combined with the use of multiple distributed workers all writing into the same replay table. We also consider a number of other, smaller changes to the algorithm. All of these simple modifications contribute to the overall performance of the D4PG algorithm; the biggest performance gain of these simple changes is arguably the use of N -step returns. Interestingly we found that the use of priority was less crucial to the overall D4PG algorithm especially on harder problems. While the use of prioritization was definitely able to increase the performance of the D3PG algorithm, we found that it can also lead to unstable updates. This was most apparent in the manipulation tasks. Finally, as our can attest, the D4PG algorithm is capable of state-of-the-art performance on a number of very difficult continuous control problems. In this section we consider two potential parameterized distributions for D4PG. Parameterized distributions, in this framework, are implemented as a neural network layer mapping the output of the critic torso (see FIG0 to the parameters of a given distribution (e.g. mean and variance). In what follows we will detail the distributions and their corresponding losses. Categorical Following Bellemare et al. FORMULA0, we first consider the categorical parameterization, a layer whose parameters are the logits ω i of a discrete-valued distribution defined over a fixed set of atoms z i. This distribution has hyperparameters for the number of atoms, and the bounds on the support (V min, V max). Given these, ∆ "Vmax´Vmin ´1corresponds to the distance between atoms, and z i " V min`i ∆ gives the location of each atom. We can then define the action-value distribution as DISPLAYFORM0 Observe that this distributional layer simply corresponds to a linear layer from the critic torso to the logits ω, followed by a softmax activation (see FIG5, left).However, this distribution is not closed under the Bellman operator defined earlier, due to the fact that adding and scaling these values will no longer lie on the support defined by the atoms. This support is explicitly defined by the (V min, V max) hyperparameters. As a we instead use a projected version of the distributional Bellman operator BID0; see Appendix B for more details. Letting p 1 be the probabilities of the projected distributional Bellman operator ΦT π applied to some target distribution Z target, we can write the loss in terms of the cross-entropy DISPLAYFORM1 Mixture of Gaussians We can also consider parameterizing the action-value distribution using a mixture of Gaussians; here the random variable Z has density given by DISPLAYFORM2 Thus, the distribution layer maps, through a linear layer, from the critic torso to the mixture weight ω i, mean µ i, and variance σ 2 i for each mixture component 0 ď i ď ´1 (see FIG5, center). We can then specify a loss corresponding to the cross-entropy portion of the KL divergence between two distributions. Given a sample transition px, a, r, x 1 q we can take samples from the target density z j " p target and approximate the cross-entropy term using DISPLAYFORM3 B CATEGORICAL PROJECTION OPERATORThe categorical parameterized distribution has finite support. Thus, the of applying the distributional Bellman equation will generally not coincide with this support. Therefore, some projection Figure 8: Results for using a mixture of Gaussians distribution on select control suite tasks. Shown are two learning rates as denoted in the legends as well as Categorical.step is required before minimizing the cross-entropy. The categorical projection of Bellemare et al. FORMULA0 is given by pΦpq i " ř ´1 j"0 h zi pz j qp j, @i, where h is a piecewise linear'hat' function, DISPLAYFORM4 In Figure 8 we display of running D4PG on a selection of control suite tasks using a mixture of Gaussians output distribution for two choices of learning rates. Here the distributional TD loss is minimized using the sample-based KL introduced earlier. While this is definitely a technique that is worth further exploration, we found in initial experiments that this choice of distribution underperformed the Categorical distribution by a fair margin. This lends further credence to the choice of distribution made in BID0. In this section we provide further details for the control suite domains. In particular see Figure 9 for images of the control suite tasks. The physics state S, action A, and observation X dimensionalities for each task are provided in Table 1. For the dexterous manipulation tasks we used a simulated model of the Johns Hopkins Modular Prosthetic Limb hand BID6 implemented in MuJoCo BID8. This anthropomorphic hand has a total of 22 degrees of freedom (19 in the fingers, 3 in the wrist), which are driven by a set of 13 position actuators (PD-controllers). The underactuation of the hand is due to coupling between some of the finger joints. For these experiments the wrist was positioned in a fixed location above a table, such that rotation and flexion about the wrist joints allowed the hand to pick up objects from the table, rotate into a palm-up position, and then manipulate them. We focused on a set of three tasks where the agent must learn to manipulate a cylindrical object FIG0 ). In each of these tasks, the observations contain the positions and velocities of all of the joints in the hand, the current position targets for the actuators in the hand, the position and quaternion of the object being manipulated, and its translational and rotational velocities. The Table 2: Observation components given in each of the manipulation tasks, and their corresponding dimensionalities. Here sin z, cos z refers to the sine and cosine of the target frame's angle of rotation about the z-axis. Figure 10: Sequences of frames illustrating the dexterous manipulation tasks we attempt to solve using D4PG. Top to bottom:'catch','pick-up-and-orient','rotate-in-hand'. The translucent objects shown in'pick-up-and-orient' and'rotate-in-hand' represent the goal states. observations given in each task are summarized in Table 2. The agent's actions are increments applied to the position targets for the actuators. In the'catch' task the agent must learn to catch a falling object before it strikes the table below. The position, height, and orientation of the object are randomly initialized at the start of each episode. The reward is given by r " ψppalm height´o bj height; c, mqwhere ψp; c, mq is a soft indicator function similar to one described by BID2 ψp; c, mq " " 1´tanhp w m q 2 if ą c, 1 otherwise. Here w " tanh´1p? 0.95q, and the tolerance c and margin m parameters are 0 cm and 5 cm respectively. Contact between the object and the table causes the current episode to terminate immediately with no reward, otherwise it will continue until a 500 step limit is reached. In the'pick-up-and-orient' task, the agent must pick up a cylindrical object from the table and maneuver it into a target position and orientation. Both the initial position and orientation of the object, and the position and orientation of the target are randomized between episodes. The reward function consists of two additive components that depend on the distance from the object to the target position, and on the angle between the z-axes of the object and target body frames where c pos =1 cm, m pos =5 cm, c ori =5˝, m ori =10˝. Note that the distance-dependent component of the reward multiplicatively gates the orientation component. This helps to encourage the agent to pick up the object before attempting to orient it to match the target. Each episode has a fixed duration of 500 steps. Finally, in the'rotate-in-hand' task the agent begins with a broad cylinder in its palm, and must rotate it axially in order to match a moving target. This requires dynamically forming and breaking contacts with the object being manipulated. The target angle is initialized uniformly, and then incremented on each time step using temporally correlated noise drawn from an Ornstein-Uhlenbeck process (σ=0.025˝, θ=0.01; Uhlenbeck & Ornstein 1930). The reward consists of two multiplicative components r " ψpcos´1pobj yaxis||xy, obj target yaxis||xy qq; c rot, m rot qψpcos´1pobj zaxis, obj target zaxis q; c ori, m ori q FORMULA0 where c rot =5˝, m rot =40˝, c ori =45˝, m ori =45˝, and ||xy denotes projection onto the global xy plane. The first component provides an incentive to match the axial rotation of the target, and the second component penalizes the agent for allowing the orientation of the cylinder's long axis to deviate too far from that of the target. The maximum episode duration is 1000 steps, with early termination if the object makes contact with the table.

We develop an agent that we call the Distributional Deterministic Deep Policy Gradient algorithm, which achieves state of the art performance on a number of challenging continuous control problems.

We isolate one factor of RL generalization by analyzing the case when the agent only overfits to the observations. We show that architectural implicit regularizations occur in this regime.


Question: Represent the scientific article to find a short TLDR

Question: State-action value functions (i.e., Q-values) are ubiquitous in reinforcement learning (RL), giving rise to popular algorithms such as SARSA and Q-learning. We propose a new notion of action value defined by a Gaussian smoothed version of the expected Q-value used in SARSA. We show that such smoothed Q-values still satisfy a Bellman equation, making them naturally learnable from experience sampled from an environment. Moreover, the gradients of expected reward with respect to the mean and covariance of a parameterized Gaussian policy can be recovered from the gradient and Hessian of the smoothed Q-value function. Based on these relationships we develop new algorithms for training a Gaussian policy directly from a learned Q-value approximator. The approach is also amenable to proximal optimization techniques by augmenting the objective with a penalty on KL-divergence from a previous policy. We find that the ability to learn both a mean and covariance during training allows this approach to achieve strong on standard continuous control benchmarks. Model-free reinforcement learning algorithms often alternate between two concurrent but interacting processes: policy evaluation, where an action value function (i.e., a Q-value) is updated to obtain a better estimate of the return associated with taking a specific action, and policy improvement, where the policy is updated aiming to maximize the current value function. In the past, different notions of Q-value have led to distinct but important families of RL methods. For example, SARSA BID18 BID22 BID26 ) uses the expected Q-value, defined as the expected return of following the current policy. Q-learning BID28 ) exploits a hard-max notion of Q-value, defined as the expected return of following an optimal policy. Soft Q-learning BID7 and PCL BID14 both use a soft-max form of Q-value, defined as the future return of following an optimal entropy regularized policy. Clearly, the choice of Q-value function has a considerable effect on the ing algorithm; for example, restricting the types of policies that can be expressed, and determining the type of exploration that can be naturally applied. In this work we introduce a new notion of action value: the smoothed action value functionQ π. Unlike previous notions, which associate a value with a specific action at each state, the smoothed Qvalue associates a value with a specific distribution over actions. In particular, the smoothed Q-value of a state-action pair (s, a) is defined as the expected return of first taking an action sampled from a normal distribution N (a, Σ(s)), centered at a, then following actions sampled from the current policy thereafter. In this way, the smoothed Q-value can also be interpreted as a Gaussian-smoothed or noisy version of the expected Q-value. We show that smoothed Q-values possess a number of interesting properties that make them attractive for use in RL algorithms. For one, the smoothed Q-values satisfy a single-step Bellman consistency, which allows bootstrapping to be used to train a function approximator. Secondly, for Gaussian policies, the standard optimization objective (expected return) can be expressed in terms of smoothed Q-values. Moreover, the gradient of this objective with respect to the mean and covariance of the Gaussian policy is equivalent to the gradient and the Hessian of the smoothed Q-value function, which allows one to derive updates to the policy parameters by having access to the derivatives of a sufficiently accurate smoothed Q-value function. This observation leads us to propose an algorithm called Smoothie, which in the spirit of (Deep) Deterministic Policy Gradient (DDPG) BID21 BID11, trains a policy using the derivatives of a trained (smoothed) Q-value function, thus avoiding the high-variance of stochastic updates used in standard policy gradient algorithms BID29 BID10. Unlike DDPG, which is well-known to have poor exploratory behavior BID7, the approach we develop is able to utilize a non-deterministic Gaussian policy parameterized by both a mean and a covariance, thus allowing the policy to be exploratory by default and alleviating the need for excessive hyperparameter tuning. Furthermore, we show that Smoothie can be easily adapted to incorporate proximal policy optimization techniques by augmenting the objective with a penalty on KL-divergence from a previous version of the policy. The inclusion of a KL-penalty is not feasible in the standard DDPG algorithm, but we show that it is possible with our formulation, and it significantly improves stability and overall performance. On standard continuous control benchmarks, our are competitive with or exceed state-of-the-art, especially for more difficult tasks in the low-data regime. We consider the standard model-free RL framework, where an agent interacts with a stochastic black-box environment by sequentially observing the state of the environment, emitting an action, and receiving a reward feedback; the goal is to find an agent that achieves maximal cumulative discounted reward. This problem can be expressed in terms of a Markov decision process (MDP) that consists of a state space S and an action space A, where at iteration t the agent encounters a state s t ∈ S and emits an action a t ∈ A, after which the environment returns a scalar reward r t ∼ R(s t, a t) and places the agent in a new state s t+1 ∼ P (s t, a t).We model the behavior of the agent using a stochastic policy π that produces a distribution over feasible actions at each state s as π(a | s). The optimization objective (expected discounted return), as a function of the policy, can then be expressed in terms of the expected action value function Q π (s, a) by, DISPLAYFORM0 where ρ π (s) is the stationary distribution of the states under π, and Q π (s, a) is recursively defined using the Bellman equation, DISPLAYFORM1 where γ ∈ is the discount factor. For brevity, we will often suppress explicit denotation of the sampling distribution R over immediate rewards and the distribution P over state transitions. The policy gradient theorem BID23 expresses the gradient of O ER (π θ) w.r.t. θ, the tunable parameters of a policy π θ, as, DISPLAYFORM2 Many reinforcement learning algorithms, including policy gradient and actor-critic variants, trade off variance and bias when estimating the random variable inside the expectation in; for example, by attempting to estimate Q π (s, a) accurately using function approximation. In the simplest scenario, an unbiased estimate of Q π (s, a) is formed by accumulating discounted rewards from each state forward using a single Monte Carlo sample. In this paper, we focus on multivariate Gaussian policies over continuous action spaces, A ≡ R da. We represent the observed state of the MDP as a d s -dimensional feature vector Φ(s) ∈ R ds, and parametrize the Gaussian policy by a mean and covariance function, respectively µ(s): DISPLAYFORM3 These map the observed state of the environment to a Gaussian distribution, DISPLAYFORM4 where DISPLAYFORM5 Below we develop new RL training methods for this family of parametric policies, but some of the ideas presented may generalize to other families of policies as well. We begin the formulation by reviewing some prior work on learning Gaussian policies. BID21 present a new formulation of the policy gradient, called the deterministic policy gradient, for the family of Gaussian policies in the limit where the policy covariance approaches zero. In such a scenario, the policy becomes deterministic because sampling from the policy always returns the Gaussian mean. The key observation of BID21 is that under a deterministic policy π ≡ (µ, Σ → 0), one can estimate the expected future return from a state s as, Then, one can express the gradient of the optimization objective (expected discounted return) for a parameterized π θ ≡ µ θ as, DISPLAYFORM0 This can be thought of as a characterization of the policy gradient theorem for deterministic policies. In the limit of Σ → 0, one can also re-express the Bellman equation FORMULA1 as, DISPLAYFORM1 Therefore, a value function approximator Q π w can be optimized by minimizing the Bellman error, DISPLAYFORM2 for transitions (s, a, r, s) sampled from a dataset D of interactions of the agent with the environment. Algorithms like DDPG BID11 ) alternate between improving the value function by gradient descent on, and improving the policy based on.In practice, to gain better sample efficiency, BID5 and BID21 replace the on-policy state distribution ρ π (s) in with an off-policy distribution ρ β (s) based on a replay buffer. After this substitution, the policy gradient identity in does not hold exactly, however, prior work finds that this works well in practice and improves sample efficiency. We also adopt a similar approximation in our method to make use of off-policy data. In this paper, we introduce smoothed action value functions, the gradients of which provide an effective signal for optimizing the parameters of a Gaussian policy. Our notion of smoothed Qvalues, denotedQ π (s, a), differs from ordinary Q-values Q π (s, a) in that smoothed Q-values do not assume the first action of the agent is fully specified, but rather they assume that only the mean of the distribution of the first action is known. Hence, to computeQ π (s, a), one has to perform an expectation of Q π (s,ã) for actionsã drawn in the vicinity of a. More formally, smoothed action values are defined as, DISPLAYFORM0 With this definition ofQ π, one can re-express the expected reward objective for a Gaussian policy π ≡ (µ, Σ) as, DISPLAYFORM1 The insight that differentiates this approach from prior work including BID8; BID4 is that instead of learning a function approximator for Q π (s, a) and then drawing samples to approximate the expectation in and its derivative, we directly learn a function approximator forQ π (s, a).The key observation that enables direct bootstrapping of smoothed Q-values,Q π (s, a), is that their form allows a notion of Bellman consistency. First, note that for Gaussian policies π ≡ (µ, Σ) we have DISPLAYFORM2 Then, combining FORMULA0 and FORMULA0, one can derive the following one-step Bellman equation for smoothed Q-values, DISPLAYFORM3 wherer ands are sampled from R(s,ã) and P (s,ã). Below, we elaborate on how one can make use of the derivatives ofQ π to learn µ and Σ, and how the Bellman equation in enables direct optimization ofQ π. We parameterize a Gaussian policy π θ,φ ≡ (µ θ, Σ φ) in terms of two sets of parameters θ and φ for the mean and the covariance. The gradient of the objective w.r.t. mean parameters follows from the policy gradient theorem and is almost identical to, DISPLAYFORM0 Estimating the derivative of the objective w.r.t. covariance parameters is not as straightforward, sincẽ Q π is not a direct function of Σ. However, a key observation of this work is that the second derivative ofQ π w.r.t. actions is sufficient to exactly compute the derivative ofQ π w.r.t. Σ, DISPLAYFORM1 A proof of this identity is provided in the Appendix. The proof may be easily derived by expressing both sides of the equation using standard matrix calculus like DISPLAYFORM2 Then, the full derivative w.r.t. φ takes the form, DISPLAYFORM3 We can think of two ways to optimizeQ 2 whereã ∼ N (a, Σ(s)), using several samples. When the target values in these residuals are treated as fixed (i.e., using a target network), such a training procedure will achieve a fixed point whenQ π w (s, a) satisfies the recursion in the Bellman equation.The second approach requires a single function approximator forQ π w (s, a), ing in a simpler implementation, and thus we use this approach in our experimental evaluation. Suppose one has access to a tuple (s,ã,r,s) sampled from a replay buffer with knowledge of the sampling probability q(ã | s) (possibly unnormalized). Then assuming that this sampling distribution has a full support, we draw a phantom action a ∼ N (ã, Σ(s)) and optimizeQ π w (s, a) by minimizing a weighted Bellman error DISPLAYFORM0 2. For a specific pair of state and action (s, a) the expected value of the objective is, DISPLAYFORM1 Note that N (a|ã, Σ(s)) = N (ã|a, Σ(s)). Therefore, when the target valuer + γQ π w (s, µ(s)) is treated as fixed (e.g., when using target networks) this training procedure reaches an optimum wheñ Q π w (s, a) satisfies the recursion in the Bellman equation FORMULA0. In practice, we find that it is unnecessary to keep track of the probabilities q(ã | s), and assume the replay buffer provides a near-uniform distribution of actions conditioned on states. Other recent work has also benefited from ignoring or heavily damping importance weights BID13 BID27 BID20. However, it is possible when interacting with the environment to save the probability of sampled actions along with their transitions, and thus have access to q(ã | s) ≈ N (ã | µ old (s), Σ old (s)). Policy gradient algorithms are notoriously unstable, particularly in continuous control problems. Such instability has motivated the development of trust region methods that attempt to mitigate the issue by constraining each gradient step to lie within a trust region BID19, or augmenting the expected reward objective with a penalty on KL-divergence from a previous policy BID15 BID20 BID0. These stabilizing techniques have thus far not been applicable to algorithms like DDPG, since the policy is deterministic. The formulation we propose in this paper, however, is easily amenable to trust region optimization. Specifically, we may augment the objective with a penalty DISPLAYFORM0 where π old ≡ (µ old, Σ old) is a previous parameterization of the policy. The optimization is straightforward, since the KL-divergence of two Gaussians can be expressed analytically. This paper follows a long line of work that uses Q-value functions to stably learn a policy, which in the past has been used to either approximate expected BID18 BID26 BID6 or optimal BID28 BID21 BID14 BID7 BID12 future value. Work that is most similar to what we present are methods that exploit gradient information from the Q-value function to train a policy. Deterministic policy gradient BID21 is perhaps the best known of these. The method we propose can be interpreted as a generalization of the deterministic policy gradient. Indeed, if one takes the limit of the policy covariance Σ(s) as it goes to 0, the proposed Q-value function becomes the deterministic value function of DDPG, and the updates for training the Q-value approximator and the policy mean are identical. Stochastic Value Gradient (SVG) BID8 ) also trains stochastic policies using an update that is similar to DDPG (i.e., SVG with replay). The key differences with our approach are that SVG does not provide an update for the covariance, and the mean update in SVG estimates the gradient with a noisy Monte Carlo sample, which we avoid by estimating the smoothed Q-value function. Although a covariance update could be derived using the same reparameterization trick as in the mean update, that would also require a noisy Monte Carlo estimate. Methods for updating the covariance along the gradient of expected reward are essential for applying the subsequent trust region and proximal policy techniques. More recently, BID4 introduced expected policy gradients (EPG), a generalization of DDPG that provides updates for the mean and covariance of a stochastic Gaussian policy using gradients of an estimated Q-value function. In that work, the expected Q-value used in standard policy gradient algorithms such as SARSA BID22 BID18 BID26 ) is estimated. The updates in EPG therefore require approximating an integral of the expected Q-value function. Our analogous process directly estimates an integral (via the smoothed Q-value function) and avoids approximate integrals, thereby making the updates simpler. Moreover, while BID4 rely on a quadratic Taylor expansion of the estimated Q-value function, we instead rely on the strength of neural network function approximators to directly estimate the smoothed Q-value function. The novel training scheme we propose for learning the covariance of a Gaussian policy relies on properties of Gaussian integrals BID2 BID16. Similar identities have been used in the past to derive updates for variational auto-encoders BID9 and Gaussian back-propagation BID17.Finally, the perspective presented in this paper, where Q-values represent the averaged return of a distribution of actions rather than a single action, is distinct from recent advances in distributional RL BID1. Those approaches focus on the distribution of returns of a single action, whereas we consider the single average return of a distribution of actions. Although we restrict our attention in this paper to Gaussian policies, an interesting topic for further investigation is to study the applicability of this new perspective to a wider class of policy distributions. We utilize the insights from Section 3 to introduce a new RL algorithm, Smoothie. Smoothie maintains a parameterizedQ π w trained via the procedure described in Section 3.2. It then uses the gradient and Hessian of this approximation to train a Gaussian policy µ θ, Σ φ using the updates stated in FORMULA0 and FORMULA0. See Algorithm 1 for a simplified pseudocode of our algorithm. Input: Environment EN V, learning rates η π, η Q, discount factor γ, KL-penalty λ, batch size B, number of training steps N, target network lag τ.Initialize θ, φ, w, set θ = θ, φ = φ, w = w. for i = 0 to N − 1 do // Collect experience Sample action a ∼ N (µ θ (s), Σ φ (s)) and apply to EN V to yield r and s. Insert transition (s, a, r, s) to replay buffer. DISPLAYFORM0 We perform a number of evaluations of Smoothie compared to DDPG. We choose DDPG as a baseline because it utilizes gradient information of a Q-value approximator, much like our algorithm; and is a standard algorithm well-known to have achieve good, sample-efficient performance on continuous control benchmarks. To evaluate Smoothie we begin with a simple synthetic task which allows us to study its behavior in a restricted setting. We devised a simple single-action one-shot environment in which the reward function is a mixture of two Gaussians, one better than the other (see FIG1). We initialize the policy mean to be centered on the worse of the two Gaussians. We plot the learnable policy mean and standard deviation during training for Smoothie and DDPG in FIG1 (Left). Smoothie learns both the mean and variance, while DDPG learns only the mean and the variance plotted is the exploratory noise, whose scale is kept fixed during training. As expected we observe that DDPG cannot escape the local optimum. At the beginning of training it exhibits some movement away from the local optimum (likely due to the initial noisy approximation given by Q π w), it is unable to progress very far from the initial mean. Note that this is not an issue of exploration. The exploration scale is high enough that Q π w is aware of the better Gaussian. The issue is in the update for µ θ, which is only with regard to the derivative of Q π w at the current mean. On the other hand, we find Smoothie is successfully able to solve the task. This is because the smoothed reward function approximated byQ π w has a derivative which clearly points µ θ towards the better Gaussian. We also observe that Smoothie is able to suitably adjust the covariance Σ φ during training. Initially, Σ φ decreases due to the concavity of the smoothed reward function. As a region of convexity is entered, it begins to increase, before again decreasing to near-zero as µ θ approaches the global optimum. The learnable policy mean and standard deviation during training for Smoothie and DDPG on a simple one-shot synthetic task. The standard deviation for DDPG is the exploratory noise kept constant during training. Right: The reward function for the synthetic task along with its Gaussian-smoothed version. We find that Smoothie can successfully escape the lower-reward local optimum. We also notice Smoothie increases and decreases its policy variance as the convexity/concavity of the smoothed reward function changes. We now turn our attention to standard continuous control benchmarks available on OpenAI Gym BID3 utilizing the MuJoCo environment BID24.Our implementations utilize feed forward neural networks for policy and Q-values. We parameterize the covariance Σ φ as a diagonal given by e φ. The exploration for DDPG is determined by an Ornstein-Uhlenbeck process BID25 BID11. Additional implementation details are provided in the Appendix. Each plot shows the average reward and standard deviation clipped at the min and max of six randomly seeded runs after choosing best hyperparameters. We see that Smoothie is competitive with DDPG even when DDPG uses a hyperparameter-tuned noise scale, and Smoothie learns the optimal noise scale (the covariance) during training. Moreoever, we observe significant advantages in terms of final reward performance, especially in the more difficult tasks like Hopper, Walker2d, and Humanoid. Across all tasks, TRPO is not sufficiently sampleefficient to provide a competitive baseline. We compare the of Smoothie and DDPG in FIG2. For each task we performed a hyperparameter search over actor learning rate, critic learning rate and reward scale, and plot the average of six runs for the best hyperparameters. For DDPG we extended the hyperparameter search to also consider the scale and damping of exploratory noise provided by the Ornstein-Uhlenbeck process. Smoothie, on the other hand, contains an additional hyperparameter to determine the weight on KL-penalty. Despite DDPG having the advantage of its exploration decided by a hyperparameter search while Smoothie must learn its exploration without supervision, we find that Smoothie performs competitively or better across all tasks, exhibiting a slight advantage in Swimmer and Ant, while showing more dramatic improvements in Hopper, Walker2d, and Humanoid. The improvement is especially dramatic for Hopper, where the average reward is doubled. We also highlight the for Humanoid, which as far as we know, are the best published for a method that only trains on the order of millions of environment steps. In contrast, TRPO, which to the best of our knowledge is the only other algorithm which can achieve better performance, requires on the order of tens of millions of environment steps to achieve comparable reward. This gives added evidence to the benefits of using a learnable covariance and not restricting a policy to be deterministic. Empirically, we found the introduction of a KL-penalty to improve performance of Smoothie, especially on harder tasks. We present a comparison of of Smoothie with and without the KL-penalty on the four harder tasks in FIG3. A KL-penalty to encourage stability is not possible in DDPG. Thus, our algorithm provides a much needed solution to the inherent instability in DDPG training. We observe benefits of using a proximal policy optimization method, especially in Hopper and Humanoid, where the performance improvement is significant without sacrificing sample efficiency. We have presented a new Q-value function,Q π, that is a Gaussian-smoothed version of the standard expected Q-value, Q π. The advantage of usingQ π over Q π is that its gradient and Hessian possess an intimate relationship with the gradient of expected reward with respect to mean and covariance of a Gaussian policy. The ing algorithm, Smoothie, is able to successfully learn both mean and covariance during training, leading to performance that can match or surpass that of DDPG, especially when incorporating a penalty on divergence from a previous policy. The success ofQ π is encouraging. Intuitively it may be argued that learningQ π is more sensible than learning Q π. The smoothed Q-values by definition make the true reward surface smoother, thus possibly easier to learn; moreover the smoothed Q-values have a more direct relationship with the expected discounted return objective. We encourage future work to further investigate these claims as well as techniques to apply the underlying motivations forQ π to other types of policies. A PROOF OF EQUATION FORMULA0 We note that similar identities for Gaussian integrals exist in the literature BID16 BID17 and point the reader to these works for further information. The specific identity we state may be derived using standard matrix calculus. We make use of the fact that DISPLAYFORM0 and for symmetric A, ∂ ∂A ||v|| DISPLAYFORM1 We omit s from Σ(s) in the following equations for succinctness. The LHS of FORMULA0 Meanwhile, towards tackling the RHS of FORMULA0 we note that DISPLAYFORM2 Thus we have DISPLAYFORM3

Represent the short summary for finding the corresponding scientific article (long)

We propose a new Q-value function that enables better learning of Gaussian policies.

represent the short summary for finding the corresponding scientific article (long)

We propose Dual Actor-Critic algorithm, which is derived in a principled way from the Lagrangian dual form of the Bellman optimality equation. The algorithm achieves the state-of-the-art performances across several benchmarks.


Question:
represent the scientific article to find a short tldr

E.g. 'Plan recognition aims to look for target plans to best explain the observed actions based on plan libraries and/or domain models. Despite the success of previous approaches on plan recognition, they mostly rely on correct action observations. Recent advances in visual activity recognition have the potential of enabling applications such as automated video surveillance. Effective approaches for such problems would require the ability to recognize the plans of agents from video information. Traditional plan recognition algorithms rely on access to detailed planning domain models. One recent promising direction involves learning approximate (or shallow) domain models directly from the observed activity sequences. Such plan recognition approaches expect observed action sequences as inputs. However, visual inference are often noisy and uncertain, typically represented as a distribution over possible actions. In this work, we develop a visual plan recognition framework that recognizes plans with an approximate domain model learned from uncertain visual data.' == 'Handling Uncertainty in Visual Perception for Plan Recognition' != 'We investigate a framework for discovery: curating a large collection of predictions, which are used to construct the agent’s representation in partially observable domains.'

Question: Interactive Fiction games are text-based simulations in which an agent interacts with the world purely through natural language. They are ideal environments for studying how to extend reinforcement learning agents to meet the challenges of natural language understanding, partial observability, and action generation in combinatorially-large text-based action spaces. We present KG-A2C, an agent that builds a dynamic knowledge graph while exploring and generates actions using a template-based action space. We contend that the dual uses of the knowledge graph to reason about game state and to constrain natural language generation are the keys to scalable exploration of combinatorially large natural language actions. Results across a wide variety of IF games show that KG-A2C outperforms current IF agents despite the exponential increase in action space size. Natural language communication has long been considered a defining characteristic of human intelligence. We are motivated by the question of how learning agents can understand and generate contextually relevant natural language in service of achieving a goal. In pursuit of this objective we study Interactive Fiction (IF) games, or text-adventures: simulations in which an agent interacts with the world purely through natural language-"seeing" and "talking" to the world using textual descriptions and commands. To progress in these games, an agent must generate natural language actions that are coherent, contextually relevant, and able to effect the desired change in the world. Complicating the problem of generating contextually relevant language in these games is the issue of partial observability: the fact that the agent never has access to the true underlying world state. IF games are structured as puzzles and often consist of an complex, interconnected web of distinct locations, objects, and characters. The agent needs to thus reason about the complexities of such a world solely through the textual descriptions that it receives, descriptions that are often incomplete. Further, an agent must be able to perform commonsense reasoning-IF games assume that human players possess prior commonsense and thematic knowledge (e.g. knowing that swords can kill trolls or that trolls live in dark places). Knowledge graphs provide us with an intuitive way of representing these partially observable worlds. Prior works have shown how using knowledge graphs aids in the twin issues of partial observability (a) and commonsense reasoning (b), but do not use them in the context of generating natural language. To gain a sense for the challenges surrounding natural language generation, we need to first understand how large this space really is. In order to solve solve a popular IF game such as Zork1 it's necessary to generate actions consisting of up to five-words from a relatively modest vocabulary of 697 words recognized by Zork's parser. Even this modestly sized vocabulary leads to O(697 5) = 1.64 × 10 the structure required to further constrain our action space via our knowledge graph-and make the argument that the combination of these approaches allows us to generate meaningful natural language commands. Our contributions are as follows: We introduce an novel agent that utilizes both a knowledge graph based state space and template based action space and show how to train such an agent. We then conduct an empirical study evaluating our agent across a diverse set of IF games followed by an ablation analysis studying the effectiveness of various components of our algorithm as well as its overall generalizability. Remarkably we show that our agent achieves state-of-the-art performance on a large proportion of the games despite the exponential increase in action space size. We examine prior work in three broad categories: text-based game playing agents and frameworks as well as knowledge graphs used for natural language generation and game playing agents. LSTM-DQN , considers verb-noun actions up to two-words in length. Separate Q-Value estimates are produced for each possible verb and object, and the action consists of pairing the maximally valued verb combined with the maximally valued object. The DRRN algorithm for choice-based games estimates Q-Values for a particular action from a particular state. Fulda et al. use Word2Vec (to aid in extracting affordances for items in these games and use this information to produce relevant action verbs. reduce the combinatorially-sized action space into a discrete form using a walkthrough of the game and introduce the Action Elimination DQN, which learns to eliminate actions unlikely to cause a world change. Côté et al. introduce TextWorld, a framework for procedurally generating parser-based games, allowing a user to control the difficulty of a generated game. , an optimized interface for playing human-made IF games-formalizing this task. They further provide a comparative study of various types of agents on their set of games, testing the performance of heuristic based agents such as NAIL (b) and various reinforcement learning agents are benchmarked. We use Jericho and the tools that it provides to develop our agents. Knowledge graphs have been shown to be useful representations for a variety of tasks surrounding natural language generation and interactive fiction. and effectively use knowledge graph representations to improve neural conversational and story ending prediction models respectively. explore procedural content generation in text-adventure games-looking at constructing a quest for a given game world, and use knowledge graphs to ground generative systems trained to produce quest content. From the perspective of text-game playing agent and most in line with the spirit of our work, Ammanabrolu & Riedl (2019a) present the Knowledge Graph DQN or KG-DQN, an approach where a knowledge graph built during exploration is used as a state representation for a deep reinforcement learning based agent. Ammanabrolu & Riedl (2019b) further expand on this work, exploring methods of transferring control policies in text-games, using knowledge graphs to seed an agent with useful commonsense knowledge and to transfer knowledge between different games within a domain. Both of these works, however, identify a discrete set of actions required to play the game beforehand and so do not fully tackle the issue of the combinatorial action space. Formally, IF games are partially observable Markov decision processes (POMDP), represented as a 7-tuple of S, T, A, Ω, O, R, γ representing the set of environment states, mostly deterministic conditional transition probabilities between states, the vocabulary or words used to compose text commands, observations returned by the game, observation conditional probabilities, reward function, and the discount factor respectively (Côté et al., 2018; a). To deal with the ing twin challenges of partial observability and combinatorial actions, we use a knowledge graph based state space and a template-based action space-each described in detail below. Knowledge Graph State Space. Building on Ammanabrolu & Riedl (2019a), we use a knowledge graph as a state representation that is learnt during exploration. The knowledge graph is stored as a set of 3-tuples of subject, relation, object. These triples are extracted from the observations using Stanford's Open Information Extraction (OpenIE) . Human-made IF games often contain relatively complex semi-structured information that OpenIE is not designed to parse and so we add additional rules to ensure that we are parsing the relevant information. Updated after every action, the knowledge graph helps the agent form a map of the world that it is exploring, in addition to retaining information that it has learned such as the affordances associated with an object, the properties of a character, current inventory, etc. Nodes relating to such information are shown on the basis of their relation to the agent which is presented on the graph using a "you" node (see example in Fig. 2a). Ammanabrolu & Riedl (2019a) build a knowledge graph in a similar manner but restrict themselves to a single domain. In contrast, we test our methods on a much more diverse set of games defined in the Jericho framework (a). These games are each structured differentlycovering a wider variety of genres-and so to be able to extract the same information from all of them in a general manner, we relax many of the rules found in Ammanabrolu & Riedl (2019a). To aid in the generalizability of graph building, we introduce the concept of interactive objects-items that an agent is able to directly interact with in the surrounding environment. These items are directly linked to the "you" node, indicating that the agent can interact with them, and the node for the current room, showing their relative position. All other triples built from the graph are extracted by OpenIE. Further details regarding knowledge graph updates are found in Appendix A.1 An example of a graph built using these rules is seen in Fig. 2a. Template Action Space. Templates are subroutines used by the game's parser to interpret the player's action. They consist of interchangeable verbs phrases (V P) optionally followed by prepositional phrases (V P P P), e.g. Figure 2b, actions may be constructed from templates by filling in the template's blanks using words in the game's vocabulary. Templates and vocabulary words are programmatically accessible through the Jericho framework and are thus available for every IF game. Further details about how we prioritize interchangeable verbs and prepositions are available in Appendix A.2. Combining the knowledge-graph state space with the template action space, Knowledge Graph Advantage Actor Critic or KG-A2C, is an on-policy reinforcement learning agent that collects experience from many parallel environments. We first discuss the architecture of KG-A2C, then detail the training algorithm. As seen in Fig. 1, KG-A2C's architecture can broadly be described in terms of encoding a state representation and then using this encoded representation to decode an action. We describe each of these processes below. Input Representation. The input representation network is broadly divided into three parts: an observation encoder, a score encoder, and the knowledge graph. At every step an observation consisting of several components is received: o t = (o t desc, o tgame, o tinv, a t−1) corresponding to the room description, game feedback, inventory, and previous action, and total score R t. The room description o t desc is a textual description of the agent's location, obtained by executing the command "look." The game feedback o tgame is the simulators response to the agent's previous action and consists of narrative and flavor text. The inventory o tinv and previous action a t−1 components inform the agent about the contents of its inventory and the last action taken respectively. The observation encoder processes each component of o t using a separate GRU encoder. As we are not given the vocabulary that o t is comprised of, we use subword tokenization-specifically using the unigram subword tokenization method described in. This method predicts the most likely sequence of subword tokens for a given input using a unigram language and contains a total vocabulary of size 8000. For each of the GRUs, we pass in the final hidden state of the GRU at step t − 1 to initialize the hidden state at step t. We concatenate each of the encoded components and use a linear layer to combine them into the final encoded observation o t. At each step, we update our knowledge graph G t using o t as described in Sec. 3 and it is then embedded into a single vector g t. Following Ammanabrolu & Riedl (2019a) we use Graph Attention networks or GATs (Veličković et al., 2018) with an attention mechanism similar to that described in. Node features are computed as, where N is the number of nodes and F the number of features in each node, consist of the average subword embeddings of the entity and of the relations for all incoming edges using our unigram language model. Self-attention is then used after a learnable linear transformation W ∈ IR 2F×F applied to all the node features. Attention coefficients α ij are then computed by softmaxing k ∈ N with N being the neighborhood in which we compute the attention coefficients and consists of all edges in G t. where p ∈ IR is a learnable parameter. The final knowledge graph embedding vector g t is computed as: where k refers to the parameters of the k th independent attention mechanism, W g and b g the weights and biases of the output linear layer, and represents concatenation. The final component of state embedding vector is a binary encoding c t of the total score obtained so far in the game-giving the agent a sense for how far it has progressed in the game even when it is not collecting reward. The state embedding vector is then calculated as s t = g t ⊕ o t ⊕ c t. Action Decoder. The state embedding vector s t is then used to sequentially construct an action by first predicting a template and then picking the objects to fill into the template using a series of Decoder GRUs. This gives rise to a template policy π T and a policy for each object π Oi. Architecture You are in the living room. There is a doorway to the east, a wooden door with strange gothic lettering to the west, which appears to be nailed shut, a trophy case, and a large oriental rug in the center of the room. Above the trophy case hangs an elvish sword of great antiquity. A batterypowered brass lantern is on the trophy case. You are carrying: A glass bottle The glass bottle contains: A quantity of water. Figure 2: An overall example of the knowledge graph building and subsequent action decoding process for a given state in Zork1, illustrating the use of interactive objects and the graph mask. wise, at every decoding step all previously predicted parts of the action are encoded and passed along with s t through an attention layer which learns to attend over these representations-conditioning every predicted object on all the previously predicted objects and template. All the object decoder GRUs share parameters while the template decoder GRU T remains separate. To effectively constrain the space of template-actions, we introduce the concept of a graph mask, leveraging our knowledge graph at that timestep G t to streamline the object decoding process. Formally, the graph mask m t = {o : o ∈ G t ∧ o ∈ V}, consists of all the entities found within the knowledge graph G t and vocabulary V and is applied to the outputs of the object decoder GRUsrestricting them to predict objects in the mask. Generally, in an IF game, it is impossible to interact with an object that you never seen or that are not in your inventory and so the mask lets us explore the action space more efficiently. To account for cases where this assumption does not hold, i.e. when an object that the agent has never interacted with before must be referenced in order to progress in the game, we randomly add objects o ∈ V to m t with a probability p m. An example of the graph-constrained action decoding process is illustrated in Fig. 2b. We adapt the Advantage Actor Critic (A2C) method to train our network, using multiple workers to gather experiences from the simulator, making several significant changes along the way-as described below. Valid Actions. Using a template-action space there are millions of possible actions at each step. Most of these actions do not make sense, are ungrammatical, etc. and an even fewer number of them actually cause the agent effect change in the world. Without any sense for which actions present valid interactions with the world, the combinatorial action space becomes prohibitively large for effective exploration. We thus use the concept of valid actions, actions that can change the world in a particular state. These actions can usually be recognized through the game feedback, with responses like "Nothing happens" or "That phrase is not recognized." In practice, we follow Hausknecht et al. (2019a) and use the valid action detection algorithm provided by Jericho. Formally, V alid(s t) = a 0, a 1...a N and from this we can construct the corresponding set of valid templates We further define a set of valid objects O valid (s t) = o 0, o 1...o M which consists of all objects in the graph mask as defined in Sec. 4. This lets us introduce two cross-entropy loss terms to aid the action decoding process. The template loss given a particular state and current network parameters is applied to the decoder GRU T. Similarly, the object loss is applied across the decoder GRU O and is calculated by summing cross-entropy loss from all the object decoding steps. Updates. A2C training starts with calculating the advantage of taking an action in a state A(s t, a t), defined as the value of taking an action Q(s t, a t) compared to the average value of taking all possible valid actions in that state V (s t): V (s t) is predicted by the critic as shown in Fig. 1 and r t is the reward received at step t. The action decoder or actor is then updated according to the gradient: updating the template policy π T and object policies π Oi based on the fact that each step in the action decoding process is conditioned on all the previously decoded portions. The critic is updated with respect to the gradient: bringing the critic's prediction of the value of being in a state closer to its true underlying value. We further add an entropy loss over the valid actions, designed to prevent the agent from prematurely converging on a trajectory. Our experiments are structured into two parts: We first present a comprehensive set of ablations designed to test the relative effectiveness of the various parts of our algorithm. The full KG-A2C is then tested on a suite of Jericho supported games and is compared to strong, established baselines. Additionally, as encouraged by Hausknecht et al. (2019a), we present the set of handicaps used by our agents: Jericho's ability to identify valid actions and the Load, Save handicap in order to acquire o t desc and o tinv using the look and inventory commands without changing the game state. Hyperparameters are provided in Appendix B. Ablation Study. Our ablation study is performed on Zork1, identified by Hausknecht et al. (2019a) to be one of the most difficult games in their suite and the subject of much prior work . Zork1 is one of the earliest IF games and is a dungeon-crawler-a player must explore a vast labyrinth while fighting off enemies and complete puzzles in order to collect treasures. It features a relatively sparse reward for collecting a treasure or moving along the right path to one, and stochasticity in terms of random enemy movements. In order to understand the contributions of different components of KG-A2C's architecture, ablate KG-A2C's knowledge graph, template-action space, and valid-action loss. In order to understand the effects of using a knowledge graph, LSTM-A2C removes all components of KG-A2C's knowledge graph. In particular, the state embedding vector is now computed as s t = o t ⊕ c t and the graph mask is not used to constrain action decoding. LSTM-A2C-masked The same as LSTM-A2C but we use interactive objects to provide an object mask that is used in the same manner as the graph mask of KG-A2C. KG-A2C-seq discards the template action space and instead decodes actions word by word up to a maximum of four words. A supervised cross-entropy-based valid action loss L V alid is now calculated by selecting a random valid action a t valid ∈ V alid(s t) and using each token in it as a target label. As this action space is orders of magnitude larger than template actions, we use teacher-forcing to enable more effective exploration while training the agent-executing a t valid with a probability p valid = 0.5 and the decoded action otherwise. In order to understand the importance of training with valid-actions, KG-A2C-unsupervised is not allowed to access the list of valid actions-the valid-action-losses L T and L O are disabled and L E now based on the full action set. Thus, the agent must explore the template action space manually. Template DQN Baseline. TDQN (a) is an extension of LSTM-DQN to template-based action spaces. This is accomplished using three output heads: one for estimating the Q-Values over templates Q(s t, u)∀u ∈ T and two for estimating Q-Values Q(s t, o 1), Q(s t, o 2)∀o i ∈ O over vocabulary to fill in the blanks of the template. The final executed action is constructed by greedily sampling from the predicted Q-values. To understand how humans progress in Zork1, a group of 10 human players-familiar with IF games-were asked to play Zork1 for the first time (with no access to walkthroughs). Half of the players reached a game score of around 40 before dying to the first beatable NPC, a troll, mostly due to neglecting to collect a weapon to fight it with beforehand. Three of the remaining players died to hidden traps even before reaching this point, achieving scores between 5 and 15. The final two players made it significantly past the troll gaining scores of around 70. A map of Zork1 with annotated rewards can be found in Appendix C and additional learning curves can be found in Appendix B. With this in mind, we first discuss the of the ablation study and then KG-A2C's performance over a much wider set of games found in Jericho. On Zork1, the full KG-A2C significantly outperforms all baselines and ablations as seen in Table 3b -indicating that all components of the full algorithm previously introduced are crucial for its performance. The first two possible rewards that can be received in this game are of magnitude 5 and 10, both requiring 4 steps to reach from the starting point when following an optimal policy. This is the extent of the progress of both the LSTM-A2C and the KG-A2C-seq. The LSTM-A2C more often than not collects both of these rewards while the KG-A2C-seq usually only collects one or the other in the span of an episode. KG-A2C-seq, using a action space consisting of the full vocabulary, performs significantly worse than the rest of the agents even when given the handicaps of teacher forcing and being allowed to train for significantly longer-indicating that the template based action space is necessary for effective exploration. The LSTM-A2C-masked progresses significantly further in the game due to the object mask cutting down the action space, allowing for more efficient exploration. It does not, however, perform as well as KG-A2C likely due to the lack of the graph component g t in the state embedding. LSTM-A2C and TDQN, which use the template-based action space without a knowledge graph, also struggle to progress beyond the initial rewards. Without a knowledge graph to maintain a belief over the world state and constrain the action generation, the agent is unable to produce contextually relevant commands. Thus both the templates and the knowledge graph are critical for the agent to attain state-of-the-art performance. The final component being tested, the valid action supervised loss does not appear to be a important as our choice of state and action spaces: KG-A2C-unsupervised achieves nearly comparable performance to the full algorithm, also achieving state-of-the-art when compared to prior agents, Tabula rasa reinforcement learning offers an intuitive paradigm for exploring goal driven, contextually aware natural language generation. The sheer size of the natural language action space, however, has proven to be out of the reach of existing algorithms. In this paper we introduced KG-A2C, a novel learning agent that demonstrates the feasibility of scaling reinforcement learning towards natural language actions spaces with hundreds of millions of actions. The key insight to being able to efficiently explore such large spaces is the combination of a knowledge-graph-based state space and a template-based action space. The knowledge graph serves as a means for the agent to understand its surroundings, accumulate information about the game, and disambiguate similar textual observations while the templates lend a measure of structure that enables us to exploit that same knowledge graph for language generation. Together they constrain the vast space of possible actions into the compact space of sensible ones. An ablation study on Zork1 shows state-of-the-art performance with respect to any currently existing general reinforcement learning agent, including those with action spaces six orders of magnitude smaller than what we consider-indicating the overall efficacy of our combined state-action space. Further, a suite of experiments shows wide improvement over TDQN, the current state-of-the-art template based agent, across a diverse set of 26 human-made IF games covering multiple genres and game structures demonstrate that our agent is able to generalize effectively. A IMPLEMENTATION DETAILS Candidate interactive objects are identified by performing part-of-speech tagging on the current observation, identifying singular and proper nouns as well as adjectives, and are then filtered by checking if they can be examined using the command examine OBJ. Only the interactive objects not found in the inventory are linked to the node corresponding to the current room and the inventory items are linked to the "you" node. The only other rule applied uses the navigational actions performed by the agent to infer the relative positions of rooms, e.g. kitchen, down, cellar when the agent performs go down when in the kitchen to move to the cellar. Templates are processed by selecting a single verb and preposition from the aliases. For the sake of agent explainability, we pick the verb and preposition that are most likely to be used by humans when playing IF games. This is done by assessing token frequencies from a dataset of human playthroughs such as those given in ClubFloyd [at/against/on/onto] ), would then be converted to take and put on. Episodes are terminated after 100 valid steps or game over/victory. Agents that decode invalid actions often wouldn't make it very far into the game, and so we only count valid-actions against the hundred step limit. All agents are trained individually on each game and then evaluated on that game. All A2C based agents are trained using data collected from 32 parallel environments. TDQN was trained using a single environment. Hyperparameters for all agents were tuned on the game of Zork1 and held constant across all other games. Final reported scores are an average over 5 runs of each algorithm. Interactive objects: tree, path, branches, forest, large, all Action: up Score: 0 ---Obs: Desc: Up a Tree You are about 10 feet above the ground nestled among some large branches. The nearest branch above you is above your reach. Beside you on the branch is a small birds nest. In the birds nest is a large egg encrusted with precious jewels, apparently scavenged by a childless songbird. The egg is covered with fine gold inlay, and ornamented in lapis lazuli and motherofpearl. Unlike most eggs, this one is hinged and closed with a delicate looking clasp. The egg appears extremely fragile. Inv: You are emptyhanded. Feedback: Up a Tree You are about 10 feet above the ground nestled among some large branches. The nearest branch above you is above your reach. Beside you on the branch is a small birds nest. In the birds nest is a large egg encrusted with precious jewels, apparently scavenged by a childless songbird. The egg is covered with fine gold inlay, and ornamented in lapis lazuli and motherofpearl. Unlike most eggs, this one is hinged and closed with a delicate looking clasp. The egg appears extremely fragile.

We present KG-A2C, a reinforcement learning agent that builds a dynamic knowledge graph while exploring and generates natural language using a template-based action space - outperforming all current agents on a wide set of text-based games.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Introduce an approach to allow agents to learn  PPDDL action models incrementally over multiple planning problems under the framework of reinforcement learning.