Question: Represent the article for finding a single-sentence TLDR

Question: Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning. Both of these challenges severely limit the applicability of such methods to complex, real-world domains. In this paper, we propose soft actor-critic, an off-policy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework. In this framework, the actor aims to maximize expected reward while also maximizing entropy - that is, succeed at the task while acting as randomly as possible. Prior deep RL methods based on this framework have been formulated as either off-policy Q-learning, or on-policy policy gradient methods. By combining off-policy updates with a stable stochastic actor-critic formulation, our method achieves state-of-the-art performance on a range of continuous control benchmark tasks, outperforming prior on-policy and off-policy methods. Furthermore, we demonstrate that, in contrast to other off-policy algorithms, our approach is very stable, achieving very similar performance across different random seeds. We explore how to design an efficient and stable model-free deep RL algorithm for continuous state and action spaces. To that end, we draw on the maximum entropy framework, which augments the standard maximum reward reinforcement learning objective with an entropy maximization term BID31 BID28 BID20 BID4 BID8. Maximum entropy reinforcement learning alters the RL objective, though the original objective can be recovered by using a temperature parameter BID8. More importantly, the maximum entropy formulation provides a substantial improvement in exploration and robustness: as discussed by BID30, maximum entropy policies are robust in the face of modeling and estimation errors, and as demonstrated by BID8, they improve exploration by acquiring diverse behaviors. Prior work has proposed model-free deep RL algorithms that perform on-policy learning with entropy maximization BID17, as well as off-policy methods based on soft Q-learning and its variants BID22 BID15 BID8. However, the on-policy variants suffer from poor sample complexity for the reasons discussed above, while the off-policy variants require complex approximate inference procedures in continuous action spaces. In this paper, we demonstrate that we can devise an off-policy maximum entropy actor-critic algorithm, which we call soft actor-critic, which provides for both sample-efficient learning and stability. This algorithm extends readily to very complex, high-dimensional tasks, such as the 21-DoF Humanoid benchmark BID3, where off-policy methods such as DDPG typically struggle to obtain good BID6, while avoiding the complexity and potential instability associated with approximate inference in prior off-policy maximum entropy algorithms based on soft Q-learning BID8. In particular, we present a novel convergence proof for policy iteration in the maximum entropy framework. We then introduce a new algorithm based on an approximation to this procedure that can be practically implemented with deep neural networks, which we call soft actor-critic. We present empirical that show that soft actor-critic attains a substantial improvement in both performance and sample efficiency over both off-policy and on-policy prior methods. Our soft actor-critic algorithm incorporates three key ingredients: an actor-critic architecture with separate policy and value function networks, an off-policy formulation that enables reuse of previously collected data for efficiency, and entropy maximization to enable stability and exploration. We review prior works that draw on some of these ideas in this section. Actor-critic algorithms are typically derived starting from policy iteration, which alternates between policy evaluation-computing the value function for a policy-and policy improvement-using the value function to obtain a better policy BID0 BID25. In large-scale reinforcement learning problems, it is typically impractical to run either of these steps to convergence, and instead the value function and policy are optimized jointly. In this case, the policy is referred to as the actor, and the value function as the critic. Many actor-critic algorithms build on the standard, on-policy policy gradient formulation to update the actor BID18, and many of them also consider the entropy of the policy, but instead of maximizing the entropy, they use it as an regularizer BID21 BID5. This tends to improve stability, but in very poor sample complexity. There have been efforts to increase the sample efficiency while retaining the robustness properties by incorporating off-policy samples and by using higher order variance reduction techniques BID17 BID6. However, fully off-policy algorithms still attain better efficiency. A particularly popular off-policy actor-critic method, , which is a deep variant of the deterministic policy gradient BID23 algorithm, uses a Q-function estimator to enable off-policy learning, and a deterministic actor that maximizes this Q-function. As such, this method can be viewed both as a deterministic actor-critic algorithm and an approximate Q-learning algorithm. Unfortunately, the interplay between the deterministic actor network and the Q-function typically makes DDPG extremely difficult to stabilize and brittle to hyperparameter settings BID3 BID9. As a consequence, it is difficult to extend DDPG to very complex, high-dimensional tasks, and on-policy policy gradient methods still tend to produce the best in such settings BID6. Our method instead combines off-policy actor-critic training with a stochastic actor, and further aims to maximize the entropy of this actor with an entropy maximization objective. We find that this actually in a substantially more stable and scalable algorithm that, in practice, exceeds both the efficiency and final performance of DDPG.Maximum entropy reinforcement learning optimizes policies to maximize both the expected return and the expected entropy of the policy. This framework has been used in many contexts, from inverse reinforcement learning BID31 to optimal control BID27 BID28 BID20. In guided policy search BID10, maximum entropy distribution is used to guide policy learning towards high-reward regions. More recently, several papers have noted the connection between Q-learning and policy gradient methods in the framework of maximum entropy learning BID17 BID8 BID15 BID22. While most of the prior works assume a discrete action space, BID16 approximate the maximum entropy distribution with a Gaussian and BID8 with a sampling network trained to draw samples from the optimal policy. Although the soft Q-learning algorithm proposed by BID8 has a value function and actor network, it is not a true actor-critic algorithm: the Q-function is estimating the optimal Q-function, and the actor does not directly affect the Q-function except through the data distribution. Hence, BID8 motivates the actor network as an approximate sampler, rather than the actor in an actor-critic algorithm. Crucially, the convergence of this method hinges on how well this sampler approximates the true posterior. In contrast, we prove that our method converges to the optimal policy from a given policy class, regardless of the policy parameterization. Furthermore, these previously proposed maximum entropy methods generally do not exceed the performance of state-of-the-art off-policy algorithms, such as DDPG, when learning from scratch, though they may have other benefits, such as improved exploration and ease of finetuning. In our experiments, we demonstrate that our soft actor-critic algorithm does in fact exceed the performance of state-of-theart off-policy deep RL methods by a wide margin. In this section, we introduce notation and summarize the standard and maximum entropy reinforcement learning frameworks. We address policy learning in continuous action spaces. To that end, we consider infinite-horizon Markov decision processes (MDP), defined by the tuple (S, A, p s, r), where the state space S and the action space A are assumed to be continuous, and the unknown state transition probability p s: S × S × A → [0, ∞) represents the probability density of the next state s t+1 ∈ S given the current state s t ∈ S and action a t ∈ A. The environment emits a bounded reward r: S × A → [r min, r max] on each transition, which we will abbreviate as r t r(s t, a t) to simplify notation. We will also use ρ π (s t) and ρ π (s t, a t) to denote the state and state-action marginals of the trajectory distribution induced by a policy π(a t |s t). The standard objective used in reinforcement learning is to maximize the expected sum of rewards t E (st,at)∼ρπ [r t]. We will consider a more general maximum entropy objective (see e.g. BID30), which favors stochastic policies by augmenting the objective with the expected entropy of the policy over ρ π (s t): DISPLAYFORM0 The temperature parameter α determines the relative importance of the entropy term against the reward, and thus controls the stochasticity of the optimal policy. The maximum entropy objective differs from the standard maximum expected reward objective used in conventional reinforcement learning, though the conventional objective can be recovered in the limit as α → 0. For the rest of this paper, we will omit writing the temperature explicitly, as it can always be subsumed into the reward by scaling it by α −1. The maximum entropy objective has a number of conceptual and practical advantages. First, the policy is incentivized to explore more widely, while giving up on clearly unpromising avenues. Second, the policy can capture multiple modes of near-optimal behavior. In particular, in problem settings where multiple actions seem equally attractive, the policy will commit equal probability mass to those actions. Lastly, prior work has observed substantially improved exploration from this objective BID8 BID22, and in our experiments, we observe that it considerably improves learning speed over state-of-art methods that optimize the conventional objective function. We can extend the objective to infinite horizon problems by introducing a discount factor γ to ensure that the sum of expected rewards and entropies is finite. Writing down the precise maximum entropy objective for the infinite horizon discounted case is more involved BID26 and is deferred to Appendix A.Prior methods have proposed directly solving for the optimal Q-function, from which the optimal policy can be recovered BID31 BID4 BID8. In the next section, we will discuss how we can devise a soft actor-critic algorithm through a policy iteration formulation, where we instead evaluate the Q-function of the current policy and update the policy through an off-policy gradient update. Though such algorithms have previously been proposed for conventional reinforcement learning, our method is, to our knowledge, the first off-policy actor-critic method in the maximum entropy reinforcement learning framework. Our off-policy soft actor-critic algorithm can be derived starting from a novel maximum entropy variant of the policy iteration method. In this section, we will first present this derivation, verify that the corresponding algorithm converges to the optimal policy from its density class, and then present a practical deep reinforcement learning algorithm based on this theory. We will begin by deriving soft policy iteration, a general algorithm for learning optimal maximum entropy policies that alternates between policy evaluation and policy improvement in the maximum entropy framework. Our derivation is based on a tabular setting, to enable theoretical analysis and convergence guarantees, and we extend this method into the general continuous setting in the next section. We will show that soft policy iteration converges to the optimal policy within a set of policies which might correspond, for instance, to a set of parameterized densities. In the policy evaluation step of soft policy iteration, we wish to compute the value of a policy π according to the maximum entropy objective in Equation 1. For a fixed policy, the soft Q-value can be computed iteratively, by starting from any function Q: S × A → R and iteratively applying a modified version of the Bellman backup operator T π defined by DISPLAYFORM0 where DISPLAYFORM1 is the soft state value function. We can find the soft value of an arbitrary policy π by repeatingly applying T π as formalized in Lemma 1.Lemma 1 (Soft Policy Evaluation). Consider the soft Bellman backup operator T π in Equation 2 and a mapping Q 0: S × A → R, and define DISPLAYFORM2 Then the sequence Q k will converge to the soft Q-value of π as k → ∞.Proof. See Appendix B.1.In the policy improvement step, we update the policy towards the exponential of the new Q-function. This particular choice of update can be guaranteed to into an improved policy in terms of its soft value, as we show in this section. Since in practice we prefer policies that are tractable, we will additionally restrict the policy to some set of policies Π, which can correspond, for example, to a parameterized family of distributions such as Gaussians. To account for the constraint that π ∈ Π, we project the improved policy into the desired set of policies. While in principle we could choose any projection, it will turn out to be convenient to use the information projection defined in terms of the Kullback-Leibler divergence. In the other words, in the policy improvement step, for each state, we update the policy according to DISPLAYFORM3 The partition function Z π old (s t) normalizes the second KL argument, and while it is intractable in general, it does not contribute to the gradient with respect to the new policy and can thus be ignored as noted in the next section. For this choice of projection, we can show that the new, projected policy has a higher value than the old policy with respect to the objective in Equation 1. We formalize this in Lemma 2. Lemma 2 (Soft Policy Improvement). Let π old ∈ Π and let π new be the optimizer of the minimization problem defined in Equation 4. Then DISPLAYFORM4 Proof. See Appendix B.2.The full soft policy iteration algorithm alternates between the soft policy evaluation and the soft policy improvement steps, and it will provably converge to the optimal maximum entropy policy among the policies in Π (Theorem 1). Although this algorithm will provably find the optimal solution, we can perform it in its exact form only in the tabular case. Therefore, we will next approximate the algorithm for continuous domains, where we need to rely on a function approximator to represent the Q-values, and running the two steps until convergence would be computationally too expensive. The approximation gives rise to a new practical algorithm, called soft actor-critic. Theorem 1 (Soft Policy Iteration). Repeated application of soft policy evaluation and soft policy improvement to any π ∈ Π converges to a policy π DISPLAYFORM5 Proof. See Appendix B.3. As discussed above, large, continuous domains require us to derive a practical approximation to soft policy iteration. To that end, we will use function approximators for both the Q-function and policy, and instead of running evaluation and improvement to convergence, alternate between optimizing both networks with stochastic gradient descent. For the remainder of this paper, we will consider a parameterized state value function V ψ (s t), soft Q-function Q θ (s t, a t), and a tractable policy π φ (a t |s t). The parameters of these networks are ψ, θ, and φ. In the following, we will derive update rules for these parameter vectors. The state value function approximates the soft value. There is no need in principle to include a separate function approximator for the state value, since it is related to the Q-function and policy according to Equation 3. This quantity can be estimated from a single action sample from the current policy without introducing a bias, but in practice, including a separate function approximator for the soft value can stabilize training-and as discussed later, can be used as a state-dependent baseline for learning the policy-and is convenient to train simultaneously with the other networks. The soft value function is trained to minimize the squared residual error DISPLAYFORM0 where D is the distribution of previously sampled states and actions, or a replay buffer. The gradient of Equation 5 can be estimated with an unbiased estimator DISPLAYFORM1 where the actions are sampled according to the current policy, instead of the replay buffer. The soft Q-function parameters can be trained to minimize the soft Bellman residual DISPLAYFORM2 which again can be optimized with stochastic unbiased gradientŝ DISPLAYFORM3 The update makes use of a target value network Vψ whereψ is an exponentially moving average of the value network weights, which has been shown to stabilize training BID13, although we found soft actor-critic to be able to learn robustly also in the absense of a target network. Finally, the policy parameters can be learned by directly minimizing the KL-divergence in Equation 4, which we reproduce here in parametrized form for completeness DISPLAYFORM4 There are several options for minimizing J π, depending on the choice of the policy class. For simple distributions, such as Gaussians, we can use the reparametrization trick, which allows us to backpropagate the gradient from the critic network and leads to a DDPG-style estimator. However, if the policy depends on discrete latent variables, such as is the case for mixture models, the reparametrization trick cannot be used. We therefore propose to use a likelihood ratio gradient estimator: DISPLAYFORM5 where b(s t) is a state-dependent baseline BID18. We can approximately center the learning signal and eliminate the intractable log-partition function by choosing b(s t) = V ψ (s t) − log Z θ (s t) − 1, which yields the final gradient estimator for each gradient step do DISPLAYFORM6 DISPLAYFORM7 The complete algorithm is described in Algorithm 1. The method alternates between collecting experience from the environment with the current policy and updating the function approximators using the stochastic gradients on batches sampled from a replay buffer. In practice we found a combination of a single environment step and multiple gradient steps to work best (see Appendix E). Using off-policy data from a replay buffer is feasible because both value estimators and the policy can be trained entirely on offpolicy data. The algorithm is agnostic to the parameterization of the policy, as long as it can be evaluated for any arbitrary state-action tuple. We will next suggest a practical parameterization for the policy, based on Gaussian mixtures. Although we could use a simple policy represented by a Gaussian, as is common in prior work, the maximum entropy framework aims to maximize the randomness of the learned policy. Therefore, a more expressive but still tractable distribution can endow our method with more effective exploration and robustness, which are the typically cited benefits of entropy maximization BID30. To that end, we propose a practical multimodal representation based on a mixture of K Gaussians. This can approximate any distribution to arbitrary precision as K → ∞, but even for practical numbers of mixture elements, it can provide a very expressive distribution in medium-dimensional action spaces. Although the complexity of evaluating or sampling from the ing distribution scales linearly in K, our experiments indicates that a relatively small number of mixture components, typically just two or four, is sufficient to achieve high performance, thus making the algorithm scalable to complex domains with high-dimensional action spaces. We define the policy as DISPLAYFORM0 where DISPLAYFORM1 are the unnormalized mixture weights, means, and covariances, respectively, which all can depend on s t in complex ways if expressed as neural networks. We also apply a squashing function to limit the actions to a bounded interval as explained in Appendix C. Note that, in contrast to soft Q-learning BID8, our algorithm does not assume that the policy can accurately approximate the optimal exponentiated Q-function distribution. The convergence for soft policy iteration holds even for policies that are restricted to a policy class, in contrast to prior methods of this type. The goal of our experimental evaluation is to understand how the sample complexity and stability of our method compares with prior off-policy and on-policy deep reinforcement learning algorithms. To that end, we evaluate on a range of challenging continuous control tasks from the OpenAI gym benchmark suite BID2. For the Swimmer and Humanoid environments, we use the rllab implementations BID3, which have more well-behaved observation spaces. Although the easier tasks in this benchmark suite can be solved by a wide range of different algorithms, the more complex benchmarks, such as the 21-dimensional Humanoid, are exceptionally difficult to solve with off-policy algorithms BID3. The stability of the algorithm also plays a large role in performance: easier tasks make it more practical to tune hyperparameters to achieve good , while the already narrow basins of effective hyperparameters become prohibitively small for the more sensitive algorithms on the most high-dimensional benchmarks, leading to poor performance BID6.In our comparisons, we compare to trust region policy optimization (TRPO) BID21, a stable and effective on-policy policy gradient algorithm; deep deterministic policy gradient (DDGP) BID11, an algorithm that is regarded as one of the more efficient off-policy deep RL methods BID3; as well as soft Q-learning (SQL) BID8, an off-policy algorithm for learning maximum entropy policies. Although DDPG is very efficient, it is also known to be more sensitive to hyperparameter settings, which limits its effectiveness on complex tasks BID6 BID9. FIG0 shows the total average reward of evaluation rollouts during training for the various methods. Exploration noise was turned off for evaluation for DDPG and TRPO. For maximum entropy algorithms, which do not explicitly inject exploration noise, we either evaluated with the exploration noise (SQL) or approximate the maximum a posteriori action by choosing the mean of the mixture component that achieves the highest Q-value (SAC). Performance across tasks. The show that, overall, SAC substantially outperforms DDPG on all of the benchmark tasks in terms of final performance, and learns faster than any of the baselines in most environments. The only exceptions are Swimmer, where DDPG is slightly faster, and Ant-v1, where SQL is initially faster, but plateaus at a lower final performance. On the hardest tasks, Ant-v1 and Humanoid (rllab), DDPG is unable to make any progress, a that is corroborated by prior work BID6 BID3. SAC still learns substantially faster than TRPO on these tasks, likely as a consequence of improved stability. Poor stability is exacerbated by highdimensional and complex tasks. Even though TRPO does not make perceivable progress for some tasks within the range depicted in the figures, it will eventually solve all of them after substantially more iterations. The quantitative attained by SAC in our experiments also compare very favorably to reported by other methods in prior work BID3 BID6 BID9, indicating that both the sample efficiency and final performance of SAC on these benchmark tasks exceeds the state of the art. All hyperparameters used in this experiment for SAC are listed in Appendix E. Sensitivity to random seeds. In FIG1, we show the performance for multiple individual seeds on the HalfCheetah-v1 benchmark of DDPG, TRPO, and SAC (FIG0 shows the average over seeds). The individual seeds attain much more consistent performance with SAC, while DDPG exhibits very high variability across seeds, indicating substantially worse stability. TRPO is unable to make progress within the first 1000 episodes. For SAC, we performed 16 gradient steps between each environment step, which becomes prohibitively slow in terms of wall clock time for longer experiments (in FIG0 we used we used 4 gradient steps). The in the previous section suggest that algorithms based on the maximum entropy principle can outperform conventional RL methods on challenging tasks such as the Ant and Humanoid. In this section, we further examine which particular components of SAC are important for good performance. To that end, we consider several ablations to pinpoint the most essential differences between our method and standard RL methods that do not maximize entropy. We use DDPG as the representative non-entropy-maximizing algorithm, since it is the most structurally similar to SAC. This section examines specific algorithm components, while a study of the sensitivity to hyperparameter settings-most importantly the reward scale and the number of gradient steps per environment step-is deferred to Appendix D.Importance of entropy maximization. The main difference of SAC and DDPG is the inclusion of entropy maximization in the objective. The entropy appears in both the policy and value function. In the policy, it prevents premature convergence of the policy variance (Equation 9). In the value function, it encourages exploration by increase the value of regions of state space that lead to highentropy behavior (Equation 5). FIG2 compares how inclusion of this term in the policy and value updates affects the performance, by removing the entropy from each one in turn. As evident from the figure, including the entropy in the policy objective is crucial, while maximizing the entropy as part of the value function is less important, and can even hinder learning in the beginning of training. Exploration noise. Another difference is the exploration noise: the maximum entropy framework naturally gives rise to Boltzmann exploration for SAC, whereas DDPG uses external OU noise BID29 BID11. For the next experiment, we used a policy with single mixture component and compared Boltzmann exploration to exploration via external OU noise FIG2. We repeated the experiment with the OU noise parameters that we used with DDPG (θ = 0.15, σ = 0.3), and parameters that we found worked best for SAC without Boltzmann exploration (θ = 1, σ = 0.3). The indicate that Boltzmann exploration outperforms external OU noise. Separate value network. Our method also differs from DDPG in that it uses a separate network to predict the state values, in addition to the Q-values. The value network serves two purposes: it is used to bootstrap the TD updates for learning the Q-function (Equation 7), and it serves as a baseline to reduce variance of the policy gradients (Equation 11). The in FIG2 indicates that the value network has an important role in the policy gradient, but only has a minor effect on the TD updates. Nevertheless, the best performance is achieved when the value network is employed for both purposes. From SAC to DDPG. Soft actor-critic closely resembles DDPG with a stochastic actor. To study which of the components that differs between the two algorithms are most important, we evaluated a range of SAC variants by swapping components of SAC with their DDPG counterparts. In FIG3 we evaluate four versions obtained through incremental modifications: 1) original SAC, 2) replaced likelihood ratio policy gradient estimator with an estimator obtained through the reparametrization trick, 3) replaced the stochastic policy with a deterministic policy and Boltzmann exploration with OU noise, and 4) eliminated the separate value network by using directly the Q-function evaluated at the mean action. The figure suggests two interesting points: First, the reparametrization trick yields somewhat faster and more stable learning compared to the use of likelihood policy gradient estimator, though this precludes the use of multimodal mixtures of Gaussians. Second, use of a deterministic policy degrades performance substantially in terms of both average return (thick line) and variance across random seeds (shaded region). This observation indicates that entropy maximization, and particularly the use of a stochastic policy, can make offpolicy deep RL significantly more robust and efficient compared to algorithms that maximizes the expected return, especially with deterministic policies. We presented soft actor-critic (SAC), an off-policy maximum entropy deep reinforcement learning algorithm that provides sample-efficient learning while retaining the benefits of entropy maximization and stability. Our theoretical derive soft policy iteration, which we show to converge to the optimal policy. From this , we can formulate a soft actor-critic algorithm, and we empirically show that it outperforms state-of-the-art model-free deep RL methods, including the off-policy DDPG algorithm and the on-policy TRPO algorithm. In fact, the sample efficiency of this approach actually exceeds that of DDPG by a substantial margin. Our suggest that stochastic, entropy maximizing reinforcement learning algorithms can provide a promising avenue for improved robustness and stability, and further exploration of maximum entropy methods, including methods that incorporate second order information (e.g., trust regions BID21) or more expressive policy classes is an exciting avenue for future work. The exact definition of the discounted maximum entropy objective is complicated by the fact that, when using a discount factor for policy gradient methods, we typically do not discount the state distribution, only the rewards. In that sense, discounted policy gradients typically do not optimize the true discounted objective. Instead, they optimize average reward, with the discount serving to reduce variance, as discussed by BID26. However, we can define the objective that is optimized under a discount factor as DISPLAYFORM0 This objective corresponds to maximizing the discounted expected reward and entropy for future states originating from every state-action tuple (s t, a t) weighted by its probability ρ π under the current policy. B.1 LEMMA 1Lemma 1 (Soft Policy Evaluation). Consider the soft Bellman backup operator T π in Equation 2 and a mapping Q 0: S × A → R, and define Q k+1 = T π Q k. Then the sequence Q k will converge to the soft Q-value of π as k → ∞.Proof. Define the entropy augmented reward as r π (s t, a t) r(s t, a t) + E st+1∼ps [H (π( · |s t+1))] and rewrite the update rule as DISPLAYFORM0 and apply the standard convergence for policy evaluation BID25. Lemma 2 (Soft Policy Improvement). Let π old ∈ Π and let π new be the optimizer of the minimization problem defined in Equation 4. Then Q πnew (s t, a t) ≥ Q π old (s t, a t) for all (s t, a t) ∈ S × A.Proof. Let π old ∈ Π and let Q π old and V π old be the corresponding soft state-action value and soft state value, and let π new be defined as DISPLAYFORM0 It must be the case that J π old (π new ( · |s t)) ≤ J π old (π old ( · |s t)), since we can always choose π new = π old ∈ Π. Hence DISPLAYFORM1 and since partition function Z π old depends only on the state, the inequality reduces to DISPLAYFORM2 Next, consider the soft Bellman equation: DISPLAYFORM3... DISPLAYFORM4 where we have repeatedly expanded Q π old on the RHS by applying the soft Bellman equation and the bound in Equation 17. Convergence to Q πnew follows from Lemma 1. Theorem 1 (Soft Policy Iteration). Repeated application of soft policy evaluation and soft policy improvement to any π ∈ Π converges to a policy π * (· |s t) such that Q π * (s t, a t) > Q π (s t, a t) for all π ∈ Π, π = π * and for all (s t, a t) ∈ S × A.Proof. Let π i be the policy at iteration i. By Lemma 2, the sequence Q πi is monotonically increasing. Since Q π is bounded above for π ∈ Π (both the reward and entropy are bounded), the sequence converges to some π *. We will still need to show that π * is indeed optimal. At convergence, it must be case that J π * (π * ( · |s t)) < J π * (π( · |s t)) for all π ∈ Π, π = π *. Using the same iterative argument as in the proof of Lemma 2, we get Q π * (s t, a t) > Q π (s t, a t) for all (s t, a t) ∈ S × A, that is, the soft value of any other policy in Π is lower than that of the converged policy. Hence π * is optimal in Π. We use an unbounded GMM as the action distribution. However, in practice, the actions needs to be bounded to a finite interval. To that end, we apply an invertible squashing function (tanh) to the GMM samples, and employ the change of variables formula to compute the likelihoods of the bounded actions. In the other words, let u ∈ R D be a random variable and µ(u|s) the corresponding density with infinite support. Then a = tanh(u), where tanh is applied elementwise, is a random variable with support in (−1, 1) with a density given by DISPLAYFORM0 Since the Jacobian da /du = diag(1 − tanh 2 (u)) is diagonal, the log-likelihood has a simple form DISPLAYFORM1 where u i is the i th element of u. This section discusses the sensitivity to hyperparameters. All of the following experiments are on the HalfCheetah-v1 benchmark, and although we found these to be representative in most cases, they may differ between the environments. For other environments, we only swept over the reward scale only. Reward scale. Soft actor-critic is particularly sensitive to the reward magnitude, because it serves the role of the temperature of the energy-based optimal policy and thus controls its stochasticity. FIG4 shows how learning performance changes when the reward scale is varied: For small reward magnitudes, the policy becomes nearly uniform, and consequently the fails to exploit the reward signal, ing in substantial degradation of performance. For large reward magnitudes, the model learns quickly at first, but the policy quickly becomes nearly deterministic, leading to poor exploration and subsequent failure due to lack of adequate exploration. With the right reward magnitude, the model balances exploration and exploitation, leading to fast and stable learning. In practice, we found reward scale to be the only hyperparameter that requires substantial tuning, and since this parameter has a natural interpretation in the maximum entropy framework, this provides good intuition for how to adjust this parameter. Mixture components. We experimented with different numbers of mixture components in the Gaussian mixture policy FIG4 ). We found that the number of components did not affect the learning performance very much, though larger numbers of components consistently attained somewhat better . Policy evaluation. Since SAC converges to stochastic policies, it is generally beneficial to make the final policy deterministic at the end for best performance. For evaluation, we approximate the maximum a posteriori action by choosing the action that maximizes the Q-function among the mixture component means. FIG4 compares training returns to test returns obtained with this strategy. The test returns are substantially better. It should be noted that all of the training curves depict the sum of rewards, which is different from the objective optimized by SAC and SQL which both maximize also the entropy. Target network update. In FIG4 we varied the smoothing constant, τ, for the target value network updates. A value of one corresponds to a hard update where the weights are copied directly at every iteration, and values close to zero correspond to exponentially moving average of the network weights. Interestingly, SAC is relatively insensitive to the smoothing constant: smaller τ tends to yield higher performance at the end with the cost of marginally slower progress in the beginning, but the difference is small. In prior work, the main use of the target network was to stabilize training BID13 BID11, but interestingly, SAC is able to learn stably also when effectively no separate target network is used, but instead the weights are copied directly (τ = 1), although at the cost of minor degradation of performance. Replay buffer size. Next, we experimented with the size of the experience replay buffer FIG4 ), which we found to be important for environments where the optimal policy becomes nearly deterministic at convergence, such as HalfCheetah-v1. For such environments, the exploration noise becomes negligible at the end, making the content of the replay buffer less diverse and ing in overfitting and instability. Use of a higher capacity buffer solves this problem by allowing the method to remember the failures from the beginning of learning, but slows down learning by allocating some of the network capacity to modeling the suboptimal initial experience. Note that a return of just 4800 is considered as the threshold level for solving this benchmark task and that the performance in all cases is well beyond the threshold. We used a buffer size of 10 million samples for HalfCheetah-v1, and 1 million samples for the other environments. Gradient steps per time step. Finally, we experimented with the number of actor and critic gradient steps per time step of the algorithm FIG4 ). Prior work has observed that increasing the number of gradient steps for DDPG tends to improve sample efficiency BID7 BID19, but only up to a point. We found 4 gradient updates per time step to be optimal for DDPG, before we saw degradation of performance. SAC is able to effectively benefit from substantially larger numbers of gradient updates in most environments, with performance increasing steadily until 16 gradient updates per step. In some environments, such as Humanoid, we did not observe as much benefit. The ability to take multiple gradient steps without negatively affecting the algorithm is important especially for learning in the real world, where experience collection is the bottleneck, and it is typical to run the learning script asynchronously in a separate thread. In this experiment, we used a replay buffer size of 1 million samples, and therefore the performance suddenly drops after reaching a return of approximately 10 thousand due to the reason discussed in the previous section. E HYPERPARAMETERS TAB1 lists the common SAC parameters used in the comparative evaluation in FIG0, and TAB2 lists the parameters that varied across the environments.

represent the tldr for finding the corresponding scientific article

We propose soft actor-critic, an off-policy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework.

Parameter space noise allows reinforcement learning algorithms to explore by perturbing parameters instead of actions, often leading to significantly improved exploration performance.


Question: Represent the scientific article to find a short TLDR.

Question: In many settings, it is desirable to learn decision-making and control policies through learning or from expert demonstrations. The most common approaches under this framework are Behaviour Cloning (BC), and Inverse Reinforcement Learning (IRL). Recent methods for IRL have demonstrated the capacity to learn effective policies with access to a very limited set of demonstrations, a scenario in which BC methods often fail. Unfortunately, directly comparing the algorithms for these methods does not provide adequate intuition for understanding this difference in performance. This is the motivating factor for our work. We begin by presenting $f$-MAX, a generalization of AIRL , a state-of-the-art IRL method. $f$-MAX provides grounds for more directly comparing the objectives for LfD. We demonstrate that $f$-MAX, and by inheritance AIRL, is a subset of the cost-regularized IRL framework laid out by. We conclude by empirically evaluating the factors of difference between various LfD objectives in the continuous control domain. Modern advances in reinforcement learning aim to alleviate the need for hand-engineered decisionmaking and control algorithms by designing general purpose methods that learn to optimize provided reward functions. In many cases however, it is either too challenging to optimize a given reward (e.g. due to sparsity of signal), or it is simply impossible to design a reward function that captures the intricate details of desired outcomes. One approach to overcoming such hurdles is Learning from Demonstrations (LfD), where algorithms are provided with expert demonstrations of how to accomplish desired tasks. The most common approaches in the LfD framework are Behaviour Cloning (BC) and Inverse Reinforcement Learning (IRL) BID22 BID15. In standard BC, learning from demonstrations is treated as a supervised learning problem and policies are trained to regress expert actions from a dataset of expert demonstrations. Other forms of Behaviour Cloning, such as DAgger BID21, consider how to make use of an expert in a more optimal fashion. On the other hand, in IRL the aim is to infer the reward function of the expert, and subsequently train a policy to optimize this reward. The motivation for IRL stems from the intuition that the reward function is the most concise and portable representation of a task BID15 BID0.Unfortunately, the standard IRL formulation BID15 faces degeneracy issues 1. A successful framework for overcoming such challenges is the Maximum-Entropy (Max-Ent) IRL method BID28 BID27. A line of research stemming from the Max-Ent IRL framework has lead to recent "adversarial" methods BID12 BID4 BID7 1 for example, any policy is optimal for the constant reward function r(s, a) = 0 2 Consider a Markov Decision Process (MDP) represented as a tuple (S, A, P, r, ρ 0, γ) with statespace S, action-space A, dynamics P: S × A × S →, reward function r(s, a), initial state distribution ρ 0, and discount factor γ ∈. In Maximum Entropy (Max-Ent) reinforcement learning BID23 BID25 BID20 BID6 BID10, the goal is to find a policy π such that trajectories sampled using this policy follow the distribution DISPLAYFORM0 where τ = (s 0, a 0, s 1, a 1, ...) denotes a trajectory, and R(τ) = t r(s t, a t) and Z is the partition function. Hence, trajectories that accumulate more reward are exponentially more likely to be sampled. Converse to the standard RL setting, in Max-Ent IRL BID28 BID27 we are instead presented with an optimal policy π exp, or more realistically, sample trajectories from such a policy, and we seek to find a reward function r that maximizes the likelihood of the trajectories sampled from π exp. Formally, our objective is: DISPLAYFORM1 Being an energy-based modelling objective, the difficulty in performing this optimization arises from estimating the partition function Z. Initial methods addressed this problem using dynamic programming BID28 BID27, and recent approaches present methods aimed at intractable domains with unknown dynamics BID5 BID12 BID4 BID7 BID13.Instead of recovering the expert's reward function and policy, recent successful methods in Max-Ent IRL aim to directly recover the policy that would from the full process. Since such methods only recover the policy, it would be more accurate to refer to them as Imitation Learning algorithms. However, to avoid confusion with Behaviour Cloning methods, in this work we will refer to them as direct methods for Max-Ent IRL.GAIL: Generative Adversarial Imitation Learning Before describing the work of BID12, we establish the definition of causal entropy BID2. Intuitively, causal entropy can be thought of as the "amount of options" the policy has in each state, in expectation. DISPLAYFORM2 Let C denote a class of cost functions (negative reward functions).Furthermore, let ρ exp (s, a), ρ π (s, a) denote the state-action marginal distributions of the expert and student policy respectively. BID12 begin with a regularized Max-Ent IRL objective, DISPLAYFORM3 where ψ: C → R is a convex regularization function on the space of cost functions, and IRL ψ (π exp) returns the optimal cost function given the expert and choice of regularization. Also, while not immediately clear, note that DISPLAYFORM4, be a function that returns the optimal Max-Ent policy given cost c(s, a). BID12 show that DISPLAYFORM5 where ψ * denotes the convex conjugate of ψ. This tells us that if we were to find the cost function c(s, a) using the regularized Max-Ent IRL objective 3, and subsequently find the optimal Max-Ent policy for this cost, we would arrive at the same policy had we directly optimized objective 4 by searching for the policy. Directly optimizing 4 is challenging for many choices of ψ. Interestingly however, BID12 show that for any symmetric f -divergences BID14, there exists a choice of ψ such that equation 4 is equivalent to RL ) ). In such settings, due to a close connection between binary classifiers and symmetric f -divergences BID16, efficient algorithms can be formed. DISPLAYFORM6 The special case for Jensen-Shannon divergence leads to the successful method dubbed Generative Adversarial Imitation Learning (GAIL). As before, let ρ exp (s, a), ρ π (s, a) denote the state-action marginal distributions of the expert and student policy respectively. Let D(s, a): S × A → be a binary classifier -often referred to as the discriminator -for identifying positive samples (sampled from ρ exp (s, a)) from negative samples (sampled from ρ π (s, a)). Using RL, the student policy is trained to maximize E τ ∼π [t log D(s t, a t)] − λH causal (π), where λ is a hyperparameter. The training procedure alternates between optimizing the discriminator and updating the policy. As noted, it is shown that this training procedure minimizes the Jensen-Shannon divergence between ρ exp (s, a) and ρ π (s, a) BID12.AIRL: Adversarial Inverse Reinforcement Learning Subsequent to the advent of GAIL BID12, BID4 present a theoretical discussion relating Generative Adversarial Networks (GANs) BID8, IRL, and energy-based models. They demonstrate how an adversarial training approach could recover the Max-Ent reward function and simultaneously train the Max-Ent policy corresponding to that reward. Building on this discussion, BID7 present a practical implementation of this method, named Adversarial Inverse Reinforcement Learning (AIRL).As before, let ρ exp (s, a), ρ π (s, a) denote the state-action marginal distributions of the expert and student policy respectively and let D(s, a): S × A → be the discriminator. In AIRL, the discriminator is parameterized as, DISPLAYFORM7 where f (s, a): S × A → R, and π(a|s) denotes the likelihood of the action under the policy. AIRL defines the reward function, r(s, a):= log D(s, a) − log (1 − D(s, a)), and sets the objective for the student policy to be the RL objective, max π E τ ∼π [t r(s t, a t)]. As in GAIL, this leads to an iterative optimization process alternating between optimizing the discriminator and the policy. At convergence, the advantage function of the expert is recovered. Given this observation, important considerations are made regarding how to extract the true reward function from f (s, a). When the objective is only to perform Imitation Learning, and we do not care to recover the reward function, the discriminator does not use the special parameterization discussed above and is instead direclty represented as a function D(s, a): S × A →, as done in GAIL BID12.Performance With Respect to BC Methods such as GAIL and AIRL have demonstrated significant performance gains compared to Behaviour Cloning. In particular, in standard Mujoco benchmarks BID24 BID3, adversarial methods for Max-Ent IRL achieve strong performance using a very limited amount of demonstrations from an expert policy, an important failure scenario for standard Behaviour Cloning. Ho & Ermon FORMULA0 demonstrate that Max-Ent IRL is the dual problem of matching ρ π (s, a) to ρ exp (s, a); indeed as noted above, GAIL BID12 optimizes the Jensen-Shannon divergence between the two distributions. In section 3 we present f -MAX, a method for matching ρ π (s, a) to ρ exp (s, a) using arbitrary f -divergences BID14. Hence, in this section we recall this class of statistical divergences as well as methods for using them for training generative models. Let P, Q be two distributions with density functions p, q. For any convex, lower-semicontinuous function f: R + → R a statistical divergence can be defined as: DISPLAYFORM0 q(x). Divergences derived in this manner are called f-divergences and amongst many interesting divergences, include the forward and reverse KL. BID17 present a variational estimation method for f -divergences between arbitrary distributions P, Q. Using the notation of BID18 we can write, DISPLAYFORM1 where T is an arbitrary class of functions T ω: X → R, and f * is the convex conjugate of f. Under mild conditions BID17 equality holds between the two sides. Motivated by this variational approximation as well as Generative Adversarial Networks (GANs) BID8, BID18 present an iterative optimization scheme for matching an implicit distribution 2 Q to a fixed distribution P using any f -divergence. For a given f -divergence, the corresponding minimax optimization is, DISPLAYFORM2 Nowozin et al. FORMULA0 discuss practical parameterizations of T ω, but to avoid notational clutter we will use the form above. DISPLAYFORM3 We begin by presenting f -MAX, a generalization of AIRL BID7 which provides a more intuitive interpretation of what similar algorithms accomplish. Imagine, for some f, we aim to train a policy by optimizing the f -divergence DISPLAYFORM4 To do so, we propose the following iterative optimization procedure, DISPLAYFORM5 DISPLAYFORM6 where f * and T ω are as defined in section 2.2. Equation 8 is the same as the inner maximization of the f -GAN objective in equation 7; this objective optimizes T ω so that equation 8 best approximates DISPLAYFORM7 On the other hand, for the policy objective, using the identities in appendix A we have, DISPLAYFORM8 which implies that the policy objective is equivalent to minimizing equation 8 with respect to π. With an identical proof as in Goodfellow et al. (2014, Proposition 2), if in each iteration the optimal T ω is found, the described optimization procedure converges to the global optimum where the policy's state-action marginal distribution matches that of the expert's. This is equivalent to iteratively computing D f (ρ exp (s, a)||ρ π (s, a)) and optimiizing the policy to minimize it. Choosing DISPLAYFORM0 ). This divergence is commonly referred to as the "reverse" KL divergence. In this setting we have, BID18. Hence, given T π ω, the policy objective in equation 9 takes the form, DISPLAYFORM1 DISPLAYFORM2 On the other hand, plugging the optimal discriminator BID8 into the AIRL BID7 policy objective, we get, DISPLAYFORM3 DISPLAYFORM4 As can be seen, the right hand side of equation 12 matches that of equation 11 up to a constant 3, meaning that AIRL is solving the Max-Ent IRL problem by minimizing the reverse KL divergence, DISPLAYFORM5 As discussed above, present a class of methods for Max-Ent IRL that directly retrieve the expert policy without explicitly finding the reward function of the expert (sec. 2.1).Using an interesting connection between surrogate cost functions for binary classification and fdivergences BID16 ), BID12 derive a special case of their method for minimizing any symmetric 4 f -divergence between ρ exp (s, a) and ρ π (s, a). Choosing the symmetric f -divergence to be the Jensen-Shannon divergence leads to the successful special case, GAIL (sec 2.1).Surprisingly, we now show that f -MAX is a subset of the cost-regularized Max-Ent IRL framework laid out in BID12! Recall the following equations from this framework, DISPLAYFORM6 DISPLAYFORM7 Method Optimized Objective (Minimization) DISPLAYFORM8 is the expert, and ρ DISPLAYFORM9 where ψ(c): C → R was a closed, proper, and convex regularization function on the space of cost function, and ψ * its convex conjugate. For our proof we will operate in the finite state-action space, as in the original work BID12. In this setting, cost functions can be represented as vectors in R S×A, and joint state-action distributions can be represented as vectors in S×A. Let f be the function defining some fdivergence. Given the expert for the task, we can define the following cost function regularizer, DISPLAYFORM10 where f * is the convex conjugate of f. Given this choice, with simple algebraic manipulation done in appendix B we have, DISPLAYFORM11 DISPLAYFORM12 Typically, the causal entropy term is considered a policy regularizer, and is weighted by 0 ≤ λ ≤ 1. Therefore, modulo the term H causal (π), our derivations show that f -MAX, and by inheritance AIRL BID7, all fall under the cost-regularized Max-Ent IRL framework of BID12 Given derived in the prior section, we can now begin to populate table 1, writing various Imitation Learning algorithms in a common form, as the minimization of some statistical divergence between ρ exp (s, a) and ρ π (s, a). In Behaviour Cloning we minimize DISPLAYFORM0 On the other hand, the corollary in section 3.1 demonstrates that AIRL BID7 minimizes KL (ρ π (s, a)||ρ exp (s, a)), while GAIL BID12 DISPLAYFORM1 Hence, there are two ways in which the direct IRL methods differ from BC. First, in standard BC the policy is optimized to match the conditional distribution ρ exp (a|s), whereas in the other two the policy is explicitly encouraged to match the marginal state distributions as well. Second, in BC we make use of the forward KL divergence, whereas AIRL and GAIL use divergences that exhibit more mode-seeking behaviour. These observations allow us to generate the following two hypotheses about why direct IRL methods outperform BC, particularly in the low-data regime, Hypothesis 1 In common MDPs of interest, the reward function depends more on the state than the action. Hence it is plausible that matching state marginals is more useful than matching action conditional marginals. Hypothesis 2 It is known that optimization using the forward KL divergence in distributions with a mode-covering behaviour, whereas using the reverse KL in modeseeking behaviour BID1. Therefore, since in Reinforcement Learning we care about the "quality of trajectories", being mode-seeking is more beneficial than mode-covering, particularly in the low-data regime. In what follows, we seek to experimentally evaluate our hypotheses. To tease apart the differences between the direct Max-Ent IRL methods and BC, we present an algorithm that optimizes KL (ρ exp (s, a)||ρ π (s, a)). We then compare its performance to Behaviour Cloning and the standard AIRL algorithm using varying amounts of expert demonstrations. While f -MAX is a general algorithm, useful for most choices of f, it unfortunately cannot be used for the special case of forward KL, i.e. KL (ρ exp (s, a)||ρ π (s, a)). In the following sections we identify the problem and present a separate direct Max-Ent IRL method that optimizes this divergence. Let T π ω denote the maximizer of equation 8 for a given policy π. For the case of forward KL, drawing upon equations from BID18 we have, DISPLAYFORM0 Given this, the objective for the policy (equation 9) under the optimal T π ω becomes, DISPLAYFORM1 Hence, there is no signal to train the policy! 6 In this section we derive an algorithm for optimizing KL (ρ exp (s, a)||ρ π (s, a)). Similar to AIRL BID7, let us have a discriminator, D(s, a) whose objective is to discriminate between expert and policy state-action pairs, DISPLAYFORM0 Figure 1: r(s, a) as the function of the logits of the optimal discriminator, π (s, a) = log DISPLAYFORM1 We now define the objective for the policy to be, DISPLAYFORM2 In appendix C we show, DISPLAYFORM3 This is a refreshing since it demonstrates that we can convert the AIRL algorithm BID7 into its forward KL counterpart by simply modifying the reward function used; in AIRL (reverse KL) the reward is defined as r(s, a):= log D(s, a) − log (1 − D(s, a)), whereas for forward KL it is defined as r(s, a):= s,a). We refer to this forward KL version of AIRL as FAIRL. DISPLAYFORM4 If we parameterize the discriminator as D(s, a):= σ ((s, a) ), where σ represents the sigmoid activation function, the logit of the discriminator, (s, a), is equal to log D(s, a) − log (1 − D(s, a) ). Hence, for an optimal discriminator, D π, we have π (s, a) = log s,a). It is instructive to plot the reward functions under the two different settings as a function of π (s, a); figure 1 presents these plots. As can be seen, in the forward KL version of AIRL, if for a state-action pair the expert puts more probability mass than the policy, the policy is severely punished. However, if for some stateaction pairs the policy places a lot more mass than the expert, it almost does not matter. As a , the policy spreads its mass. On the other hand, in the original AIRL formulation (reverse KL), the policy is always encouraged to put less mass than the expert. These observations are in line with standard intuitions about the mode-covering/mode-seeking behaviours of the two KL divergences BID1. DISPLAYFORM5 In this section we provide empirical comparisons between AIRL, FAIRL, and standard BC in the Ant and Halfcheetah environments found in Open-AI Gym BID3. Figure 2: Average return on 50 evaluation trajectories as a function of number of expert demonstrations (higher is better). Models evaluated deterministically. As we ran two seeds per experiment, we do not present standard deviations. While FAIRL performs comparably to AIRL, Behaviour cloning lags behind quite significantly. Considering the form of their objectives (table 1), this demonstrates that the advantage of direct Max-Ent IRL methods over BC is a of the additional aspect of their objectives explicitly matching marginal state distributions. Expert Policy To simulate access to expert demonstrations we train an expert policy using SoftActor-Critic (SAC) BID11, a state-of-the-art reinforcement learning algorithm for continuous control. The expert policy consists of a 2-layer MLP with 256-dim layers, ReLU activations, and two output streams for the mean and the diagonal covariance of a Tanh(Normal(µ, σ)) distribution 7. We use the default hyperparameter settings for training the expert. Evaluation Setup Using a trained expert policy, we generated 4 sets of expert demonstrations of that contain {4, 8, 16, 32} trajectories. Starting from a random offset, each trajectory is subsampled by a factor of 20. This is standard protocol employed in prior direct methods for Max-Ent IRL BID12 BID7. Also note that when generating demonstrations we sample from the expert's action distribution rather than taking the mode. This way, since the expert was trained using Soft-Actor-Critic, the expert should correspond to the Max-Ent optimal policy for the reward function 1 τ r g (s, a), where τ is the SAC temperature used and r g (s, a) is the groundtruth reward function. To compare the various learning-from-demonstration algorithms we train each method at each amount of expert demonstrations using 2 random seeds. For each seed, we checkpoint the model at its best validation loss 8 throughout training. At the end of training, the ing checkpoints are evaluated on 50 test episodes. Details for AIRL & FAIRL For AIRL and FAIRL, the student policy has an identical architecture to that of the expert, and the discriminator is a 2-layer MLP with 256-dim layers and Tanh activations. We normalize the observations from the environment by computing the mean and standard deviations of the expert demonstrations. The RL algorithm used for the student policies is SAC BID11, and the temperature parameter is tuned separately for AIRL & FAIRL.Details for BC For BC, we use an identical architecture as the expert. The model was fit using Maximum Likelihood Estimation 9. As before, the observations from the environment are normalized using the mean and standard deviation of the expert demonstrations. To match state-action marginals, the optimal student policy must sample actions from the stateconditional distribution, π(a|s). On the other hand, when we deploy a trained policy it is reasonable to instead choose the mode of this distribution, which we call the deterministic setting. Here, we present evaluation under the former setting, and defer the for the deterministic setting to the appendix. DISPLAYFORM0 Figure 3: Validation curves throughout training using stochastic evaluation (refer to appendix D for deterministic evaluation ). Top row Ant, Bottom row Halfcheetah. n represents the number of expert demonstrations provided. Due to its mode-covering behaviour, FAIRL does not perform as well as AIRL when evaluated stochastically. However, with determinisitc evaluation FAIRL outperforms AIRL in the Ant environment. Figure 4 demonstrates that both AIRL and FAIRL outperform BC by a large margin, especially in the low data regime. Specifically, the fact that FAIRL outperforms BC supports hypothesis 1 that the performance gain of Max-Ent IRL is not necessarily due to the direction of KL divergence used, but is the of explicitly encouraging the policy to match the marginal state distribution of the expert in addition to the matching of conditional action distribution. To compare AIRL and FAIRL, in figure 3 we plot the validation curves throughout training using stochastic evaluation. Across the two tasks and various number of expert demonstrations, AIRL consistently outperforms FAIRL. When using deterministic evaluation (figure 5), FAIRL achieves a significant performance gain to the point that it outperforms AIRL on the Ant environment across all demonstrations set sizes. Such observations provide initial positive support for hypothesis 2; as more expert demonstrations are provided, the policy trained with FAIRL broadens its distribution to cover the data-distribution, ing in trajectories accumulating less reward in expectation. We note however that more detailed experiments are necessary for adequately comparing the two methods. The motivation for this work stemmed from the superior performance of recent direct Max-Ent IRL methods BID12 BID7 compared to BC in the low-data regime, and the desire to understand the relation between various approaches for Learning from Demonstrations. We first presented f -MAX, a generalization of AIRL BID7, which allowed us to interpret AIRL as optimizing for KL (ρ π (s, a)||ρ exp (s, a)). We demonstrated that f -MAX, and by inhertance AIRL, is a subset of the cost-regularized IRL framework laid out by BID12. Comparing to the standard BC objective, E ρ exp (s) [KL (ρ exp (a|s)||ρ π (a|s))], we hypothesized two reasons for the superior performance of AIRL: 1) the additional terms in the objective encouraging the matching of marginal state distributions, and 2) the direction of the KL divergence being optimized. Setting out to empirically evaluate these claims we presented FAIRL, a one-line modification of the AIRL algorithm that optimizes KL (ρ exp (s, a)||ρ π (s, a)). FAIRL outperformed BC in a similar fashion to AIRL, which allowed us to conclude the key factor being the matching of state marginals. Additional comparisons between FAIRL and AIRL provided initial understanding about the role of the direction of the KL being optimized. In future work we aim to produce on a more diverse set of more challenging environments. Additionally, evaluating other choices of f -divergence beyond forward and reverse KL may present interesting avenues for improvement BID26. Lastly, but importantly, we would like to understand whether the mode-covering behaviour of FAIRL could in more robust policies BID19.A SOME USEFUL IDENTITIES Let h: S × A → R be an arbitrary function. If all episodes have the same length T, we have, DISPLAYFORM0 DISPLAYFORM1 In a somewhat similar fashion, in the infinite horizon case with fixed probability γ ∈ of transitioning to a terminal state, for the discounted sum below we have, DISPLAYFORM2 DISPLAYFORM3 where Γ:= 1 1−γ is the normalizer of the sum t γ t. Since the integral of an infinite series is not always equal to the infinite series of integrals, some analytic considerations must be made to go from equation 34 to 35. But, one simple case in which it holds is when the ranges of h and all ρ π (s t, a t) are bounded. and assuming the discriminator is optimal 10, we have, Figure 4: Average return on 50 evaluation trajectories as a function of number of expert demonstrations (higher is better). Models evaluated stochastically. As we ran two seeds per experiment, we do not present standard deviations. While FAIRL performs comparably to AIRL, Behaviour cloning lags behind quite significantly. Considering the form of their objectives (table 1), this demonstrates that the advantage of direct Max-Ent IRL methods over BC is a of the additional aspect of their objectives explicitly matching marginal state distributions.10 As a reminder, the optimal discriminator has the form, D(s, a) = ρ exp (s,a) ρ exp (s,a)+ρ π (s,a). A simple proof of which can be found in BID8.

Distribution matching through divergence minimization provides a common ground for comparing adversarial Maximum-Entropy Inverse Reinforcement Learning methods to Behaviour Cloning.

Represent the TLDR for finding the corresponding scientific article

An empirical study of variational inference based on chi-square divergence minimization, showing that minimizing the CUBO is trickier than maximizing the ELBO