Question: Represent the article for finding a single-sentence TLDR

Question: Actor-critic methods solve reinforcement learning problems by updating a parameterized policy known as an actor in a direction that increases an estimate of the expected return known as a critic. However, existing actor-critic methods only use values or gradients of the critic to update the policy parameter. In this paper, we propose a novel actor-critic method called the guide actor-critic (GAC). GAC firstly learns a guide actor that locally maximizes the critic and then it updates the policy parameter based on the guide actor by supervised learning. Our main theoretical contributions are two folds. First, we show that GAC updates the guide actor by performing second-order optimization in the action space where the curvature matrix is based on the Hessians of the critic. Second, we show that the deterministic policy gradient method is a special case of GAC when the Hessians are ignored. Through experiments, we show that our method is a promising reinforcement learning method for continuous controls. The goal of reinforcement learning (RL) is to learn an optimal policy that lets an agent achieve the maximum cumulative rewards known as the return BID31. Reinforcement learning has been shown to be effective in solving challenging artificial intelligence tasks such as playing games BID20 and controlling robots BID6.Reinforcement learning methods can be classified into three categories: value-based, policy-based, and actor-critic methods. Value-based methods learn an optimal policy by firstly learning a value function that estimates the expected return. Then, they infer an optimal policy by choosing an action that maximizes the learned value function. Choosing an action in this way requires solving a maximization problem which is not trivial for continuous controls. While extensions to continuous controls were considered recently, they are restrictive since specific structures of the value function are assumed BID10 BID3.On the other hand, policy-based methods, also called policy search methods BID6, learn a parameterized policy maximizing a sample approximation of the expected return without learning the value function. For instance, policy gradient methods such as REIN-FORCE BID34 use gradient ascent to update the policy parameter so that the probability of observing high sample returns increases. Compared with value-based methods, policy search methods are simpler and naturally applicable to continuous problems. Moreover, the sample return is an unbiased estimator of the expected return and methods such as policy gradients are guaranteed to converge to a locally optimal policy under standard regularity conditions BID32. However, sample returns usually have high variance and this makes such policy search methods converge too slowly. Actor-critic methods combine the advantages of value-based and policy search methods. In these methods, the parameterized policy is called an actor and the learned value-function is called a critic. The goal of these methods is to learn an actor that maximizes the critic. Since the critic is a low variance estimator of the expected return, these methods often converge much faster than policy search methods. Prominent examples of these methods are actor-critic BID32 BID15, natural actor-critic BID25, trust-region policy optimization BID27, and asynchronous advantage actor-critic BID21. While their approaches to learn the actor are different, they share a common property that they only use the value of the critic, i.e., the zero-th order information, and ignore higher-order ones such as gradients and Hessians w.r.t. actions of the critic 1. To the best of our knowledge, the only actor-critic methods that use gradients of the critic to update the actor are deterministic policy gradients (DPG) BID29 and stochastic value gradients. However, these two methods do not utilize the second-order information of the critic. In this paper, we argue that the second-order information of the critic is useful and should not be ignored. A motivating example can be seen by comparing gradient ascent to the Newton method: the Newton method which also uses the Hessian converges to a local optimum in a fewer iterations when compared to gradient ascent which only uses the gradient BID24. This suggests that the Hessian of the critic can accelerate actor learning which leads to higher data efficiency. However, the computational complexity of second-order methods is at least quadratic in terms of the number of optimization variables. For this reason, applying second-order methods to optimize the parameterized actor directly is prohibitively expensive and impractical for deep reinforcement learning which represents the actor by deep neural networks. Our contribution in this paper is a novel actor-critic method for continuous controls which we call guide actor-critic (GAC). Unlike existing methods, the actor update of GAC utilizes the secondorder information of the critic in a computationally efficient manner. This is achieved by separating actor learning into two steps. In the first step, we learn a non-parameterized Gaussian actor that locally maximizes the critic under a Kullback-Leibler (KL) divergence constraint. Then, the Gaussian actor is used as a guide for learning a parameterized actor by supervised learning. Our analysis shows that learning the mean of the Gaussian actor is equivalent to performing a second-order update in the action space where the curvature matrix is given by Hessians of the critic and the step-size is controlled by the KL constraint. Furthermore, we establish a connection between GAC and DPG where we show that DPG is a special case of GAC when the Hessians and KL constraint are ignored. In this section, we firstly give a of reinforcement learning. Then, we discuss existing second-order methods for policy learning and their issue in deep reinforcement learning. We consider discrete-time Markov decision processes (MDPs) with continuous state space S Ď R ds and continuous action space A Ď R da. We denote the state and action at time step t P N by s t and a t, respectively. The initial state s 1 is determined by the initial state density s 1 " ppsq. At time step t, the agent in state s t takes an action a t according to a policy a t " πpa|s t q and obtains a reward r t " rps t, a t q. Then, the next state s t`1 is determined by the transition function s t`1 " pps 1 |s t, a t q. A trajectory τ " ps 1, a 1, r 1, s 2,... q gives us the cumulative rewards or return defined as ř 8 t"1 γ t´1 rps t, a t q, where the discount factor 0 ă γ ă 1 assigns different weights to rewards given at different time steps. The expected return of π after executing an action a in a state s can be expressed through the action-value function which is defined as DISPLAYFORM0 where E p r¨s denotes the expectation over the density p and the subscript t ě 1 indicates that the expectation is taken over the densities at time steps t ě 1. We can define the expected return as DISPLAYFORM1 where G P R d θˆdθ is a curvature matrix. The behavior of second-order methods depend on the definition of a curvature matrix. The most well-known second-order method is the Newton method where its curvature matrix is the Hessian of the objective function w.r.t. the optimization variables: DISPLAYFORM2 The natural gradient method is another well-known second-order method which uses the Fisher information matrix (FIM) as the curvature matrix : DISPLAYFORM3 Unlike the Hessian matrix, FIM provides information about changes of the policy measured by an approximated KL divergence: E ppsq rKLpπ θ pa|sq||π θ 1 pa|sqqs « pθ´θ 1 q J G FIM pθ´θ 1 q BID13. We can see that G Hessian and G FIM are very similar but the former also contains the critic and the Hessian of the actor while the latter does not. This suggests that the Hessian provides more information than that in FIM. However, FIM is always positive semi-definite while the Hessian may be indefinite. Please see BID7 for detailed comparisons between the two curvature matrices in policy search 4. Nonetheless, actor-critic methods based on natural gradient were shown to be very efficient BID25 BID27 ).We are not aware of existing work that considers second-order updates for DPG or SVG. However, their second-order updates can be trivially derived. For example, a Newton update for DPG is DISPLAYFORM4 where the pi, jq-th entry of the Hessian matrix DISPLAYFORM5 Note that Bπ θ psq{Bθ and B 2 π θ psq{BθBθ 1 are vectors since π θ psq is a vector-valued function. Interestingly, the Hessian of DPG contains the Hessians of the actor and the critic. In contrast, the Hessian of the actor-critic method contains the Hessian of the actor and the value of the critic. Second-order methods are appealing in reinforcement learning because they have high data efficiency. However, inverting the curvature matrix (or solving a linear system) requires cubic computational complexity in terms of the number of optimization variables. For this reason, the secondorder updates in Eq. and Eq. are impractical in deep reinforcement learning due to a large number of weight parameters in deep neural networks. In such a scenario, an approximation of the curvature matrix is required to reduce the computational burden. For instance, BID7 proposed to use only diagonal entries of an approximated Hessian matrix. However, this approximation clearly leads to a loss of useful curvature information since the gradient is scaled but not rotated. More recently, BID35 proposed a natural actor-critic method that approximates block-diagonal entries of FIM. However, this approximation corresponds to ignoring useful correlations between weight parameters in different layers of neural networks. In this section, we propose the guide actor-critic (GAC) method that performs second-order updates without the previously discussed computational issue. Unlike existing methods that directly learn the parameterized actor from the critic, GAC separates the problem of learning the parameterized actor into problems of 1) learning a guide actor that locally maximizes the critic, and 2) learning a parameterized actor based on the guide actor. This separation allows us to perform a second-order update for the guide actor where the dimensionality of the curvature matrix is independent of the parameterization of the actor. We formulate an optimization problem for learning the guide actor in Section 3.1 and present its solution in Section 3.2. Then in Section 3.3 and Section 3.4, we show that the solution corresponds to performing second-order updates. Finally, Section 3.5 presents the learning step for the parameterized actor using supervised learning. The pseudo-code of our method is provided in Appendix B and the source code is available at https://github.com/voot-t/guide-actor-critic. Our first goal is to learn a guide actor that maximizes the critic. However, greedy maximization should be avoided since the critic is a noisy estimate of the expected return and a greedy actor may change too abruptly across learning iterations. Such a behavior is undesirable in real-world problems, especially in robotics BID6. Instead, we maximize the critic with additional constraints: DISPLAYFORM0 subject to E p β psq rKLpπpa|sq||π θ pa|sqqs ď, DISPLAYFORM1 whereπpa|sq is the guide actor to be learned, π θ pa|sq is the current parameterized actor that we want to improve upon, and p β psq is the state distribution induced by past trajectories. The objective function differs from the one in Eq. in two important aspects. First, we maximize for a policy functionπ and not for the policy parameter. This is more advantageous than optimizing for a policy parameter since the policy function can be obtained in a closed form, as will be shown in the next subsection. Second, the expectation is defined over a state distribution from past trajectories and this gives us off-policy methods with higher data efficiency. The first constraint is the Kullback-Leibler (KL) divergence constraint where KLpppxq||qpxqq " E ppxq rlog ppxq´log qpxqs. The second constraint is the Shannon entropy constraint where Hpppxqq "´E ppxq rlog ppxqs. The KL constraint is commonly used in reinforcement learning to prevent unstable behavior due to excessively greedy update BID25 BID26 BID16 BID27. The entropy constraint is crucial for maintaining stochastic behavior and preventing premature convergence BID36 BID0 BID21 BID11. The final constraint ensures that the guide actor is a proper probability density. The KL bound ą 0 and the entropy bound´8 ă κ ă 8 are hyper-parameters which control the explorationexploitation trade-off of the method. In practice, we fix the value of and adaptively reduce the value of κ based on the current actor's entropy, as suggested by BID0. More details of these tuning parameters are given in Appendix C.This optimization problem can be solved by the method of Lagrange multipliers. The solution is DISPLAYFORM2 where η ‹ ą 0 and ω ‹ ą 0 are dual variables corresponding to the KL and entropy constraints, respectively. The dual variable corresponding to the probability density constraint is contained in the normalization term and is determined by η ‹ and ω ‹. These dual variables are obtained by minimizing the dual function: DISPLAYFORM3 All derivations and proofs are given in Appendix A. The solution in Eq. FORMULA0 tells us that the guide actor is obtained by weighting the current actor with p Qps, aq. If we set Ñ 0 then we haveπ « π θ and the actor is not updated. On the other hand, if we set Ñ 8 then we havẽ πpa|sq 9 expp p Qps, aq{ω ‹ q, which is a softmax policy where ω ‹ is the temperature parameter. Computingπpa|sq and evaluating gpη, ωq are intractable for an arbitrary π θ pa|sq. We overcome this issue by imposing two assumptions. First, we assume that the actor is the Gaussian distribution: DISPLAYFORM0 where the mean φ θ psq and covariance Σ θ psq are functions parameterized by a policy parameter θ. Second, we assume that Taylor's approximation of p Qps, aq is locally accurate up to the secondorder. More concretely, the second-order Taylor's approximation using an arbitrary action a 0 is given by DISPLAYFORM1 where g 0 psq " ∇ a p Qps, aq| a"a0 and H 0 psq " ∇ 2 a p Qps, aq| a"a0 are the gradient and Hessian of the critic w.r.t. a evaluated at a 0, respectively. By assuming that the higher order term Op}a} 3 q is sufficiently small, we can rewrite Taylor's approximation at a 0 as DISPLAYFORM2 where ψ 0 psq " g 0 psq´H 0 psqa 0 and ξ 0 psq " DISPLAYFORM3 Note that H 0 psq, ψ 0 psq, and ξ 0 psq depend on the value of a 0 and do not depend on the value of a. This dependency is explicitly denoted by the subscript. The choice of a 0 will be discussed in Section 3.3.Substituting the Gaussian distribution and Taylor's approximation into Eq. FORMULA0 yields another Gaussian distributionπpa|sq " N pa|φ`psq, Σ`psqq, where the mean and covariance are given by DISPLAYFORM4 The matrix F psq P R daˆda and vector Lpsq P R da are defined as DISPLAYFORM5 The dual variables η ‹ and ω ‹ are obtained by minimizing the following dual function: DISPLAYFORM6 where F η psq and L η psq are defined similarly to F psq and Lpsq but with η instead of η ‹.The practical advantage of using the Gaussian distribution and Taylor's approximation is that the guide actor can be obtained in a closed form and the dual function can be evaluated through matrixvector products. The expectation over p β psq can be approximated by e.g., samples drawn from a replay buffer BID20. We require inverting F η psq to evaluate the dual function. However, these matrices are computationally cheap to invert when the dimension of actions is not large. As shown in Eq., the mean and covariance of the guide actor is computed using both the gradient and Hessian of the critic. Yet, these computations do not resemble second-order updates discussed previously in Section 2.2. Below, we show that for a particular choice of a 0, the mean computation corresponds to a second-order update that rotates gradients by a curvature matrix. For now we assume that the critic is an accurate estimator of the true action-value function. In this case, the quality of the guide actor depends on the accuracy of sample approximation in p gpη, ωq and the accuracy of Taylor's approximation. To obtain an accurate Taylor's approximation of p Qps, aq using an action a 0, the action a 0 should be in the vicinity of a. However, we did not directly use any individual a to compute the guide actor, but we weight π θ pa|sq by expp p Qps, aqq (see Eq. FORMULA0). Thus, to obtain an accurate Taylor's approximation of the critic, the action a 0 needs to be similar to actions sampled from π θ pa|sq. Based on this observation, we propose two approaches to perform Taylor's approximation. Taylor's approximation around the mean. In this approach, we perform Taylor's approximation using the mean of π θ pa|sq. More specifically, we use a 0 " E π θ pa|sq ras " φ θ psq for Eq.. In this case, we can show that the mean update in Eq. corresponds to performing a second-order update in the action space to maximize p Qps, aq: DISPLAYFORM0 where DISPLAYFORM1 Qps, aq| a"φ θ psq. This equivalence can be shown by substitution and the proof is given in Appendix A.2. This update equation reveals that the guide actor maximizes the critic by taking a step in the action space similarly to the Newton method. However, the main difference lies in the curvature matrix where the Newton method uses Hessians H φ θ psq but we use a damped Hessian F φ θ psq. The damping term η ‹ Σ´1 θ psq corresponds to the effect of the KL constraint and can be viewed as a trust-region that controls the step-size. This damping term is particularly important since Taylor's approximation is accurate only locally and we should not take a large step in each update BID24.Expectation of Taylor's approximations. Instead of using Taylor's approximation around the mean, we may use an expectation of Taylor's approximation over the distribution. More concretely, we define r Qps, aq to be an expectation of p Q 0 ps, aq over π θ pa 0 |sq: DISPLAYFORM2 Note that E π θ pa0|sq rH 0 psqs " E π θ pa0|sq r∇ 2 a p Qps, aq| a"a0 s and the expectation is computed w.r.t. the distribution π θ of a 0. We use this notation to avoid confusion even though π θ pa 0 |sq and π θ pa|sq are the same distribution. When Eq. FORMULA1 is used, the mean update does not directly correspond to any second-order optimization step. However, under an (unrealistic) independence assumption E π θ pa0|sq rH 0 psqa 0 s " E π θ pa0|sq rH 0 psqsE π θ pa0|sq ra 0 s, we can show that the mean update corresponds to the following second-order optimization step: DISPLAYFORM3 where E π θ pa0|sq rF 0 psqs " η ‹ Σ´1 θ psq´E π θ pa0|sq rH 0 psqs. Interestingly, the mean is updated by rotating an expected gradient using an expected Hessians. In practice, the expectations can be approximated using sampled actions ta 0,i u S i"1 " π θ pa|sq. We believe that this sampling can be advantageous for avoiding local optima. Note that when the expectation is approximated by a single sample a 0 " π θ pa|sq, we obtain the update in Eq. regardless of the independence assumption. In the remainder, we use F psq to denote both of F φ θ psq and E π θ pa0|sq rF 0 psqs, and use Hpsq to denote both of H φ θ psq and E π θ pa0|sq rH 0 psqs. In the experiments, we use GAC-0 to refer to GAC with Taylor's approximation around the mean, and we use GAC-1 to refer to GAC with Taylor's approximation by a single sample a 0 " π θ pa|sq. The covariance update in Eq. indicates that F psq " η ‹ Σ´1 θ psq´Hpsq needs to be positive definite. The matrix F psq is guaranteed to be positive definite if the Hessian matrix Hpsq is negative semi-definite. However, this is not guaranteed in practice unless p Qps, aq is a concave function in terms of a. To overcome this issue, we firstly consider the following identity: DISPLAYFORM0 The proof is given in Appendix A.3. The first term is always negative semi-definite while the second term is indefinite. Therefore, a negative semi-definite approximation of the Hessian can be obtained as DISPLAYFORM1 The second term in Eq. FORMULA1 is proportional to expp´p Qps, aqq and it will be small for high values of p Qps, aq. This implies that the approximation should gets more accurate as the policy approach a local maxima of p Qps, aq. We call this approximation Gauss-Newton approximation since it is similar to the Gauss-Newton approximation for the Newton method BID24. The second step of GAC is to learn a parameterized actor that well represents the guide actor. Below, we discuss two supervised learning approaches for learning a parameterized actor. Since the guide actor is a Gaussian distribution with a state-dependent mean and covariance, a natural choice for the parameterized actor is again a parameterized Gaussian distribution with a state-dependent mean and covariance: π θ pa|sq " N pa|φ θ psq, Σ θ psqq. The parameter θ can be learned by minimizing the expected KL divergence to the guide actor: DISPLAYFORM0 where DISPLAYFORM1 ı is the weighted-mean-squared-error (WMSE) which only depends on θ of the mean function. The const term does not depend on θ. Minimizing the KL divergence reveals connections between GAC and deterministic policy gradients (DPG) BID29. By computing the gradient of the WMSE, it can be shown that DISPLAYFORM2 The proof is given in Appendix A.4. The negative of the first term is precisely equivalent to DPG. Thus, updating the mean parameter by minimizing the KL loss with gradient descent can be regarded as updating the mean parameter with biased DPG where the bias terms depend on η ‹. We can verify that ∇ a p Qps, aq| a"φ`psq " 0 when η ‹ " 0 and this is the case of Ñ 8. Thus, all bias terms vanish when the KL constraint is ignored and the mean update of GAC coincides with DPG. However, unlike DPG which learns a deterministic policy, we can learn both the mean and covariance in GAC. While a state-dependent parameterized covariance function is flexible, we observe that learning performance is sensitive to the initial parameter of the covariance function. For practical purposes, we propose using a parametrized Gaussian distribution with state-independent covariance: π θ pa|sq " N pa|φ θ psq, Σq. This class of policies subsumes deterministic policies with additive independent Gaussian noise for exploration. To learn θ, we minimize the mean-squared-error (MSE): DISPLAYFORM0 For the covariance, we use the average of the guide covariances: Σ " pη ‹`ω‹ qE p β psq " F´1psq ‰. For computational efficiency, we execute a single gradient update in each learning iteration instead of optimizing this loss function until convergence. Similarly to the above analysis, the gradient of the MSE w.r.t. θ can be expanded and rewritten into DISPLAYFORM1 Again, the mean update of GAC coincides with DPG when we minimize the MSE and set η ‹ " 0 and Hpsq "´I where I is the identity matrix. We can also substitute these values back into Eq.. By doing so, we can interpret DPG as a method that performs first-order optimization in the action space: DISPLAYFORM2 and then uses the gradient in Eq. to update the policy parameter. This interpretation shows that DPG is a first-order method that only uses the first-order information of the critic for actor learning. Therefore in principle, GAC, which uses the second-order information of the critic, should learn faster than DPG. Beside actor learning, the performance of actor-critic methods also depends on the accuracy of the critic. We assume that the critic p Q ν ps, aq is represented by neural networks with a parameter ν. We adopt the approach proposed by with some adjustment to learn ν. More concretely, we use gradient descent to minimize the squared Bellman error with a slowly moving target critic: DISPLAYFORM0 where α ą 0 is the step-size. The target value y " rps, aq`γE πpa 1 |s 1 q r p Qν ps 1, a 1 qs is computed by the target critic p Qν ps 1, a 1 q whose parameterν is updated byν Ð τ ν`p1´τ qν for 0 ă τ ă 1. As suggested by, the target critic improves the learning stability and we set τ " 0.001 in experiments. The expectation for the squared error is approximated using mini-batch samples tps n, a n, r n, s 1 n qu N n"1 drawn from a replay buffer. The expectation over the current actor πpa 1 |s 1 q is approximated using samples ta DISPLAYFORM1 We do not use a target actor to compute y since the KL upper-bound already constrains the actor update and a target actor will further slow it down. Note that we are not restricted to this evaluation method and more efficient methods such as Retrace BID22 can also be used. Our method requires computing ∇ a p Q ν ps, aq and its outer product for the Gauss-Newton approximation. The computational complexity of the outer product operation is Opd 2 a q and is inexpensive when compared to the dimension of ν. For a linear-in-parameter model p Q ν ps, aq " ν J µps, aq, the gradient can be efficiently computed for common choices of the basis function µ such as the Gaussian function. For deep neural network models, the gradient can be computed by the automaticdifferentiation BID9 where its cost depends on the network architecture. Besides the connections to DPG, our method is also related to existing methods as follows. A similar optimization problem to Eq. was considered by the model-free trajectory optimization (MOTO) method BID1. Our method can be viewed as a non-trivial extension of MOTO with two significant novelties. First, MOTO learns a sequence of time-dependent log-linear Gaussian policies π t pa|sq " N pa|B t s`b t, Σ t q, while our method learns a log-nonlinear Gaussian policy. Second, MOTO learns a time-dependent critic given by p Q t ps, aq " 1 2 a J C t a`a J D t sà J c t`ξt psq and performs policy update with these functions. In contrast, our method learns a more complex critic and performs Taylor's approximation in each training step. Besides MOTO, the optimization problem also resembles that of trust region policy optimization (TRPO) BID27. TRPO solves the following optimization problem: DISPLAYFORM0 where p Qps, aq may be replaced by an estimate of the advantage function BID28 ). There are two major differences between the two problems. First, TRPO optimizes the policy parameter while we optimize the guide actor. Second, TRPO solves the optimization problem by conjugate gradient where the KL divergence is approximated by the Fisher information matrix, while we solve the optimization problem in a closed form with a quadratic approximation of the critic. Our method is also related to maximum-entropy RL BID36 BID4 BID11 BID23, which maximizes the expected cumulative reward with an additional entropy bonus: ř 8 t"1 E p π psq rrps t, a t q`αHpπpa t |s t qqs, where α ą 0 is a trade-off parameter. The optimal policy in maximum-entropy RL is the softmax policy given by DISPLAYFORM1 where Q ‹ soft ps, aq and V ‹ soft psq are the optimal soft action-value and state-value functions, respectively BID11 BID23. For a policy π, these are defined as Q π soft ps, aq " rps, aq`γE pps 1 |s,aq DISPLAYFORM2 DISPLAYFORM3 The softmax policy and the soft state-value function in maximum-entropy RL closely resemble the guide actor in Eq. when η ‹ " 0 and the log-integral term in Eq. when η " 0, respectively, except for the definition of action-value functions. To learn the optimal policy of maximum-entropy RL, BID11 proposed soft Q-learning which uses importance sampling to compute the soft value functions and approximates the intractable policy using a separate policy function. Our method largely differs from soft Q-learning since we use Taylor's approximation to convert the intractable integral into more convenient matrix-vector products. The idea of firstly learning a non-parameterized policy and then later learning a parameterized policy by supervised learning was considered previously in guided policy search (GPS) BID16. However, GPS learns the guide policy by trajectory optimization methods such as an iterative linear-quadratic Gaussian regulator BID18, which requires a model of the transition function. In contrast, we learn the guide policy via the critic without learning the transition function. We evaluate GAC on the OpenAI gym platform BID5 with the Mujoco Physics simulator BID33. The actor and critic are neural networks with two hidden layers of 400 and 300 units, as described in Appendix C. We compare GAC-0 and GAC-1 against deep DPG (DDPG), Q-learning with a normalized advantage function (Q-NAF) BID10, and TRPO BID27 BID35. FIG0 shows the learning performance on 9 continuous control tasks. Overall, both GAC-0 and GAC-1 perform comparably with existing methods and they clearly outperform the other methods in Half-Cheetah. The performance of GAC-0 and GAC-1 is comparable on these tasks, except on Humanoid where GAC-1 learns faster. We expect GAC-0 to be more stable and reliable but easier to get stuck at local optima. On the other hand, the randomness introduced by GAC-1 leads to high variance approximation but this could help escape poor local optima. We conjecture GAC-S that uses S ą 1 samples for the averaged Taylor's approximation should outperform both GAC-0 and GAC-1. While this is computationally expensive, we can use parallel computation to reduce the computation time. The expected returns of both GAC-0 and GAC-1 have high fluctuations on the Hopper and Walker2D tasks when compared to TRPO as can be seen in FIG0 and FIG0. We observe that they can learn good policies for these tasks in the middle of learning. However, the policies quickly diverge to poor ones and then they are quickly improved to be good policies again. We believe that this happens because the step-size F´1psq "`η ‹ Σ´1´Hpsq˘´1 of the guide actor in Eq. FORMULA1 can be very large near local optima for Gauss-Newton approximation. That is, the gradients near local optima have small magnitude and this makes the approximation Hpsq " ∇ a p Qps, aq∇ a p Qps, aq J small as well. If η ‹ Σ´1 is also relatively small then the matrix F´1psq can be very large. Thus, under these conditions, GAC may use too large step sizes to compute the guide actor and this in high fluctuations in performance. We expect that this scenario can be avoided by reducing the KL bound or adding a regularization constant to the Gauss-Newton approximation. TAB1 in Appendix C shows the wall-clock computation time. DDPG is computationally the most efficient method on all tasks. GAC has low computation costs on tasks with low dimensional actions and its cost increases as the dimensionality of action increases. This high computation cost is due to the dual optimization for finding the step-size parameters η and ω. We believe that the computation cost of GAC can be significantly reduced by letting η and ω be external tuning parameters. Actor-critic methods are appealing for real-world problems due to their good data efficiency and learning speed. However, existing actor-critic methods do not use second-order information of the critic. In this paper, we established a novel framework that distinguishes itself from existing work by utilizing Hessians of the critic for actor learning. Within this framework, we proposed a practical method that uses Gauss-Newton approximation instead of the Hessians. We showed through experiments that our method is promising and thus the framework should be further investigated. Our analysis showed that the proposed method is closely related to deterministic policy gradients (DPG). However, DPG was also shown to be a limiting case of the stochastic policy gradients when the policy variance approaches zero BID29. It is currently unknown whether our framework has a connection to the stochastic policy gradients as well, and finding such a connection is our future work. Our main goal in this paper was to provide a new actor-critic framework and we do not claim that our method achieves the state-of-the-art performance. However, its performance can still be improved in many directions. For instance, we may impose a KL constraint for a parameterized actor to improve its stability, similarly to TRPO BID27. We can also apply more efficient policy evaluation methods such as Retrace BID22 ) to achieve better critic learning. The solution of the optimization problem: DISPLAYFORM0 subject to E p β psq rKLpπpa|sq||π θ pa|sqqs ď, E p β psq rHpπpa|sqqs ě κ, DISPLAYFORM1 can be obtained by the method of Lagrange multipliers. The derivation here follows the derivation of similar optimization problems by BID26 and BID0. The Lagrangian of this optimization problem is DISPLAYFORM2 where η, ω, and ν are the dual variables. Then, by taking derivative of L w.r.t. r π we obtain DISPLAYFORM3 We set this derivation to zero in order to obtain 0 " E p β psq "ż´p Qps, aq´pη`ωq log r πpa|sq`η log π θ pa|sq¯da ´p η`ω´νq " p Qps, aq´pη`ωq log r πpa|sq`η log π θ pa|sq´pη`ω´νq. Then the solution is given by DISPLAYFORM4 To obtain the dual function gpη, ωq, we substitute the solution to the constraint terms of the Lagrangian and this gives us DISPLAYFORM5 After some calculation, we obtain Lpη, ω, νq " η ´ωκ`E p β psq rη`ω´νs DISPLAYFORM6 where in the second line we use the fact that expp´η`ω´ν η`ω q is the normalization term of r πpa|sq. Firstly, we show that GAC performs second-order optimization in the action space when Taylor's approximation is performed with a 0 " E πpa|sq ras " φ θ psq. Recall that Taylor's approximation with φ θ is given by DISPLAYFORM0 where ψ φ θ psq " ∇ a p Qps, aq| a"φ θ psq´H φ θ psqφ θ psq. By substituting ψ φ θ psq into Lpsq " η ‹ Σ´1 θ psqφ θ psq´ψ θ psq, we obtain DISPLAYFORM1 Therefore, the mean update is equivalent to φ`psq " F´1psqLpsq DISPLAYFORM2 which is a second-order optimization step with a curvature matrix F psq " η ‹ Σ´1 θ psq´H φ θ psq. Similarly, for the case where a set of samples ta 0 u " π θ pa 0 |sq " N pa 0 |φ θ psq, Σpsqq is used to compute the averaged Taylor's approximation, we obtain DISPLAYFORM3 Then, by assuming that E π θ pa0|sq rH 0 psqa 0 psqs " E π θ pa0|sq rH 0 s E π θ pa0|sq ra 0 s, we obtain DISPLAYFORM4 Therefore, we have a second-order optimization step DISPLAYFORM5 where F´1psq " η ‹ Σ´1 θ psq´E π θ pa0|sq rH 0 psqs is a curvature matrix. As described in the main paper, this interpretation is only valid when the equality E π θ pa0|sq rH 0 psqa 0 psqs " E π θ pa0|sq rH 0 s E π θ pa0|sq ra 0 s holds. While this equality does not hold in general, it holds when only one sample a 0 " π θ pa 0 |sq is used. Nonetheless, we can still use the expectation of Taylor's approximation to perform policy update regardless of this assumption. Let f ps, aq " expp p Qps, aqq, then the Hessian Hpsq " ∇ 2 a p Qps, aq can be expressed as Hpsq " ∇ a r∇ a log f ps, aqs DISPLAYFORM0 a f ps, aqf ps, aq´1 " ∇ a f ps, aq∇ f f ps, aq´1 p∇ a f ps, aqq J`∇2 a f ps, aqf ps, aq´1 "´∇ a f ps, aqf ps, aq´2 p∇ a f ps, aqq J`∇2 a f ps, aqf ps, aq´1 "´`∇ a f ps, aqf ps, aq´1˘`∇ a f ps, aqf ps, aq´1˘J`∇ 2 a f ps, aqf ps, aq´1 "´∇ a log f ps, aq∇ a log f ps, aq J`∇2 a f ps, aqf ps, aq´1 "´∇ a p Qps, aq∇ a p Qps, aq DISPLAYFORM1 which concludes the proof. Beside Gauss-Newton approximation, an alternative approach is to impose a special structure on p Qps, aq so that Hessians are always negative semi-definite. In literature, there exists two special structures that satisfies this requirement. Normalized advantage function (NAF) BID10: NAF represents the critic by a quadratic function with a negative curvature: DISPLAYFORM2 where a negative-definite matrix-valued function W psq, a vector-valued function bpsq and a scalarvalued function V psq are parameterized functions whose their parameters are learned by policy evaluation methods such as Q-learning BID31. With NAF, negative definite Hessians can be simply obtained as Hpsq " W psq. However, a significant disadvantage of NAF is that it assumes the action-value function is quadratic regardless of states and this is generally not true for most reward functions. Moreover, the Hessians become action-independent even though the critic is a function of actions. Input convex neural networks (ICNNs) BID3: ICNNs are neural networks with special structures which make them convex w.r.t. their inputs. Since Hessians of concave functions are always negative semi-definite, we may use ICNNs to represent a negative critic and directly use its Hessians. However, similarly to NAF, ICNNs implicitly assume that the action-value function is concave w.r.t. actions regardless of states and this is generally not true for most reward functions. We first consider the weight mean-squared-error loss function where the guide actor is N pa|φ`psq, Σ`psqq and the current actor is N pa|φ θ psq, Σ θ psqq. Taylor's approximation of DISPLAYFORM0 By assuming that H φ θ psq is strictly negative definite 5, we can take a derivative of this approximation w.r.t. a and set it to zero to obtain a " H´1 φ θ psq∇ a p Qps, aq´H´1 φ θ psqψ φ θ psq. Replacing a by φ θ psq and φ`psq yields DISPLAYFORM1 φ`psq " H´1 φ θ psq∇ a p Qps, aq| a"φ`psq´H´1 φ θ psqψ φ θ psq. Recall that the weight mean-squared-error is defined as DISPLAYFORM2 Published as a conference paper at ICLR 2018Firstly, we expand the quadratic term of NAF as follows: DISPLAYFORM3 where ψpsq "´W psqbpsq and ξpsq " 1 2 bpsq J W psqbpsq`V psq. By substituting the quadratic model obtained by NAF into the GAC framework, the guide actor is now given byπpa|sq " N pa|φ`psq, Σ`psqqq with DISPLAYFORM4 To obtain Q-learning with NAF, we set η ‹ " 0, i.e., we perform a greedy maximization where the KL upper-bound approaches infinity, and this yields φ`psq "´W psq´1p´W psqbpsqq DISPLAYFORM5 which is the policy obtained by performing Q-learning with NAF. Thus, NAF with Q-learning is a special case of GAC if Q-learning is also used in GAC to learn the critic. The pseudo-code of GAC is given in Algorithm 1. The source code is available at https:// github.com/voot-t/guide-actor-critic. We try to follow the network architecture proposed by the authors of each baseline method as close as possible. For GAC and DDPG, we use neural networks with two hidden layers for the actor network and the critic network. For both networks the first layer has 400 hidden units and the second layer has 300 units. For NAF, we use neural networks with two hidden layers to represent each of the functions bpsq, W psq and V psq where each layer has 200 hidden units. All hidden units use the relu activation function except for the output of the actor network where we use the tanh activation function to bound actions. We use the Adam optimizer BID14 with learning rate 0.001 and 0.0001 for the critic network and the actor network, respectively. The moving average step for target networks is set to τ " 0.001. The maximum size of the replay buffer is set to 1000000. The mini-batches size is set to N " 256. The weights of the actor and critic networks are initialized as described by BID8, except for the output layers where the initial weights are drawn uniformly from Up´0.003, 0.003q, as described by. The initial covariance Σ in GAC is set to be an identity matrix. DDPG and QNAF use the OU-process with noise parameters θ " 0.15 and σ " 0.2 for exploration.For TRPO, we use the implementation publicly available at https://github.com/openai/ baselines. We also use the provided network architecture and hyper-parameters except the batch size where we use 1000 instead of 1024 since this is more suitable in our test setup. For GAC, the KL upper-bound is fixed to " 0.0001. The entropy lower-bound κ is adjusted heuristically by κ " maxp0.99pE´E 0 q`E 0, E 0 q,where E « E p β psq rHpπ θ pa|sqqs denotes the expected entropy of the current policy and E 0 denotes the entropy of a base policy N pa|0, 0.01Iq. This heuristic ensures that the lower-bound gradually decreases but the lower-bound cannot be too small. We apply this heuristic update once every 5000 training steps. The dual function is minimize by the sequential least-squares quadratic programming (SLSQP) method with an initial values η " 0.05 and ω " 0.05. The number of samples for computing the target critic value is M " 10. Sample N mini-batch samples tps n, a n, r n, s Compute y n, update ν by, e.g., Adam, and updateν by moving average: DISPLAYFORM0 DISPLAYFORM1 DISPLAYFORM2 13:end procedure procedure LEARN GUIDE ACTOR 15:Compute a n,0 for each s n by a n,0 " φ θ ps n q or a n,0 " N pa|φ θ ps n q, Σq. Compute g 0 psq " ∇ a p Qps n, aq| a"an,0 and H 0 ps s q "´g 0 ps n qg 0 ps n q J. Solve for pη ‹, ω ‹ q " argmin ηą0,ωą0 p gpη, ωq by a non-linear optimization method. Compute the guide actor r πpa|s n q " N pa|φ`ps n q, Σ`ps n qq for each s n. end procedure procedure UPDATE PARAMETERIZED ACTOR Update policy parameter by, e.g., Adam, to minimize the MSE: DISPLAYFORM0 ∇ θ }φ θ ps n q´φ`ps n q} 2 2.22:Update policy covariance by averaging the guide covariances: DISPLAYFORM1 23:end procedure end procedure 25: end for 26: Output: Learned actor π θ pa|sq. We perform experiments on the OpenAI gym platform BID5 with Mujoco Physics simulator BID33 where all environments are v1. We use the state space, action space and the reward function as provided and did not perform any normalization or gradient clipping. The maximum time horizon in each episode is set to 1000. The discount factor γ " 0.99 is only used for learning and the test returns are computed without it. Experiments are repeated for 10 times with different random seeds. The total computation time are reported in TAB1. The figures below show the averaged over 10 trials. The y-axis indicates the averaged test returns where the test returns in each trial are computed once every 5000 training time steps by executing 10 test episodes without exploration. The error bar indicates standard error. The total computation time for training the policy for 1 million steps (0.1 million steps for the Invert-Pendulum task). The mean and standard error are computed over 10 trials with the unit in hours. TRPO is not included since it performs a lesser amount of update using batch data samples.

This paper proposes a novel actor-critic method that uses Hessians of a critic to update an actor.

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.


Question: Represent the article for finding a single-sentence TLDR

Question: Deep Infomax~(DIM) is an unsupervised representation learning framework by maximizing the mutual information between the inputs and the outputs of an encoder, while probabilistic constraints are imposed on the outputs. In this paper, we propose Supervised Deep InfoMax~(SDIM), which introduces supervised probabilistic constraints to the encoder outputs. The supervised probabilistic constraints are equivalent to a generative classifier on high-level data representations, where class conditional log-likelihoods of samples can be evaluated. Unlike other works building generative classifiers with conditional generative models, SDIMs scale on complex datasets, and can achieve comparable performance with discriminative counterparts. With SDIM, we could perform \emph{classification with rejection}. Instead of always reporting a class label, SDIM only makes predictions when test samples' largest logits surpass some pre-chosen thresholds, otherwise they will be deemed as out of the data distributions, and be rejected. Our experiments show that SDIM with rejection policy can effectively reject illegal inputs including out-of-distribution samples and adversarial examples. Non-robustness of neural network models emerges as a pressing concern since they are observed to be vulnerable to adversarial examples . Many attack methods have been developed to find imperceptible perturbations to fool the target classifiers (; ;). Meanwhile, many defense schemes have also been proposed to improve the robustnesses of the target models (; Tramèr et al., 2017; ;). An important fact about these works is that they focus on discriminative classifiers, which directly model the conditional probabilities of labels given samples. Another promising direction, which is almost neglected so far, is to explore robustness of generative classifiers . A generative classifier explicitly model conditional distributions of inputs given the class labels. During inference, it evaluates all the class conditional likelihoods of the test input, and outputs the class label corresponding to the maximum. Conditional generative models are powerful and natural choices to model the class conditional distributions, but they suffer from two big problems: it is hard to scale generative classifiers on high-dimensional tasks, like natural images classification, with comparable performance to the discriminative counterparts. Though generative classifiers have shown promising of adversarial robustness, they hardly achieve acceptable classification performance even on CIFAR10; ). The behaviors of likelihood-based generative models can be counter-intuitive and brittle. They may assign surprisingly higher likelihoods to out-of-distribution (OoD) samples . discuss the issues of likelihood as a metric for density modeling, which may be the reason of non-robust classification, e.g. OoD samples detection. In this paper, we propose supervised deep infomax (SDIM) by introducing supervised statistical constraints into deep infomax , an unsupervised learning framework by maximizing the mutual information between representations and data. SDIM is trained by optimizing two objectives: maximizing the mutual information (MI) between the inputs and the high-level data representations from encoder; ensuring that the representations satisfy the supervised statistical constraints. The supervised statistical constraints can be interpreted as a generative classifier on high-level data representations giving up the full generative process. Unlike full generative models making implicit manifold assumptions, the supervised statistical constraints of SDIM serve as explicit enforcement of manifold assumption: data representations (low-dimensional) are trained to form clusters corresponding to their class labels. With SDIM, we could perform classification with rejection . SDIMs reject illegal inputs based on off-manifold conjecture , where illegal inputs, e.g. adversarial examples, lie far away from the data manifold. Samples whose class conditionals are smaller than the pre-chosen thresholds will be deemed as off-manifold, and prediction requests on them will be rejected. The contributions of this paper are: • We propose Supervised Deep Infomax (SDIM), an end-to-end framework whose probabilistic constraints are equivalent to a generative classifier. SDIMs can achieve comparable classification performance with similar discrinimative counterparts at the cost of small over-parameterization. • We propose a simple but novel rejection policy based on off-manifold conjecture: SDIM outputs a class label only if the test sample's largest class conditional surpasses the prechosen class threshold, otherwise outputs rejection. The choice of thresholds relies only on training set, and takes no additional computations. • Experiments show that SDIM with rejection policy can effectively reject illegal inputs, including OoD samples and adversarial examples generated by a comprehensive group of adversarial attacks. Deep InfoMax is an unsupervised representation learning framework by maximizing the mutual information (MI) of the inputs and outputs of an encoder. The computation of MI takes only input-output pairs with the deep neural networks based esimator MINE . Let E φ be an encoder parameterized by φ, working on the training set, and generating. DIM is trained to find the set of parameters φ such that: the mutual information I(X, Y) is maximized over sample sets X and Y. the representations, depending on the potential downstream tasks, match some prior distribution. Denote J and M the joint and product of marginals of random variables X, Y respectively. MINE estimates a lower-bound of MI with Donsker-Varadhan representation of KL-divergence: where T ω (x, y) ∈ R is a family of functions with parameters ω represented by a neural network. Since in representation learning we are more interested in maximizing MI, than its exact value, non-KL divergences are also favorable candidates. We can get a family of variational lower-bounds using f -divergence representations : where f * is the Fenchel conjugate of a specific divergence f. For KL-divergence, f * (t) = e (t−1). A full f * list is provided in Tab. 6 of. Noise-Contrastive Estimation (Gutmann & Hyvärinen, 2010) can also be used as lower-bound of MI in "infoNCE" . All the components of SDIM framework are summurized in Fig. 1. The focus of Supervised Deep InfoMax (SDIM) is on introducing supervision to probabilistic constraints of DIM for (generative) classification. We choose to maximize the local MI, which has shown to be more effective in classification tasks than maximizing global MI. Equivalently, we minimize J MI: where L φ (x) is a local M × M feature map of x extracted from some intermediate layer of encoder E, andĨ can be any possible MI lower-bounds. By adopting a generative approach p(x, y) = p(y)p(x|y), we assume that the data follows the manifold assumption: the (high-dimensional) data lies on low-dimensional manifolds corresponding to their class labels. Denotex the compact representation generated with encoder E φ (x). In order to explicitly enforce the manifold assumption, we admit the existence of data manifold in the representation space. Assume that y is a discrete random variable representing class labels, and p(x|y) is the real class conditional distribution of the data manifold given y. Let p θ (x|y) be the class conditionals we model parameterized with θ. We approximate p(x|y) by minimizing the KL-divergence between p(x|y) and our model p θ (x|y), which is given by: where the first item on RHS is a constant independent of the model parameters θ. Eq. 4 equals to maximize the expectation Ex,y∼p(x,y) [log p θ (x|y)]. In practice, we minimize the following loss J NLL, equivalent to empicically maximize the above expectation over: Besides the introduction of supervision, SDIM differs from DIM in its way of enforcing the statistical constraints: DIM use adversarial learning to push the representations to the desired priors, while SDIM directly maximizes the parameterized class conditional probability. Maximize Likelihood Margins Since a generative classifier, at inference, decides which class a test input x belongs to according to its class conditional probability. On one hand, we maximize samples' true class conditional probabilities (classes they belong to) using J NLL; On the other hand, we also hope that samples' false class conditional probabilities (classes they do not belong to) can be minimized. This is assured by the following likelihood margin loss J LM: where K is a positive constant to control the margin. For each encoder outputx i, the C − 1 truefalse class conditional gaps are squared 1, which quadratically increases the penalties when the gap becomes large, then are averaged. Putting all these together, the complete loss function we minimize is: Parameterization of Class Conditional Probability Each of the class conditional distribution is represented as an isotropic Gaussian. So the generative classifier is simply a embedding layer with C entries, and each entry contains the trainable mean and variance of a Gaussian. This minimized parameterization encourages the encoder to learn simple and stable low-dimensional representations that can be easily explained by even unimodal distributions. Considering that we maximize the true class conditional probability, and minimize the false class conditional probability at the same time, we do not choose conditional normalizing flows, since the parameters are shared across class labels, and the training can be very difficult. , each class conditional probability is represented with a VAE, thus scaling to complex datasets with huge number of classes, e.g. ImageNet, is almost impossible. A generative approach models the class-conditional distributions p(x|y), as well as the class priors p(y). For classification, we compute the posterior probabilities p(y|x) through Bayes' rule: The prior p(y) can be computed from the training set, or we simply use uniform class prior for all class labels by default. Then the prediction of test sample x * from posteriors is: The drawback of the above decision function is that it always gives a prediction even for illegal inputs. Instead of simply outputting the class label that maximizes class conditional probability of x *, we set a threshold for each class conditional probability, and define our decision function with rejection to be: The model gives a rejection when log p(x * |y *) is smaller than the threshold δ y *. Note that here we can use p(x * |y *) and p(x * |y *) interchangeably. This is also known as selective classification or classification with reject option (See Supp. A) 4 RELATED WORKS Robustness of Likelihood-based Generative Models Though likelihood-based generative models have achieved great success in samples synthesis, the behaviors of their likelihoods can be counter-intuitive. Flow-based models and as well as VAEs , surprisingly assign even higher likelihoods to out-of-distribution samples than the samples in the training set . Pixel-level statistical analyses in show that OoD dataset may "sit inside of" the in-distribution dataset (i.e. with roughly the same mean but smaller variance). observe that adversarial examples are outside the training distribution via statistical testing. DefenseGAN models real data distribution with the generator G of GAN. At inference, instead of feeding the test input x to the target classifier directly, it searches for the "closest" sample G(z *) from generator distribution to x as the final input to the classifier. It ensures that the classifier only make predictions on the data manifold represented by the generator, ruling out the potential adversarial perturbations in x. PixelDefend takes a similar approach which uses likelihood-based generative model -PixelCNN to model the data distribution. Both DefenseGAN and PixelDefend are additionally trained as peripheral defense schemes agnostic to the target classifiers. Training generative models on complex datasets notoriously takes huge amount of computational resources . In contrast, the training of SDIM is computationally similar to its discriminative counterpart. The verification of whether inputs are offmanifold is a built-in property of the SDIM generative classifier. The class conditionals of SDIM are modeled on low-dimensional data representations with simple Gaussians, which is much easier, and incurs very small computations. Datasets We evaluate the effectiveness of the rejection policy of SDIM on four image datasets: MNIST, FashionMNIST (both resized to 32×32 from 28×28); and CIFAR10, SVHN. See App. B.1 for details of data processing. For out-of-distribution samples detection, we use the dataset pairs on which likelihood-based generative models fail : FashionMNIST (in)-MNIST (out) and CIFAR10 (in)-SVHN (out). Adversarial examples detection are evaluated on MNIST and CIFAR10. Choice of thresholds It is natural that choosing thresholds based on what the model knows, i.e. training set, and can reject what the model does not know, i.e. possible illegal inputs. We set one threshold for each class conditional. For each class conditional probability, we choose to evaluate on two different thresholds: 1st and 2nd percentiles of class conditional log-likelihoods of the correctly classified training samples. Compared to the detection methods proposed in, our choice of thresholds is much simpler, and takes no additional computations. Models A typical SDIM instance consists of three networks: an encoder, parameterized by φ, which outputs a d-dimensional representation; mutual information evaluation networks, i.e. T ω in Eqn. and Eqn.; and C-way class conditional embedding layer, parameterized by θ, with each entry a 2d-dimensional vector. We set d = 64 in all our experiments. For encoder of SDIM, we use ResNet on 32 × 32 with a stack of 8n + 2 layers, and 4 filter sizes {32, 64, 128, 256}. The architecture is summarized as: The last layer of encoder is a d-way fully-connected layer. To construct a discriminative counterpart, we simply set the output size of the encoder's last layer to C for classification. We use ResNet10 (n = 1) on MNIST, FashionMNIST, and ResNet26 (n = 3) on CIFAR10, SVHN. We report the classification accuracies (see Tab. 1) of SDIMs and the discriminative counterparts on clean test sets. Results show that SDIMs achieve the same level of accuracy as the discriminative counterparts with slightly increased number of parameters (17% increase for ResNet10, and 5% increase for ResNet26). We are aware of the existence of better reported on these datasets using more complex models or automatically designed architectures , but pushing the state-of-the-art is not the focus of this paper. , both model class conditional probability with VAE , and achieve acceptable accuracies (> 98%) on MNIST. However, it is hard for fully conditional generative models to achieve satisfactory classification accuracies even on CIFAR10. On CIFAR10, methods in achieve only < 50% accuracy. They also point out that the classification accuracy of a conditional PixelCNN++ is only 72.4%. The test accuracy of ABS in is only 54%. In contrast, SDIM could achieve almost the same performance with similar discriminative classifier by giving up the full generative process, and building generative classifier on high-level representations. improves the accuracy to 92% by feeding the features learned by powerful discriminative classifier-VGG16 to their generative classifiers, which also suggests that modeling likelihood on high-level representation (features) is more favorable for generative classification than pixel-level likelihood of fully generative classifiers. For classification tasks, discovering discriminative features is much more important than reconstructing the all the image pixels. Thus performing generative classification with full generative models may not be the right choice. We also investigate the implications of the proposed decision function with rejection under different thresholds. The in Tab. 2 show that choosing a higher percentile as threshold will reject more prediction requests. At the same time, the classification accuracies of SDIM on the left test sets become increasingly better. This demonstrate that out rejection policy tend to reject the ones on which SDIMs make wrong predictions. Table 2: Classification performances of SDIMs using the proposed decision function with rejection. We report the rejection rates of the test sets and the accuracies on the left test sets for each threshold. Class-wise OoD detections are performed, and mean detection rates over all in-distribution classes are reported in Tab. 3. For each in-distribution class c, we evaluate the log-likelihoods of the whole OoD dataset. Samples whose log-likelihoods are lower the class threshold δ c will be detected as OoD samples. Same evaluations are applied on conditional Glows with 10th percentile thresholds, but the are not good. The are clear and confirm that SDIMs, generative classifiers on high-level representations, are more effective on classification tasks than fully conditional generative models on raw pixels. Note that fully generative models including VAE used in; fail on OoD detection. The stark difference between SDIM and full generative models (flows or VAEs) is that SDIM models samples' likelihoods in the high-level representation spaces, while generative models evaluate directly on the raw pixels. See Supp. C for more about the histograms of the class conditionals of in-out distributions. Table 3: Mean detection rates of SDIMs and Glows with different thresholds on OoD detection. We comprehensively evaluate the robustness of SDIMs against various attacks: • gradient-based attacks: one-step gradient attack FGSM , its iterative variant projected gradient descent , CW-L 2 attack , deepfool . • score-based attacks: local search attack . • decision-based attack: boundary attack . Attacks Using Cross-Entropy We find that SDIMs are much more robust to gradient-based attacks using cross-entropy, e.g. FGSM and PGD, since the gradients numerically vanish as a side effect of the likelihood margin loss J LM of SDIM. This phenomenon is similar to some defences that try to hinder generations of adversarial examples by masking the gradients on inputs. While full generative classifiers in still suffer from these attacks. See Supp. D.1 for detailed . Conservative Adversarial Examples Adversarial attacks aim to find the minimal perturbations that sufficiently change the classification labels, i.e. flip other logits to be the largest one. We show case examples on MNIST generated by untargeted attacks and their logits in Tab. 4 (See Supp. D.2 for examples of CIFAR10). Though these attacks successfully flip the logits, they are designed to be conservative to avoid more distortions to the original images. As a , the largest logits of adversarial examples are still much lower than the thresholds, so they can be detected by our rejection policy. We find that our rejection policy performs perfectly on MNIST, but fails to detect all adversarial examples on CIFAR10 except for Boundary attack (See Tab. 5). It seems to be a well-known observation that models trained on CIFAR10 are more vulnerable than one trained on MNIST. connects this observation to the generalization of models. They found that many test samples, though correctly classified, are close to the misclassfied samples, which implies the existence of adversarial examples. If a model has higher error rate, it would take smaller perturbations to move correctly classified samples to misclassified areas. Table 4: Full logits of the adversarial examples generated with different attacks. The original image is the first sample of class 0 of MNIST test set. The first row gives the 1st percentile thresholds, and the second row shows the logits of the original image. The largest logits are marked in bold. Adversarial examples with more confidence Based on the observations above, a natural question we should ask is: can we generate adversarial examples with not only successfully flipped logits, Table 5: Detection rates of our rejection policies. We perform untargeted adversarial evaluation on the first 1000 images of test sets. CW-L 2 is not involved here, but carefully investigated below. but also the largest logit larger than some threshold value? Unlike the conservativeness on paying more distortions of other attacks, CW attack allows us to control the gap between largest and second largest logits with some confidence value κ. We perform targeted CW attacks with confidences κ = {0, 500, 1000} (Tab. 6). We find that increasing the confidences help increasing the largest logits of adversarial examples to some extent, but may lead to failures of generation. The sensitivity to confidence values is also different given different targets. The success rates of generating adversarial examples monotonically decreases with the confidences increasing (Tab. 7). Note that on discriminative counterparts, CW-L 2 with the same settings easily achieves 100% success rates. This means that explicitly forcing data representations to form clusters with maximum margins between them help increase average distances between normal samples and the nearest misclassified areas, thus increase the hardness of finding minimal adversarial perturbations. In this case, it takes a large enough adversarial perturbation to move a sample from its cluster to the other. Meanwhile, detection rates remain satisfactory on MNIST, but obviously decline on CIFAR10. For victim generative classifiers in under CW-L 2 attack, the detection rates of adversarial examples using the proposed detection methods can be > 95% on MNIST, but fall < 50% on even CIFAR10-binary (their models don't scale on CIFAR10, and CW-L 2 with non-zero confidences are also not evaluated). Table 7: Targeted adversarial evaluations of our rejection policies on the first 1000 test samples. We report the detection rates with different thresholds and success rates of generating adversarial examples. Discussions on off-manifold conjecture challenges whether the off-manifold conjecture holds in general. They experiment on synthetic dataset-two high-dimensional concentric spheres with theoretical analyses, showing that even for a trained classifier with close to zero test error, there may be a constant fraction of the data manifold misclassified, which indicates the existence of adversarial examples within the manifold. But there are still several concerns to be addressed: First, as also pointed out by the authors, the manifolds in natural datasets can be quite complex than that of simple synthesized dataset. draws similar from analyses on synthesized data with particular geometry. So the big concern is whether the in; still hold for the manifolds in natural datasets. A practical obstacle to verify this is that works modeling the full generative processes are based on manifold assumption, but provide no explicit manifolds for analytical analyses like;. While SDIM enables explicit and customized manifolds on high-level data representations via probabilistic constraints, thus enables analytical analyses. In this paper, samples of different classes are trained to form isotropic Gaussians corresponding to their classes in representation space (other choices are possible). The relation between the adversarial robustness and the forms and dimensionalities of data manifolds is to be explored. Second, in their experiments, all models evaluated are discriminative classifiers. Considering the recent promising of generative classifiers against adversarial examples, would using generative classifiers lead to different ? One thing making us feel optimistic is that even though the existence of adversarial examples is inevitable, suggest that adversarial robustness can be improved by minimizing the test errors, which is also supported by our experimental differences on MNIST and CIFAR10. We introduce supervised probabilistic constraints to DIM. Giving up the full generative process, SDIMs are equivalent to generative classifiers on high-level data representations. Unlike full conditional generative models which achieve poor classification performance even on CIFAR10, SDIMs attain comparable performance as the discriminative counterparts on complex datasets. The training of SDIM is also computationally similar to discriminative classifiers, and does not require prohibitive computational resources. Our proposed rejection policy based on off-manifold conjecture, a built-in property of SDIM, can effectively reject illegal inputs including OoD samples and adversarial examples. We demonstrate that likelihoods modeled on high-level data representations, rather than raw pixel intensities, are more robust on downstream tasks without the requirement of generating real samples. We make comparisons between SDIM and GBZ, which consistently performs best in Deep Bayes. FGSM and PGD-L ∞ The in Fig 4 and Fig 5 show that SDIM performs consistently better than the baseline. We find that increasing the distortion factor of FGSM has no influences of SDIM's accuracy, and the adversarial examples keep the same. Recall that the class conditionals are optimized to keep a considerable margin. Before evaluating the cross entropy loss, softmax is applied on the class conditionals log p(x|c) to generate a even sharper distribution. So for the samples that are correctly classified, their losses are numerically zeros, and the gradient on inputs ∇J x (x, y) are also numerically zeros. The PGD-L ∞ we use here is the randomized version 2, which adds a small random perturbation before the iterative loop. The randomness is originally introduced to generate different adversarial examples for adversarial training, but here it breaks the zero loss so that the gradient on inputs ∇J x (x, y) will not be zeros in the loop. FGSM can also be randomized (Tramèr et al., 2017), which can be seen as a one-step variant of randomized PGD. This phenomena is similar to what some defenses using gradient obfuscation want to achieve. Defensive distillation masks the gradients of cross-entropy by increasing the temperature of softmax. But for CW attacks, which do not use cross-entropy, and operate on logits directly, this could be ineffective. 4 -353.1 -471.6 -400.3 -342.7 -367.2 -486.4 -326.4 Boundary 213.9 -417.4 -458.3 -548.0 -587.4 -236.3 214.0 -1246.1 -171.2 -555.6 LocalSearch 165.2 -485.7 190.9 -325.6 -439.0 -379.0 -318.8 -327.5 -357.9 -272.3 Table 9: Full logits of adversarial examples generated with different attacks. The original image is the fist sample of class 0 of CIFAR10 test set. The first row gives the 1st percentile thresholds, and the second row shows the logits of the original image. The largest logits are marked in bold.

Represent the TLDR for finding the corresponding scientific article.

scale generative classifiers  on complex datasets, and evaluate their effectiveness to reject illegal inputs including out-of-distribution samples and adversarial examples.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a method based on the adversarial training strategy to learn discriminative features unbiased and invariant to the confounder(s) by incorporating a loss function that encourages a vanished correlation between the bias and learned features.