Question: represent this text:

Question: We propose a new sample-efficient methodology, called Supervised Policy Update (SPU), for deep reinforcement learning. Starting with data generated by the current policy, SPU formulates and solves a constrained optimization problem in the non-parameterized proximal policy space. Using supervised regression, it then converts the optimal non-parameterized policy to a parameterized policy, from which it draws new samples. The methodology is general in that it applies to both discrete and continuous action spaces, and can handle a wide variety of proximity constraints for the non-parameterized optimization problem. We show how the Natural Policy Gradient and Trust Region Policy Optimization (NPG/TRPO) problems, and the Proximal Policy Optimization (PPO) problem can be addressed by this methodology. The SPU implementation is much simpler than TRPO. In terms of sample efficiency, our extensive experiments show SPU outperforms TRPO in Mujoco simulated robotic tasks and outperforms PPO in Atari video game tasks. The policy gradient problem in deep reinforcement learning (DRL) can be defined as seeking a parameterized policy with high expected reward. An issue with policy gradient methods is poor sample efficiency BID10 BID21 BID27 BID29. In algorithms such as REINFORCE BID28, new samples are needed for every gradient step. When generating samples is expensive (such as robotic environments), sample efficiency is of central concern. The sample efficiency of an algorithm is defined to be the number of calls to the environment required to attain a specified performance level BID10.Thus, given the current policy and a fixed number of trajectories (samples) generated, the goal of the sample efficiency problem is to construct a new policy with the highest performance improvement possible. To do so, it is desirable to limit the search to policies that are close to the original policy π θ k BID21 BID29 BID24. Intuitively, if the candidate new policy π θ is far from the original policy π θ k, it may not perform better than the original policy because too much emphasis is being placed on the relatively small batch of new data generated by π θ k, and not enough emphasis is being placed on the relatively large amount of data and effort previously used to construct π θ k.This guideline of limiting the search to nearby policies seems reasonable in principle, but requires a distance η(π θ, π θ k) between the current policy π θ k and the candidate new policy π θ, and then attempt to solve the constrained optimization problem: DISPLAYFORM0 subject to η(π θ, π θ k) ≤ δwhereĴ(π θ | π θ k, new data) is an estimate of J(π θ), the performance of policy π θ, based on the previous policy π θ k and the batch of fresh data generated by π θ k. The objective attempts to maximize the performance of the updated policy, and the constraint ensures that the updated policy is not too far from the policy π θ k that was used to generate the data. Several recent papers BID21 BID24 belong to the framework-.Our work also strikes the right balance between performance and simplicity. The implementation is only slightly more involved than PPO. Simplicity in RL algorithms has its own merits. This is especially useful when RL algorithms are used to solve problems outside of traditional RL testbeds, which is becoming a trend BID30 BID16.We propose a new methodology, called Supervised Policy Update (SPU), for this sample efficiency problem. The methodology is general in that it applies to both discrete and continuous action spaces, and can address a wide variety of constraint types for. Starting with data generated by the current policy, SPU optimizes over a proximal policy space to find an optimal non-parameterized policy. It then solves a supervised regression problem to convert the non-parameterized policy to a parameterized policy, from which it draws new samples. We develop a general methodology for finding an optimal policy in the non-parameterized policy space, and then illustrate the methodology for three different definitions of proximity. We also show how the Natural Policy Gradient and Trust Region Policy Optimization (NPG/TRPO) problems and the Proximal Policy Optimization (PPO) problem can be addressed by this methodology. While SPU is substantially simpler than NPG/TRPO in terms of mathematics and implementation, our extensive experiments show that SPU is more sample efficient than TRPO in Mujoco simulated robotic tasks and PPO in Atari video game tasks. Off-policy RL algorithms generally achieve better sample efficiency than on-policy algorithms BID8. However, the performance of an on-policy algorithm can usually be substantially improved by incorporating off-policy training BID17, BID26 ). Our paper focuses on igniting interests in separating finding the optimal policy into a two-step process: finding the optimal non-parameterized policy, and then parameterizing this optimal policy. We also wanted to deeply understand the on-policy case before adding off-policy training. We thus compare with algorithms operating under the same algorithmic constraints, one of which is being on-policy. We leave the extension to off-policy to future work. We do not claim state-of-the-art . We consider a Markov Decision Process (MDP) with state space S, action space A, and reward function r(s, a), s ∈ S, a ∈ A. Let π = {π(a|s): s ∈ S, a ∈ A} denote a policy, let Π be the set of all policies, and let the expected discounted reward be: DISPLAYFORM0 where γ ∈ is a discount factor and τ = (s 0, a 0, s 1, . . .) is a sample trajectory. Let A π (s, a) be the advantage function for policy π BID14. Deep reinforcement learning considers a set of parameterized policies Π DL = {π θ |θ ∈ Θ} ⊂ Π, where each policy is parameterized by a neural network called the policy network. In this paper, we will consider optimizing over the parameterized policies in Π DL as well as over the non-parameterized policies in Π. For concreteness, we assume that the state and action spaces are finite. However, our methodology also applies to continuous state and action spaces, as shown in the Appendix. One popular approach to maximizing J(π θ) over Π DL is to apply stochastic gradient ascent. The gradient of J(π θ) evaluated at a specific θ = θ k can be shown to be BID28: DISPLAYFORM1 We can approximate by sampling N trajectories of length T from π θ k: DISPLAYFORM2 for the the future state probability distribution for policy π, and denote π(·|s) for the probability distribution over the action space A when in state s and using policy π. Further denote D KL (π π θ k)[s] for the KL divergence from π(·|s) to π θ k (·|s), and denote the following as the "aggregated KL divergence". DISPLAYFORM3 For the sample efficiency problem, the objective J(π θ) is typically approximated using samples generated from π θ k BID21. Two different approaches are typically used to approximate J(π θ) − J(π θ k). We can make a first order approximation of J(π θ) around θ k BID18 BID29 BID21: DISPLAYFORM0 where g k is the sample estimate. The second approach is to approximate the state distribution BID1: DISPLAYFORM1 DISPLAYFORM2 There is a well-known bound for the approximation BID11. Furthermore, the approximation DISPLAYFORM3 to the first order with respect to the parameter θ. Natural gradient was first introduced to policy gradient by and then in BID18 BID29 BID1 BID21. referred to collectively here as NPG/TRPO. Algorithmically, NPG/TRPO finds the gradient update by solving the sample efficiency problem- with η(π θ, π θ k) =D KL (π θ π θ k), i.e., use the aggregate KL-divergence for the policy proximity constraint. NPG/TRPO addresses this problem in the parameter space θ ∈ Θ. First, it approximates J(π θ) with the first-order approximation andD KL (π θ π θ k) using a similar second-order method. Second, it uses samples from π θ k to form estimates of these two approximations. Third, using these estimates (which are functions of θ), it solves for the optimal θ *. The optimal θ * is a function of g k and of h k, the sample average of the Hessian evaluated at θ k. TRPO also limits the magnitude of the update to ensureD KL (π θ π θ k) ≤ δ (i.e., ensuring the sampled estimate of the aggregated KL constraint is met without the second-order approximation).SPU takes a very different approach by first (i) posing and solving the optimization problem in the non-parameterized policy space, and then (ii) solving a supervised regression problem to find a parameterized policy that is near the optimal non-parameterized policy. A recent paper, Guided Actor Critic (GAC), independently proposed a similar decomposition BID24. However, GAC is much more restricted in that it considers only one specific constraint criterion (aggregated reverse-KL divergence) and applies only to continuous action spaces. Furthermore, GAC incurs significantly higher computational complexity, e.g. at every update, it minimizes the dual function to obtain the dual variables using SLSQP. MPO also independently propose a similar decomposition BID24. MPO uses much more complex machinery, namely, Expectation Maximization to address the DRL problem. However, MPO has only demonstrates preliminary on problems with discrete actions whereas our approach naturally applies to problems with either discrete or continuous actions. In both GAC and MPO, working in the non-parameterized space is a by-product of applying the main ideas in those papers to DRL. Our paper demonstrates that the decomposition alone is a general and useful technique for solving constrained policy optimization. Clipped-PPO ) takes a very different approach to TRPO. At each iteration, PPO makes many gradient steps while only using the data from π θ k. Without the clipping, PPO is the approximation. The clipping is analogous to the constraint in that it has the goal of keeping π θ close to π θ k. Indeed, the clipping keeps π θ (a t |s t) from becoming neither much larger than (1 +)π θ k (a t |s t) nor much smaller than (1 −)π θ k (a t |s t). Thus, although the clipped PPO objective does not squarely fit into the optimization framework FORMULA0 - FORMULA1, it is quite similar in spirit. We note that the PPO paper considers adding the KL penalty to the objective function, whose gradient is similar to ours. However, this form of gradient was demonstrated to be inferior to Clipped-PPO. To the best of our knowledge, it is only until our work that such form of gradient is demonstrated to outperform Clipped-PPO.Actor-Critic using Kronecker-Factored Trust Region (ACKTR) BID29 proposed using Kronecker-factored approximation curvature (K-FAC) to update both the policy gradient and critic terms, giving a more computationally efficient method of calculating the natural gradients. ACER BID26 ) exploits past episodes, linearizes the KL divergence constraint, and maintains an average policy network to enforce the KL divergence constraint. In future work, it would of interest to extend the SPU methodology to handle past episodes. In contrast to bounding the KL divergence on the action distribution as we have done in this work, Relative Entropy Policy Search considers bounding the joint distribution of state and action and was only demonstrated to work for small problems . The SPU methodology has two steps. In the first step, for a given constraint criterion η(π, π θ k) ≤ δ, we find the optimal solution to the non-parameterized problem: DISPLAYFORM0 Note that π is not restricted to the set of parameterized policies Π DL. As commonly done, we approximate the objective function. However, unlike PPO/TRPO, we are not approximating the constraint. We will show below the optimal solution π * for the non-parameterized problem- can be determined nearly in closed form for many natural constraint criteria η(π, π θ k) ≤ δ. In the second step, we attempt to find a policy π θ in the parameterized space Π DL that is close to the target policy π *. Concretely, to advance from θ k to θ k+1, we perform the following steps:(i) We first sample N trajectories using policy π θ k, giving sample data DISPLAYFORM1 Here A i is an estimate of the advantage value A π θ k (s i, a i). (For simplicity, we index the samples with i rather than with (i, t) corresponding to the tth sample in the ith trajectory.)(ii) For each s i, we define the target distribution π * to be the optimal solution to the constrained optimization problem- for a specific constraint η.(iii) We then fit the policy network π θ to the target distributions π * (·|s i), i = 1,.., m. Specifically, to find θ k+1, we minimize the following supervised loss function: DISPLAYFORM2 For this step, we initialize with the weights for π θ k. We minimize the loss function L(θ) with stochastic gradient descent methods. The ing θ becomes our θ k+1. To illustrate the SPU methodology, for three different but natural types of proximity constraints, we solve the corresponding non-parameterized optimization problem and derive the ing gradient for the SPU supervised learning problem. We also demonstrate that different constraints lead to very different but intuitive forms of the gradient update. We first consider constraint criteria of the form: DISPLAYFORM0 subject to DISPLAYFORM1 Note that this problem is equivalent to minimizing L π θ k (π) subject to the constraints FORMULA0 and FORMULA0. We refer to as the "aggregated KL constraint" and to FORMULA0 as the "disaggregated KL constraint". These two constraints taken together restrict π from deviating too much from π θ k. We shall refer to FORMULA0 - FORMULA0 as the forward-KL non-parameterized optimization problem. Note that this problem without the disaggregated constraints is analogous to the TRPO problem. The TRPO paper actually prefers enforcing the disaggregated constraint to enforcing the aggregated constraints. However, for mathematical conveniences, they worked with the aggregated constraints: "While it is motivated by the theory, this problem is impractical to solve due to the large number of constraints. Instead, we can use a heuristic approximation which considers the average KL divergence" BID21. The SPU framework allows us to solve the optimization problem with the disaggregated constraints exactly. Experimentally, we compared against TRPO in a controlled experimental setting, e.g. using the same advantage estimation scheme, etc. Since we clearly outperform TRPO, we argue that SPU's two-process procedure has significant potentials. DISPLAYFORM2 Note that π λ (a|s) is a function of λ. Further, for each s, let λ s be such that DISPLAYFORM3 Theorem 1 The optimal solution to the problem- FORMULA0 is given by: DISPLAYFORM4 where λ is chosen so that DISPLAYFORM5 Equation FORMULA0 provides the structure of the optimal non-parameterized policy. As part of the SPU framework, we then seek a parameterized policy π θ that is close toπ λ (a|s), that is, minimizes the loss function. For each sampled state s i, a straightforward calculation shows (Appendix B): DISPLAYFORM6 where DISPLAYFORM7 We estimate the expectation in with the sampled action a i and approximate A π θ k (s i, a i) as A i (obtained from the critic network), giving: DISPLAYFORM8 To simplify the algorithm, we slightly modify. We replace the hyper-parameter δ with the hyper-parameter λ and tune λ rather than δ. Further, we set ∼ λ si = λ for all s i in and introduce per-state acceptance to enforce the disaggregated constraints, giving the approximate gradient: DISPLAYFORM9 We make the approximation that the disaggregated constraints are only enforced on the states in the sampled trajectories. We use as our gradient for supervised training of the policy network. The equation FORMULA0 has an intuitive interpretation: the gradient represents a trade-off between the approximate performance of π θ (as captured by 1 λ DISPLAYFORM10 For the stopping criterion, we train until DISPLAYFORM11 In a similar manner, we can derive the structure of the optimal policy when using the reverse KL-divergence as the constraint. For simplicity, we provide the for when there are only disaggregated constraints. We seek to find the non-parameterized optimal policy by solving: DISPLAYFORM0 DISPLAYFORM1 Theorem 2 The optimal solution to the problem- FORMULA1 is given by: DISPLAYFORM2 where λ(s) > 0 and DISPLAYFORM3 Note that the structure of the optimal policy with the backward KL constraint is quite different from that with the forward KL constraint. A straight forward calculation shows (Appendix B): DISPLAYFORM4 The equation FORMULA1 has an intuitive interpretation. It increases the probability of action a if A π θ k (s, a) > λ (s) − 1 and decreases the probability of action a if A π θ k (s, a) < λ (s) − 1. also tries to keep π θ close to π θ k by minimizing their KL divergence. In this section we show how a PPO-like objective can be formulated in the context of SPU. Recall from Section 3 that the the clipping in PPO can be seen as an attempt at keeping π θ (a i |s i) from becoming neither much larger than (1 +)π θ k (a i |s i) nor much smaller than (1 −)π θ k (a i |s i) for i = 1,..., m. In this subsection, we consider the constraint function DISPLAYFORM0 which leads us to the following optimization problem: DISPLAYFORM1 DISPLAYFORM2 Note that here we are using a variation of the SPU methodology described in Section 4 since here we first create estimates of the expectations in the objective and constraints and then solve the optimization problem (rather than first solve the optimization problem and then take samples as done for Theorems 1 and 2). Note that we have also included an aggregated constraint in addition to the PPO-like constraint, which further ensures that the updated policy is close to π θ k. The optimal solution to the optimization problem is given by: DISPLAYFORM0 for some λ > 0 where DISPLAYFORM1 To simplify the algorithm, we treat λ as a hyper-parameter rather than δ. After solving for π *, we seek a parameterized policy π θ that is close to π * by minimizing their mean square error over sampled states and actions, i.e. by updating θ in the negative direction of ∇ θ i (π θ (a i |s i) − π * (a i |s i)) 2. This loss is used for supervised training instead of the KL because we take estimates before forming the optimization problem. Thus, the optimal values for the decision variables do not completely characterize a distribution. We refer to this approach as SPU with the L ∞ constraint. Although we consider three classes of proximity constraint, there may be yet another class that leads to even better performance. The methodology allows researchers to explore other proximity constraints in the future. Extensive experimental demonstrate SPU outperforms recent state-of-the-art methods for environments with continuous or discrete action spaces. We provide ablation studies to show the importance of the different algorithmic components, and a sensitivity analysis to show that SPU's performance is relatively insensitive to hyper-parameter choices. There are two definitions we use to conclude A is more sample efficient than B: (i) A takes fewer environment interactions to achieve a pre-defined performance threshold BID10; (ii) the averaged final performance of A is higher than that of B given the same number environment interactions. Implementation details are provided in Appendix D. The Mujoco BID25 ) simulated robotics environments provided by OpenAI gym BID5 have become a popular benchmark for control problems with continuous action spaces. In terms of final performance averaged over all available ten Mujoco environments and ten different seeds in each, SPU with L ∞ constraint (Section 5.3) and SPU with forward KL constraints (Section 5.1) outperform TRPO by 6% and 27% respectively. Since the forward-KL approach is our best performing approach, we focus subsequent analysis on it and hereafter refer to it as SPU. SPU also outperforms PPO by 17%. FIG0 illustrates the performance of SPU versus TRPO, PPO.To ensure that SPU is not only better than TRPO in terms of performance gain early during training, we further retrain both policies for 3 million timesteps. Again here, SPU outperforms TRPO by 28%. FIG3 in the Appendix illustrates the performance for each environment. Code for the Mujoco experiments is at https://github.com/quanvuong/Supervised_Policy_Update. The indicator variable in enforces the disaggregated constraint. We refer to it as per-state acceptance. Removing this component is equivalent to removing the indicator variable. We refer to using i D KL (π θ π θ k)[s i] to determine the number of training epochs as dynamic stopping. Without this component, the number of training epochs is a hyper-parameter. We also tried removing DISPLAYFORM0 ] from the gradient update step in. TAB0 illustrates the contribution of the different components of SPU to the overall performance. The third row shows that the term DISPLAYFORM1 ] makes a crucially important contribution to SPU. Furthermore, per-state acceptance and dynamic stopping are both also important for obtaining high performance, with the former playing a more central role. When a component is removed, the hyper-parameters are retuned to ensure that the best possible performance is obtained with the alternative (simpler) algorithm. To demonstrate the practicality of SPU, we show that its high performance is insensitive to hyperparameter choice. One way to show this is as follows: for each SPU hyper-parameter, select a reasonably large interval, randomly sample the value of the hyper parameter from this interval, and then compare SPU (using the randomly chosen hyper-parameter values) with TRPO. We sampled 100 SPU hyper-parameter vectors (each vector including δ,, λ), and for each one determined the relative performance with respect to TRPO. First, we found that for all 100 random hyper-parameter value samples, SPU performed better than TRPO. 75% and 50% of the samples outperformed TRPO by at least 18% and 21% respectively. The full CDF is given in Figure 4 in the Appendix. We can conclude that SPU's superior performance is largely insensitive to hyper-parameter values.6.4 ON ATARI BID20 BID15 demonstrates that neural networks are not needed to obtain high performance in many Mujoco environments. To conclusively evaluate SPU, we compare it against PPO on the Arcade Learning Environments exposed through OpenAI gym BID5. Using the same network architecture and hyper-parameters, we learn to play 60 Atari games from raw pixels and rewards. This is highly challenging because of the diversity in the games and the high dimensionality of the observations. Here, we compare SPU against PPO because PPO outperforms TRPO by 9% in Mujoco. Averaged over 60 Atari environments and 20 seeds, SPU is 55% better than PPO in terms of averaged final performance. FIG1 provides a high-level overview of the . The dots in the shaded area represent environments where their performances are roughly similar. The dots to the right of the shaded area represent environment where SPU is more sample efficient than PPO. We can draw two : (i) In 36 environments, SPU and PPO perform roughly the same; SPU clearly outperforms PPO in 15 environments while PPO clearly outperforms SPU in 9; (ii) In those 15+9 environments, the extent to which SPU outperforms PPO is much larger than the extent to which PPO outperforms SPU. FIG5, Figure 6 and FIG6 in the Appendix illustrate the performance of SPU vs PPO throughout training. SPU's high performance in both the Mujoco and Atari domains demonstrates its high performance and generality. We first show that FORMULA0 - FORMULA0 is a convex optimization. To this end, first note that the objective FORMULA0 is a linear function of the decision variables π = {π(a|s): s ∈ S, a ∈ A}. The LHS of FORMULA0 can be rewritten as: a∈A π(a|s) log π(a|s) − a∈A π(a|s) log π θ k (a|s). The second term is a linear function of π. The first term is a convex function since the second derivative of each summand is always positive. The LHS of FORMULA0 is thus a convex function. By extension, the LHS of FORMULA0 is also a convex function since it is a nonnegative weighted sum of convex functions. The problem FORMULA0 - FORMULA0 is thus a convex optimization problem. According to Slater's constraint qualification, strong duality holds since π θ k is a feasible solution to FORMULA0 - FORMULA0 where the inequality holds strictly. We can therefore solve FORMULA0 - FORMULA0 by solving the related Lagrangian problem. For a fixed λ consider: DISPLAYFORM0 The above problem decomposes into separate problems, one for each state s: DISPLAYFORM1 DISPLAYFORM2 Further consider the unconstrained problem without the constraint: DISPLAYFORM3 subject to DISPLAYFORM4 A simple Lagrange-multiplier argument shows that the opimal solution to FORMULA1 - FORMULA2 is given by: DISPLAYFORM5 where Z λ (s) is defined so that π λ (·|s) is a valid distribution. Now returning to the decomposed constrained problem-, there are two cases to consider. The first case is when DISPLAYFORM6 In this case, the optimal solution to- FORMULA0 is π λ (a|s). The second case is when DISPLAYFORM7 In this case the optimal is π λ (a|s) with λ replaced with λ s, where λ s is the solution to DISPLAYFORM8 Thus, an optimal solution to- is given by: DISPLAYFORM9 where DISPLAYFORM10 To find the Lagrange multiplier λ, we can then do a line search to find the λ that satisfies: DISPLAYFORM11 A.2 BACKWARD KL CONSTRAINTThe problem FORMULA0 - FORMULA1 decomposes into separate problems, one for each state s ∈ S: DISPLAYFORM12 subject to E DISPLAYFORM13 After some algebra, we see that above optimization problem is equivalent to: DISPLAYFORM14 DISPLAYFORM15 where = + entropy(π θ k). FORMULA2 - FORMULA1 is a convex optimization problem with Slater's condition holding. Strong duality thus holds for the problem FORMULA2 - FORMULA1. Applying standard Lagrange multiplier arguments, it is easily seen that the solution to FORMULA2 - FORMULA1 is DISPLAYFORM16 where λ(s) and λ (s) are constants chosen such that the disaggregegated KL constraint is binding and the sum of the probabilities equals 1. It is easily seen λ(s) > 0 and DISPLAYFORM17 The problem is equivalent to: DISPLAYFORM18 DISPLAYFORM19 This problem is clearly convex. π θ k (a i |s i), i = 1,..., m is a feasible solution where the inequality constraint holds strictly. Strong duality thus holds according to Slater's constraint qualification. To solve FORMULA2 - FORMULA3, we can therefore solve the related Lagrangian problem for fixed λ: DISPLAYFORM20 DISPLAYFORM21 which is separable and decomposes into m separate problems, one for each s i: DISPLAYFORM22 DISPLAYFORM23 DISPLAYFORM24 λ s (Log of product is sum of log) DISPLAYFORM25 (Adding the gradient of the entropy on both sides and collapse the sum of gradients of cross entropy and entropy into the gradient of the KL) DISPLAYFORM26 (Taking gradient on both sides) DISPLAYFORM27 (Adding the gradient of the entropy on both sides and collapse the sum of gradients of cross entropy and entropy into the gradient of the KL) The methodology developed in the body of this paper also applies to continuous state and action spaces. In this section, we outline the modifications that are necessary for the continuous case. We first modify the definition of d π (s) by replacing P π (s t = s) with d ds P π (s t ≤ s) so that d π (s) becomes a density function over the state space. With this modification, the definition ofD KL (π π k) and the approximation are unchanged. The SPU framework described in Section 4 is also unchanged. Consider now the non-parameterized optimization problem with aggregate and disaggregate constraints, but with continuous state and action space: DISPLAYFORM0 Theorem 1 holds although its proof needs to be slightly modified as follows. It is straightforward to show that remains a convex optimization problem. We can therefore solve by solving the Lagrangian with the sum replaced with an integral. This problem again decomposes with separate problems for each s ∈ S giving exactly the same equations. The proof then proceeds as in the remainder of the proof of Theorem 1.Theorem 2 and 3 are also unchanged for continuous action spaces. Their proofs require slight modifications, as in the proof of Theorem 1. As in, for Mujoco environments, the policy is parameterized by a fullyconnected feed-forward neural network with two hidden layers, each with 64 units and tanh nonlinearities. The policy outputs the mean of a Gaussian distribution with state-independent variable standard deviations, following BID21 BID7. The action dimensions are assumed to be independent. The probability of an action is given by the multivariate Gaussian probability distribution function. The baseline used in the advantage value calculation is parameterized by a similarly sized neural network, trained to minimize the MSE between the sampled states TD−λ returns and the their predicted values. For both the policy and baseline network, SPU and TRPO use the same architecture. To calculate the advantage values, we use Generalized Advantage Estimation BID22. States are normalized by dividing the running mean and dividing by the running standard deviation before being fed to any neural networks. The advantage values are normalized by dividing the batch mean and dividing by the batch standard deviation before being used for policy update. The TRPO is obtained by running the TRPO implementation provided by OpenAI, commit 3cc7df060800a45890908045b79821a13c4babdb. At every iteration, SPU collects 2048 samples before updating the policy and the baseline network. For both networks, gradient descent is performed using Adam with step size 0.0003, minibatch size of 64. The step size is linearly annealed to 0 over the course of training. γ and λ for GAE BID22 BID17. The output of the network is passed through a relu, linear and softmax layer in that order to give the action distribution. The output of the network is also passed through a different linear layer to give the baseline value. States are normalized by dividing by 255 before being fed into any network. The TRPO is obtained by running the PPO implementation provided by OpenAI, commit 3cc7df060800a45890908045b79821a13c4babdb. 8 different processes run in parallel to collect timesteps. At every iteration, each process collects 256 samples before updating the policy and the baseline network. Each process calculates its own update to the network's parameters and the updates are averaged over all processes before being used to update the network's parameters. Gradient descent is performed using Adam with step size 0.0001. In each process, random number generators are initialized with a different seed according to the formula process_seed = experiment_seed + 10000 * process_rank. Training is performed for 10 million timesteps for both SPU and PPO. For SPU, δ,, λ and the maximum number of epochs per iteration are set to 0.02, δ/1.3, 1.1 and 9 respectively. Algorithm 1 Algorithmic description of forward-KL non-parameterized SPU Require: A neural net π θ that parameterizes the policy. Require: A neural net V φ that approximates V π θ. Require: General hyperparameters: γ, β (advantage estimation using GAE), α (learning rate), N (number of trajectory per iteration), T (size of each trajectory), M (size of training minibatch). Require: Algorithm-specific hyperparameters: δ (aggregated KL constraint), (disaggregated constraint), λ, ζ (max number of epoch). 1: for k = 1, 2,... do under policy π θ k, sample N trajectories, each of size T (s it, a it, r it, s i(t+1) ), i = 1,..., N, t = 1,..., T Using any advantage value estimation scheme, estimate A it, i = 1,..., N, t = 1,..., T DISPLAYFORM0 10: DISPLAYFORM1 θ ← θ − αL(θ) TRPO and SPU were trained for 1 million timesteps to obtain the in section 6. To ensure that SPU is not only better than TRPO in terms of performance gain early during training, we further retrain both policies for 3 million timesteps. Again here, SPU outperforms TRPO by 28%. FIG3 illustrates the performance on each environment. When values for SPU hyper-parameter are randomly sampled as is explained in subsection 6.3, the percentage improvement of SPU over TRPO becomes a random variable. Figure 4 illustrates the CDF of this random variable.

Represent the short summary for finding the corresponding scientific article (long)

first posing and solving the sample efficiency optimization problem in the non-parameterized policy space, and then solving a supervised regression problem to find a parameterized policy that is near the optimal non-parameterized policy.

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.


Question: Represent the article for finding a single-sentence TLDR

Question: Recent work on modeling neural responses in the primate visual system has benefited from deep neural networks trained on large-scale object recognition, and found a hierarchical correspondence between layers of the artificial neural network and brain areas along the ventral visual stream. However, we neither know whether such task-optimized networks enable equally good models of the rodent visual system, nor if a similar hierarchical correspondence exists. Here, we address these questions in the mouse visual system by extracting features at several layers of a convolutional neural network (CNN) trained on ImageNet to predict the responses of thousands of neurons in four visual areas (V1, LM, AL, RL) to natural images. We found that the CNN features outperform classical subunit energy models, but found no evidence for an order of the areas we recorded via a correspondence to the hierarchy of CNN layers. Moreover, the same CNN but with random weights provided an equivalently useful feature space for predicting neural responses. Our suggest that object recognition as a high-level task does not provide more discriminative features to characterize the mouse visual system than a random network. Unlike in the primate, training on ethologically relevant visually guided behaviors -- beyond static object recognition -- may be needed to unveil the functional organization of the mouse visual cortex. Visual object recognition is a fundamental and difficult task performed by the primate brain via a hierarchy of visual areas (the ventral stream) that progressively untangles object identity information, gaining invariance to a wide range of object-preserving visual transformations. Fueled by the advances of deep learning, recent work on modeling neural responses in sensory brain areas builds upon hierarchical convolutional neural networks (CNNs) trained to solve complex tasks like object recognition. Interestingly, these models have not only achieved unprecedented performance in predicting neural responses in several brain areas of macaques and humans, but they also revealed a hierarchical correspondence between the layers of the CNNs and areas of the ventral stream: the higher the area in the ventral stream, the higher the CNN layer that explained it best. The same approach also provided a quantitative signature of a previously unclear hierarchical organization of A1 and A2 in the human auditory cortex. These discoveries about the primate have sparked a still unresolved question: to what extent is visual object processing also hierarchically organized in the mouse visual cortex and how well can the mouse visual system be modeled using goal-driven deep neural networks trained on static object classification? This question is important since mice are increasingly used to study vision due to the plethora of available experimental techniques such as the ability to genetically identify and manipulate neural circuits that are not easily available in primates. Recent work suggests that rats are capable of complex visual discrimination tasks and recordings from extrastriate areas show a gradual increase in the ability of neurons in higher visual areas to support discrimination of visual objects. Here, we set out to study how well the mouse visual system can be characterized by goal-driven deep neural networks. We extracted features from the hidden layers of a standard CNN (VGG16, ) trained on object categorization, to predict responses of thousands of neurons in four mouse visual areas (V1, LM, AL, RL) to static natural images. We found that VGG16 yields powerful features for predicting neural activity, outperforming a Gabor filter bank energy model in these four visual areas. However, VGG16 does not significantly outperform a feature space produced by a network with an identical architecture but random weights. In contrast to previous work in primates, our data provide no evidence so far for a hierarchical correspondence between the deep network layers and the visual areas we recorded. trough the core (A) network (first n layers of VGG16) to produce a feature space shared by all neurons. Then, the spatial transformer readout (B) finds a mapping between these features and the neural responses for each neuron separately. The shifter network (an MLP with one hidden layer) corrects for eye movements. The output of the readout is multiplied by a gain predicted by the modulator network (an MLP with one hidden layer) that uses running speed and pupil dilation. A static nonlinearity converts the into the predicted spike rate. All components of the model are trained jointly end-to-end to minimize the difference between predicted and observed neural responses. Our network (Fig.1) builds upon earlier work. It consist of four main network components: a core that provides nonlinear features of input images, a readout that maps those features to each neuron's responses, a shifter that predicts receptive field shifts from pupil position, and a modulator that provides a gain factor for each neuron based on running speed and pupil dilation of the mouse. For the core we use VGG16 up to one of the first eight convolutional layers. We chose VGG16 due to its simple feed-forward architecture, competitive object classification performance, and increasing popularity to characterize rodent visual areas. The collection of output feature maps of a VGG16 layer -the shared feature space -was then fed into a spatial transformer readout for each neuron (Fig.1B, see for details). This readout learns one (x, y) location for each neuron (its receptive field location, RF) and extracts a feature vector at this location from multiple downsampled versions (scales) of the feature maps. The output of the readout is a linear combination of the concatenated feature vectors. We regularized the feature weights with an L 1 penalty to encourage sparsity. Shifter and modulator are multi-layer perceptrons (MLP) with one hidden layer. The shifter takes the tracked pupil position in camera coordinates and predicts a global receptive field shift (∆x, ∆y) in monitor coordinates. The modulator uses the mouse's running speed, its pupil diameter, and the derivative to predict a gain for each neuron by which the neuron's predicted response is multiplied. A soft-thresholding nonlinearity turns the into a non-negative spike rate prediction (Fig.1). All components of the model (excluding the core, which is pre-trained on ImageNet) are trained jointly end-to-end to minimize the difference between predicted and observed neural responses using Adam with a learning rate of 10 −4, a batch size of 125 and early stopping. Neural data. We recorded responses of excitatory neurons in areas V1, LM, AL, and RL (layer 2/3) from two scans from one mouse and a third scan from a second mouse with a large-field-of-view two-photon mesoscope (see for details) at a frame rate of 6.7 Hz. We selected cells based on a classifier for somata on the segmented cell masks and deconvolved their fluorescence traces, yielding 7393, 4674, 4680, 5797 neurons from areas V1, LM, AL, and RL, respectively. We further monitored pupil position, pupil dilation, and absolute running speed of the animal. Visual stimuli. Stimuli consisted of 5100 images taken from ImageNet, cropped to 16:9 and converted to grayscale. The screen was 55 × 31 cm at a distance of 15 cm, covering roughly 120 • × 90 •. In each scan, we showed 5000 of these images once (training and validation set) and the remaining 100 images 10 times each (test set). Each image was presented for 500 ms followed by a blank screen lasting between 300 ms and 500 ms. For each neuron, we extract the accumulated activity between 50 ms and 550 ms after stimulus onset using a Hamming window. We fitted one model (that of Fig.1 ; see for training details) for each combination of scan, brain area, VGG16 layer (out of the first eight), random initialization (out of three seeds), and input resolution. We considered several resolutions of the input images because the right scale at which VGG16 layers extract relevant features that best match the representation in the brain is unknown. Optimizing the scale for each layer was critical, since the correspondence between a single layer and a brain area (in terms of best correlation performance) strongly depends on the input resolution (e.g. see Fig 3A for V1 data). For further analyses (Fig 3B & 4), we pick for each case the best performing input scale in the validation set. No hierarchical correspondence. Previous in primates show that a brain area higher in the hierarchy is better matched (i.e has a peak in prediction performance) by a higher network layer. In contrast, when comparing the average performance across cells and scans for each convolutional layer and brain area, we find no clear evidence for a hierarchy (Fig.3B) since there is no clear ordering of the brain areas. VGG16 outperforms classical models. We then investigate whether the lack of an evident hierarchy was due to an overall poor performance of our model. Thus, we first revise how much of the explainable stimulus-driven variability the VGG16-based model captures. To this end we calculate the oracle correlation (the conditional mean of n − 1 responses without the model) obtaining an upper bound of the achievable performance. Then we evaluate the test correlation of our model restricted to visual input information (no shifter and no modulator), against the oracle (Fig.4B) and find that VGG16 features explain a substantial fraction of the oracle for the for areas (70-78%) Second, we consider a subunit energy model with Gabor quadrature pairs as a baseline due to its competitive predictive performance of macaque V1 responses. We replace the core from Fig. 1 with a Gabor filter bank (GFB) consisting of a large number of Gabor filters with different orientations, sizes and spatial frequencies arranged in quadrature pairs, and followed by a squaring nonlinearity. We find that for all areas and scans, the VGG16 core outperformed the GFB (Fig.4C). Core with random weights performs similarly. The so far show that VGG16 provides a powerful feature space to predict responses, which may suggest that static object recognition could be a useful high-level goal to describe the function of the mouse visual system. However, we were surprised that most VGG layers led to similar performance. To understand this better, we also evaluated a core with identical architecture but random weights. This random core performed similarly well as its pre-trained counterpart (Fig.4D), suggesting that training on static object recognition as a high-level goal is not necessary to achieve state-of-the-art performance in predicting neural responses in those four visual areas. Instead, a sufficiently large collection of random features followed by rectification provides a similarly powerful feature space. The number of LN layers is critical to best match neural activity. Since random features produced by a linearnonlinear (LN) hierarchy closely match the performance of the pretrained VGG16, we then asked if the number of LN steps -when accounting for multiple input resolutions -was the key common aspect of these networks that yielded the best predictions. Effectively, similar to the case of the pretrained VGG16 core, we found that the fourth and fifth rectified convolutional layers of the random core are the best predictive layers for the four areas we studied (Fig. 5). However, it is important to note that in both cases the increase in performance after the second convolutional layer is only marginal. Overall, we conclude that the nonlinear degree -number of LN stages -rather than the static object recognition training goal dictates how close the representations are to the neural activity. In contrast to similar work in the primate, we find no match between the hierarchy of mouse visual cortical areas and the layers of CNNs trained on object categorization. Although VGG16 achieves state-of-the-art performance, it is matched by random weights. There are three implications of our : First, our work is in line with previous work in machine learning that shows the power of random features. Therefore, we argue that models based on random features should always be reported as baselines in studies on neural system identification. Second, which VGG layer best predicted any given brain area depended strongly on the image resolution we used to feed into VGG16. We observed a similar effect in our earlier work on primate V1. Thus, the studies reporting a hierarchical correspondence between goal-driven deep neural networks and the primate ventral stream should be taken with a grain of salt, as they -to the best of our knowledge -do not include this control. Third, optimizing the network for static object recognition alone as a high-level goal does not appear to be the right approximation to describe representations and the visual hierarchy in the mouse cortex. Although our do not exclude a potential object processing hierarchy in the mouse visual system, they suggest that training with more ethologically relevant visually guided tasks for the mouse could be a more fruitful goal-driven approach to characterize the mouse visual system. For instance, an approach with dynamic stimuli such as those found during prey capture tasks could yield more meaningful features to unveil the functional organization of the mouse visual system.

Represent the short summary for finding the corresponding scientific article (long)!

A goal-driven approach to model four mouse visual areas (V1, LM, AL, RL) based on deep neural networks trained on static object recognition does not unveil a functional organization of visual cortex unlike in primates

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

We propose a novel architecture that traverses an image pyramid in a top-down fashion, while it visits only the most informative regions along the way.


Question: The reparameterization trick has become one of the most useful tools in the field of variational inference. However, the reparameterization trick is based on the standardization transformation which restricts the scope of application of this method to distributions that have tractable inverse cumulative distribution functions or are expressible as deterministic transformations of such distributions. In this paper, we generalized the reparameterization trick by allowing a general transformation. We discover that the proposed model is a special case of control variate indicating that the proposed model can combine the advantages of CV and generalized reparameterization. Based on the proposed gradient model, we propose a new polynomial-based gradient estimator which has better theoretical performance than the reparameterization trick under certain condition and can be applied to a larger class of variational distributions. In studies of synthetic and real data, we show that our proposed gradient estimator has a significantly lower gradient variance than other state-of-the-art methods thus enabling a faster inference procedure. Most machine learning objective function can be rewritten in the form of an expectation: where θ is a parameter vector. However, due to the intractability of the expectation, it's often impossible or too expensive to calculate the exact gradient w.r.t θ, therefore it's inevitable to estimate the gradient ∇ θ L in practical applications. Stochastic optmization methods such as reparameterization trick and score function methods have been widely applied to address the stochastic gradient estimation problem. Many recent advances in large-scale machine learning tasks have been brought by these stochastic optimization tricks. Like in other stochastic optimzation related works, our paper mainly focus on variational inference tasks. The primary goal of variational inference (VI) task is to approximate the posterior distribution in probabilistic models . To approximate the intractable posterior p(z|x) with the joint probability distribution p(x, z) over observed data x and latent random variables z given, VI introduces a parameteric family of distribution q θ (z) and find the best parameter θ by optimizing the Kullback-Leibler (KL) divergence D KL (q(z; θ) p(z|x)). The performance of VI methods depends on the capacity of the parameteric family of distributions (often measured by Rademacher complexity) and the ability of the optimizer. In this paper, our method tries to introduce a better optimizer for a larger class of parameteric family of distributions. The main idea of our work is to replace the parameter-independent transformation in reparameterization trick with generalized transformation and construct the generalized transformation-based (G-TRANS) gradient with the velocity field which is related to the characteristic curve of the sublinear partial differential equation associated with the generalized transformation. Our gradient model further generalizes the G-REP and provides a more elegant and flexible way to construct gradient estimators. We mainly make the following contributions: 1. We develop a generalized transformation-based gradient model based on the velocity field related to the generalized transformation and explicitly propose the unbiasedness constraint on the G-TRANS gradient. The proposed gradient model provides a more poweful and flexible way to construct gradient estimators. 2. We show that our model is a generalization of the score function method and the reparameterization trick. Our gradient model can reduce to the reparameterization trick by enforcing a transport equation constraint on the velocity field. We also show our model's connection to control variate method. 3. We propose a polynomial-based gradient estimator that cannot be induced by any other existing generalized reparameterization gradient framework, and show its superiority over similar works on several experiments. The rest of this paper is organized as follows. In Sec.2 we review the stochastic gradient variational inference (SGVI) and stochastic gradient estimators. In Sec.3 we propose the generalized transformation-based gradient. In Sec.4 we propose the polynomial-based G-TRANS gradient estimator. In Sec.5 we study the performance of our gradient estimator on synthetic and real data. In Sec.6 we review the related works. In Sec.7 we conclude this paper and discuss future work. To obtain the best variational parameter θ, rather than minimize the KL divergence D KL (q(z; θ) p(z|x)), we usually choose to maximize the evidence lower bound (ELBO) , where The entropy term H[q(z; θ)] is often assumed to be available analytically and usually omitted in the procedure of stochastic optimization. This stochastic optimization problem is the basic setting for our method and experiments. Without extra description, we only consider the simplified version of the ELBO: Generally, this expectation is intractable to compute, let alone its gradient. Therefore, a common stochastic optimization method for VI task is to construct a Monte Carlo estimator for the exact gradient of the ELBO w.r.t θ. Among those gradient estimators, the score function method and the reparamterization trick are most popular and widely applied. Score function method. The score function estimator, also called log-derivative trick or reinforce; is a general way to obtain unbiased stochastic gradients of the ELBO (; ;). The simplest variant of the score function gradient estimator is defined as: and then we can build the Monte Carlo estimator by drawing samples from the variational distribution q θ (z) independently. Although the score function method is very general, the ing gradient estimator suffers from high variance. Therefore, it's necessary to apply variance reduction (VR) methods such as RaoBlackwellization and control variates in practice. Reparameterization trick. In reparameterization trick, we assume that there is an invertible and continuously differentiable standardization function φ(z, θ) that can transform the variational distribution q(z; θ) into a distribution s(ρ) that don't depend on the variational parameter θ as follows, θ (ρ) Then the reparameterization trick can turn the computation of the gradient of the expectation into the expectation of the gradient: Although this reparameterization can be done for many commonly used distributions, such as the Gaussian distribution, it's hard to find appropriate standardization functions for a number of standard distributions, such as Gamma, Beta or Dirichlet because the standardization functions will inevitably involve special functions. On the other hand, though the reparameterization trick is not as generally applicable as the score function method, it does in a gradient estimator with lower variance. Define a random variable ρ by an invertible differentiable transformation ρ = φ(z, θ), where φ is commonly called generalized standardization transformation since it's dependent on the variational parameter θ. Theorem 3.1. Let θ be any component of the variational parameter θ, the probability density function of ρ be w(ρ, θ) and where The proof details of the Theorem.3.1 are included in the Appendix. A.1. We refer to the gradient ∂L ∂θ with v θ satisfying the unbiasedness constraint as generalized transformation-based (G-TRANS) gradient. We can construct the G-TRANS gradient estimator by choosing v θ of specific form. In the following, we demonstrate that the score function gradient and reparameterization gradient are special cases of our G-TRANS gradient model associating with special velocity fields. Remark. The score function method is a special case of the G-TRANS model when The standardization function φ doesn't depend on the parameter θ when v θ = 0 according to the velocity field equation (Equ.5). Conversely, for any φ that doesn't depend on θ, we have = 0, thus the ing gradient estimator has the same variance as the score function estimator. Remark. The reparameterization trick is a special case when The detailed computation to obtain the transport equation (Equ.7) is included in the Appendix. A.1. The transport equation is firstly introduced by , however, their work derive this equation by an analog to the optimal transport theory. In 1-dimensional case, for any standardization distributions w(ρ) that doesn't depend on the parameter θ, the variance of the ing gradient estimator is some constant (for fixed θ) determined by the unique 1-dimensional solution of the transport equation. For the existence of the velocity field v θ and the generalized standardization transformation φ(z, θ), g(z, θ) must satisfy some strong differential constraints . We can see that the G-TRANS model is a special case of the control variate method with a complex differential structure. This connection to CV means our gradient model can combine the advantages of CV and generalized reparameterization. Theorem.3.1 transforms the generalized unbiased reparameterization procedure into finding the appropriate velocity field that satisfy the unbiasedness constraint. It's possible to apply variational optimization theory to find the velocity field with the least estimate variance, however, the solution to the Euler-Lagrange equation contains f (z) in the integrand which makes it impractical to use in real-world model (See Appendix. A.2 for details). By introducing the notion of velocity field, we provide a more elegant and flexible way to construct gradient estimator without the need to compute the Jacobian matrix for a specific transformation. In the next section, we introduce a polynomial-based G-TRANS gradient estimator that cannot be incorporated into any other existing generalized reparameterized gradient framework and is better than the reparameterization gradient estimator theoretically. In this section, we always assume that the base distribution q(z, θ) can be factorized as where N is the dimension of the random variable z, θ i is a slice of θ and θ i share no component with θ j if i = j. We consider an ad-hoc velocity field family: We always assume v θ ah to be continuous which guarantees the existence of the solution to the velocity field equation. We verify in the Appendix. A.3 that v θ ah (z, θ) satisfy the unbiasedness constraint if h(z, θ) is bounded. It's easy to see that the gradient estimator that from v θ ah is more general than the score function method or reparameterization trick since they are two special cases when h(z, θ) = 0 or h(z, θ) = f (z) respectively. In this paper, we mainly consider a more special family of the v θ ah (z, θ): where, but their properties are similar (we present some theoretical of v θ dp in the Appendix. A.4). Therefore we only consider v θ poly (z, θ) here. We refer to v θ poly as polynomial velocity field. Proposition 4.1. For distributions with analytical high order moments such as Gamma, Beta or Dirichlet distribution, the expectation are polynomials of random variable z. Therefore, for distribution with analytical high order moments, With Proposition.4.1, we can write the G-TRANS gradient for the polynomial velocity field as: Thus we can construct a G-TRANS gradient estimator based upon the polynomial velocity field with a samplez drawn from q(z, θ): The polynomial-based G-TRANS gradient estimator has a form close to control variate, thus cannot be induced by any other existing generalized reparameterized gradient framework. In the following, we show that the polynomial-based G-TRANS gradient estimator performs better than the reparameterization gradient estimator under some condition. dz N ), then the gradient estimator ed from polynomial velocity field has a smaller variance than the reparameterization gradient estimator. Proof. Since E q [P k (z, θ) ∂ ∂θ log q] can be resolved analytically, we have then by reorganizing the expression Var(− ∂f ∂zi), we can prove this proposition. As an example about how to choose a good polynomial, for ), we can obtain a polynomial-based G-TRANS gradient estimator that is better than the reparameterization gradient estimator according to the Proposition.4.2. And we can adjust the value of C i (θ) to obtain better performance. According to the approximation theory, we can always find a polynomial P k (z, θ) that is close enough to f (z), and in this case, we can dramatically reduce the variance of the ing gradient estimator. For example, within the convergence radius, we can choose P k (z, θ) to be the k-th degree Taylor polynomial of f (z) with the remainder |f (z) − P k (z, θ)| being small. In the practical situation, however, it's often difficult to estimate the coefficients of the polynomial P k (z, θ). And when k is large, we need to estimate O(N k) coefficients which is almost impossible in real-world applications. Therefore in the following experiments, we only consider k < 2. In this section, we use a Dirichlet distribution to approximate the posterior distribution for a probilistic model which has a multinomial likelihood with a Dirichlet prior. We use Gamma distributions to simulate Dirichlet distributions. If Then the problem we study here can be written as: with f (z) being the multinomial log-likelihood. We use shape parameter α = (α 1, . . ., α K) to parameterize the variational Dirichlet distribution. To construct polynomial-based G-TRANS gradient estimator for the factorized distribution K k=1 Gamma(z k ; α k, 1), we need an accurate and fast way to approximate the derivative of the lower incomplete gamma function (part of the gamma CDF) w.r.t the shape parameter. The lower incomplete gamma function γ(α, z) is a special function and does not admit analytical expression for derivative w.r.t. the shape parameter. However, for small α and z, we have In practice, we take the first 200 terms from this power series. And the approximation error is smaller than 10 −9 when α < 5 and z < 20 with double precision floating point number. For large α, we use central finite difference to approximate the derivative. This approximation scheme for lower incomplete gamma function can also be used to construct polynomial-based G-TRANS gradient estimator for distributions that can be simulated by the Gamma distribution such as Beta distribution and Dirichlet distribution. We follow the experiment setting in. Fig.1 shows the ing variance of the first component of the gradient based on samples simulated from a Dirichlet distribution with K = 100 components, and gradients are computed with N = 100 trials. We use P 1 (z) = c · z to construct the G-TRANS gradient estimator, and we assign 0.2,0 and −0.1 to c successively as α 1 increases. Results. From Fig.1, we can see that the IRG method and our G-TRANS gradient estimator has obviously lower gradient variance than the RSVI (even with the shape augmentation trick ) or G-REP method. Further, our G-TRANS gradient estimator outperforms the IRG method when α 1 is large though there is no obvious difference between these two methods when α 1 is small. In this section, we study the performance of our G-TRANS gradient estimator on the Sparse Gamma deep exponential family (DEF) model with the Olivetti faces dataset that consists of 64 × 64 gray-scale images of human faces in 8 bits. We follow the Sparse Gamma DEF setting in where the DEF model is specified by: . C is the polynomial coefficient, B denotes shape augmentation and optimal concentration is α = 2. Here n is the number of observations, is the layer number, k denotes the k-th component in a specific layer and d is the dimension of the output layer (layer 0). z n,k is local random variable, w k,k is global weight that connects different layers like deep neural networks, and x n,d denotes the set of observations. We use the experiment setting in. α z is set to 0.1, all priors on the weights are set to Gamma(0.1, 0.3), and the top-layer local variables priors are set to Gamma(0.1, 0.1). The model consists of 3 layers, with 100, 40, and 15 components in each. All variational Gamma distributions are parameterized by the shape and mean. For non-negative variational parameters θ, the transfomration θ = log(1 + exp(ϑ)) is applied to avoid constrained optimization. In this experiment, we use the step-size sequence ρ n proposed by: δ = 10 −16, t = 0.1, η = 0.75 is used in this experiment. The best of RSVI is reproduced with B = 4 . We still use P 1 (z) = c · z to construct the G-TRANS gradient estimator and we use c = −10.0 for all time. Results. From Fig.2, We can see that G-TRANS achieves significant improvements in the first 1000 runs and exceeds RSVI though with a slower initial improvement. G-TRANS achieves obviously better accuracy than ADVI, BBVI, G-REP and RSVI, and keeps improving the ELBO even after 75000 runs. G-TRANS is faster than the IRG in early training stage which means G-TRANS has a lower gradient variance. However, this speed advantage of G-TRANS gradually decreases as the step size goes down in the later training stage. There are already some lines of research focusing on extending the reparameterization trick to a larger class of distributions. The G-REP generalizes the reparameterization gradient by using a standardization transformation that allows the standardization distribution to depend weakly on variational parameters. Our gradient model gives a more elegant expression of the generalized reparameterized gradient than that of G-REP which decomposes the gradient as g rep + g cor. Different from G-REP, our model hides the transformation behind the velocity field thus the expensive computation of the Jacobian matrix of the transformation is evaded. And it's more flexible to construct gradient estimator with the velocity field than the very detailed transformation. The RSVI develops a similar generalized reparameterized gradient model with the tools from rejection sampling literatures. RSVI introduces a score function gradient term to compensate the gap that is caused by employing the proposal distribution of a rejection sampler as a surrogate distribution for reparameterization gradient, although the score function gradient term can often be ignored in practice to reduce the gradient variance at the cost of small bias. Unlike RSVI, our gradient estimator can be constructed with deterministic procedure which avoids the additional stochasticity introduced by the accept-reject steps thus lower gradient variance. The path-wise derivative is closely related to our model. They obtain the transport equation by an analog to the displacement of particles, while we derive the transport equation for reparameterization gradient by rigorous mathematical deduction. The path-wise gradient model can be seen as a special case of our G-TRANS gradient model. Their work only focus on standard reparameterization gradient while our model can admit generalized transformation-based gradient. The velocity field used in their work must conform to the transport equation while we only require the velocity field to satisfy the unbiasedness constraint. The implicit reparameterization gradient (IRG) differentiates from the path-wise derivative only by adopting a different method for multivariate distributions. There are also some other works trying to address the limitations of standard reparameterization. applies implicit reparameterization for mixture distributions and uses approximations to the inverse CDF to derive gradient estimators. Both work involve expensive computation that cannot be extended to large-scale variational inference. expressed the gradient in a similar way to G-REP and automatically estimate the gradient in the context of stochastic computation graphs, but their work is short of necessary details therefore cannot be applied to general variational inference task directly. ADVI transforms the random variables such that their support are on the reals and then approximates transformed random variables with Gaussian variational posteriors. However, ADVI struggles to approximate probability densities with singularities as noted by. We proposed a generalized transformation-based (G-TRANS) gradient model which extends the reparameterization trick to a larger class of variational distributions. Our gradient model hides the details of transformation by introducing the velocity field and provides a flexible way to construct gradient estimators. Based on the proposed gradient model, we introduced a polynomial-based G-TRANS gradient estimator that cannot be induced by any other existing generalized reparameterization gradient framework. In practice, our gradient estimator provides a lower gradient variance than other state-of-the-art methods, leading to a fast converging process. For future work, We can consider how to construct G-TRANS gradient estimators for distributions that don't have analytical high-order moments. We can also utilize the from the approximation theory to find certain kinds of high-order polynomial functions that can approximate the test function effectively with cheap computations for the coefficients. Constructing velocity fields with the optimal transport theory is also a promising direction. A.1 PROOF OF THEOREM.3.1 We assume that transformed random variable ρ = φ(z, θ) is of the same dimension as z. And we assume that there exists ψ(ρ, θ) that satisfy the constraint z = ψ(φ(z, θ), θ). Firstly, by the change-of-variable technique, we have Take derivative w.r.t θ (any component of θ) at both sizes, we have With the rule of determinant derivation, we have Substitute the Equ.19 into Equ.18, we have, we obtain the first of the Theorem.3.1. As for the second part, we have Thus we obtain the second part of the Theorem.3.1. Proof ends. As a by-product, if we make ∂ ∂θ w(φ(z, θ), θ) = 0, we can obtain the transport equation for the reparameterization trick: And ∂ ∂θ w(φ(z, θ), θ) = 0 also means that the standardization distribution is independent with θ which is the core of the reparameterization trick. For the simplicity of the proof, we only consider the 1-dimensional here. And denote where with the unbiased constraint, we have E q(z,θ) [r(z, θ)] = E q(z,θ) [f (z) ∂q(z,θ) ∂θ q(z,θ) ] = const, so we need to consider the term E q(z,θ) [(r θ (z, θ)) 2 ] only. According to the Euler-Lagrange equation, we have Simplify it, we have (f ∂q ∂θ Then we have Thus we have which is usually intractable in real world practice. Here we verify that v θ ah (z, θ) satisfy the unbiasedness constraint if h(z, θ) is bounded. Let θ i be any component of θ i, then If h(z, θ) is bounded, we have Therefore, E q θ [If we take the dual polynomial velocity field v θ dp in the G-TRANS framework, we can reach a dual to the Proposition.4.2: Proposition A.1. If Cov(P k ∂ log q(z,θ) ∂θ, (2f − P k) ∂ log q(z,θ) ∂θ ) > 0, then the gradient estimator ed from dual polynomial velocity field has a smaller gradient variance than the score function gradient estimator. The proof is similar to that of Proposition.4.2.

Represent the TLDR for finding the corresponding scientific article.

a generalized transformation-based gradient model for variational inference

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.