Question: Represent the scientific article to find a short TLDR.

Question: We study two types of preconditioners and preconditioned stochastic gradient descent (SGD) methods in a unified framework. We call the first one the Newton type due to its close relationship to the Newton method, and the second one the Fisher type as its preconditioner is closely related to the inverse of Fisher information matrix. Both preconditioners can be derived from one framework, and efficiently estimated on any matrix Lie groups designated by the user using natural or relative gradient descent minimizing certain preconditioner estimation criteria. Many existing preconditioners and methods, e.g., RMSProp, Adam, KFAC, equilibrated SGD, batch normalization, etc., are special cases of or closely related to either the Newton type or the Fisher type ones. Experimental on relatively large scale machine learning problems are reported for performance study. This paper investigates the use of preconditioner for accelerating gradient descent, especially in large scale machine learning problems. Stochastic gradient descent (SGD) and its variations, e.g., momentum BID11 BID9, RMSProp and Adagrad BID5, Adam BID6, etc., are popular choices due to their simplicity and wide applicability. These simple methods do not use well normalized step size, could converge slow, and might involve more controlling parameters requiring fine tweaking. Convex optimization is a well studied field BID2. Many off-the-shelf methods there, e.g., (nonlinear) conjugate gradient descent, quasi-Newton methods, Hessian-free optimizations, etc., can be applied to small and middle scale machine learning problems without much modifications. However, these convex optimization methods may have difficulty in handling gradient noise and scaling up to problems with hundreds of millions of free parameters. For a large family of machine learning problems, natural gradient with the Fisher information metric is equivalent to a preconditioned gradient using inverse of the Fisher information matrix as the preconditioner BID1. Natural gradient and its variations, e.g., Kronecker-factored approximate curvature (KFAC) BID8 and the one in BID10, all use such preconditioners. Other less popular choices are the equilibrated preconditioner BID4 and the one proposed in BID7. Momentum or the heavy-ball method provides another independent way to accelerate converge BID9 BID11. Furthermore, momentum and preconditioner can be combined to further accelerate convergence as shown in Adam BID6. This paper groups the above mentioned preconditioners and preconditioned SGD methods into two classes, the Newton type and the Fisher type. The Newton type is closely related to the Newton method, and is suitable for general purpose optimizations. The Fisher type preconditioner relates to the inverse of Fisher information matrix, and is limited to a large subclass of stochastic optimization problems where the Fish information metric can be well defined. Both preconditioners can be derived from one framework, and estimated on any matrix Lie groups designated by the user with almost the same natural or relative gradient descent methods minimizing specific preconditioner estimation criteria. We consider the minimization of cost function f (θ θ θ) = E z [(θ θ θ, z z z)]where E z takes expectation over random variable z z z, is a loss function, and θ θ θ is the model parameter vector to be optimized. For example, in a classification problem, could be the cross entropy loss, z z z is a pair of input feature vector and class label, vector θ θ θ consists of all the trainable parameters in the classification model, and E z takes average over all samples from the training data set. By assuming second order differentiable model and loss, we could approximate (θ θ θ, z z z) as a quadratic function of θ θ θ within a trust region around θ θ θ, i.e., (θ θ θ, z z z) = b b b T z θ θ θ + 0.5θ θ θ T H H H z θ θ θ + a z, where a z is the sum of higher order approximation errors and constant term independent of θ θ θ, H H H z is a symmetric matrix, and subscript z in b b b z, H H H z and a z reminds us that these three terms depend on z z z. Clearly, these three terms depend on θ θ θ as well, although we do not explicitly show this dependence to simplify our notations since we just consider parameter updates in the same trust region. Now, we may rewrite DISPLAYFORM0 We do not impose any assumption, e.g., positive definiteness, on H H H except for being symmetric. Thus, the quadratic surface in the trust region could be non-convex. To simplify our notations, we no longer consider the higher order approximation errors included in a, and simply assume that f (θ θ θ) is a quadratic function of θ θ θ in the trust region. Let us consider a certain iteration. Preconditioned SGD updates θ θ θ as DISPLAYFORM0 where µ > 0 is the step size,f (θ θ θ) is an estimate of f (θ θ θ) obtained by replacing expectation with sample average, and positive definite matrix P P P could be a fixed or adaptive preconditioner. By letting θ θ θ = P P P −0.5 θ θ θ, we can rewrite as DISPLAYFORM1 where P P P −0.5 denotes the principal square root of P P P. Hence, suggests that preconditioned SGD is equivalent to SGD in a transformed parameter domain. Within the considered trust region, let us write the stochastic gradient, ∂f (θ θ θ)/∂θ θ θ, explicitly as DISPLAYFORM2 whereĤ H H andb b b are estimates of H H H and b b b, respectively. Combining and gives the following linear system θ θ θ ← (I I I − µP P PĤ H H)θ θ θ − µP P Pb b b for updating θ θ θ within the assumed trust region, where I I I is the identity matrix. A properly determined P P P could significantly accelerate convergence of the locally linear system in.We review a few facts shown in BID7 before introducing our main contributions. Let δθ θ θ be a random perturbation of θ θ θ, and be small enough such that θ θ θ + δθ θ θ still resides in the same trust region. Then, suggests the following ant perturbation of stochastic gradient, DISPLAYFORM3 where ε ε ε accounts for the error due to replacingĤ H H with H H H. Note that by definition, δĝ g g is a random vector dependent on both z z z and δθ θ θ. The preconditioner in BID7 is pursued by minimizing criterion c(P P P) = E z,δθ [δĝ g g T P P P δĝ g g + δθ θ θ T P P P −1 δθ θ θ] where subscript δθ in E z,δθ denotes taking expectation over δθ θ θ. Under mild conditions, criterion determines a unique positive definite P P P, which is optimal in the sense that it preconditions the stochastic gradient such that DISPLAYFORM4 which is comparable to relationship H H H −1 δg g gδg g g T H H H −1 = δθ θ θδθ θ θ T, where δg g g = H H Hδθ θ θ is the perturbation of noiseless gradient, and we assume that H H H is invertible, but not necessarily positive definite. Clearly, this preconditioner is comparable to H H H −1. It perfectly preconditions the gradient such that the amplitudes of parameter perturbations match that of the associated preconditioned gradient, regardless of the amount of gradient noise. Naturally, preconditioned SGD with this preconditioner inherits the scale-invariance property of the Newton method. Note that in the presence of gradient noise, the optimal P P P and P P P −1 given by are not unbiased estimates of H H H −1 and H H H, respectively. Actually, even if H H H is positive definite and available, H H H −1 may not always be a good preconditioner when H H H is ill-conditioned since it could significantly amplify the gradient noise along the directions of the eigenvectors of H H H associated with small eigenvalues, and lead to divergence. More specifically, it is shown in BID7 that Preconditioner estimation criterion requires δθ θ θ to be small enough such that θ θ θ and θ θ θ + δθ θ θ reside in the same trust region. In practice, numerical error might be an issue when handling small numbers with floating point arithmetic. This concern becomes more grave with the popularity of single and even half precision math in large scale neural network training. Luckily, relates δĝ g g to the Hessianvector product, which can be efficiently evaluated with automatic differentiation software tools. Let v v v be a random vector with the same dimension as θ θ θ. Then, suggests the following method for Hessian-vector product evaluation, DISPLAYFORM5 DISPLAYFORM6 Now, replacing (δθ θ θ, δĝ g g) in with (v v v,Ĥ H Hv v v) leads to our following new preconditioner estimation criterion, c n (DISPLAYFORM7 where the subscript v in E z,v suggests taking expectation over v v v. We no longer have the need to assume v v v to be an arbitrarily small vector. It is important to note that this criterion only requires the Hessian-vector product. The Hessian itself is not of interest. We call the Newton type preconditioner estimation criterion as the ant preconditioned SGD method is closely related to the Newton method. We consider the machine learning problems where the empirical Fisher information matrix can be well defined by DISPLAYFORM0 whereĝ g g = ∂f (θ θ θ)/∂θ θ θ is a shorthand for stochastic gradient, and λ ≥ 0 is a damping factor. Clearly, v v v is independent ofĝ g g. Let us further assume that v v v is drawn from standard multivariate normal distribution N (0 0 0, DISPLAYFORM1 Then, we could simplify c f (P P P) as DISPLAYFORM2 By letting the derivative of c f (P P P) with respect to P P P be zero, the optimal positive definite solution for c f (P P P) is readily shown to be DISPLAYFORM3 Whenĝ g g is a gradient estimation obtained by taking average over B independent samples, E z [ĝ g gĝ g g T] is related to the Fisher information matrix by DISPLAYFORM4 We call this preconditioner the Fisher type one due to its close relationship to the Fisher information matrix. One can easily modify this preconditioner to obtain an unbiased estimation of F F F −1. Let s s s be an exponential moving average ofĝ g g. Then, after replacing theĝ g g in withĝ g g − s s s + s s s/ √ B and setting λ = 0, P P P 2 /B will be an unbiased estimation of F F F −1. Generally, it might be acceptable to keep the bias term, (B − 1)g g gg g g T /B, in for two reasons: it is nonnegative definite and regularizes the inversion in FORMULA0; it vanishes when the parameters approach a stationary point. Actually, the Fisher information matrix could be singular for many commonly used models, e.g., finite mixture models, neural networks, hidden Markov models. We might not be able to inverse F F F for these singular statistical models without using regularization or damping. A Fisher type preconditioner with λ > 0 loses the scale-invariance property of a Newton type preconditioner. Both P P P and P P P 2 can be useful preconditioners when the step size µ and damping factor λ are set properly. Following the ideas in BID7, we can show that determines a unique positive definite preconditioner if and only if DISPLAYFORM0 has distinct eigenvalues. Other minimum solutions of criterion FORMULA0 are either indefinite or negative definite, and are not interested for our purpose. The proof itself has limited novelty. We omit it here. Instead, let us consider the simplest case, where θ θ θ is a scalar parameter, to gain some intuitive understandings of criterion. For scalar parameter, it is trivial to show that the optimal solutions minimizing are DISPLAYFORM1 whereĤ H H, H H H, P P P, and v v v are replaced with their plain lower case letters, and we have used the fact that H H H −Ĥ H H and v v v are independent. For gradient descent, we choose the positive solution, although the negative one gives the global minimum of. With the positive preconditioner, eigenvalue of the locally linear system in is DISPLAYFORM2 Now, it is clear that this optimal preconditioner damps the gradient noise when E z [(h−ĥ) 2 ] is large, and preconditions the locally linear system in such that its eigenvalue has unitary amplitude when the gradient noise vanishes. Convergence is ensured when a normalized step size, i.e., 0 < µ < 1, is used. For θ θ θ with higher dimensions, eigenvalues of the locally linear system in is normalized into range [−1, 1] as well, in a way similar to. Let us take the Newton type preconditioner as an example to derive its updating rule. Updating rule for the Fisher type preconditioner is the same except for replacing the Hessian-vector product with stochastic gradient. Here, Lie group always refers to the matrix Lie group. It is inconvenient to optimize P P P directly as it must be a symmetric and positive definite matrix. Instead, we represent the preconditioner as P P P = Q Q Q T Q Q Q, and estimate Q Q Q. Now, Q Q Q must be a nonsingular matrix as both c n (P P P) and c f (P P P) diverge when P P P is singular. Invertible matrices with the same dimension form a Lie group. In practice, we are more interested in Lie groups with sparse representations. Examples of such groups are given in the next section. Let us consider a proper small perturbation of Q Q Q, δQ Q Q, such that Q Q Q + δQ Q Q still lives on the same Lie group. The distance between Q Q Q and Q Q Q + δQ Q Q can be naturally defined as dist(Q Q Q, Q Q Q + δQ Q Q) = tr(δQ Q QQ Q Q −1 Q Q Q −T δQ Q Q T) BID1. Intuitively, this distance is larger for the same amount of perturbation when Q Q Q is closer to a singular matrix. With the above tensor metric definition, natural gradient for optimizing Q Q Q has form DISPLAYFORM0 For example, when Q Q Q lives on the group of invertible upper triangular matrices, R R R is given by DISPLAYFORM1 where triu takes the upper triangular part of a matrix. Another way to derive is to let δQ Q Q = EQ Q Q, and consider the derivative with respect to E, where E is a proper small matrix such that Q Q Q + EQ Q Q still lives on the same Lie group. Gradient derived in this way is known as relative gradient BID3. For our preconditioner learning problem, relative gradient and natural gradient have the same form. Now, Q Q Q can be updated using natural or relative gradient descent as DISPLAYFORM2 In practice, it is convenient to use the following updating rule with normalized step size, DISPLAYFORM3 where 0 < µ 0 < 1, and · takes the norm of a matrix. One simple choice for matrix norm is the maximum absolute value of a matrix. Note that natural gradient can take different forms. One should not confuse the natural gradient on the Lie group derived from a tensor metric with the natural gradient for model parameter learning derived from a Fisher information metric. One iteration of the Newton type preconditioned SGD consists of the following steps.1. Evaluate stochastic gradientĝ g g. The two step sizes, µ and µ 0, are normalized. They should take values in range with typical value 0.01. We usually initialize Q Q Q to a scaled identity matrix with proper dimension. The specific form ofR R R depends on the Lie group to be considered. For example, for upper triangular Q Q Q, we havê DISPLAYFORM0, where Q Q Q −T v v v can be efficiently calculated with back substitution. We only need to replaceĤ H Hv v v in the Newton type preconditioned SGD withĝ g g+λv v v to obtain the Fisher type one. Thus, we do not list its main steps here. Note that only its step size for the preconditioner updating is normalized. There is no simple way to jointly determine the proper ranges for step size µ and damping factor λ. Again,R R R may take different forms on different Lie groups. For upper triangular Q Q Q, we haveR DISPLAYFORM0, where v v v ∼ N (0 0 0, I I I). Here, it is important to note that the natural or relative gradient for c f (P P P) with the form given in involves explicit matrix inversion. However, matrix inversion can be avoided by using the c f (P P P) in, which includes v v v as an auxiliary variable. It is highly recommended to avoid explicit matrix inversion for large Q Q Q. There are many ways to modify the above preconditioned SGD methods. Since curvatures typically evolves slower than gradients, one can update the preconditioner less frequently to save computations per iteration. With parallel computing available, one might update the preconditioner and model parameters simultaneously and asynchronously to save wall time per iteration. Combining preconditioner and momentum may further accelerate convergence. For recurrent neural network learning, we may need to clip the norm of preconditioned gradients to avoid excessively large parameter updates. In general, preconditioned gradient clipping relates to the trust region method by θ θ θ[new] − θ θ θ = µP P Pĝ g g/max(1, P P Pĝ g g /Ω) ≤ µΩwhere Ω > 0 is a clipping threshold, comparable to the size of trust region. One may set Ω to a positive number proportional to the square root of the number of model parameters. Most importantly, we can choose different Lie groups for estimating our preconditioners to achieve a good trade off between performance and complexity. In practice, we seldom consider the Lie group consisting of dense invertible matrices for preconditioner estimation when the problem size is large. Lie groups with sparse structures are of more interests. To begin with, let us recall a few facts about Lie group. If A A A and B B B are two Lie groups, then A A A T, A A A ⊗B B B, and A A A ⊕B B B all are Lie groups, where ⊗ and ⊕ denote Kronecker product and direct sum, respectively. Furthermore, for any matrix C C C with compatible dimensions, block matrix DISPLAYFORM0 still forms a Lie group. We do not show proofs of the above statements here as they are no more than a few lines of algebraic operations. These simple rules can be used to design many useful Lie groups for constructing sparse preconditioners. We already know that invertible upper triangular matrices form a Lie group. Here, we list a few useful ones with sparse representations. Diagonal matrices with the same dimension and positive diagonal entries form a Lie group with reducible representation. Preconditioners learned on this group are called diagonal preconditioners. For matrix parameter Θ Θ Θ, we can flatten Θ Θ Θ into a vector, and precondition its gradient using a Kronecker product preconditioner with Q Q Q having form Q Q Q = Q Q Q 2 ⊗ Q Q Q 1. Clearly, Q Q Q is a Lie group as long as Q Q Q 1 and Q Q Q 2 are two Lie groups. Let us check its role in learning the following affine transformation DISPLAYFORM0 where x x x is the input feature vector augmented with 1, and y y y is the output feature vector. After reverting the flattened Θ Θ Θ back to its matrix form, the preconditioned SGD learning rule for Θ Θ Θ is DISPLAYFORM1 Similar to, we introduce coordinate transformation DISPLAYFORM2 2, and rewrite as DISPLAYFORM3 Correspondingly, the affine transformation in FORMULA2 is rewritten as y y y = Θ Θ Θ x x x, where y y y = Q Q Q −T 1 y y y and x x x = Q Q Q 2 x x x are the transformed input and output feature vectors, respectively. Hence, the preconditioned SGD in is equivalent to the SGD in with transformed feature vectors x x x and y y y. We know that feature whitening and normalization could significantly accelerate convergence. A Kronecker product preconditioner plays a similar role in learning the affine transformation in. This is a special Kronecker product preconditioner by constraining Q Q Q 1 to be a diagonal matrix, and Q Q Q 2 to be a sparse matrix where only its diagonal and last column can have nonzero values. Note that Q Q Q 2 with nonzero diagonal entries forms a Lie group. Hence, Q Q Q = Q Q Q 2 ⊗ Q Q Q 1 is a Lie group as well. We call it a scaling and normalization preconditioner as it resembles a preconditioner that scales the output features and normalizes the input features. Let us check the transformed features y y y = Q Q Q −T 1 y y y and x x x = Q Q Q 2 x x x. It is clear that y y y is an element-wisely scaled version of y y y as Q Q Q 1 is a diagonal matrix. To make x x x a "normalized" feature vector, x x x needs to be an input feature vector augmented with 1.Let us check a simple example to verify this point. We consider an input vector with two features, and write down its normalized features explicitly as below, DISPLAYFORM0 where m i and σ i are the mean and standard deviation of x i, respectively. It is straightforward to show that the feature normalization operation in forms a Lie group with four freedoms. For the scaling-and-normalization preconditioner, we have no need to force the last diagonal entry of Q Q Q 2 to be 1. Hence, the group of feature normalization operation is a subgroup of Q Q Q 2. This is another special Kronecker product preconditioner by constraining Q Q Q 1 to be a diagonal matrix, and Q Q Q 2 to be an upper triangular matrix with positive diagonal entries. We call it a scaling-andwhitening preconditioner since it resembles a preconditioner that scales the output features and whitens the input features. Again, the input feature vector x x x must be augmented with 1 such that the whitening operation forms a Lie group represented by upper triangular matrices with 1 being its last diagonal entry. This is a subgroup of Q Q Q 2 as we have no need to fix Q Q Q 2's last diagonal entry to 1.It is not possible to enumerate all kinds of Lie groups suitable for constructing preconditioners. For example, Kronecker product preconditioner with form Q Q Q = Q Q Q 3 ⊗ Q Q Q 2 ⊗ Q Q Q 1 could be used for preconditioning gradients of a third order tensor. The normalization and whitening groups are just two special cases of the groups with the form shown in FORMULA0, and there are numerous more choices having sparsities between that of these two. Regardless of the detailed form of Q Q Q, all such preconditioners share the same form of learning rule shown in FORMULA2, and they all can be efficiently learned using natural or relative gradient descent without much tuning effort. Adagrad, RMSProp and Adam all use the Fisher type preconditioner living on the group of diagonal matrices with positive diagonal entries. This is a simple group. Optimal solution for c f (P P P) has closed-form solution P P P = diag(1 E z [ĝ g g ĝ g g] + λ 2 ), where and denote element wise multiplication and division, respectively. In practice, simple exponential moving average is used to replace the expectation when using this preconditioner. For diagonal preconditioner, the optimal solution minimizing c n (P P P) has closed-form solution DISPLAYFORM0 reduces to a vector with unit entries. Then, this optimal solution gives the equilibration preconditioner in BID4. The preconditioners considered in BID10 and BID8 are closely related to the Fisher type Kronecker product preconditioners. While KFAC approximates the Fisher metric of a matrix parameter as a Kronecker product to obtain its approximated inverse in closedform solution, our method turns to an iterative solution to approximate this same inverse. Theoretically, our method's accuracy is only limited by the expressive power of the Lie group since no intermediate approximation is made. In practice, one distinct advantage of our method over KFAC is that explicit matrix inversion is avoided by introducing auxiliary vector v v v and using back substitution, while KFAC typically requires inversion of symmetric matrices. On graphics processing units (GPU) and with large matrices, parallel back substitution is as computationally cheap as matrix multiplication, and could be several orders of magnitude faster than inversion of symmetric matrix. Another advantage is that our method is derived from a unified framework. There is no need to invent different preconditioner learning rules when we switch the Lie group representations. Batch normalization can be viewed as preconditioned SGD using a specific scaling-andnormalization preconditioner with constraint Q Q Q 1 = I I I and Q Q Q 2 from the feature normalization Lie group. However, we should be aware that explicit input feature normalization is only empirically shown to accelerate convergence, and has little meaning in certain scenarios, e.g., recurrent neural network learning where features may not have any stationary first or second order statistic. Both the Newton and Fisher type preconditioned SGD methods provide a more general and principled approach to find the optimal preconditioner, and apply to a broader range of applications. Generally, a scaling-and-normalization preconditioner does not necessarily "normalize" the input features in the sense of mean removal and variance normalization. We use the square root Fisher type preconditioners in the following experiments since they are less picky on the damping factor, and seem to be more numerically robust on large scale problems. Still, as shown in our Pytorch implementation package, the original Fisher type preconditioners could perform better on small scale problems like the MNIST handwritten digit recognition task. Let us consider the minimization of Rosenbrock function, f (θ θ θ) = 100(θ 2 − θ 2 1) 2 + (1 − θ 1) 2, starting from initial guess θ θ θ = [−1, 1]. This is a well known benchmark problem for mathematical optimization. The compared methods use fixed step size. For each method, the best step size is selected from sequence {. . ., 1, 0.5, 0.2, 0.1, 0.05, 0.02, 0.01, . . .}. For gradient descent, the best step size is 0.002. For momentum method, the moving average factor is 0.9, and the best step size is 0.002. For Nesterov momentum, the best step size is 0.001. For preconditioned SGD, Q Q Q is initialized to 0.1I I I and lives on the group of triangular matrices. For the Fisher type method, we set λ = 0.1, and step sizes 0.01 and 0.001 for preconditioner and parameter updates, respectively. For the Newton type method, we set step sizes 0.2 and 0.5 for preconditioner and parameter updates, respectively. FIG1 summarizes the . The Newton type method performs the best, converging to the optimal solution using about 200 iterations. The Fisher type method does not fit into this problem, and performs poorly as expected. Mathematical optimization is not our focus. Still, this example shows that the Newton type preconditioned SGD works well for mathematical optimization. We consider the ImageNet ILSVRC2012 database for the image classification task. The well known AlexNet is considered. We follow the descriptions in BID0 as closely as possible to set up our experiment. One main difference is that we do not augment the training data. Another big difference is that we use a modified local response normalization (LRN). The LRN function from TensorFlow implementation is not second order differentiable. We have to approximate the local energy used for LRN with a properly scaled global energy to facilitate Hessian-vector product evaluation. Note that convolution can be rewritten as correlation between the flattened input image patches and filter coefficients. In this way, we find that there are eight matrices to be optimized in the AlexNet, and their shapes are: each matrix. Denser preconditioners, e.g., the Kronecker product one, require hundreds of millions parameters for representations, and are expensive to run on our platform. Each compared method is trained with 40 epochs, mini-batch size 128, step size µ for the first 20 epochs, and 0.1µ for the last 20 epochs. We have compared several methods with multiple settings, and only report the ones with reasonably good here. For Adam, the initial step size is set to 0.00005. For batch normalization, initial step size is 0.002, and its moving average factors for momentum and statistics used for feature normalization are 0.9 and 0.99, respectively. The momentum method uses initial step size 0.002, and moving average factor 0.9 for momentum. Preconditioned SGD performs better with the scaling-and-normalization preconditioner. Its Q Q Q is initialized to 0.1I I I, and updated with normalized step size 0.01. For the Fisher type preconditioner, we set λ = 0.001 and initial step size 0.00005. For the Newton type preconditioner, its initial step size is 0.01. FIG0 summarizes the . Training loss for batch normalization is only for reference purpose as normalization alters the L2-regularization term. Batch normalization does not perform well under this setup, maybe due to its conflict with certain settings like the LRN and L2-regularization. We see that the scalingand-normalization preconditioner does accelerate convergence, although it is super sparse. The Newton type preconditioned SGD performs the best, and achieves top-1 validation accuracy about 56% when using only one crop for testing, while the momentum method may require 90 epochs to achieve similar performance. We consider the world level language modeling problem with reference implementation available from https://github.com/pytorch/examples. The Wikitext-2 database with 33278 tokens is considered. The task is to predict the next token from history observations. Our tested network consists of six layers, i.e., encoding layer, LSTM layer, dropout layer, LSTM layer, dropout layer, and decoding layer. For each LSTM layer, we put all its coefficients into a single matrix Θ Θ Θ by defining output and augmented input feature vectors as in DISPLAYFORM0, where t is a discrete time index, x x x is the input, h h h is the hidden state, and c c c is the cell state. The encoding layer's weight matrix is the transpose of that of the decoding layer. Thus, we totally get three matrices to be optimized. With hidden layer size 200, shapes of these three matrices are respectively. For all methods, the step size is reduced to one fourth of the current value whenever the current perplexity on validation set is larger than the best one ever found. For SGD, the initial step size is 20, and the gradient is clipped with threshold 0.25. The momentum method diverges easily without clipping. We set momentum 0.9, initial step size 1, and clipping threshold 0.25. We set initial step size 0.005 and damping factor λ 2 = 10 −12 for Adam and sparse Adam. Sparse Adam updates its moments and model parameters only when the corresponding stochastic gradients are not zeros. We have tried diagonal, scaling-and-normalization and scaling-and-whitening preconditioners for each matrix. The encoding (decoding) matrix is too large to consider KFAC like preconditioner. The diagonal preconditioner performs the worst, and the other two have comparable performance. For both types of preconditioned SGD, the clipping threshold for preconditioned gradient is 100, the initial step size is 0.1, and Q Q Q is initialized to I I I. We set λ = 0 for the Fisher Figure 3 summarizes the when the dropout rate is 0.35. Methods involving momentum, including Adam and sparse Adam, perform poorly. Note that our preconditioners preserve the sparsity property of gradients from the encoding and decoding layers (Appendix A). This saves considerable computations by avoiding update parameters with zero gradients. Again, both preconditioners accelerate convergence significantly despite their high sparsity. Compared with SGD, the Fisher type preconditioned SGD adds limited computational complexity when sparse preconditioners are adopted. The Newton type preconditioned SGD requires Hessianvector product, which typically has complexity comparable to that of gradient evaluation. Thus, using SGD as the base line, the Newton type preconditioned SGD approximately doubles the computational complexity per iteration, while the Fisher type SGD has similar complexity. Wall time per iteration of preconditioned SGD highly depends on the implementations. Ideally, the preconditioners and parameters could be updated in a parallel and asynchronous way such that SGD and preconditioned SGD have comparable wall time per iteration. We have put our TensorFlow and Pytorch implementations on https://github.com/ lixilinx. More experimental comparing different preconditioners and optimization methods on diverse benchmark problems can be found there. For the ImageNet experiment, all compared methods are implemented in Tensorflow, and require two days and a few hours to finish 40 epochs on a GeForce GTX 1080 Ti GPU. The word level language modeling experiment is implemented in Pytorch. We have rewritten the word embedding function to enable second order derivative. For this task, SGD and the Fisher type preconditioned SGD have similar wall time per iteration, while the Newton type method requires about 80% more wall time per iteration than SGD when running on the same GPU. Two types of preconditioners and preconditioned SGD methods are studied. The one requiring Hessian-vector product for preconditioner estimation is suitable for general purpose optimization. We call it the Newton type preconditioned SGD due to its close relationship to the Newton method. The other one only requires gradient for preconditioner estimation. We call it the Fisher type preconditioned SGD as its preconditioner is closely related to the inverse of Fisher information matrix. Both preconditioners can be efficiently learned using natural or relative gradient descent on any matrix Lie groups designated by the user. The Fisher type preconditioned SGD has lower computational complexity per iteration, but may require more tuning efforts on selecting its step size and damping factor. The Newton type preconditioned SGD has higher computational complexity per iteration, but is more user friendly due to its use of normalized step size and built-in gradient noise damping ability. Both preconditioners, even with very sparse representations, are shown to considerably accelerate convergence on relatively large scale problems.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a new framework for preconditioner learning, derive new forms of preconditioners and learning methods, and reveal the relationship to methods like RMSProp, Adam, Adagrad, ESGD, KFAC, batch normalization, etc.

Represent the TLDR for finding the corresponding scientific article

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: Represent the text to find a very short summary (1-2 sentences max).

Question: We present EDA: easy data augmentation techniques for boosting performance on text classification tasks. EDA consists of four simple but powerful operations: synonym replacement, random insertion, random swap, and random deletion. On five text classification tasks, we show that EDA improves performance for both convolutional and recurrent neural networks. EDA demonstrates particularly strong for smaller datasets; on average, across five datasets, training with EDA while using only 50% of the available training set achieved the same accuracy as normal training with all available data. We also performed extensive ablation studies and suggest parameters for practical use. Text classification is a fundamental task in natural language processing (NLP). Machine learning and deep learning have achieved high accuracy on tasks ranging from sentiment analysis to topic classification BID24, but high performance is often dependent on the size and quality of training data, which is often tedious to collect. Automatic data augmentation is commonly used in vision BID20 BID22 BID10 and speech (; BID7 and can help train more robust models, particularly when using smaller datasets. However, because it is difficult to come up with generalized rules for language transformation, universal data augmentation techniques in NLP have not been explored. Previous work has proposed techniques for data augmentation in NLP. One popular study generated new data by translating sentences into French and back into English BID28 . Other works have used predictive language models for synonym replacement BID8 and data noising as smoothing BID27 . Although these techniques are valid, they are not often used in practice because they have a high cost of implementation relative to performance gain. In this paper, we present a simple set of universal data augmentation techniques for NLP called EDA (easy data augmentation). To the best of our knowledge, we are the first to comprehensively explore text editing techniques for data augmentation. We systematically evaluate EDA on five benchmark classification tasks, and show that EDA provides substantial improvements on all five tasks and is particularly helpful for smaller datasets. Code will be made publicly available. Operation Sentence None A sad, superior human comedy played out on the back roads of life. SR A lamentable, superior human comedy played out on the backward road of life. RI A sad, superior human comedy played out on funniness the back roads of life. RS A sad, superior human comedy played out on roads back the of life. RD A sad, superior human out on the roads of life. Frustrated by the measly performance of text classifiers trained on small datasets, we tested a number of augmentation operations loosely inspired by those used in vision and found that they helped train more robust models. Here, we present the full details of EDA. For a given sentence in the training set, we perform the following operations: 1. Synonym Replacement (SR): Randomly choose n words from the sentence that are not stop words. Replace each of these words with one of its synonyms chosen at random. 2. Random Insertion (RI): Find a random synonym of a random word in the sentence that is not a stop word. Insert that synonym into a random position in the sentence. Do this n times. 3. Random Swap (RS): Randomly choose two words in the sentence and swap their positions. Do this n times. 4. Random Deletion (RD): Randomly remove each word in the sentence with probability p. Since long sentences have more words than short ones, they can absorb more noise while maintaining their original class label. To compensate, we vary the number of words changed, n, for SR, RI, and RS based on the sentence length l with the formula n=α l, where α is a parameter that indicates the percent of the words in a sentence are changed (we use p=α for RD). Furthermore, for each original sentence, we generate n aug augmented sentences. Examples of augmented sentences are shown in TAB0. We note that synonym replacement has been used previously BID9 BID26, but to our knowledge, random insertions, swaps, and deletions have not been studied. We conduct experiments on five benchmark text classification tasks: SST-2: Stanford Sentiment Treebank BID21, CR: customer reviews BID3 BID13, SUBJ: subjectivity/objectivity dataset BID15 ), TREC: question type dataset BID11, and PC: Pro-Con dataset BID2. Summary statistics are shown in TAB3 in the Appendix. Furthermore, we hypothesize that EDA is more helpful for smaller datasets, so we delegate the following sized datasets by selecting a random subset of the full training set with N train ={500, 2,000, 5,000, all available data}.We run experiments for two state-of-the-art models in text classification. Recurrent neural networks (RNNs) are suitable for sequential data. We use a LSTM-RNN BID12. Convolutional neural networks (CNNs) have also achieved high performance for text classification. We implement them as described in BID6. Details are in Section 6.1 in the Appendix. We run both CNN and RNN models with and without EDA across all five datasets for varying training set sizes. Average performances (%) are shown in Table 2. Of note, average improvement was 0.8% for full datasets and 3.0% for N train =500. Table 2: Average performances (%) across five text classification tasks for models without and without EDA on different training set sizes. Overfitting tends to be more severe when training on smaller datasets. By conducting experiments using a restricted fraction of the available training data, we show that EDA has more significant improvements for smaller training sets. We run both normal training and EDA training for the following training set fractions (%): {1, 5, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100}. FIG0 shows average performance across all datasets. The best average accuracy without augmentation, 88.3%, was achieved using 100% of the training data. Models trained using EDA surpassed this number by achieving an average accuracy of 88.6% while only using 50% of the available training data. Results for individual datasets are displayed in FIG3 (Appendix). In data augmentation, input data is altered while class labels are maintained. However, if sentences are significantly changed, then original class labels may no longer be valid. We take a visualization approach to examine whether EDA operations significantly change the meanings of augmented sentences. First, we train an RNN on the pro-con classification task (PC) without augmentation. Then, we apply EDA to the test set by generating nine augmented sentences per original sentence. These are fed into the RNN along with the original sentences, and we extract the outputs from the last dense layer. We apply t-SNE to these vectors and plot their 2-D representations FIG1 ). We found that the ing latent space representations for augmented sentences closely surrounded those of the original sentences. 3.4 ABLATION STUDIES So far, we have shown encouraging empirical . In this section, we perform ablation studies to explore the effects of each component in EDA. Synonym replacement has been previously used BID9 BID26, but the other three EDA operations have not yet been explored. One could hypothesize that the bulk of EDA's performance gain is from synonym replacement, so we isolate each of the EDA operations to determine their individual ability to boost performance. For all four operations, we ran models using a single operation while varying the augmentation parameter α={0.05, 0.1, 0.2, 0.3, 0.4, 0.5} FIG2 ).It turns out that all four EDA operations contribute to performance gain. For SR, improvement was good for small α, but high α hurt performance, likely because replacing too many words in a sentence changed the identity of the sentence. For RI, performance gains were more stable for different α values, possibly because the original words in the sentence and their relative order were maintained in this operation. RS yielded high performance gains at α≤0.2, but declined at α≥0.3 since performing too many swaps is equivalent to shuffling the entire order of the sentence. RD had the highest gains for low α but severely hurt performance at high α, as sentences are likely unintelligible if up to half the words are removed. Improvements were more substantial on smaller datasets for all operations, and α=0.1 appeared to be a "sweet spot" across the board. The natural next step is to determine how the number of generated augmented sentences per original sentence, n aug, affects performance. We calculate average performances over all datasets for n aug ={1, 2, 4, 8, 16, 32}, as shown in FIG2 (middle).On smaller training sets, overfitting was more likely, so generating many augmented sentences yielded large performance boosts. For larger training sets, adding more than four augmented sentences per original sentence was unhelpful since models tend to generalize properly when large quantities of real data are available. Based on these , we recommend parameters for practical use in FIG2 (right). Related work is creative but often complex. Back-translation BID18, translational data augmentation BID1, and noising BID27 have shown improvements in BLEU measure for machine translation. For other tasks, previous approaches include task-specific heuristics BID5 and back-translation BID19 BID28. Regarding synonym replacement (SR), one study showed a 1.4% F1-score boost for tweet classification by finding synonyms with k-nearest neighbors using word embeddings BID26 ). Another study found no improvement in temporal analysis when replacing headwords with synonyms BID9, and mixed were reported for using SR in character-level text classification; however, neither work conducted extensive ablation studies. Most studies explore data augmentation as a complementary for translation or in a taskspecific context, so it is hard to directly compare EDA to previous literature. But there are two studies similar to ours that evaluate augmentation techniques on multiple datasets. BID4 proposed a generative model that combines a variational auto-encoder (VAE) and attribute discriminator to generate fake data, demonstrating a 3% gain in accuracy on two datasets. BID8 showed that replacing words with other words that were predicted from the sentence context using a bi-directional language model yielded a 0.5% gain on five datasets. However, training a variational auto-encoder or bidirectional LSTM language model is a lot of work. EDA yields similar but is much easier to use because it does not require training a language model and does not use external datasets. In TAB5 (Appendix), we show EDA's ease of use compared to other techniques. We have shown that simple data augmentation operations can boost performance on text classification tasks. Although improvement is at times marginal, EDA substantially boosts performance and reduces overfitting when training on smaller datasets. Continued work on this topic could include exploring the theoretical underpinning of the EDA operations. We hope that EDA's simplicity makes a compelling case for its widespread use in NLP. This section contains implementation details, dataset statistics, and detailed not included in the main text. All code for EDA and the experiments in this paper will be made available. The following implementation details were omitted from the main text:Synonym thesaurus. All synonyms for synonym replacements and random insertions were generated using WordNet BID14. We suspect that EDA will work with any thesaurus. Word embeddings. We use 300-dimensional Common-Crawl word embeddings trained using GloVe BID16. We suspect that EDA will work with any pre-trained word embeddings. CNN. We use the following architecture: input layer, 1-D convolutional layer of 128 filters of size 5, global 1D max pool layer, dense layer of 20 hidden units with ReLU activation function, softmax output layer. We initialize this network with random normal weights and train against the categorical cross-entropy loss function with the adam optimizer. We use early stopping with a patience of 3 epochs. RNN. The architecture used in this paper is as follows: input layer, bi-directional hidden layer with 64 LSTM cells, dropout layer with p=0.5, bi-directional layer of 32 LSTM cells, dropout layer with p=0.5, dense layer of 20 hidden units with ReLU activation, softmax output layer. We initialize this network with random normal weights and train against the categorical cross-entropy loss function with the adam optimizer. We use early stopping with a patience of 3 epochs. Summary statistics for the five datasets used are shown in TAB3. DISPLAYFORM0 In FIG3, we show performance on individual text classification tasks for both normal training and training with EDA, with respect to percent of dataset used for training. In FIG3, we compare the EDA's ease of use to that of related work.1 BID4 How does using EDA improve text classification performance? While it is hard to identify exactly how EDA improves the performance of classifiers, we believe there are two main reasons. The first is that generating augmented data similar to original data introduces some degree of noise that helps prevent overfitting. The second is that using EDA can introduce new vocabulary through the synonym replacement and random insertion operations, allowing models to generalize to words in the test set that were not in the training set. Both these effects are more pronounced for smaller datasets. Why should I use EDA instead of other techniques such as contextual augmentation, noising, GAN, or back-translation? All of the above are valid techniques for data augmentation, and we encourage you to try them, as they may actually work better than EDA, depending on the dataset. But because these techniques require the use of a deep learning model in itself to generate augmented sentences, there is often a high cost of implementing these techniques relative to the expected performance gain. With EDA, we aim to provide a set of simple techniques that are generalizable to a range of NLP tasks. Is there a chance that using EDA will actually hurt my performance? Considering our across five classification tasks, it's unlikely but there's always a chance. It's possible that one of the EDA operations can change the class of some augmented sentences and create mislabeled data. But even so, "deep learning is robust to massive label noise" BID17.For random insertions, why do you only insert words that are synonyms, as opposed to inserting any random words? Data augmentation operations should not change the true label of a sentence, as that would introduce unnecessary noise into the data. Inserting a synonym of a word in a sentence, opposed to a random word, is more likely to be relevant to the context and retain the original label of the sentence.

Simple text augmentation techniques can significantly boost performance on text classification tasks, especially for small datasets.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the natural language

Question: We propose a model that is able to perform physical parameter estimation of systems from video, where the differential equations governing the scene dynamics are known, but labeled states or objects are not available. Existing physical scene understanding methods require either object state supervision, or do not integrate with differentiable physics to learn interpretable system parameters and states. We address this problem through a \textit{physics-as-inverse-graphics} approach that brings together vision-as-inverse-graphics and differentiable physics engines, where objects and explicit state and velocity representations are discovered by the model. This framework allows us to perform long term extrapolative video prediction, as well as vision-based model-predictive control. Our approach significantly outperforms related unsupervised methods in long-term future frame prediction of systems with interacting objects (such as ball-spring or 3-body gravitational systems), due to its ability to build dynamics into the model as an inductive bias. We further show the value of this tight vision-physics integration by demonstrating data-efficient learning of vision-actuated model-based control for a pendulum system. We also show that the controller's interpretability provides unique capabilities in goal-driven control and physical reasoning for zero-data adaptation. System identification or physical parameter estimation is commonly required for control or state estimation for physical modelling, and typically relies on dedicated sensing equipment and carefully constructed experiments. Current machine learning approaches to physical modeling from video either require training by supervised regression from video to object coordinates before estimating explicit physics (; b;), or are able to discover and segment objects from video in an unsupervised manner, but do not naturally integrate with a physics engine for long-term predictions or generation of interpretable locations and physical parameters for physical reasoning (; van). In this work, we bridge the gap between unsupervised discovery of objects from video and learning the physical dynamics of a system, by learning unknown physical parameters and explicit trajectory coordinates. Our approach, called physics-as-inverse-graphics, solves the physical modeling problem via a novel vision-as-inverse-graphics encoder-decoder system that can render and de-render image components using Spatial Transformers (ST) in a way that makes it possible for the latent representation to generate disentangled interpretable states (position/velocity). These can be used directly by a differentiable physics engine to learn the parameters of a scene where the family of differential equations governing the system are known (e.g. objects connected by a spring), but the corresponding parameters are not (e.g. spring constant). This allows us to to identify physical parameters and learn vision components of the model jointly in an end-to-end fashion. Our contribution is a solution to unsupervised learning of physical parameters from video, without having access to ground-truth appearance, position or velocities of the objects, a task that had so far remained unsolved . In addition to showing that our model can learn physical parameters without object or state supervision (a task with intrinsic scientific interest in and of itself), we show that incorporating dynamics priors in the form of known physical equations of motion with learnable parameters together with learnable vision and graphics can improve model performance in two challenging tasks: long term video prediction and visual model predictive control. We first evaluate physical parameter estimation accuracy and future video frame prediction on 4 datasets with different non-linear interactions and visual difficulty. We then demonstrate the value of our method by applying it for data-efficient learning of vision-based control of an under-actuated pendulum. Notably our unique ability to extract interpretable states and parameters from pixels without supervision enables end-to-end vision-based control to exploit goal-paramaterized policies and physical reasoning for zero-shot adaptation. The ability to build inductive bias into models through structure is a key factor behind the success of modern neural architectures. Convolutional operations capture spatial correlations in images, recurrency allows for temporal reasoning , and spatial transformers provide spatial invariance in learning. However, many aspects of common data generation processes are not yet considered by these simple inductive biases. Importantly, they typically ignore the physical interactions underpinning data generation. For example, it is often the case that the underlying physics of a dynamic visual scene is known, even if specific parameters and objects are not. Incorporation of this information would be beneficial for learning, predicting the future of the visual scene, or control. Physics-as-inverse graphics introduces a framework that allows such high-level physical interaction knowledge to be incorporated into learning, even when ground-truth object appearance, positions and velocities are not available. In recent years there has been increased interest in physical scene understanding from video (; ; ; ; ; ;). In order to learn explicit physical dynamics from video our system must discover and model the objects in a scene, having position as an explicit latent variable. Here we build on the long literature of neural vision-as-inverse-graphics (; ; ; ; ; a), particularly on the use of spatial transformers (ST) for rendering . There are several models that assume knowledge of the family of equations governing system dynamics, but where the individual objects are either pre-segmented or their ground-truth positions/velocities are known (; b;). In terms of learning physical parameters, our work is directly inspired by the Galileo model and Physics 101 dataset (; 2016), which fits the dynamics equations to a scene with interacting objects. However, the Galileo model makes use of custom trackers which estimate the position and velocity of each object of interest, and is incapable of end-to-end learning from video, thus bypasses the difficulty of recognizing and tracking objects from video using a neural system. To the best of our knowledge, our model is the first to offer end-to-end unsupervised physical parameter and state estimation. Within the differentiable physics literature , observed that a multi-layer perceptron (MLP) encoder-decoder architecture with a physics engine was not able to learn without supervising the physics engine's output with position/velocity labels (c.f. Fig. 4 in). While in their case 2% labeled data is enough to allow learning, the transition to no labels causes the model to not learn at all. The key contribution of our work is the incorporation of vision-as-inverse-graphics with physics, which makes the transition possible. Another related area of increasing interest is unsupervised discovery of objects and/or dynamics from video (; van ; ; . Though powerful, such models do not typically use interpretable latent representations that can be directly used by a physics engine, reasoned about for physical problem solving, or that are of explicit interest to model users. For example, and use ST's to locate/place objects in a scene and predict their motion, but this work differs from ours in that our coordinate-consistent design obtains explicit cartesian, angular or scale coordinates, allowing us to feed state vectors directly into a differentiable physics engine. Under a similar motivation as our work, but without an inverse-graphics approach, developed an unsupervised model to obtain consistent object locations. However, this only applies to cartesian coordinates, not angles or scale. High-level view of our architecture. The encoder (top-right) estimates the position of N objects in each input frame. These are passed to the velocity estimator which estimates objects' velocities at the last input frame. The positions and velocities of the last input frame are passed as initial conditions to the physics engine. At every time-step, the physics engine outputs a set of positions, which are used by the decoder (bottom-right) to output a predicted image. If the system is actuated, an input action is passed to the physics engine at every time-step. See Section 3 for detailed descriptions of the encoder and decoder architectures. Despite recent interest in model-free reinforcement learning, model-based control systems have repeatedly shown to be more robust and sample efficient (; ; a). learn a latent dynamics model (PlaNet) that allows for planning from pixels, which is significantly more sample efficient than model-free learning strategies A3C and D4PG . However, when used for control, there is often a desire for visually grounded controllers operating under known dynamics, as these are verifiable and interpretable , and provide transferability and generality. However, system identification is challenging in vision-based control settings. use supervised learning to segment objects, controlling these using known rigid body dynamics. learn feedforward models with REINFORCE to predict physical states used by a known controller and dynamical model, but this is extremely sample inefficient. In contrast, we learn parameter and state estimation modules jointly to perform unsupervised system identification from pixels, enabling data-efficient vision-actuated model-based control. In order to learn explicit physics from video, several components have to be in place. First, the model must be able to learn to identify and represent the objects in an image. In order to perform dynamics prediction with a physics engine, the position and velocity of the object must be represented as explicit latent states (whereas appearance can be represented through some latent vector or, in our case, as a set of learned object templates). Our sequence-to-sequence video prediction architecture consists of 4 modules trained jointly: an encoder, a velocity estimator, a differentiable physics engine, and a graphics decoder. The architecture is shown in Figure 1. Encoder The encoder net takes a single frame I t as input and outputs a vector p t ∈ R N ×D corresponding to the D-dimensional coordinates of each of N objects in the scene, To extract each object's coordinates we use a 2-stage localization approach 1. First, the input frame is passed through a U-Net to produce N unnormalized masks. These masks (plus a learnable mask) are stacked and passed through a softmax to produce N + 1 masks, where each input pixel is softly assigned to a mask. The input image is then multiplied by each mask, and a 2-layer location network produces coordinate outputs from each masked input component. For a 2D system where the coordinates of each object are its (x, y) position (the polar coordinates case is analogous) and the images have dimensions H × H, the encoder output represents (x, y) coordinates with values in [0, H]. To do this, the activation of the encoder's output layer is a saturating non-linearity H/2 · tanh(·) + H/2. Velocity estimator The velocity estimator computes the velocity vector of each object at the L-th input frame given the coordinates produced by the encoder for this object at the first L input frames, v We implement this as a 3 hidden layer MLP with 100 tanh activated units. Differentiable physics engine The physics engine contains the differential equations governing the system, with unknown physical parameters to be learned -such as spring constants, gravity, mass, etc. Given initial positions and velocities produced by the encoder and velocity estimator, the physics engine rolls out the objects' trajectories. In this work we use a simple physics engine with Euler integration, where p t, v t is computed from p t−1, v t−1 by repeating for i ∈ [1..M]: where ∆t is the integration step, θ are the physical parameters and F is the force applied to each object, according to the equations in Appendix A. We use M = 5 in all experiments. In principle, more complex physics engines could be used . Coordinate-Consistent Decoder The decoder takes as input the positions given by the encoder or physics engine, and outputs a predicted imageĨ t. The decoder is the most critical part of this system, and is what allows the encoder, velocity estimator and physics engine to train correctly in a fully unsupervised manner. We therefore describe its design and motivation in greater detail. While an encoder with outputs in the range [0, H] can represent coordinates in pixel space, it does not mean that the decoder will learn to correctly associate an input vector (x, y) with an object located at pixel (x, y). If the decoder is unconstrained, like a standard MLP, it can very easily learn erroneous, non-linear representations of this Cartesian space. For example, given two different inputs, (x 1, y 1) and (x 1, y 2), with y 1 = y 2, the decoder may render those two objects at different horizontal positions in the image. While having a correct Cartesian coordinate representation is not strictly necessary to allow physical parameters of the physics engine to be learned from video, it is critical to ensure correct future predictions. This is because the relationship between position vector and pixel space position must be fixed: if the position vector changes by (∆x, ∆y), the object's position in the output image must change by (∆x, ∆y). This is the key concept that allows us to improve on , in order to learn an encoder, decoder and physics engine without state labels. In order to impose a correct latent-coordinate to pixel-coordinate correspondence, we use spatial transformers (ST) with inverse parameters as the decoder's writing attention mechanism. We want transformer parameters ω to be such that a decoder input of p n t = [x, y] n t, places the center of the writing attention window at (x, y) in the image, or that a decoder input of p n t = θ n t rotates the attention window by θ. In the original ST formulation , the matrix ω represents the affine transformation applied to the output image to obtain the source image. This means that the elements of ω in Eq. 1 of do not directly represent translation, scale or angle of the writing attention window. To achieve this representation, we use a ST with inverse transformation parameters. For a general affine transformation with translation (x, y), angle θ and scale s, we want to modify the source image coordinates according to: 1 Though any other architecture capable of effectively extracting object locations from images would work. This transformation can be obtained with a ST by inverting: Therefore, to obtain a decoder with coordinate-consistent outputs, we simply use a ST with parameters ω as given in Each object is represented by a learnable content c n ∈ H×H×C and mask m n ∈ R H×H×1 tensor, n = 1..N. Additionally, we learn content c bkg ∈ H×H×C and mask m bkg ∈ R H×H×1, that do not undergo spatial transformation. One may think of the content as an RGB image containing the texture of an object and the mask as a grayscale image containing the shape and z-order of the object. In order to produce an output image, the content and mask are transformed according to [ĉ The decoder architecture is shown in Fig. 1, bottom-right. The combined use of ST's and masks provides a natural way to model depth ordering, allowing us to capture occlusions between objects. Auxiliary autoencoder loss Using a constrained decoder ensures the encoder and decoder produces objects in consistent locations. However, it is hard to learn the full model from future frame prediction alone, since the encoder's training signal comes exclusively from the physics engine. To alleviate this and quickly build a good encoder/decoder representation, we add a static per-frame autoencoder loss. Training During training we use L input frames and predict the next T pred frames. Defining the frames produced by the decoder via the physics engine asĨ pred t and the frames produced by the decoder using the output of the encoder directly asĨ ae t, the total loss is: where α is a hyper-parameter. We use mean-squared error loss throughout. During testing we predict an additional T ext frames in order to evaluate long term prediction beyond the length seen for training. Figure 3: Frame prediction accuracy (SSI, higher is better) for the balls datasets. Left of the green dashed line corresponds to the training range, T pred, right corresponds to extrapolation, T ext. We outperform Interaction Networks (IN) , DDPAE and VideoLSTM in extrapolation due to incorporating explicit physics. Dataset 2-balls spring 2-digits spring 3-balls gravity 3-balls gravity Setup To explore learning physical parameters and evaluate long-term prediction we train our model on scenes with 5 different settings: two colored balls bouncing off the image edges; two colored balls connected by a spring; three colored balls with gravitational pull -all on a black ; and to test greater visual complexity, we also use 2 MNSIT digits connected by a spring, on a CIFAR . We train using values of (L, T pred, T ext) set to,,, and, respectively. For the spring systems the physical parameters to be learned are the spring constant k and equilibrium distance l, and for the gravitational system it is the gravity constant g or mass of the objects m (when learning gravity the mass if fixed, and vice versa). In all cases we use objects with mass m = 1. We provide the exact equations of motion used in these systems and other training details in Appendices A and B, respectively. All datasets consist of 5000 sequences for training, 500 for validation, and 500 for testing. We use a learnable ST scale parameter initialized at s = 2 in the balls datasets and s = 1 in the digits dataset. In these datasets we set θ = 0. Baselines We compare our model to 3 strong baselines: DDPAE 2, which is a generative model that uses an inverse-graphics model with black-box dynamics; VideoLSTM , which uses black-box encoding, decoding and dynamics; Interaction Network + Inverse-Graphics, which uses the same encoder and decoder as our Physics-as-Inverse-Graphics model, but where the dynamics module is an Interaction Network . The latter model allows us to compare explicit physics with relational dynamics networks, in terms of their ability to correctly capture object interactions 3. Results Table 1 shows that our model finds physical parameters close to the ground-truth values used to generate the datasets, and Figure 4 shows the contents and masks learned by the decoder. This highlights the fact that the proposed model can successfully perform unsupervised system identification from pixels. Future frame predictions for two of the systems are shown in Figure 2, and per-step Structural Similarity Index (SSI) 4 of the models on the prediction and extrapolation range are shown in Figure 3. While all models obtain low error in the prediction range (left of the green dashed line), our model is significantly better in the extrapolation range. Even many steps into the future, our model's predictions are still highly accurate, unlike those of other black-box models (Figure 2). This shows the value of using an explicit physics model in systems where the dynamics are non-linear yet well defined. Further rollouts are shown in Appendix C, and we encourage the reader to watch the videos for all the datasets at https://sites.google.com/ view/physicsasinversegraphics. This difference in performance is explained in part by the fact that in some of these systems the harder-to-predict parts of the dynamics do not appear during training. For example, in the gravitational system, whiplash from objects coming in close contact is seldom present in the first K + T pred steps given in the training set, but it happens frequently in the T ext extrapolation steps evaluated during testing. We do not consider this to be a failure of black-box model, but rather a consequence of the generality vs specificity tradeoff: a model without a sufficiently strong inductive bias on the dynamics is simply not able to correctly infer close distance dynamics from long distance dynamics. Table 2: Test loss under different training conditions. Separate gradients: Train encoder/decoder on L rec, and velocity estimator and physics engine on L pred. Black-box decoder, joint: Joint training using a standard MLP network as the decoder. Only joint training using our coordinate-consistent decoder succeeds. Ablation studies Since the encoder and decoder must discover the objects present in the image and the corresponding locations, one might assume that the velocity estimator and physics engine could be learned using only the prediction loss, and encoder/decoder using only the static autoencoder loss, i.e., without joint training. In Table 2 we compare the performance of four variants on the 3-ball gravity dataset: joint training using only the prediction loss; joint training using the prediction and autoencoder losses; training the encoder/decoder on the autoencoder loss and the velocity estimator and physics engine on the prediction loss; and joint training, but using an MLP black-box decoder. We can see that only joint prediction and autoencoder loss obtain satisfactory performance, and that the use of the proposed coordinate-consistent decoder is critical. The prediction loss is essential in order for the model to learn encoders/decoders whose content and masks can be correctly used by the physics engine. This can be understood by considering how object interaction influences the decoder. In the gravitational system, the forces between objects depend on their distances, so if the objects swap locations, the forces must be the same. If the content/mask learned for each object are centered differently relative to its template center, rendering the objects at positions [x, y] and [w, z], or [w, z] and [x, y] will produce different distances between these two objects in image space. This violates the permutation invariance property of the system. Learning the encoder/decoder along with the velocity estimator and physics engine on the prediction loss allows the encoder and decoder to learn locations and contents/masks that satisfy the characteristics of the system and allows the physics to be learned correctly. In Appendix D we perform further ablations on the decoder architecture and its ability to correctly render objects in regions of the image not seen during training. in terms of learning sample efficiency (left). Explicit physics allows reasoning for zero-shot adaptation to domain-shift in gravity (center) and goal-driven control to balance the pendulum in any position (right). DDPG (VAE) corresponds to a DDPG agent trained on the latent space of an autoencoder (trained with 320k images) after 80k steps. DDPG (proprio) corresponds to an agent trained from proprioception after 30k steps. Bottom: The first 3 rows show a zero-shot counterfactual episode with a gravity multiplier of 1.4 for an oracle, our model and planet, with vertical as the target position (as trained). The last row shows an episode using a goal image to infer the non-vertical goal state. Tasks One of the main applications of our method is to identify the (actuated) dynamical parameters and states of a physical system from video, which enables vision-based planning and control. Here we apply it to the pendulum from OpenAI Gym -one typically solved from proprioceptive state, not pixels. For training we collect 5000 sequences of 14 frames with random initialization (θ 0 ∼ Unif(−6, 6)) and actions (u t ∼ Unif(−2, 2)). The physical parameters to learn are gravity g = 10.0 and actuation coefficient a = 1.0. We use K = 4 and T pred = 10. We use the trained MPC model as follows. At every step, the previous 4 frames are passed to the encoder and velocity nets to estimate [θ t,θ t]. This is passed to the physics engine with learned parameters g and a. We perform 100-step model-predictive control using the cross entropy method , exactly as described in , setting vertical position and zero velocity as the goal. Baselines We compare our model to an oracle model, which has the true physical parameters and access to the true pendulum position and velocity (not vision-based), as well as a concurrent state-of-the art model-based RL method (PlaNet ), and a model-free 5 deep deterministic policy gradient (DDPG) agent. To provide an equivalent comparison to our model, we train PlaNet on random episodes. Results In terms of system identification, our model recovers the correct gravity (g = 9.95) and force coefficient (a = 0.99) values from vision alone, which is a prerequisite to perform correct planning and control. Figure 5 (top-left) highlights the data efficiency of our method, which is comparable to PlaNet, while being dramatically faster than DDPG from pixels. Importantly, the interpretability of the explicit physics in our model provides some unique capabilities. We can perform simple counter-factual physical reasoning such as' How should I adapt my control policy if gravity was increased?', which enables zero-shot adaptation to new environmental parameters. Figure 5 (top-middle) shows that our model can exploit such reasoning to succeed immediately over a wide range of gravities with no re-training. Similarly, while the typical inverted pendulum goal is to balance the pendulum upright, interpretable physics means that this is only one point in a space of potential goals. Figure 5 (top-right) evaluates the goal-paramaterized control enabled by our model. Any feasible target angle specified can be directly reached by the controller. There is extrapolative generalisation across the space of goals even though only one goal (vertical) was seen during training. Importantly, these last two capabilities are provided automatically by our model due to its disentangled interpretable representation, but cannot be achieved without further adaptive learning by alternatives that are reward-based or rely on implicit physics . Alhough the approach presented here shows promising in terms of physical parameter estimation, long-term video prediction and MPC, a number of limitations need to be overcome for real-world application. Templates as object representation Though the assumption that every scene in a dataset is a combination of learnable templates is a common one in the literature (c.f. for an extensive study on this), this is insufficient to model real-world scenes. For example, applying physics-as-inverse-graphics to the Physics101 dataset would require representing objects using a latent appearance representation that could be used by the decoder . This would introduce new modelling challenges, requiring object tracking to keep correct object identity associations . In this work we simplify this problem by assuming that objects are visually distinct throughout the dataset, though this does not detract from the essential contributions of the paper. Rigid sequence to sequence architecture In this work we used a sequence-to-sequence architecture, with a fixed number of input steps. This architectural choice (inspired by), prevents the model from updating its state beliefs if given additional input frames later in the sequence. Formulating the current model in a probabilistic manner that would allow for state/parameter filtering and smoothing at inference time is a promising direction of future work. Static assumption Many scenes of interest do not follow the assumption that the only moving objects in the scene are the objects of interest (even though this assumption is widely used). Adapting our model to varying scene s would require additional components to discern which parts of the scene follow the dynamics assumed by the physics engine, in order to correctly perform object discovery. This is a challenging problem, but we believe it would greatly increase the range of applications of the ideas presented here. Physics-as-inverse graphics provides a valuable mechanism to integrate inductive bias about physical data generating processes into learning. This allows unsupervised object tracking and system identification, in addition to sample efficient, generalisable and flexible control. However, incorporating this structure into lightly supervised deep learning models has proven challenging to date. We introduced a model that accomplishes this, relying on a coordinate-consistent decoder that enables image reconstruction from physics. We have shown that our model is able to perform accurate long term prediction and that it can be used to learn the dynamics of an actuated system, allowing us to perform vision-based model-predictive control. In this section we describe the equations of motion used for each system. 2-balls and 2-digits spring The force applied on object i by object j follows Hooke's law: Each step corresponds to an interval ∆t = 0.3. 3-balls gravity The force applied on object i by object j follows Newton's law of gravity: where the masses are set to 1. Each step corresponds to an interval ∆t = 0.5. Pendulum The pendulum follows the equations used by the OpenAI Gym environment: where u is the action. Each step corresponds to an interval ∆t = 0.05. In the physics engine used by the model we introduce an extra actuation coefficient a to be learned along with g: For all datasets we use with an initial learning rate of 3 × 10 −4. For the balls and digits datasets we train for 500 epochs with α = 2, and divide the learning rate by 5 after 375 epochs. For the pendulum data we train for 1000 epochs using α = 3, but divide the learning rate by 5 after 500 epochs. The image sizes are 32 × 32 for the 2-balls bouncing and spring, 36 × 36 for the 3-balls gravity, 64 × 64 for the 2-digits spring, and 64 × 64 grayscale for the pendulum. The content and mask variables are the output of a neural network with a constant array of 1s as input and 1 hidden layer with 200 units and tanh activation. We found this easier to train rather than having the contents and masks as trainable variables themselves. One limitation of standard fully-connected or deconvolutional decoders is the inability to decode states corresponding to object poses or locations not seen during training. For example, if in the training set no objects appear in the bottom half of the image, a fully-connected decoder will simply learn to output zeros in that region. If in the test set objects move into the bottom half of the image, the decoder lacks the inductive bias necessary to correctly extrapolate in image space. To test this hypothesis, we replaced our model's decoder with a Deconv and Spatial Broadcast (b) decoder, and compared them in a spatial extrapolation experiment. In this experiments, objects never enter the bottom half of the image in the input and prediction range, though in the extrapolation range in the test set objects move to this region of the scene. In the rollouts shown in Fig. D, Broadcast performs better than Deconv, but they both fail to maintain object integrity when the balls move to the bottom half of the image in the extrapolation steps, validating our hypothesis that a black-box decoder has insufficient extrapolation ability. In contrast, our rendering decoder is be able to correctly decode states not seen during training. In the limit that our renderer corresponds to a full-blown graphics-engine, any pose, location, color, etc. not seen during training can still be rendered correctly. This property gives models using rendering decoders, such as ours and , an important advantage in terms of dataefficiency. We note, however, that in general this advantage does not apply to correctly inferring the states from images whose objects are located in regions not seen during training. This is because the encoders used are typically composed simply of convolutional and fully-connected layers, with limited de-rendering inductive biases. Figure 6: Comparison between graphics decoder and two black-box decoders, trained on data where objects only appear in the top half of the scene. Only the graphics decoder is able to correctly render the objects in the bottom half of the scene at test time. Broadcast: spatial broadcast decoder (b); Deconv: standard deconvolutional network. The model proposed assumes we know the number of objects present in the scene. Here we briefly explore how to the model behaves when we use an incorrect number of slots N. We use the gravitational system, since interaction forces between objects are easy to generalize for any N. Fig. 7, left, shows that when using only 2 object slots, two of the objects are found, since the model does not have capacity to find more. Fig. 7, right, shows that when using more slots than the number of objects in the scene, all objects are discovered, and extra slots are left empty. However, in both cases we found predictive performance to be subpar, since in one case there are objects missing to correctly infer interactions, and in the other there are interactions between object slots and empty slots, confusing the dynamics. Results for incorrect number of object slots in the physics engien for the 3-body gravitational system Left: Contents and masks learned for 2 object slots. Right: Contents and objects learned for 4 object slots.

represent the short summary for finding the corresponding scientific article (long)

We propose a model that is able to perform physical parameter estimation of systems from video, where the differential equations governing the scene dynamics are known, but labeled states or objects are not available.

We present an agent that uses a beta-vae to extract visual features and an attention mechanism to ignore irrelevant features from visual observations to enable robust transfer between visual domains.