Question: Represent the scientific article to find a short TLDR

Question: Evolutionary Strategies (ES) are a popular family of black-box zeroth-order optimization algorithms which rely on search distributions to efficiently optimize a large variety of objective functions. This paper investigates the potential benefits of using highly flexible search distributions in ES algorithms, in contrast to standard ones (typically Gaussians). We model such distributions with Generative Neural Networks (GNNs) and introduce a new ES algorithm that leverages their expressiveness to accelerate the stochastic search. Because it acts as a plug-in, our approach allows to augment virtually any standard ES algorithm with flexible search distributions. We demonstrate the empirical advantages of this method on a diversity of objective functions. We are interested in the global minimization of a black-box objective function, only accessible through a zeroth-order oracle. In many instances of this problem the objective is expensive to evaluate, which excludes brute force methods as a reasonable mean of optimization. Also, as the objective is potentially non-convex and multi-modal, its global optimization cannot be done greedily but requires a careful balance between exploitation and exploration of the optimization landscape (the surface defined by the objective). The family of algorithms used to tackle such a problem is usually dictated by the cost of one evaluation of the objective function (or equivalently, by the maximum number of function evaluations that are reasonable to make) and by a precision requirement. For instance, Bayesian Optimization targets problems of very high evaluation cost, where the global minimum must be approximately discovered after a few hundreds of function evaluations. When aiming for a higher precision and hence having a larger budget (e.g. thousands of function evaluations), a popular algorithm class is the one of Evolutionary Strategies (ES) , a family of heuristic search procedures. ES algorithms rely on a search distribution, which role is to propose queries of potentially small value of the objective function. This search distribution is almost always chosen to be a multivariate Gaussian. It is namely the case of the Covariance Matrix Adaptation Evolution Strategies (CMA-ES) , a state-of-the-art ES algorithm made popular in the machine learning community by its good on hyper-parameter tuning . It is also the case for Natural Evolution Strategies (NES) algorithms, which were recently used for direct policy search in Reinforcement Learning (RL) and shown to compete with state-of-the-art MDP-based RL techniques . Occasionally, other distributions have been used; e.g. fat-tails distributions like the Cauchy were shown to outperform the Gaussian for highly multi-modal objectives . We argue in this paper that in ES algorithms, the choice of a standard parametric search distribution (Gaussian, Cauchy, ..) constitutes a potentially harmful implicit constraint for the stochastic search of a global minimum. To overcome the limitations of classical parametric search distributions, we propose using flexible distributions generated by bijective Generative Neural Networks (GNNs), with computable and differentiable log-probabilities. We discuss why common existing optimization methods in ES algorithms cannot be directly used to train such models and design a tailored algorithm that efficiently train GNNs for an ES objective. We show how this new algorithm can readily incorporate existing ES algorithms that operates on simple search distributions, Algorithm 1: Generic ES procedure input: zeroth-order oracle on f, distribution π 0, population size λ repeat (Sampling) Sample x 1,..., x λ i.i.d ∼ π t (Evaluation) Evaluate f (x 1),..., f (x n). (Update) Update π t to produce x of potentially smaller objective values. until convergence; like the Gaussian. On a variety of objective functions, we show that this extension can significantly accelerate ES algorithms. We formally introduce the problem and provide on Evolutionary Strategies in Section 2. We discuss the role of GNNs in generating flexible search distributions in Section 3. We explain why usual algorithms fail to train GNNs for an ES objective and introduce a new algorithm in Section 4. Finally we report experimental in Section 5. In what follows, the real-valued objective function f is defined over a compact X and π will generically denote a probability density function over X. We consider the global optimization of f: The generic procedure followed by ES algorithms is presented in Algorithm 1. To make the update step tractable, the search distribution is tied to a family of distributions and parametrized by a realvalued parameter vector θ (e.g. the mean and covariance matrix of a Gaussian), and is referred to as π θ. This update step constitutes the main difference between ES algorithms. One principled way to perform that update is to minimize the expected objective value over samples x drawn from π θ. Indeed, when the search distribution is parametric and tied to a parameter θ, this objective can be differentiated with respect to θ thanks to the log-trick: This quantity can be approximated from samples -it is known as the score-function or REINFORCE estimator, and provides a direction of update for θ. Unfortunately, naively following a stochastic version of the gradient -a procedure called Plain Gradient Evolutionary Strategies (PGES) -is known to be highly ineffective. PGES main limitation resides in its instability when the search distribution is concentrating, making it unable to precisely locate any local minimum. To improve over the PGES algorithm the authors of proposed to descend J(θ) along its natural gradient . More precisely, they introduce a trust-region optimization scheme to limit the instability of PGES, and minimize a linear approximation of J(θ) under a Kullback-Leibler (KL) divergence constraint: To avoid solving analytically the trust region problem, shows that its solution can be approximated by: is the Fischer Information Matrix (FIM) of π θ. The parameter θ is therefore not updated along the negative gradient of J but rather along F −1 θ ∇ θ J(θ), a quantity known as the natural gradient. The FIM F θ is known analytically when π θ is a multivariate Gaussian and the ing algorithm, Exponential Natural Evolutionary Strategies (xNES) has been shown to reach state-of-the-art performances on a large ES benchmark. Figure 2: Example of an undesirable behavior of a Gaussian search distribution. The dashed lines represent density level lines of the search distribution. Because the latter cannot have a curved profile, it is forced to drastically reduce its entropy until it reaches the straight part of the valley. CMA-ES Naturally, there exist other strategies to update the search distribution π θ. For instance, CMA-ES relies on a variety of heuristic mechanisms like covariance matrix adaptation and evolution paths, but is only defined when π θ is a multivariate Gaussian. Explaining such mechanisms would be out of the scope of this paper, but the interested reader is referred to the work of for a detailed tutorial on CMA-ES. ES implicitly balance the need for exploration and exploitation of the optimization landscape. The exploitation phase consists in updating the search distribution, and exploration happens when samples are drawn from the search distribution's tails. The key role of the search distribution is therefore to produce a support adapted to the landscape's structure, so that new points are likely to improve over previous samples. We argue here that the choice of a given parametric distribution (the multivariate Gaussian distribution being overwhelmingly represented in state-of-the-art ES algorithms) constitutes a potentially harmful implicit constraint for the stochastic search of a global minimum. For instance, a Gaussian distribution is not adapted to navigate a curved valley because of its inability to continuously curve its density. This lack of flexibility will lead it to drastically reduce its entropy, until the curved valley looks locally straight. At this point, the ES algorithm resembles a hill-climber and barely takes advantage of the exploration abilities of the search distribution. An illustration of this phenomenon is presented in Figure 2 on the Rosenbrock function. Another limitation of classical search distribution is their inability to follow multiple hypothesis, that is to explore at the same time different local minima. Even if mixture models can show such flexibility, hyper-parameters like the number of mixtures have optimal values that are impossible to guess a priori. We want to introduce flexible search distributions to overcome these limitations. Such distributions should, despite their expressiveness, be easily trainable. We should also be concerned when designing them with their role in the exploration/exploitation trade off: a search distribution with too much capacity could over-fit some seemingly good samples, leading to premature convergence. To sum-up, we want to design search-distributions that are: • more flexible than classical distributions • yet easily trainable • while keeping control over the exploration / exploitation trade-off In the following section, we carefully investigate the class of Generative Neural Networks (GNNs) to find a parametric class of distributions satisfying such properties. Generative Neural Networks have been studied in the context of density estimation and shown to be able to model complex and highly multimodal distributions . We propose here to leverage their expressiveness for ES, and train them in a principled way thanks to the ES objective: As discussed in Section 2, optimizing J(π) with gradient-based methods is possible through the score-function estimator, which requires to be able to compute and efficiently differentiate the logprobabilities of π. The core idea behind a GNN is to map a latent variable z ∈ Z drawn from a known distribution ν ω to an output variable x = g η (z) where g η is the forward-pass of a neural network. The parameter η represents the weights of this neural network while ω describe the degrees of freedom of the latent space distribution ν ω. We denote θ = (ω, η) and π θ (x) the density of the output variable x. For general neural network architectures, it is impossible to compute π θ (x) for samples x drawn from the GNN. This is namely why their are often trained with adversarial methods for sample generation purposes, bypassing the need of computing densities, but at the expense of a good density estimation (mode-dropping). An alternative to adversarial methods was proposed with variational auto-encoders however at the cost of learning two neural networks (an encoder and a decoder). A less computationally expensive method consists in restricting the possible architectures to build bijective GNNs, also known as Normalizing Flows (NF) , which allows the exact computation of the distribution's density. Indeed, if g η is a bijection from Z to X with inverse h η g −1 η, the change of variable formula provides a way to compute π θ (x): To have a tractable density one therefore needs to ensure that the determinant of the Jacobian |∂h η (x)/∂x| is easily computable. Several models satisfying these two properties (i.e bijectivity and computable Jacobian) have been proposed for density estimation (; ; 2016), and proved their expressiveness despite their relatively simple structure. NFs therefore answer two of our needs when building our new search distribution: flexibility and easiness to train. In this work, we will focus on one NF model: the Non-Linear Independent Component Estimation (NICE) model, for its numerical stability and volume preserving properties. The authors of NICE proposed to build complex yet invertible transformations through the use of additive coupling layers. An additive coupling layer leaves half of its input unchanged, and adds a non-linear transformation of the first half to the second half. More formally, by noting v = [v 1, v 2] the output of a coupling layer and u = [u 1, u 2] its input, one has: where t is an arbitrarily complex transformation -modelled by a Multi-Layer Perceptron (MLP) with learnable weights and biases. This transformation has unit Jacobian determinant and is easily invertible: and only requires a feed-forward pass on the MLP t. The choice of the decomposition u = [u 1, u 2] can be arbitrary, and is performed by applying a binary filter to the input. By stacking additive coupling layers, one can create complex distributions, and the inversion of the ing mapping is independent of the complexity of the neural networks t. The density of the ing distribution is readily computable thanks to the inverse transform theorem. The transformation induced by NICE is volume preserving (it has a unitary Jacobian determinant). This is quite desirable in a ES context, as the role of concentrating the distribution on a minimum can be left to the latent space distribution ν ω. The role of the additive coupling layers is therefore only to introduce non-linearities in the inverse transform h η so that the distribution is better adapted to the optimization landscape. The fact that this fit is volume-preserving (every subset of the latent space has an image in the data space with the same probability mass) encourages the search distribution to align its tails with regions of small value of the optimization landscape, which is likely to improve the quality of future exploration steps. The NICE model therefore fits perfectly our needs for a flexible search distribution that is easy to train, and that provides enough control on the exploration / exploitation trade-off. Other bijective GNN models like the Real-NVP introduce non-volume preserving transformations, which cannot provide such a control. In practice, we observed that using such transformations for ES led to early concentration and premature convergence. We are now equipped with enough tools to use GNNs for ES: an adapted model (NICE) for our search distribution π θ, and an objective to train it with: Here, θ describes jointly the free parameters of the latent distribution ν ω and η, the weights and biases of the MLPs forming the additive coupling layers. We start this section by explaining why existing training strategies based on the objective are not sufficient to truly leverage the flexibility of GNNs for ES, before introducing a new algorithm tailored for this task. We found that the PGES algorithm (naive stochastic gradient descent of with the score-function estimator) applied to the NICE distribution suffers from the same limitations as when applied to the Gaussian; it is inable to precisely locate any local minimum. As for the Gaussian, training the NICE distribution for ES requires employing more sophisticated algorithms -such as NES. However, using the natural gradient for the GNNs distributions is not trivial. First the Fischer Information Matrix F θ is not known analytically and must be estimated via Monte-Carlo sampling, thereby introducing approximation errors. Also, we found that the approximations justifying to follow the descent direction provided by the natural gradient are not adapted to the NICE distribution. Indeed, the assumption behind the NES update is that the loss J(θ) can be (locally) well approximated by the quadratic objective: where γ is a given non-negative Lagrange multiplier. For NICE, given the highly non-linear nature of π θ this approximation is bound to fail even close to the current parameter θ and will lead to spurious updates. A classical technique to avoid such updates is to artificially increase the curvature of the quadratic term, and is known as damping. Practically, this implies using F θ + βI instead of F θ as the local curvature metric, with β a non-negative damping parameter. We found that to ensure continuous decrease of J(θ), and because of its highly non-linear nature when using the GNNs, the damping parameter β has to be set to such high values that the modifications of the search distribution are too small to quickly make progress and by no means reaches state-of-the-art performances. We observed that even if the training of the additive coupling layers is performed correctly (i.e the distribution has the correct shape), high damping of the latent space parameters prevents the distribution from quickly concentrating when a minimum is found. It is unclear how the damping parameter should be adapted to avoid spurious update, while still allowing the distribution to make large step in the latent space and ensure fast concentration when needed. In the following, we present an alternated minimization scheme to bypass the issues raised by natural gradient training for GNN distributions in a ES context. So far, we used the parameter θ to describe both ω and η (respectively, the free parameters of the latent space distribution ν ω and the degrees of freedom of the non-linear mapping g η), and the optimization over all these parameters was performed jointly. Separating the roles of ω and η, the initial objective can be rewritten as follows: Therefore, the initial objective can be rewritten as the expected value of samples drawn from the latent distribution, under the objective f • g η -that is, the representation of the objective function f in the latent space. If ν ω is a standard distribution (i.e efficiently trainable with the natural gradient) and f • g η is a well structured function (i.e one for which ν ω is an efficient search distribution), then the single optimization of ω by classical methods (such as the natural gradient) should avoid the limitations discussed in 2.2. This new representation motivates the design of a new training algorithm that optimizes the parameters ω and η separately. Alternating Minimization In the following, we will replace the notation π θ with π ω,η to refer to the NICE distribution with parameter θ = (ω, η). We want to optimize ω and η in an alternate fashion, which means performing the following updates at every step of the ES procedure: This means that at iteration t, samples are drawn from π ωt,ηt and serve to first optimize the latent space distribution parameters ω, and then the additive coupling layers parameters η. For the following iteration, the population is sampled under π ωt+1,ηt+1. The update (11a) of the latent space parameters is naturally derived from the new representation of the initial objective. Indeed, ω can be updated via natural gradient ascent of J(ω, η t) -that is with keeping η = η t fixed. Practically, this therefore reduces to applying a NES algorithm to the latent distribution ν ω on the modified objective function f • g ηt. Once the latent space parameters updated, the coupling layers parameters should be optimized with respect to: At this stage, the only available samples are drawn under π ωt,ηt. To estimate, based on these samples, expectations under π ωt+1,ηt one must use importance propensity scores: The straightforward minimization of this off-line objective is known to lead to degeneracies (, Section 4), and must therefore be regularized. For our application, it is also desirable to make sure that the update η does not undo the progress made in the latent space -in other words, we want to regularize the change in f • g η. To that extent, we adopt a technique proposed in and minimize a modification on the initial objective with clipped propensity weights: clip ε (x) clips the value of x between 1 − and 1 +. The parameter ε is an hyper-parameter that controls the change in distribution, and the program can be efficiently solved via a gradient descent type algorithm, such as Adam . To sum up, we propose optimizing the latent distribution and the coupling layers separately. The latent space is optimized by natural gradient descent, and the coupling layers via an off-policy objective with clipped propensity weights. We call this algorithm GNN-ES for Generative Neural Networks Evolutionary Strategies. Latent space optimization It turns out the GNN-ES can be readily modified to incorporate virtually any existing ES algorithms that operates on the simple distribution ν ω. For instance, if ν ω is set to be a multivariate Gaussian with learnable mean and covariance matrix, the latent space optimization (11a) can be performed by either xNES or CMA-ES. This holds for any standard distribution ν ω and any ES algorithm operating on that distribution. This remark allows us to place GNN-ES in a more general framework and to understand it as a way to improve existing ES algorithm, by providing a principled way to learn complex, non-linear transformations on top of rather standard search distributions (like the Gaussian). In what follows, we will use the GNN prefix in front of existing ES algorithm to describe its augmented version with our algorithm, working as a plug-in. Pseudo-code for this general algorithm can be found in Appendix B. Using historic data ES algorithms typically use small populations of samples to estimate expectations. Such small sample sizes don't allow for enough data exposure for the GNN to build a meaningful transformation g η. To circumvent this problem, we augment the off-line program with samples for past generations thanks to the fused importance sampling estimator . This technique is classical in similar settings like MDP-based reinforcement learning and counterfactual reasoning and proves to be essential for our problem. Formally, for a given horizon T that controls how far we look in the past, this amounts to storing the samples x drawn from π θ t−T +1,..., π θt (as well as their respective scores) in a buffer H T. The objective can then be rewritten as: This technique allows to increase the data exposure of the GNN by using past samples (and therefore does not require additional function evaluations) and to reduce the variance of the off-line estimator of the original expectation . To control the change in distribution, the fused propensity weights can then be clipped in a similar fashion than in the program. Mode preserving properties To achieve improved exploration, the search distribution should align its tails with the level sets of the objective function. This is not guaranteed when performing the update step since the GNN's update could simply move the mean of the search distribution without shaping the tails. One way to encourage the GNN's capacity to be allocated to the tails is to impose a mode-preserving property. If µ denotes the location of a mode of the latent distribution, then the mode of the distribution π θ generated by the NICE model is located in g η (µ) (see Appendix A for the proof). It is therefore easy to build a map f η based on the initial g η that is mode-preserving: where µ t denotes the mode of the latent distribution ν ω at iteration t. Defined as such, f η preserves the mode of the previous search distribution (since f ηt+1 (µ) = f ηt (µ)), is trivially still a bijection and remains volume preserving. Using the push-forward map f η instead of g η, we explicitly push the flexibility brought by the GNN to impact only the tails of the search distribution. As detailed in an ablation study presented in Appendix F, this additional tool turns out to be essential in order to use GNNs for ES. In all that follows, we build the NICE model with three coupling layers. Each coupling layer's nonlinear mapping t is built with a one hidden layer MLP, with 128 neurons and leaky ReLU activation functions. This architecture is kept constant in all our experiments. We present here two-dimensional visualizations of the behavior of a GNN distribution trained with GNN-xNES -the latent distribution is therefore Gaussian. Figure 3a Figure 3b displays the density level lines of the latent distribution, as well as the learned representation of the objective in the latent space. The search distribution is able to have curved density isolines, enabling better exploration. In the latent space, the global minimum can be reached without navigating a curved valley. Figures 4a and 4b provide similar visualizations on the Rastrigin function, a highly multimodal but symmetric objective. The GNN lowers the barriers between local minima, making it easier to escape a local minimum to the global minimum. Experimental set-up We present experiments on both unimodal and multimodal objectives for xNES and GNN-xNES. We use the official implementation of xNES 1 with default hyper-parameters (such as the population size λ), both as a baseline and as an inner optimization method for GNNxNES. All experiments are run on the COmparing Continous Optimizers (COCO) platform, a popular framework for comparing black-box optimization algorithms. It namely allows to benchmark different algorithms on translated and rotated versions of the same objectives, in order to evaluate multiple configurations with different global minimum positions. We compare xNES and GNN-xNES on functions from the 2018 Black-Box Optimization Benchmark (BBOB) suite. When comparing these two algorithms, we impose that their initial search distributions are close in order to ensure fair comparison. We insist on the fact that the xNES algorithm has the exact same configuration whether it is used by itself or as an inner-optimization algorithm for GNN-xNES. Further experimental details, including additional hyper-parameters value for GNN-xNES are provided in Appendix C. We run the different algorithms on two unimodal landscapes where we expect GNN search distributions to bring a significant improvement over the Gaussian -as discussed in 2.2. These objectives functions are the Rotated Rosenbrock function (a curved valley with high conditioning) and the Bent Cigar (an asymmetric and curved Cigar function). Extensive details on these objective functions can be found in the BBOB documentation . Results on additional unimodal functions can be found in Appendix E. Performance is measured through Empirical Cumulative Distribution Functions (ECDFs) of the runtime, also known as data profiles (Moré &). Such curves report the fraction of problems solved as a function of the number of objective evaluations. For a given precision ∆, a problem is said to be solved if the best function evaluation made so far is smaller than f (x *) + ∆. We create 200 problems, equally spaced on a log-scale from ∆ = 10 2 to ∆ = 10 −5 and, as in the COCO framework, aggregate them over 15 function instances. Results are presented in Figure 5 for the two benchmark functions and in dimensions d = 2, 5, 10. We now compare the performances of the different algorithms on a collection of three multimodal objectives: the Rastrigin function, the Griewank-Rosenbrock function and the Schwefel function. Extensive details about these objectives can be found in. When using ES algorithms to optimize multimodal functions, it is usual to augment them with restart strategies . When convergence is detected, the search distribution is re-initialized in order to search another part of the landscape, and often the population size is increased. This allows to fairly compared algorithms that converge fast to potentially bad local minima, and algorithms that converges slower to better minima. Their exist a large variety of restart strategies ; as the official implementation of xNES is not equipped with a default one, we trigger a restart whenever the algorithm makes no progress for more than 30 × d iterations. The standard deviation of the search distribution is set back to 1, and its mean sampled uniformly within the compact X of interest (defined by the COCO framework). At each restart, the population size of the algorithm is multiplied by 2, as in. This restart strategy is used for both xNES and GNN-xNES. We measure performance as the number of functions evaluations to find an objective value smaller than f (x *) + 10 −5 within a budget of d × 10 5 function evaluations, averaged over 15 function instances. When an algorithm is not able to discover the global minimum within the given budget, we use the maximum number of evaluations as its performance. For visualization purposes, this measure of performance is divided by d 2. Results are reported in Figure 6. On all objectives, and for all dimensions, GNN-xNES discovers (in average) the global minimum faster than xNES. Additional on others multimodal functions are presented in Appendix E. The goal of this section is to present additional comparison between xNES and GNN-xNES on RL-based objective functions -less synthetic than the previously considered BBOB functions. ES algorithms have recently been used for direct policy search in Reinforcement Learning (RL) and shown to reach performances comparable with state-of-the-art MDP-based techniques . Direct Policy Search ignores the MDP structure of the RL environment and rather considers it as a black-box. The search for the optimal policy is performed directly in parameter space to maximize the average reward per trajectory: where p x is the distribution of trajectories induced by the policy (the state-conditional distribution over actions) parametrized by x, and r the rewards generated by the environment. The objective can readily be approximated from samples by simply rolling out M trajectories, and optimized using ES. In our experiments 2, we set M = 10 and optimize deterministic linear policies (as in). In Figures 7a and 7b we report of the GNN-xNES algorithm compared to xNES, when run on the Mujoco locomotion tasks Swimmer and InvertedDoublePendulum, both from the OpenAI Gym . Performance is measured by the average reward per trajectory as a function of the number of evaluations of the objective f. Results are averaged over 5 random seeds (ruling the initialization of the environment and the initial distribution over the policy parameters x). In all three environments, GNN-xNES discovers behaviors of high rewards faster than xNES. In this work, we motivate the use of GNNs for improving Evolutionary Strategies by pinpointing the limitations of classical search distributions, commonly used by standard ES algorithms. We propose a new algorithm that leverages the high flexibility of distributions generated by bijective GNNs with an ES objective. We highlight that this algorithm can be seen as a plug-in extension to existing ES algorithms, and therefore can virtually incorporate any of them. Finally, we show its empirical advantages across a diversity of synthetic objective functions, as well as from objectives coming from Reinforcement Learning. Beyond the proposal of this algorithm, we believe that our work highlights the role of expressiveness in exploration for optimization tasks. This idea could be leverage in other settings where exploration is crucial, such a MDP-based policy search methods. An interesting line of future work could focus on optimizing GNN-based conditional distribution for RL tasks -an idea already developed in;. Other possible extensions to our work could focus on investigating first-order and mixed oracles, such as in; Faury et al. (2018 We prove here the fact that if µ denotes the location of the mode of the latent distribution ν ω, then g η (µ) is a mode for π ω,η. Indeed, under reasonable smoothness assumptions, one has that y is a mode for π ω,η if and only if: Since π ω,η (x) = ν ω (h η (x)), this is therefore equivalent to: In the NICE model, we have that ∂hη(x) ∂x = 1 for all x hence the matrix ∂hη(x) ∂x x=y is invertible and its kernel is reduced to the null vector. Therefore: and therefore µ = h η (y) by definition of µ (the only critical point of ν ω). Hence since h −1 η = g η, we have that y = g η (µ) which concludes the proof. We provide below the pseudo-code for the generic algorithm GNN-A-ES, where A is a generic ES algorithm operating on a parametric distribution ν ω. The additional hyper-parameters are the horizon T as well as the clipping constant ε. The function clip (x, lb, ub) clips the input x between a lower-bound lb and an upper-bound ub.: objective function f, distribution ν ω and its related ES algorithm A hyper-parameters: clipping constant ε, NICE model architecture, initial parameters ω 0, initial weights η 0, horizon T, population size λ (Initialization) Initialize NICE MLPs weights and biases with η 0. Let H be a circular buffer of length T × λ while not terminate do (Sampling) //One step ES-based optimization of the latent space Apply A to the latent distribution (GNN update) * //Many-steps gradient based optimization of the GNN η t+1 η x,f ∈H f · clip Algorithm 2 does not detail the mode-preserving addition for the sake of readability and clarity. We provide additional details on this procedure here. Let µ t be the mode of the latent distribution ν ωt. At the (Initialization) step, set α 0 = g η0 (µ 0) where g η (·) is the push-forward map on the NICE model described in Section 3.2. For all round t ≥ 1, let f η (z) = g η (z) − g η (µ t) + α t. The variable α t represent the push forward mapping of the latent distribution's mean under the current model. Every time the latent space is updated -the (ES update) step, let α t+1 = f ηt (µ t+1). Then, for the (GNN update), optimize the forward-map f η (z) = g η (z) − g η (µ t+1) + α t+1. After this update, we have f ηt+1 (µ t+1) = α t+1 = f ηt (µ t+1), which means that the mode of the search distribution (which is the image of the latent distribution mode) has not been impacted by the GNN update. C EXPERIMENTAL DETAILS Baselines We use xNES with its default (adapted) hyper-parameters (described in) for both its baselines versions and its inner optimization parts in GNN-xNES. The population size λ is one such hyper-parameters, and is therefore set to λ = 4 + 3 log(d). Also, as it is classically done in ES algorithms, we use a rank-based fitness shaping, designed to make the algorithm invariant with respect to order-preserving cost transformations. We use the same fitnessshaping function as in. Across all experiments, we use the same hyper-parameters for GNN-xNES without fine tuning for each tasks. We use three coupling layers, each with a single hidden layer MLP with 128 hidden neurons and Leaky ReLU activations. The MLPs are initialized via Glorot initialization, and the clipping constant is set to ε = 0.05. The history size T was determined experimentally, and set to T = 3 * (1 + log(d)). When restarts are used, this history size is divided by the numbers of restart so far (as the population size grows larger). Every synthetic objective we used in this work was taken from the BBOB2019 benchmark dataset. Their expression as well as additional details on the framework can be found in Hansen et al. (2010; 2016). At the beginning of each experiment, we set the Gaussian search distribution (for xNES) and the Gaussian latent distribution (for GNN-xNES) to a standard normal, with a mean uniformly sampled within the compact X of interest (defined by the COCO framework). C.3 RL ENVIRONMENTS Table 1 provides details on the RL environment used to compare GNN-xNES and xNES, like the dimensions of the state space S and action space A, the number d of the policy's degrees of freedom and the maximum number of steps m per trajectory. At the beginning of each experiment, we set the Gaussian search distribution (for xNES) and the Gaussian latent distribution (for GNN-xNES) to a standard normal with zero mean. In this particular case, where the function evaluations are noisy, we kept the default population size of the xNES algorithm. Algorithm mean(# restarts), d=2 mean(# restarts), d=5 mean(# restarts), d=10 xNES 2.3 2.7 3.4 GNN-xNES 1.3 2.5 2.9 Table 2: Mean number of restarts needed to discover the global minimum on the Rastrigin function. efficiently fit each optimization landscapes, without having to reduce its entropy like a multivariate normal would. We present here some additional on some unimodal and multimodal synthetic functions. Figure 9 present ECDFs curve obtained from the Attractive Sector function, a highly asymmetrical function around its global minimum. On such a function, GNN-xNES seems to accelerate xNES in small dimensions, however this speed-up disappears in higher dimensions. Figure 10 presents on the Rosenbrock function (without random rotations). Again, GNN-xNES accelerates the xNES algorithm. Figure 11 present on the multimodal functions Gallagher's Gaussian 101 Peaks and Gallagher's Gaussian 21 Peaks. Again, GNN-xNES discovers the global minimum faster (on average) than xNES. In our multimodal experiments, we used simulated restarts as a fair mean of comparing different algorithm (this is common practice in order to fairly compare algorithms that converge fast to potentially bad local minima to algorithms that converge slowly to the global minimum). If the empirical prove that GNN-xNES accelerate xNES in the discovery of the global minimum, it does not prove that GNN-xNES leverages the flexibility of the GNN to detect the global minimum when xNES misses it. In an attempt to prove that it is indeed the case, we report in Table 2 the number of restarts needed by both GNN-xNES and xNES to discover the global minimum on the Rastrigin function (averaged over the 15 randomly initialized run). For this instance, GNN-xNES consistently discovers the global minimum with less restarts than xNES. As detailed in Section 4, one can apply Algorithm 2 as a plug-in to any ES method. So far, we empirically evaluated the benefits of our approach by comparing xNES against its GNN extension (GNN-xNES). We present in Figure 12 additional evaluations obtained by comparing CMA-ES and its GNN extension (denoted GNN-CMA-ES) on the Rosenbrock function in dimension 2,5 and 10. CMA-ES is considered to be the state-of-the-art ES algorithm, and improving its performances is a non-trivial task. On the considered example GNN-CMA-ES improves CMA-ES, highlighting the empirical benefit of our approach for a large class of ES algorithm. One can however observe that the performance boost brought by the GNN extension is milder for GNN-CMA-ES then for GNNxNES. We suspect that this is due to the use of cumulation via an evolution path in CMA-ES 3, which basically introduces a momentum-like update when optimizing the latent distribution. While using an evolution path makes a lot of sense when optimizing a stationary objective, it can be quite harmful for non-stationary ones. We therefore believe that the cumulation step in CMA-ES (for the latent distribution) and the GNN optimization (making the objective optimized by CMA-ES in the latent space non-stationary) can lead to conflicting updates and might hinder the benefits brought by the GNN's additional flexibility. Designing a GNN update strategy complying with the use of evolution paths could therefore be a way of further improving GNN-CMA-ES, and is left for future work. We present here an ablation study for two additional tools that we introduced after the alternating optimization view: the mode preserving extension as well as the history augmentation. Figure 13 presents ECDFs curves on the Rosenbrock, Rotated Rosenbrock and Bent Cigar functions in 2D, for a version of GNN-xNES that doesn't use history but only the current population. Using history and therefore exposing the GNN to larger datasets improves the procedure. Figure 14 present similar on a version of GNN-xNES without the mode preserving property. Again, one can notice that ensuring that the GNN training is mode-preserving is crucial to improve experimental . Figure 14: ECDFs curves for xNES, GNN-xNES and GNN-xNES-nmp, which is not mode preserving. Ensuring that the training of the GNN doesn't impact the mode of the search distribution improves the optimization.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a new algorithm leveraging the expressiveness of Generative Neural Networks to improve Evolutionary Strategies algorithms.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: Represent the article for finding a single-sentence TLDR.

Question: We propose and evaluate new techniques for compressing and speeding up dense matrix multiplications as found in the fully connected and recurrent layers of neural networks for embedded large vocabulary continuous speech recognition (LVCSR). For compression, we introduce and study a trace norm regularization technique for training low rank factored versions of matrix multiplications. Compared to standard low rank training, we show that our method leads to good accuracy versus number of parameter trade-offs and can be used to speed up training of large models. For speedup, we enable faster inference on ARM processors through new open sourced kernels optimized for small batch sizes, ing in 3x to 7x speed ups over the widely used gemmlowp library. Beyond LVCSR, we expect our techniques and kernels to be more generally applicable to embedded neural networks with large fully connected or recurrent layers. For embedded applications of machine learning, we seek models that are as accurate as possible given constraints on size and on latency at inference time. For many neural networks, the parameters and computation are concentrated in two basic building blocks:1. Convolutions. These tend to dominate in, for example, image processing applications.2. Dense matrix multiplications (GEMMs) as found, for example, inside fully connected layers or recurrent layers such as GRU and LSTM. These are common in speech and natural language processing applications. These two building blocks are the natural targets for efforts to reduce parameters and speed up models for embedded applications. Much work on this topic already exists in the literature. For a brief overview, see Section 2.In this paper, we focus only on dense matrix multiplications and not on convolutions. Our two main contributions are:1. Trace norm regularization: We describe a trace norm regularization technique and an accompanying training methodology that enables the practical training of models with competitive accuracy versus number of parameter trade-offs. It automatically selects the rank and eliminates the need for any prior knowledge on suitable matrix rank. We explore the importance of optimizing for low batch sizes in on-device inference, and we introduce kernels 1 for ARM processors that vastly outperform publicly available kernels in the low batch size regime. These two topics are discussed in Sections 3 and 4, respectively. Although we conducted our experiments and report in the context of large vocabulary continuous speech recognition (LVCSR) on embedded devices, the ideas and techniques are broadly applicable to other deep learning networks. Work on compressing any neural network for which large GEMMs dominate the parameters or computation time could benefit from the insights presented in this paper. Our work is most closely related to that of, where low rank factored acoustic speech models are similarly trained by initializing weights from a truncated singular value decomposition (SVD) of pretrained weight matrices. This technique was also applied to speech recognition on mobile devices BID26. We build on this method by adding a variational form of trace norm regularization that was first proposed for collaborative prediction BID24 and also applied to recommender systems BID13 ). The use of this technique with gradient descent was recently justified theoretically BID6. argue that trace norm regularization could provide a sensible inductive bias for neural networks. To the best of our knowledge, we are the first to combine the training technique of with variational trace norm regularization. Low rank factorization of neural network weights in general has been the subject of many other works BID7 BID22 BID2 BID14. Some other approaches for dense matrix compression include sparsity BID15 BID19, hash-based parameter sharing BID3, and other parameter-sharing schemes such as circulant, Toeplitz, or more generally low-displacement-rank matrices BID23 BID17. BID14 explore splitting activations into independent groups. Doing so is akin to using block-diagonal matrices. The techniques for compressing convolutional models are different and beyond the scope of this paper. We refer the interested reader to, e.g., BID8; BID10 and references therein. Low rank factorization is a well studied and effective technique for compressing large matrices. In, low rank models are trained by first training a model with unfactored weight matrices (we refer to this as stage 1), and then initializing a model with factored weight matrices from the truncated SVD of the unfactored model (we refer to this as warmstarting a stage 2 model from a stage 1 model). The truncation is done by retaining only as many singular values as required to explain a specified percentage of the variance. If the weight matrices from stage 1 had only a few nonzero singular values, then the truncated SVD used for warmstarting stage 2 would yield a much better or even error-free approximation of the stage 1 matrix. This suggests applying a sparsity-inducing 1 penalty on the vector of singular values during stage 1 training. This is known as trace norm regularization in the literature. Unfortunately, there is no known way of directly computing the trace norm and its gradients that would be computationally feasible in the context of large deep learning models. Instead, we propose to combine the two-stage training method of with an indirect variational trace norm regularization technique BID24 BID6. We describe this technique in more detail in Section 3.1 and report experimental in Section 3.2. First we introduce some notation. Let us denote by || · || T the trace norm of a matrix, that is, the sum of the singular values of the matrix. The trace norm is also referred to as the nuclear norm or the Schatten 1-norm in the literature. Furthermore, let us denote by || · || F the Frobenius norm of a matrix, defined as DISPLAYFORM0 The Frobenius norm is identical to the Schatten 2-norm of a matrix, i.e. the 2 norm of the singular value vector of the matrix. The following lemma provides a variational characterization of the trace norm in terms of the Frobenius norm. Lemma 1 BID12; BID6 ). Let W be an m × n matrix and denote by σ its vector of singular values. Then DISPLAYFORM1 where the minimum is taken over all U: m×min(m, n) and V: min(m, n)×n such that W = U V. Furthermore, if W =Ũ ΣṼ * is a singular value decomposition of W, then equality holds in for the choice U =Ũ √ Σ and V = √ ΣṼ *.The procedure to take advantage of this characterization is as follows. First, for each large GEMM in the model, replace the m × n weight matrix W by the product W = U V where U: m × min(m, n) and V: min(m, n) × n. Second, replace the original loss function (W) by DISPLAYFORM2 where λ is a hyperparameter controlling the strength of the approximate trace norm regularization. Proposition 1 in BID6 guarantees that minimizing the modified loss equation FORMULA2 is equivalent to minimizing the actual trace norm regularized loss: DISPLAYFORM3 In Section 3.2.1 we show empirically that use of the modified loss is indeed highly effective at reducing the trace norm of the weight matrices. To summarize, we propose the following basic training scheme:• Stage 1:-For each large GEMM in the model, replace the m×n weight matrix W by the product W = U V where U: m × r, V: r × n, and r = min(m, n). -Replace the original loss function (W) by DISPLAYFORM4 where λ is a hyperparameter controlling the strength of the trace norm regularization.-Train the model to convergence. DISPLAYFORM5 -For the trained model from stage 1, recover W = U V by multiplying the two trained matrices U and V.-Train low rank models warmstarted from the truncated SVD of W. By varying the number of singular values retained, we can control the parameter versus accuracy trade-off. One modification to this is described in Section 3.2.3, where we show that it is actually not necessary to train the stage 1 model to convergence before switching to stage 2. By making the transition earlier, training time can be substantially reduced. We report here the of our experiments related to trace norm regularization. Our baseline model is a forward-only Deep Speech 2 model, and we train and evaluate on the widely used Wall Street Journal (WSJ) speech corpus. Except for a few minor modifications described in Appendix B, we follow closely the original paper describing this architecture BID1, and we refer the reader to that paper for details on the inputs, outputs, exact layers used, training methodology, and so on. For the purposes of this paper, suffice it to say that the parameters and computation are dominated by three GRU layers and a fully connected layer. It is these four layers that we compress through low-rank factorization. As described in Appendix B.2, in our factorization scheme, each Figure 1: CER dependence on λ rec and λ nonrec for trace norm regularization (left) and DISPLAYFORM0 GRU layer involves two matrix multiplications: a recurrent and a non-recurrent one. For a simple recurrent layer, we would write DISPLAYFORM1 For a GRU layer, there are also weights for reset and update gates, which we group with the recurrent matrix. See Appendix B.2 for details and the motivation for this split. Since our work focuses only on compressing acoustic models and not language models, the error metric we report is the character error rate (CER) rather than word error rate (WER). As the size and latency constraints vary widely across devices, whenever possible we compare techniques by comparing their accuracy versus number of parameter trade-off curves. All CERs reported here are computed on a validation set separate from the training set. In this section, we investigate the effects of training with the modified loss function in. For simplicity, we refer to this as trace norm regularization. As the WSJ corpus is relatively small at around 80 hours of speech, models tend to benefit substantially from regularization. To make comparisons more fair, we also trained unfactored models with an 2 regularization term and searched the hyperparameter space just as exhaustively. For both trace norm and 2 regularization, we found it beneficial to introduce separate λ rec and λ nonrec parameters for determining the strength of regularization for the recurrent and non-recurrent weight matrices, respectively. In addition to λ rec and λ nonrec in initial experiments, we also roughly tuned the learning rate. Since the same learning rate was found to be optimal for nearly all experiments, we just used that for all the experiments reported in this section. The dependence of final CER on λ rec and λ nonrec is shown in Figure 1. Separate λ rec and λ nonrec values are seen to help for both trace norm and 2 regularization. However, for trace norm regularization, it appears better to fix λ rec as a multiple of λ nonrec rather than tuning the two parameters independently. The first question we are interested in is whether our modified loss is really effective at reducing the trace norm. As we are interested in the relative concentration of singular values rather than their absolute magnitudes, we introduce the following nondimensional metric. Definition 1. Let W be a nonzero m × n matrix with d = min(m, n) ≥ 2. Denote by σ the d-dimensional vector of singular values of W. Then we define the nondimensional trace norm coefficient of W as follows: We show in Appendix A that ν is scale-invariant and ranges from 0 for rank 1 matrices to 1 for maximal-rank matrices with all singular values equal. Intuitively, the smaller ν(W), the better W can be approximated by a low rank matrix. DISPLAYFORM0 As shown in Figure 2, trace norm regularization is indeed highly effective at reducing the nondimensional trace norm coefficient compared to 2 regularization. At very high regularization strengths, 2 regularization also leads to small ν values. However, from Figure 1 it is apparent that this comes at the expense of relatively high CERs. As shown in Figure 3, this translates into requiring a much lower rank for the truncated SVD to explain, say, 90 % of the variance of the weight matrix for a given CER. Although a few 2 -regularized models occasionally achieve low rank, we observe this only at relatively high CER's and only for some of the weights. Note also that some form of regularization is very important on this dataset. The unregularized baseline model (the green points in Figure 3) achieves relatively low CER. In this section, we report the of stage 2 experiments warmstarted from either trace norm or L 2 regularized stage 1 models. For each regularization type, we took the three best stage 1 models (in terms of final CER: all were below 6.8) and used the truncated SVD of their weights to initialize the weights of stage 2 models. By varying the threshold of variance explained for the SVD truncation, each stage 1 model ed into multiple stage 2 models. The stage 2 models were trained without regularization (i.e., λ rec = λ nonrec = 0) and with the initial learning rate set to three times the final learning rate of the stage 1 model. FIG1, the best models from either trace norm or L 2 regularization exhibit similar accuracy versus number of parameter trade-offs. For comparison, we also warmstarted some stage 2 models from an unregularized stage 1 model. These models are seen to have significantly lower accuracies, accentuating the need for regularization on the WSJ corpus. In the previous sections, we trained the stage 1 models for 40 epochs to full convergence and then trained the stage 2 models for another 40 epochs, again to full convergence. Since the stage 2 models are drastically smaller than the stage 1 models, it takes less time to train them. Hence, shifting the stage 1 to stage 2 transition point to an earlier epoch could substantially reduce training time. In this section, we show that it is indeed possible to do so without hurting final accuracy. Specifically, we took the stage 1 trace norm and 2 models from Section 3.2.1 that ed in the best stage 2 models in Section 3.2.2. In that section, we were interested in the parameters vs accuracy trade-off and used each stage 1 model to warmstart a number of stage 2 models of different sizes. In this section, we instead set a fixed target of 3 M parameters and a fixed overall training budget of 80 epochs but vary the stage 1 to stage 2 transition epoch. For each of the stage 2 runs, we initialize the learning rate with the learning rate of the stage 1 model at the transition epoch. So the learning rate follows the same schedule as if we had trained a single model for 80 epochs. As before, we disable all regularization for stage 2. The are shown in FIG3. Based on the left panel, it is evident that we can lower the transition epoch number without hurting the final CER. In some cases, we even see marginal CER improvements. For transition epochs of at least 15, we also see slightly better for trace norm than 2. In the right panel, we plot the convergence of CER when the transition epoch is 15. We find that the trace norm model's CER is barely impacted by the transition whereas the huge jump in CER at the transition epoch. Furthermore, the plot suggests that a total of 60 epochs may have sufficed. However, the savings from reducing stage 2 epochs are negligible compared to the savings from reducing the transition epoch. With low rank factorization techniques similar 2 to those described in Section 3, we were able to train large vocabulary continuous speech recognition (LVCSR) models with acceptable numbers of parameters and acceptable loss of accuracy compared to a production server model (baseline). TAB1 shows the baseline along with three different compressed models with much lower number of parameters. The tier-3 model employs the techniques of Sections B.4 and B.3. Consequently, it runs significantly faster than the tier-1 model, even though they have a similar number of parameters. Unfortunately, this comes at the expense of some loss in accuracy. Although low rank factorization significantly reduces the overall computational complexity of our LVCSR system, we still require further optimization to achieve real-time inference on mobile or embedded devices. One approach to speeding up the network is to use low-precision 8-bit integer representations for weight matrices and matrix multiplications (the GEMM operation in BLAS terminology). This type of quantization after training reduces both memory as well as computation requirements of the network while only introducing 2% to 4% relative increase in WER. Quantization for embedded speech recognition has also been previously studied in BID25, and it may be possible to reduce the relative WER increase by quantizing the forward passes during training. As the relative WER losses from compressing the acoustic and language models were much larger for us, we did not pursue this further for the present study. where A is a random matrix with dimension 6144 × 320, and x is a random matrix with dimension 320× batch size. All matrices are in unsigned 8-bit integer format. To perform low precision matrix multiplications, we originally used the gemmlowp library, which provides state-of-the-art low precision GEMMs using unsigned 8-bit integer values (. However, gemmlowp's approach is not efficient for small batch sizes. Our application, LVCSR on embedded devices with single user, is dominated by low batch size GEMMs due to the sequential nature of recurrent layers and latency constraints. This can be demonstrated by looking at a simple RNN cell which has the form: DISPLAYFORM0 This cell contains two main GEMMs: The first, U h t−1, is sequential and requires a GEMM with batch size 1. The second, W x t, can in principle be performed at higher batch sizes by batching across time. However, choosing a too large batch sizes can significantly delay the output, as the system needs to wait for more future context. In practice, we found that batch sizes higher than around 4 ed in too high latencies, negatively impacting user experience. This motivated us to implement custom assembly kernels for the 64-bit ARM architecture (AArch64, also known as ARMv8 or ARM64) to further improve the performance of the GEMMs operations. We do not go through the methodological details in this paper. Instead, we are making the kernels and implementation details available at https://github.com/paddlepaddle/farm. FIG4 compares the performance of our implementation (denoted by farm) with the gemmlowp library for matrix multiplication on iPhone 7, iPhone 6, and Raspberry Pi 3 Model B. The farm kernels are significantly faster than their gemmlowp counterparts for batch sizes 1 to 4. The peak single-core theoretical performance for iPhone 7, iPhone 6, and Raspberry Pi 3 are 56.16, 22.4 and 9.6 Giga Operations per Second, respectively. The gap between the theoretical and achieved values are mostly due to kernels being limited by memory bandwidth. For a more detailed analysis, we refer to the farm website. In addition to low precision representation and customized ARM kernels, we explored other approaches to speed up our LVCSR system. These techniques are described in Appendix B.Finally, by combining low rank factorization, some techniques from Appendix B, int8 quantization and the farm kernels, as well as using smaller language models, we could create a range of speech recognition models suitably tailored to various devices. These are shown in TAB2. We worked on compressing and reducing the inference latency of LVCSR speech recognition models. To better compress models, we introduced a trace norm regularization technique and demonstrated its potential for faster training of low rank models on the WSJ speech corpus. To reduce latency at inference time, we demonstrated the importance of optimizing for low batch sizes and released optimized kernels for the ARM64 platform. Finally, by combining the various techniques in this paper, we demonstrated an effective path towards production-grade on-device speech recognition on a range of embedded devices. Figure 7: Contours of ||σ|| 1 and ||σ|| 2. ||σ|| 2 is kept constant at σ. For this case, ||σ|| 1 can vary from σ to √ 2σ. In this section, we describe some of the properties of the non-dimensional trace norm coefficient defined in Section 3.1. DISPLAYFORM0 (iii) ν(W) = 0 if and only if W has rank 1.(iv) ν(W) = 1 if and only if W has maximal rank and all singular values are equal. Proof. Since we are assuming W is nonzero, at least one singular value is nonzero and hence ||σ|| 2 = 0. Property (i) is immediate from the scaling property ||cσ|| = |c| · ||σ|| satisfied by all norms. To establish the other properties, observe that we have DISPLAYFORM1 The first inequality holds since singular values are nonnegative, and the inequality is strict unless σ i or σ j vanishes. The second inequality comes from an application of Jensen's inequality and is strict unless σ i = σ j. Thus, replacing (σ i, σ j) by (σ i + σ j, 0) preserves ||σ|| 1 while increasing ||σ|| 2 unless one of σ i or σ j is zero. Similarly, replacing (σ i, σ j) by (DISPLAYFORM2 2 σ j) preserves ||σ|| 1 while decreasing ||σ|| 2 unless σ i = σ j. By a simple argument by contradiction, it follows that the minima occur for σ = (σ 1, 0, . . ., 0), in which case ν(W) = 0 and the maxima occur for σ = (σ 1, . . ., σ 1), in which case ν(W) = 1.We can also obtain a better intuition about the minimum and maximum of ν(W) by looking at the 2D case visualized in Figure 7. For a fixed ||σ|| 2 = σ, ||σ|| 1 can vary from σ to √ 2σ. The minimum ||σ|| 1 happens when either σ 1 or σ 2 are zero. For these values ||σ|| 2 = ||σ|| 1 and as a ν(W) = 0. Similarly, the maximum ||σ|| 1 happens for σ 1 = σ 2, ing in ν(W) = 1. We describe here a few preliminary insights that informed our choice of baseline model for the experiments reported in Sections 3 and 4.Since the target domain is on-device streaming speech recognition with low latency, we chose to focus on Deep Speech 2 like models with forward-only GRU layers BID1. Across several data sets and model architectures, we consistently found that the sizes of the recurrent layers closer to the input could be shrunk without affecting accuracy much. A related phenomenon was observed in: When doing low rank approximations of the acoustic model layers using SVD, the rank required to explain a fixed threshold of explained variance grows with distance from the input layer. To reduce the number of parameters of the baseline model and speed up experiments, we thus chose to adopt growing GRU dimensions. Since the hope is that the compression techniques studied in this paper will automatically reduce layers to a near-optimal size, we chose to not tune these dimensions, but simply picked a reasonable affine increasing scheme of 768, 1024, 1280 for the GRU dimensions, and dimension 1536 for the final fully connected layer. For the recurrent layers, we employ the Gated Recurrent Unit (GRU) architecture proposed in BID5, where the hidden state h t is computed as follows: DISPLAYFORM0 where σ is the sigmoid function, z and r are update and reset gates respectively, U z, U r, U h are the three recurrent weight matrices, and W z, W r, W h are the three non-recurrent weight matrices. We consider here three ways of performing weight sharing when doing low rank factorization of the 6 weight matrices.1. Completely joint factorization. Here we concatenate the 6 weight matrices along the first dimension and apply low rank factorization to this single combined matrix. 2. Partially joint factorization. Here we concatenate the 3 recurrent matrices into a single matrix U and likewise concatenate the 3 non-recurrent matrices into a single matrix W. We then apply low rank factorization to each of U and W separately. 3. Completely split factorization. Here we apply low rank factorization to each of the 6 weight matrices separately. In BID14, the authors opted for the LSTM analog of completely joint factorization, as this choice has the most parameter sharing and thus the highest potential for compression of the model. However, we decided to go with partially joint factorization instead, largely for two reasons. First, in pilot experiments, we found that the U and W matrices behave qualitatively quite differently during training. For example, on large data sets the W matrices may be trained from scratch in factored form, whereas factored U matrices need to be either warmstarted via SVD from a trained unfactored model or trained with a significantly lowered learning rate. Second, the U and W split is advantageous in terms of computational efficiency. For the non-recurrent W GEMM, there is no sequential time dependency and thus its inputs x may be batched across time. Finally, we compared the partially joint factorization to the completely split factorization and found that the former indeed led to better accuracy versus number of parameters trade-offs. Some from this experiment are shown in TAB3. Switching from 161-dimensional linear spectrograms to 80-dimensional mel spectrograms reduces the per-timestep feature dimension by roughly a factor of 2. Furthermore, and likely owing to this switch, we could reduce the frequency-dimension size of the convolution filters by a factor of 2. In combination, this means about a 4x reduction in compute for the first and second convolution layers, and a 2x reduction in compute for the first GRU layer. On the WSJ corpus as well as an internal dataset of around 1,000 hours of speech, we saw little impact on accuracy from making this change, and hence we adopted it for all experiments in Section 3. Gram-CTC is a recently proposed extension to CTC for training models that output variable-size grams as opposed to single characters BID16. Using Gram-CTC, we were able to increase the time stride in the second convolution layer by a factor of 2 with little to no loss in CER, though we did have to double the number of filters in that same convolution layer to compensate. The net effect is a roughly 2x speedup for the second and third GRU layers, which are the largest. This speed up more than makes up for the size increase in the softmax layer and the slightly more complex language model decoding when using Gram-CTC. However, for a given target accuracy, we found that Gram-CTC models could not be shrunk as much as CTC models by means of low rank factorization. That is, the net effect of this technique is to increase model size in exchange for reduced latency. Shown in FIG5 is the parameter reduction versus relative CER increase trade-off for various techniques on an internal data set of around 1,000 hours of speech. The baseline model is a Deep Speech 2 model with three forward-GRU layers of dimension 2560, as described in BID1. This is the same baseline model used in the experiments of BID19, from which paper we also obtained the sparse data points in the plot. Shown also are versions of the baseline model but with the GRU dimension scaled down to 1536 and 1024. Overall, models with low rank factorizations on all non-recurrent and recurrent weight matrices are seen to provide the best CER vs parameters trade-off. All the low rank models use growing GRU dimensions and the partially split form of low rank factorization, as discussed in Sections B.1 and B.2. The models labeled fast in addition use Gram-CTC as described in Section B.4 and mel features and reduced convolution filter sizes as described in Section B.3.As this was more of a preliminary comparison to some past experiments, the setup was not perfectly controlled and some models were, for example, trained for more epochs than others. We suspect that, given more effort and similar adjustments like growing GRU dimensions, the sparse models could be made competitive with the low rank models. Even so, given the computational advantage of the low rank approach over unstructured sparsity, we chose to focus only on the former going forward. This does not, of course, rule out the potential usefulness of other, more structured forms of sparsity in the embedded setting.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We compress and speed up speech recognition models on embedded devices through a trace norm regularization technique and optimized kernels.

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.