Question: Represent the scientific article to find a short TLDR

Question: Adaptive regularization methods pre-multiply a descent direction by a preconditioning matrix. Due to the large number of parameters of machine learning problems, full-matrix preconditioning methods are prohibitively expensive. We show how to modify full-matrix adaptive regularization in order to make it practical and effective. We also provide novel theoretical analysis for adaptive regularization in non-convex optimization settings. The core of our algorithm, termed GGT, consists of efficient inverse computation of square roots of low-rank matrices. Our preliminary experiments underscore improved convergence rate of GGT across a variety of synthetic tasks and standard deep learning benchmarks. Stochastic gradient descent is the workhorse behind the recent deep learning revolution. This simple and age-old algorithm has been supplemented with a variety of enhancements to improve its practical performance, and sometimes its theoretical guarantees. Amongst the acceleration methods there are three main categories: momentum, adaptive regularization, and variance reduction. Momentum (in its various incarnations, like heavy-ball or Nesterov acceleration) is the oldest enhancement. It has a well-developed theory, and is known to improve practical convergence in a variety of tasks, small and large. It is also easy to implement. Variance reduction is the most recent advancement; in theory and practice, it is mostly applicable to convex optimization, and is thus less influential in deep learning. This brings us to adaptive regularization: the most sophisticated, hard to implement, and debated acceleration method. While state-of-the-art optimizers such as Adam and AdaGrad (; BID13 do use adaptive regularization, they do so in a very limited form: with diagonal matrices, often marketed as per-coordinate adaptive learning-rate methods. Despite solid theoretical guarantees, the practical value of diagonal adaptive regularization as compared to "vanilla" SGD has been the subject of much debate BID48 . However, the efficacy of full-matrix adaptive regularization has been relatively unexplored. This is due to the prohibitive computational cost associated with full-matrix operations: full AdaGrad requires taking the inverse square root of a large matrix. In this paper, we present GGT, a practical solution to the computational problems plaguing fullmatrix adaptive regularization, making this technique scalable for modern deep models. At the heart of our method is a simple, GPU-friendly way to apply the inverse square root of the low-rank second-moment matrix of recent gradients; see FIG0 . GGT's running time is comparable to state-of-the-art optimizers. We proceed to show that full-matrix preconditioning allows for much better exploitation of anisotropic curvature in loss landscapes. First, we show synthetic experiments which demonstate clear benefits of GGT over baselines, especially when the problem is ill-conditioned. Then, we implement GGT at scale, and show that the benefits translate to faster training on standard deep learning benchmarks. Our improvement is most salient in complicated landscapes like RNN training. Our algorithm comes with theoretical guarantees. We give the first proof of convergence to firstorder critical points for an algorithm with adaptive regularization in a stochastic non-convex setting, featuring a rate which is dependent on an adaptive ratio. We show examples where our bound is stronger than that for SGD, providing some theoretical basis for our empirical findings. Since the introduction of AdaGrad BID13, diagonal adaptive regularization has been a mainstay in the machine learning practitioner's toolbox. A quick perusal of the literature shows that these methods have continued to thrive in the deep learning era, and appear in all major frameworks BID0 BID39 BID9 . By citation count (or GitHub search hits), Adam is by far the most popular adaptive optimizer for training a variety of modern deep models. For this reason, this paper's exposition is targeted towards a full-matrix dropin replacement for Adam; however, our techniques extend straightforwardly to a plethora of variants, like RMSprop BID45, Adadelta , Nadam BID12, etc. Full-matrix adaptive regularization has existed alongside the more commonly used diagonal-matrix manifestation since their common inception in BID13 ); however, a major obstacle to the scalability of these methods is the need for the storage and inversion of square matrices in the model dimension. This becomes prohibitively expensive in dimension greater than 10 4, while state-of-theart models regularly exceed 10 7 parameters. Matrix sketching has been employed to approximate the AdaGrad preconditioner BID25 BID34; however, the sketched estimate for the matrix inverse can be sensitive to noise. In the former, the authors report a 5-10× overhead over AdaGrad, even with < 10 5 model parameters; we could not find a usable GPU implementation for their requisite rank-1 QR update. BID19 propose a way to do AdaGrad with Kronecker products of full-matrix preconditioners, a more limited setting which requires knowledge of the model's structure. Finally, as we argue in Section 3.1, there is intrinsic value of "forgetting" past curvature using an exponential window. With this, a low-rank preconditioning matrix naturally arises, allowing us to bypass the computational need for sketching in the model dimension or architecture-dependent restriction of the preconditioner. Our algorithm bears a superficial resemblance to L-BFGS BID28, a version of BFGS BID6 BID15 BID18 BID43 which uses a sliding window of gradient history. Although some are viable for large-scale implementation, these quasi-Newton methods, along with (subsampled, online, cubic-regularized) Newton methods BID14 BID2 BID30 BID20 BID1 BID8 exhibit very different dynamics than the standard optimizers in deep learning, and thus have not seen widespread adoption. We find recent deep learning applications of secondorder methods (e.g. BID32 BID33) to be intriguing, though outside the scope of this paper. Recently, the role of adaptive regularization has been a hotly contested topic. In BID48, the authors suggest that properly-tuned SGD exhibits superior generalization to adaptive methods. In turn, propose switching the optimizer from Adam to SGD at the end of training, to reap the advantages of each. Influentially, Adam's convergence has been the object of recent scrutiny BID42. However, Adam continues to enjoy successful convergence in practice; the problematic construction involves pathological outlier gradients. We do not use the analyses of Adam or AMSGrad. Several parallel works BID27 BID51 BID47 BID10 BID40 BID50 have studied the convergence of adaptive methods for non-convex optimization, matching the asymptotic iteration complexity of SGD. Apart from our algorithmic contribution, our work is (to our knowledge) the first attempt to characterize the advantage of adaptivity in terms of the dimension and geometry of the optimization problem. Our main algorithmic contribution is GGT, an efficient first-order algorithm for full-matrix adaptive preconditioning. In brief, GGT uses the preconditioner from full-matrix AdaGrad, with gradient history attenuated exponentially as in Adam, and truncated to a window parameter r. The name GGT acts as a convenient mnemonic for the gradient second-moment matrix GG maintained by full-matrix AdaGrad, even though we never compute this matrix. The mathematical specification of GGT is given in Algorithm 1, in the usual model of stochastic optimization (see Section 4), with gradients ∇f (x). Notice that the coordinate-wise scaling of Adam is recovered by zeroing out the off-diagonal entries of GG.Algorithm 1 GGT adaptive optimizer 1: Input: initializer x 1, window size r, learning rate schedule {η t}, DISPLAYFORM0 Receive stochastic gradient ∇f (x t). GGT provides the power of full-matrix adaptive regularization at a cost not much larger than SGD. This crucially exploits the fact only a small window of historical gradients are used for preconditioning. The intuition for using a small window, as opposed to the entire history, is clear (and time-tested, by the ubiquity of Adam): the curvature of the loss surface changes, rendering previous gradient information obsolete. We expand on the benefits of forgetting gradients in section 3.1.The fact that the preconditioning matrix is based on a small window of gradients implies that it has low rank. GGT exploits this fact by computing the inverse square root of the empirical covariance matrix indirectly, as outlined in FIG0. In effect, instead of inverting a full matrix in the dimension of parameters, using the special matrix structure GGT inverts a matrix of dimension window-size. The remainder of this section will discuss efficient implementation and some heuristics. GGT has provable guarantees even for non-convex optimization: it is guaranteed to converge to a first-order critical point. Its rate of convergence is never significantly slower than that of SGD, and in some favorable geometric conditions, can be significantly faster. These theoretical bounds are made precise in section 4. The window parameter r should be roughly the number of copies of the model that fit in RAM; in our large-scale experiments, we use r = 200. A pessimistic but principled choice is r = Θ(1/(1 − β 2)), which truncates on the time scale of the exponential attenuation. Our key observation, highlighted in FIG0, is that the inversion of the large low-rank matrix GG can be performed by diagonalizing the small matrix G G, along with some extremely GPU-friendly matrix-vector operations. The basic intuition is contained in FIG0, but it remains to include the εI term. We derive the full update here. Let DISPLAYFORM0, and let Σ r ∈ R r× r be its top left block. Let U =: [U r U d−r], so that the columns of U r ∈ R d×r are an orthonormal basis for the column space of G, and DISPLAYFORM1 The first term is none other than an SGD update step. The rest can be computed by taking the DISPLAYFORM2 We prefer this to taking the direct SVD of G, which is > 10 times slower on GPU.Using a cyclic buffer to store and update G t, the algorithm takes O(dr 2 + r 3) (sequential) time per iteration, and O(dr) memory in total. Iterating over the model parameters to update G t incurs the same overhead cost as usual adaptive optimizers. The r × d matrix multiplication and r × r SVD operations benefit from decades of extensive hardware-level optimizations. In the experiments in Section 3, we observed a ∼ 1.3× (CNN) and ∼ 2× (RNN) running-time overhead over SGD; we note that this ratio could be even smaller in reinforcement learning (where the environment causes the time bottleneck), or universally with a more optimized implementation. Below, we list some practical suggestions for applying GGT to training large-scale models. Momentum. In order to bring GGT closer to a drop-in replacement for Adam, we can add momentum to the gradient steps: let v t ← β 1 v t−1 + ∇f (x t), and apply the preconditioner to v t to compute the update step. We use momentum in all large-scale experiments, with the standard β 1 = 0.9. We also get a small performance boost by using v t instead of the gradients to update G t. On the other hand, as long as r T, it makes little difference to choose β 2 = 1, letting the window (rather than exponential attenuation) forget stale gradient information. Interpolation with SGD. We note the possibility of decoupling the scalars ε and 1/ε which appear in the efficient update step. Appealingly, this allows the user to tune GGT's behavior to be arbitrarily close to that of SGD.Numerical concerns. For greater numerical stability, it is possible to add a small multiple of the identity matrix (we suggest 10 −6) to G G before computing its eigendecomposition, without noticeable differences in training. In this section, we present an empirical study of GGT. We begin with some simple experiments, showing that adaptive methods help in the presence of ill-conditioned optimization problems, as well as the value of limited gradient memory. Next, we evaluate the performance of GGT on largerscale deep learning tasks (and provide some additional such experiments in Appendix B). Finally, we present some interesting empirical insights on the training dynamics in deep learning models. Our visualizations of gradient spectra suggest that adaptive optimizers are indeed correcting for changing anisotropic curvature in the loss landscape. The original theorems on the behavior of adaptive first-order methods are established from the perspective of online convex optimization BID13. The dynamics are less understood on realistic loss landscapes in stochastic optimization. For this reason, we begin our experimental section with some simple empirical comparisons between full-and diagonal-matrix adaptive optimizers and SGD. In each synthetic experiment, we generated an ill-conditioned landscape, and compared SGD with adaptive optimizers, excluding the typical accompanying heuristics (i.e. no momentum, regularization, or learning rate schedule). We tested diagonal-matrix preconditioners with and without exponential gradient attenuation (like Adam and AdaGrad, respectively), and their full-matrix analogues. The experiments were robust with respect to the choice of ε (we used 10 −4) and batch size. In the first synthetic experiment (left), we exhibit an instance of logistic regression in dimension 10, with 10 3 samples generated from an extremely anisotropic (σ 2 max /σ 2 min ≈ 10 4) Gaussian distribution, and binary labels determined by a random hyperplane. SGD converges the slowest, and diagonal AdaGrad consistently accelerates optimization. Finally, full-matrix preconditioning (using cubic-time matrix inversion) converges the fastest. In this setting, adding a window improved convergence, but not drastically; we elaborate below. Next, we show an optimization problem (right) which accentuates the utility of exponentially decaying gradient memory. We consider the problem of minimizing the logarithmic barrier function of a randomly generated anisotropic polytope, otherwise known as finding its analytic center: this replaces the logistic loss terms with f i (w) = − log(w x i + c i), with x i generated the same way as above, and c i generated uniformly from. We observed the same ranking of convergence rates as in the first experiment, but the improvement afforded by the window was much clearer. The primary of our synthetic experiments is to demonstrate some small-scale settings in which adaptive regularization ameliorates anisotropy in the optimization landscape. A subtler point is that the windowed variants can help with changing curvature, even for convex losses. Note that the curvature of the former landscape is constant (in that its Hessian matrix at different locations w only changes by a scalar factor). The latter setting, in contrast, features a changing curvature (its Hessians do not commute in general), necessitating "forgetfulness" in adaptive curvature estimation. In Section 3.4, we will return to these proof-of-concept optimization instances, connecting them to an empirical study of curvature in more realistic landscapes. We investigated the training dynamics of GGT on a typical deep architecture for computer vision. For this, we used a 26-layer 3-branch residual network with Shake-Shake regularization, recently proposed in BID16. Aside from its ability to reach state-of-the-art classification accuracy, this architecture also features a relatively low parameter count (∼ 3M), enabling the use of a large window parameter (r = 200).In each experiment, we kept the cosine learning rate annealing schedule proposed in the paper, originally from BID29; performance degraded consistently and significantly with a fixed learning rate. For both Adam and GGT, we chose the commonly used parameters β 1 = 0.9, β 2 = 0.999, ε = 10 −8; for SGD, we used momentum with parameter 0.9. With correctly tuned RMSprop and Adadelta, with the same window parameters, training curves were virtually identical to those for Adam. We used the standard data augmentation techniques of 4-pixel padding + random cropping and horizontal flipping. Our are shown in FIG2. In terms of training loss, GGT consistently dominated existing optimizers. We corroborate a number of observations from previous empirical studies of the generalization of optimizers. Most prominently, we found that SGD generalized slightly better than all others BID48 towards the end of training, including ours. The gap (< 0.2%) is less dramatic than that seen in for two reasons: we only show curves with a tuned and annealed learning rate; also, we use an architecture with powerful explicit regularization techniques which have gained attention since their publication. Our preliminary observation is that GGT shrinks this gap slightly (corroborated by another experiment in Appendix B), and expect that there is vastly more empirical work to be done concerning architectures synergistically tuned to existing optimizers. We also verify the long-held empirical observation that the learning rate decay of AdaGrad is too aggressive (e.g. in ), ing in convergence to a poor solution. Finally, as noted in BID48, we find that using a sufficiently low learning rate for any optimizer can in a better training loss curve, but not without significantly degrading generalization (> 3% worse). Next, we move to recurrent architectures for language modeling. We train a 3-layer LSTM with ∼ 5M parameters for character-level modeling of the Penn Treebank dataset BID31. This is the setting in which we observe the most striking improvement over baselines. The particularities of this optimization task, and why it might be especially amenable to full-matrix regularization, remain a fruitful research direction BID38. FIG2 (bottom) shows training and validation perplexities for the first 50 epochs; no optimizer makes significant progress afterwards. The state of the art for character-level language modeling is less thoroughly documented than its word-level counterpart, though we note that our end-to-end (validation perplexity 2.42 after 500 epochs) is competitive with those reported for recurrent models, like by BID24. In contrast, Adam, AdaGrad, and SGD reach 2.51, 2.65, and 2.76, respectively. Note that Adam is the de facto standard optimizer for language modeling BID35. Even with iterations taking twice the time, we outperform all baselines in wall-clock time throughout training. We also tried using GGT as a drop-in replacement for Adam in the state-of-the-art word-level language modeling code accompanying BID36. Although we were competitive with Adam, we only observed an improvement in the first ∼ 20 epochs. We hypothesize that the advantage of full-matrix regularization in this setting is more marginal, as the gradients in the embedding layers are naturally sparse in the vocabulary ("one-hot") basis. On a similar note, we found that Adam outperformed GGT on attention-based architectures for NLP; refer to Appendix B for an experiment and discussion. In this section, we unify the insights gleaned from the synthetic experiments and deep learning benchmarks. Along the way, we provide some interesting anecdotal observations on the evolution of the preconditioner matrices' singular values. We plot the density of the spectrum of the low-rank preconditioner G t G t as training progresses. Since the fast implementation of GGT takes an eigendecomposition of G t G t, we can read off the distribution of eigenvalues during training at no additional computational cost. FIG3 visualizes the of this experiment for the CNN and RNN training settings from the previous two sections. In each case, we observe that G t G t has a condition number of ∼ 10 3, noting that this can be visualized as the vertical range in the logarithmic plot. This visualization affords a new way to see how CNN and RNN landscapes are fundamentally different: their gradient spectra evolve in very distinct ways over the course of training. Interestingly, the condition number of the CNN landscape surges near the end, which may be related to the the low-rank structure of well-trained nets noted by BID5, who derive rank-dependent generalization bounds for neural networks. On recurrent models, the rapidly evolving spectral structure at the early stage of training indicates a possibly more complex landscape. Intriguingly, the enormous condition number (∼ 10 6) correlates with the massive lead of GGT over the others, confirming our intuition that full-matrix preconditioning ameliorates anisotropy. To our knowledge, this is the first empirical study of this kind, using the covariance matrix of recent gradients as a surrogate to examining the changing curvature of the loss landscape. In the spirit of recent empirical lenses of this flavor BID41 BID26, we leave this as a way to visualize deep learning dynamics, possibly of independent exploratory interest. In this section we outline our analysis of GGT, for which we show convergence to an approximate first-order critical point, in some settings faster than SGD. To obtain the strongest theory, we analyze GGT with a "hard window" instead of exponentially decaying gradient memory, explained in Section A.2.We work in the usual theoretical framework of stochastic optimization of a differentiable non-convex function f (·), equipped with an unbiased variance-bounded stochastic gradient oracle ∇f (·). The objective, as is standard in the literature (see, e.g. BID17 ; BID4), is to find an ε-approximate stationary point x; that is, ∇f (x) ≤ ε. We quantify the improvement of adaptive regularization by its advantage over the usual worst-case bound of SGD. To this end, we define the adaptive ratio µ of an algorithm A as DISPLAYFORM0, where x A is the output of the A, and x * is a comparator. For convex optimization problems x * is naturally the global minimum. For non-convex optimization it is a subtler choice, which we detail in Appendix A.This ratio for the AdaGrad algorithm was shown in BID13 to be always bounded by a quantity independent of T, and potentially much smaller. Specifically, it was shown to be inversely proportional to the dimension in certain convex optimization problems, providing a theoretical justification for the speedup of adaptive optimizers. In Section A.4, we show a new, simple, and natural setting illustrating adaptive speedup, even for a strongly convex function f. We informally state the main theorem below. We defer the full bound without suppressed smoothness constants, as well as all technical proofs, to Appendix A. Theorem 4.1. Let f: R d → R be a bounded, Lipschitz, and smooth function with stochastic gradient oracle ∇f (·), whose variance is at most σ 2. In expectation, Algorithm 3 outputs an ε- DISPLAYFORM0 This theorem matches and potentially improves the known analysis for stochastic gradient descent with the introduction of the data-dependent adaptivity constant µ into the leading-order term governing the rate of convergence. Since BID13 bounded µ by a quantity independent of T, our theorem matches the classic O ε −4 rate of convergence. This work investigates full-matrix adaptive regularization: our main contribution is to make this technique viable for large-scale optimization, by a method for efficient multiplication by the inverse square root of a full second-moment matrix over a short window of gradients. This leads to a new algorithm, GGT, a truly scalable optimization algorithm with full-matrix adaptive preconditioning. Through synthetic experiments, we have shown that GGT accelerates optimization in ill-conditioned loss landscapes; this is supported by accompanying adaptive convergence guarantees. Preliminary experiments show accelerated convergence on standard deep learning benchmarks, with very different training dynamics from existing diagonal adaptive methods. We accompany our algorithm and experiments with the first theoretical characterization of the benefits of adaptive regularization in a non-convex setting. We hope that GGT will be the first of a new class of algorithms for the modern large-scale optimization toolbox, and to foster new discussion towards an ever-elusive understanding of loss landscapes in deep learning. In this section, we give the details on the theoretical treatment of GGT outlined in Section 4. The overall goal is to develop a theory for adaptive regularization in non-convex stochastic optimization. After formalizing the setting, we will define a version of GGT that uses a hard gradient memory window. This will allow us to transfer any insight on the advantage of adaptivity in the convex case to the non-convex case, giving rise to the main theorem. We will conclude this section by with an example illustrating the advantage of adaptive optimizers in the presence of sparse gradients. A.1 SETTING: STOCHASTIC NON-CONVEX OPTIMIZATION Theorem A.2 will provide a bound on the number of stochastic gradient calls required by GGT to achieve a first-order critical point. In particular, the theorem shows that GGT can converge to an approximate first-order critical point faster than SGD, with convergence rate controlled by the adaptive ratio µ, defined in.We consider the standard setting of stochastic optimization of a differentiable non-convex function f (·), equipped with a bounded-variance stochastic gradient oracle defined as follows. Definition A.1 (stochastic gradient oracle). Given a function f: D → R we call an oracle O f, a σ-bounded stochastic gradient oracle if for any x, O f returns a a random vector ∇f (x) such that DISPLAYFORM0 The objective, as is standard in non-convex optimization, is to find a first-order critical point, i.e. a point x for which ∇f (x) ≤ ε. We will also assume that f has a Lipschitz gradient; i.e. DISPLAYFORM1 Our algorithm makes a reduction to the case of stochastic convex optimization. The setting formally is that, given a smooth convex function and a σ-bounded stochastic gradient oracle, the algorithm's aim is to minimize the convex function f. Given any algorithm A we can now define the adaptive ratio of the algorithm, referred to as µ, as DISPLAYFORM2 where x A is the output of the algorithm A and x * ∈ argmin x f (x), with a total of at most T calls to the stochastic gradient oracle. µ captures the advantage in convergence rate obtained by the algorithm as compared to the error obtained by vanilla SGD, noting that the denominator is a bound on the error obtained by SGD in the same setting. A popular algorithm for stochastic (and in general online) convex optimization is AdaGrad BID13. Due to adaptive regularization, AdaGrad can often be advantageous over SGD. We quantify this advantage by the notion of µ defined above. The bounds of BID13 imply that µ can be as small as DISPLAYFORM3, depending on the geometry of the optimization problem. An example of this was provided by BID13 for both the diagonal and the full version of Adagrad. At the end of this section, we provide a different example which shows the same phenomenon even in the case of strongly convex functions. In the rest of this section we describe Algorithm 3, which uses AdaGrad (Algorithm 2) as a subroutine during each window. In this regard, while stating the bounds for our algorithms, we use µ as an upper bound on the advantage of AdaGrad in each iteration. As mentioned in Section 4, our analysis uses a slightly idealized version of GGT, which replaces the gradient memory mechanism (governed by w and β 2) with a hard window; i.e., the gradient buffer is reset every w steps. This simple modification enables us to develop a more informative theory, in which we benefit directly from the familiar theory of AdaGrad for convex optimization, while capturing the necessity of forgetting past gradient information in adaptive non-convex optimization. First, for clarity, we restate the definition of the full-matrix AdaGrad algorithm, introduced by BID13, which accumulates the second-moment matrix of all past gradients:Algorithm 2 AdaGrad for convex optimization BID13 1: Input: initializer x 1, window length w, stochastic gradient oracle ∇f (·), ε, η > 0. 2: for t = 1,..., w do 3:Receive stochastic gradient ∇f (x t). Let G t = [g t g t−1 . . . g 1], where g t:= ∇f (x t). Update x t+1 ← x t − η · εI + (G t G t) 1/2 −1 g t. 6: end for 7: Output: Average iterate DISPLAYFORM0 The final algorithm we analyze simply runs AdaGrad between restarts. Algorithm 3 GGT with a hard gradient window 1: Input: initializer x 1, time horizon T, window length w, λ > 0. 2: for t = 1 to T: do DISPLAYFORM1 Update x t+1 to be the output of Algorithm 2 on f t (x), starting at x t, for w steps. 5: end for 6: Output: Best iterate x t *, where t *:= argmin t≤T +1 ∇f (x t).The remaining discrepancies between Algorithm 3 and Algorithm 1 from the main paper are standard. We provide some references below.• Absence of first-moment estimation. Although it is customary to use nonzero β 1 (otherwise known as momentum) when applying Adam in practice, it is orthogonal to the effect of adaptive regularization in all established theory. In fact, the convergence rates given by (and fixed by BID42) contain only factors of 1/(1 − β 1), and are thus strongest when β 1 = 0.• Model averaging. Theoretical guarantees in online and stochastic convex optimization are most naturally stated on the average iterate; see BID40 BID13. Thus, we adopt the convention that Algorithm 2 returns the average iterate. We note that model averaging is a common regularization technique in practical non-convex settings, though not the default choice for adaptive optimizers in practice.• 2 regularization. The addition of the λ x − x t 2 term in Algorithm 3 is an artifact we introduce to obtain a tight analysis for hard-window GGT. It ensures that iterates in each window do not move too far, and allows us to analyze each window as a fixed convex program, so that we can use the convex theory of AdaGrad directly. The soft-window analogue would simply to be decrease the learning rate. Interestingly, a similar technique directly appears in the algorithm proposed by BID3. Finally, we note that from a σ-bounded stochastic gradient oracle for f, it is trivial to construct one for f t, by adding −2λx t (deterministically). Theorem A.2. Consider a non-convex function f, such that for all x, ∇ 2 f (x) 2 ≤ L and a point DISPLAYFORM0 Further, suppose we have access to a σ-bounded DISPLAYFORM1. Then the point x returned by Algorithm 3 is such that E ∇f (x) ≤ ε, where µ = max t∈[T] µ t and µ t is the adaptive ratio when run on f t (as defined in FORMULA9). Further, note that choosing λ = 3L/2, the total number of stochastic gradient calls to the oracle O f, made by the algorithm is bounded by T · w = FIG2. Top: CIFAR-10 classification with a 3-branch ResNet. Bottom: PTB character-level language modeling with a 3-layer LSTM. We present some additional large-scale empirical studies in FIG5. To demonstrate a vision task with a harder optimization landscape, we use GGT to train a 19-layer "vanilla" convolutional network (VGGNet, BID44), without residual connections or batch normalization, on the same CIFAR-10 classification task. Here, we recover the same insights as found by BID48, in which diagonal-matrix adaptive methods can fail to train a network dramatically. Here, unlike diagonal-matrix adaptive optimizers, GGT stays on par with SGD throughout training, with a ∼ 1% gap remaining in generalization at the end. We use a standard fixed halving learning rate schedule; it is clear here that in the initial epochs after decaying the learning rate, GGT trains the most rapidly. We leave a careful investigation of leveraging this phenomenon, and tuning GGT's learning rate schedule, to future work. A recent significant advancement on many NLP tasks, including language modeling, is the introduction of attention-based models. We investigate the behavior of GGT on a Transformer network BID46, on the same Penn Treebank character-level language modeling task. Here, after an initial lead, GGT is outperformed by Adam in training and validation loss. The value of using gradient correlations to assist in the training of attention models seems to be limited.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

fast, truly scalable full-matrix AdaGrad/Adam, with theory for adaptive stochastic non-convex optimization

represent this text to find a much much longer article about it with say hundreds-thousands of words
Fewshot example: "Simple and effective graph neural network with mixture of random walk steps and attention" == "Graph neural networks have shown promising on representing and analyzing diverse graph-structured data such as social, citation, and protein interaction networks. Existing approaches commonly suffer from the oversmoothing issue, regardless of whether policies are edge-based or node-based for neighborhood aggregation. Most methods also focus on transductive scenarios for fixed graphs, leading to poor generalization performance for unseen graphs. To address these issues, we propose a new graph neural network model that considers both edge-based neighborhood relationships and node-based entity features, i.e. Graph Entities with Step Mixture via random walk (GESM). GESM employs a mixture of various steps through random walk to alleviate the oversmoothing problem and attention to use node information explicitly. These two mechanisms allow for a weighted neighborhood aggregation which considers the properties of entities and relations. With intensive experiments, we show that the proposed GESM achieves state-of-the-art or comparable performances on four benchmark graph datasets comprising transductive and inductive learning tasks. Furthermore, we empirically demonstrate the significance of considering global information. The source code will be publicly available in the near future. Graphs are universal data representations that exist in a wide variety of real-world problems, such as analyzing social networks , forecasting traffic flow , and recommending products based on personal preferences . Owing to breakthroughs in deep learning, recent graph neural networks (GNNs) have achieved considerable success on diverse graph problems by collectively aggregating information from graph structures; ). As a , much research in recent years has focused on how to aggregate the feature representations of neighbor nodes so that the dependence of graphs is effectively utilized. The majority of studies have predominantly depended on edges to aggregate the neighboring nodes' features. These edge-based methods are premised on the concept of relational inductive bias within graphs , which implies that two connected nodes have similar properties and are more likely to share the same label . While this approach leverages graphs' unique property of capturing relations, it appears less capable of generalizing to new or unseen graphs (b). To improve the neighborhood aggregation scheme, some studies have incorporated node information; They fully utilize node information and reduce the effects of relational (edge) information. A recent approach, graph attention networks (GAT), employs the attention mechanism so that weights used for neighborhood aggregation differ according to the feature of nodes (Veličković et al., 2018). This approach has yielded impressive performance and has shown promise in improving generalization for unseen graphs. Regardless of neighborhood aggregation schemes, most methods, however, suffer from a common problem where neighborhood information is considered to a limited degree . For example, graph convolutional networks (GCNs) only operate on data that are closely connected due to oversmoothing, which indicates the "washing out" of remote nodes' features via averaging. Consequently, information becomes localized and access to global information is restricted , leading to poor performance on datasets in which only a small portion is labeled. In order to address the aforementioned issues, we propose a novel method, Graph Entities with Step Mixture via random walk (GESM), which considers information from all nodes in the graph and can be generalized to new graphs by incorporating random walk and attention. Random walk enables our model to be applicable to previously unseen graph structures, and a mixture of random walks alleviates the oversmoothing problem, allowing global information to be included during training. Hence, our method can be effective, particularly for nodes in the periphery or a sparsely labeled dataset. The attention mechanism also advances our model by considering node information for aggregation. This enhances the generalizability of models to diverse graph structures. To validate our approach, we conducted experiments on four standard benchmark datasets: Cora, Citeseer, and Pubmed, which are citation networks for transductive learning, and protein-protein interaction (PPI) for inductive learning, in which test graphs remain unseen during training. In addition to these experiments, we verified whether our model uses information of remote nodes by reducing the percentage of labeled data. The experimental demonstrate the superior performance of GESM on inductive learning as well as transductive learning for datasets. Moreover, our model achieved enhanced accuracy for datasets with reduced label rates, indicating the contribution of global information. The key contributions of our approach are as follows: • We present graphs with step mixture via random walk, which can adaptively consider local and global information, and demonstrate its effectiveness through experiments on public benchmark datasets with few labels. • We propose Graph Entities with Step Mixture via random walk (GESM), an advanced model which incorporates attention, and experimentally show that it is applicable to both transductive and inductive learning tasks, for both nodes and edges are utilized for the neighborhood aggregation scheme. • We empirically demonstrate the importance of propagation steps by analyzing its effect on performance in terms of inference time and accuracy. Step-0 Step-1 Step- Figure 1: Random walk propagation procedure. From left to right are step-0, step-1, step-2, and stepinfinite. The values in each node indicate the distribution of a random walk. In the leftmost picture, only the starting node has a value of 100, and all other nodes are initialized to zero. As the number of steps increases, values spread throughout the graph and converge to some extent. Random walk, which is a widely used method in graph theory, mathematically models how node information propagates throughout the graph. As shown in Figure 1, random walk refers to randomly moving to neighbor nodes from the starting node in a graph. For a given graph, the transition matrix P, which describes the probabilities of transition, can be formulated as follows: where A denotes the adjacency matrix of the graph, and D the diagonal matrix with a degree of nodes. The probability of moving from one node to any of its neighbors is equal, and the sum of the probabilities of moving to a neighboring node adds up to one. Let u t be the distribution of the random walk at step t (u 0 represents the starting distribution). The t step random walk distribution is equal to multiplying P, the transition matrix, t times. In other words, The entries of the transition matrix are all positive numbers, and each column sums up to one, indicating that P is a matrix form of the Markov chain with steady-state. One of the eigenvalues is equal to 1, and its eigenvector is a steady-state . Therefore, even if the transition matrix is infinitely multiplied, convergence is guaranteed. The attention mechanism was introduced in sequence-to-sequence modeling to solve long-term dependency problems that occur in machine translation . The key idea of attention is allowing the model to learn and focus on what is important by examining features of the hidden layer. In the case of GNNs , GATs (Veličković et al., 2018) achieved stateof-the-art performance by using the attention mechanism. Because the attention mechanism considers the importance of each neighboring node, node features are given more emphasis than structural information (edges) during the propagation process. Consequently, using attention is advantageous for training and testing graphs with different node features but the same structures (edges). Given the many benefits of attention, we incorporate the attention mechanism to our model to fully utilize node information. The attention mechanism enables different importance values to be assigned to nodes of the same neighborhood, so combining attention with mixture-step random walk allows our model to adaptively highlight features with salient information in a global scope. Let G = (V, E) be a graph, where V and E denote the sets of nodes and edges, respectively. Nodes are represented as a feature matrix X ∈ R n×f, where n and f respectively denote the number of nodes and the input dimension per node. A label matrix is Y ∈ R n×c with the number of classes c, and a learnable weight matrix is denoted by W. The adjacency matrix of graph G is represented as A ∈ R n×n. The addition of self-loops to the adjacency matrix is A = A + I n, and the column normalized matrix of A isÂ = AD −1 withÂ 0 = I n. Most graph neural networks suffer from the oversmoothing issue along with localized aggregation. Although JK-Net tried to handle oversmoothing by utilizing GCN blocks with mulitple propagation, it could not completely resolve the issue as shown in Figure 4b. We therefore propose Graph Step Mixture (GSM), which not only separates the node embedding and propagation process but also tackles oversmoothing and localized aggregation issues through a mixture of random walk steps. GSM has a simple structure that is composed of three stages, as shown in Figure 2. Input X passes through a fully connected layer with a nonlinear activation. The output is then multiplied by a normalized adjacency matrixÂ for each random walk step that is to be considered. The for each step are concatenated and fed into another fully connected layer, giving the final output. The entire propagation process of GSM can be formulated as: where is the concatenation operation, s is the maximum number of steps considered for aggregation, andÂ k is the normalized adjacency matrixÂ multiplied k times. As can be seen from Equation 3, weights are shared across nodes. In our method, the adjacency matrixÂ is an asymmetric matrix, which is generated by random walks and flexible to arbitrary graphs. On the other hand, prior methods such as JK-Net and MixHop , use a symmetric Laplacian adjacency matrix, which limits graph structures to given fixed graphs. (a) Traditional global aggregation scheme Step-1 Step-2 Step-3 (b) Our step-mixture scheme For the concatenation operation, localized sub-graphs are concatenated with global graphs, which allows the neural network to adaptively select global and local information through learning (see Figure 3). While traditional graph convolution methods consider aggregated information within three steps by A(A(AXW )W )W, our method can take all previous aggregations into account To develop our base model which depends on edge information, we additionally adopt the attention mechanism so that node information is emphasized for aggregation, i.e., Graph Entity Step Mixture (GESM). We simply modify the nonlinear transformation of the first fully connected layer in GSM by replacing it with the attention mechanism denoted by H multi (see Equations 3 and 4). As described in Equation 4, we employ multi-head attention, where H multi is the concatenation of m attention layers and α is the coefficient of attention computed using concatenated features of nodes and its neighboring nodes. By incorporating attention to our base model, we can avoid or ignore noisy parts of the graph, providing a guide for random walk . Utilizing attention can also improve combinatorial generalization for inductive learning, where training and testing graphs are completely different. In particular, datasets with the same structure but different node information can benefit from our method because these datasets can only be distinguished by node information. Focusing on node features for aggregation can thus provide more reliable in inductive learning. The time complexity of our base model is O(s × l × h), where s is the maximum number of steps considered for aggregation, l is the number of non-zero entries in the adjacency matrix, and h is the hidden feature dimension. As suggested by , we can assume h << l under realistic assumptions. Our model complexity is, therefore, highly efficient with time complexity O(s × l), which is on par with vanilla GCN . Transductive learning. We utilize three benchmark datasets for node classification: Cora, Citeseer, and Pubmed . These three datasets are citation networks, in which the nodes represent documents and the edges correspond to citation links. The edge configuration is undirected, and the feature of each node consists of word representations of a document. Detailed statistics of the datasets are described in Table 1. For experiments on datasets with the public label rate, we follow the transductive experimental setup of. Although all of the nodes' feature vectors are accessible, only 20 node labels per class are used for training. Accordingly, 5.1% for Cora, 3.6% for Citeseer, and 0.3% for Pubmed can be learned. In addition to experiments with public label rate settings, we conducted experiments using datasets where labels were randomly split into a smaller set for training. To check whether our model can propagate node information to the entire graph, we reduced the label rate of Cora to 3% and 1%, Citeseer to 1% and 0.5%, Pubmed to 0.1%, and followed the experimental settings of for these datasets with low label rates. For all experiments, we report the using 1,000 test nodes and use 500 validation nodes. Inductive learning. We use the protein-protein interaction PPI dataset ,which is preprocessed by Veličković et al.. As detailed in Table 1, the PPI dataset consists of 24 different graphs, where 20 graphs are used for training, 2 for validation, and 2 for testing. The test set remains completely unobserved during training. Each node is multi-labeled with 121 labels and 50 features regarding gene sets and immunological signatures. For transductive learning, we compare our model with numbers of state-of-the-art models according to the reported in the corresponding papers. Our model is compared with baseline models specified in (Veličković et al., 2018) such as label propagation (LP) , graph embeddings via random walk (DeepWalk) , and Planetoid . We also compare our model with models that use self-supervised learning (Union), learnable graph convolution (LGCN) , GCN based multi-hop neighborhood mixing (JK-GCN and MixHop) , multi-scale graph convolutional networks (AdaLNet) and maximal entropy transition (PAN) . We further include models that utilize teleport term during propagation APPNP , conduct convolution via spectral filters such as ChebyNet , GCN , SGC (a), and GWNN and models that adopt attention between nodes, such as GAT (Veličković et al., 2018) and AGNN . For inductive learning tasks, we compare our model against four baseline models. This includes graphs that use sampling and aggregation (GraphSAGE-LSTM) , and jumping-knowledge (JK-LSTM) , along with GAT and LGCN which are used in the transductive setting. Regarding the hyperparameters of our transductive learning models, we used different settings for datasets with the public split and random split. We set the dropout probability such that 0.3 of the data were kept for the public split and 0.6 were kept for the random split. We set the number of multi-head m = 8 for GESM. The size of the hidden layer h ∈ {64, 512} and the maximum number of steps used for aggregation s ∈ {10, 30} were adjusted for each dataset. We trained for a maximum of 300 epochs with L2 regularization λ = 0.003 and learning rate lr = 0.001. We report the average classification accuracy of 20 runs. For inductive learning, the size of all hidden layers was the same with h = 256 for both GSM, which consisted of two fully connected layers at the beginning and GESM. We set the number of steps s = 10 for GSM, and s = 5, m = 15 for GESM. L2 regularization and dropout were not used for inductive learning (Veličković et al., 2018). We trained our models for a maximum of 2,000 epochs with learning rate lr = 0.008. The evaluation metric was the micro-F1 score, and we report the averaged of 10 runs. For all the models, the nonlinearity function of the first fully connected layer was an exponential linear unit (ELU) . Our models were initialized using Glorot initialization and were trained to minimize the cross-entropy loss using the Adam . We employed an early stopping strategy based on the loss and accuracy of the validation sets, with a patience of 100 epochs. Results on benchmark datasets. Table 2 summarizes the comparative evaluation experiments for transductive and inductive learning tasks. In general, not only are there a small number of methods that can perform on both transductive and inductive learning tasks, but the performance of such methods is not consistently high. Our methods, however, are ranked in the top-3 for every task, indicating that our method can be applied to any task with large predictive power. For transductive learning tasks, the experimental of our methods are higher than or equivalent to those of other methods. As can be identified from the table, our base model GSM, which is computationally efficient and simple, outperforms many existing baseline models. These indicate the significance of considering both global and local information and using random walks. It can also be observed that GESM yielded more impressive than GSM, suggesting the importance of considering node information in the aggregation process. For the inductive learning task, our base model GSM, which employs an edge-based aggregation method, does not invariably obtain the highest accuracy. However, our model with attention, GESM, significantly improves performance of GSM by learning the importance of neighborhood nodes, and surpasses the of GAT, despite the fact that GAT consists of more attention layers. These for unseen graphs are in good agreement with shown by Veličković et al., in which reducing the influence of structural information improved generalization. Results on datasets with low label rates. To demonstrate that our methods can consider global information, we experimented on sparse datasets with low label rates of transductive learning datasets. As indicated in Table 3, our models show remarkable performance even on the dataset with low label rates. In particular, we can further observe the superiority of our methods by inspecting Table 2 and 3, in which our methods trained on only 3% of the Cora dataset outperformed some other methods trained on 5.1% of the data. Because both GSM and GESM showed enhanced accuracy, it could be speculated that using a mixture of random walks played a key role in the experiments; the improved can be explained by our methods adaptively selecting node information from local and global neighborhoods, and allowing peripheral nodes to receive information. Oversmoothing and Accuracy. As shown in Figure 4a, GCN , SGC (a), and GAT (Veličković et al., 2018) suffer from oversmoothing. GCN and GAT show severe degradation in accuracy after the 8th step; The accuracy of SGC does not drop as much as GCN and GAT but nevertheless gradually decreases as the step size increases. The proposed GSM, unlike the others, maintains its performance without any degradation, because no rank loss occurs and oversmoothing is overcome by step mixture. Interestingly, JK-Net also keeps the training accuracy regardless of the step size by using GGN blocks with multiple steps according to Figure 4a. We further compared the test accuracy of GSM with JK-Net, a similar approach to our model, in regards to the step size. To investigate the adaptability to larger steps of GSM and JK-Net, we concatenated features after the 10th step. As shown in Figure 4b, GSM outperforms JK-Net, even though both methods use concatenation to alleviate the oversmoothing issue. These are in line with the fact that JK-Net obtains global information similar to GCN or GAT. Consequently, the larger the step, the more difficult it is for JK-Net to maintain performance. GSM, on the other hand, maintains a steady performance, which confirms that GSM does not collapse even for large step sizes. Test Accuracy GSM public (5.1%) 3% 1% GESM public (5.1%) 3% 1% We also observe the effect on accuracy as the number of steps increases under three labeling conditions for GSM and GESM. As represented in Figure 5, it is evident that considering remote nodes can contribute to the increase in accuracy. By taking into account more data within a larger neighborhood, our model can make reliable decisions, ing in improved performance. Inspection of the figure also indicates that the accuracy converges faster for datasets with higher label rates, presumably because a small number of walk steps can be used to explore the entire graph. Moreover, the addition of attention benefits performance in terms of higher accuracy and faster convergence. Inference time. As shown in Figure 6, the computational complexity of all models increases linearly as the step size increases. We can observe that the inference time of GSM is faster than GCN especially when the number of steps is large. The inference time of GESM is much faster than GAT (Veličković et al., 2018) while providing higher accuracies and stable (see Appendix A). Our methods are both fast and accurate due to the sophisticated design with a mixture of random walk steps. Embedding Visualization. Figure 7 visualizes the hidden features of Cora from our models by using Figure 7: t-SNE plot of the last hidden layer trained on the Cora dataset. the t-SNE algorithm . The figure illustrates the difference between edgebased and node-based aggregation. While the nodes are closely clustered in the from GSM, they are scattered in that of GESM. According to the in Table 2, more closely clustered GSM does not generally produce better than loosely clustered GESM, which supports findings that the attention mechanism aids models to ignore or avoid noisy information in graphs . Table 5: Average test set accuracy and standard deviation over 100 random train/validation/test splits with 20 runs. Top-3 for each column are highlighted in bold, and top-1 values are underlined. Coauthor CS Coauthor Physics Amazon Computers Amazon Photo MLP 88.3 ± 0.7 88.9 ± 1.1 44.9 ± 5.8 69.6 ± 3.8 LogReg 86.4 ± 0.9 86.7 ± 1.5 64.1 ± 5.7 73.0 ± 6.5 LP 73.6 ± 3.9 86.6 ± 2.0 70.8 ± 8.1 72.6 ± 11.1 GCN 91.1 ± 0.5 92.8 ± 1.0 82.6 ± 2.4 91.2 ± 1.2 GraphSAGE 91.3 ± 2.8 93.0 ± 0.8 82.4 ± 1.8 91.4 ± 1.3 GAT (Veličković et al., 2018) 90.5 ± 0.6 92.5 ± 0.9 78.0 ± 19.0 85.7 ± 20.3 GSM (our base model) 91.8 ± 0.4 93.3 ± 0.6 79.2 ± 2.0 89.3 ± 1.9 GESM (GSM+attention) 92.0 ± 0.5 93.7 ± 0.6 79.3 ± 1.7 90.0 ± 2.0 For an in-depth verification of overfitting, we extended our experiments to four types of new node classification datasets. Coauthor CS and Coauthor Physics are co-authorship graphs from the KDD Cup 2016 challenge 1, in which nodes are authors, features represent the article keyword for each author's paper, and class labels indicate each author's most active research areas. Amazon Computers and Amazon Photo are co-purchase graphs of Amazon, where nodes represent the items, and edges indicate that items have been purchased together. The node features are bag-of-words of product reviews, and class labels represent product categories. Detailed statistics of the datasets are described in Table 4 and we followed the experimental setup of. We used the same values for each hyperparameter (unified size: 64, step size: 15, multi-head for GAT and GESM: 8) without tuning. The in Table 5 prove that our proposed methods do not overfit to a particular dataset. Moreover, in comparison to GAT, the performance of GESM is more accurate, and more stable. We visualized the distribution of attention vectors. Figure 8a plots the distribution of neighbors with equal importance and Figure 8b displays the distribution of attention weighted neighbors that we trained with GESM. Although both figures look similar to some degree, we can conjecture that GESM slightly adjusts the weight values, contributing to improved performance."

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: represent the scientific article to find a short tldr

Question: Dialogue systems require a great deal of different but complementary expertise to assist, inform, and entertain humans. For example, different domains (e.g., restaurant reservation, train ticket booking) of goal-oriented dialogue systems can be viewed as different skills, and so does ordinary chatting abilities of chit-chat dialogue systems. In this paper, we propose to learn a dialogue system that independently parameterizes different dialogue skills, and learns to select and combine each of them through Attention over Parameters (AoP). The experimental show that this approach achieves competitive performance on a combined dataset of MultiWOZ , In-Car Assistant , and Persona-Chat . Finally, we demonstrate that each dialogue skill is effectively learned and can be combined with other skills to produce selective responses. Unlike humans who can do both, goal-oriented dialogues and chit-chat conversations (a;) are often learned with separate models. A more desirable approach for the users would be to have a single chat interface that can handle both casual talk and tasks such as reservation or scheduling. This can be formulated as a problem of learning different conversational skills across multiple domains. A skill can be either querying a database, generating daily conversational utterances, or interacting with users in a particular task-domain (e.g. booking a restaurant). One challenge of having multiple skills is that existing datasets either focus only on chit-chat or on goal-oriented dialogues. This is due to the fact that traditional goal-oriented systems are modularized (; ; ; ;); thus, they cannot be jointly trained with end-to-end architecture as in chit-chat. However, recently proposed end-to-end trainable models;; ) and datasets (; allow us to combine goal-oriented (; and chit-chat into a single benchmark dataset with multiple conversational skills as shown in Table 1. A straight forward solution would be to have a single model for all the conversational skills, which has shown to be effective to a certain extent by and . Putting aside the performance in the tasks, such fixed shared-parameter framework, without any task-specific designs, would lose controllability and interpretability in the response generation. In this paper, instead, we propose to model multiple conversational skills using the Mixture of Experts (MoE) paradigm, i.e., a model that learns and combine independent specialized experts using a gating function. For instance, each expert could specialize in different dialogues domains (e.g., Hotel, Train, ChitChat etc.) and skills (e.g., generate SQL query). A popular implementation of MoE ) uses a set of linear transformation (i.e., experts) in between two LSTM layers. However, several problems arise with this implementation: 1) the model is computationally expensive as it has to decode multiple times each expert and make the combination at the representation-level; 2) no prior knowledge is injected in the expert selection (e.g., domains); 3) Seq2Seq model has limited ability in extracting information from a Knowledge Base (KB) (i.e., generated by the SQL query), as required in end-to-end task-oriented dialogues Table 1: An example from the dataset which includes both chit-chat and task-oriented conversations. The model has to predict all the Sys turn, which includes SQL query and generating response from a the Memory content, which is dynamically updated with the queries . The skills are the prior knowledge needed for the response, where Persona refers to chit-chat. Spk. Conversation Skills Usr: Can you help me find a cheap 2 star hotel? SELECT * FROM hotel WHERE pricerange='cheap' AND stars=2 AND type='hotel' SQL+HOTEL Mem: (Result table from . The latter can be solved by using more advanced multi-hop models like the Transformer, but the remaining two need to be addressed. Hence, in this paper we: • propose a novel Transformer-based architecture called Attention over Parameters (AoP). This model parameterize the conversational skills of end-to-end dialogue systems with independent decoder parameters (experts), and learns how to dynamically select and combine the appropriate decoder parameter sets by leveraging prior knowledge from the data such as domains and skill types; • proof that AoP is algorithmically more efficient (Appendix A1) compared to forwarding all the Transformer decoders and then mix their output representation, like is normally done in MoE. Figure 1 illustrates the high-level intuition of the difference; • empirically show the effectiveness of using specialized parameters in a combined dataset of Multi-WOZ , In-Car Assistant, and Persona-Chat , which to the best of our knowledge, is the first evaluation of this genre i.e. end-to-end large-scale multi-domains/skills. Moreover, we show that our model is highly interpretable and is able to combine different learned skills to produce compositional responses. Dialogue Task-oriented dialogue models can be categorized in two types: modulebased (; ; ; ;) and end-to-end. In this paper, we focus on the latter which are systems that train a single model directly on text transcripts of dialogues. These tasks are tackled by selecting a set of predefined utterances (; ; ;) or by generating a sequence of tokens; b;; ). Especially in the latter, copy-augmented models;; ) are very effective since extracting entities from a knowledge base is fundamental. On the other hand, end-to-end open domain chit-chat models have been widely studied (a; ;). Several works improved on the initially reported baselines with various methodologies (; ; ; ; ; ;). Finally, was the first attempt of having an end-to-end system for both task-oriented models and chit-chat. However, the dataset used for the evaluation was small, evaluated only in single domain, and the chit-chat ability was added manually through rules. The idea of having specialized parameters, or so-called experts, has been widely studied topics in the last two decades . For instance, different architecture and methodologies have been used such as SVM , Gaussian Processes (; ;), Dirichlet Processes , Hierarchical Experts , Infinity Number of Experts , and sequential expert addition . More recently, the Mixture Of Expert model was proposed which added a large number of experts between two LSTMs . To the best of our knowledge, none of these previous works applied the of the gating function to the parameters itself. On the other hand, there are Conditional Computational models which learn to dynamically select their computation graph . Several methods have been used such as reinforcement learning , a halting function (; ;), by pruning and routing/controller function . However, this line of work focuses more on optimizing the inference performance of the model more than specializing parts of it for computing a certain task. Multi-task Learning Even though our model processes only input sequence and output sequences of text, it actually jointly learns multiple tasks (e.g. SQL and BOOK query, memory retrieval, and response generation), thus it is also related to multi-task learning . Interested readers may refer to for a general overview on the topic. In Natural Language Processing, multi-task learning has been applied in a wide range of applications such as parsing (; ;, machine translation in multiple languages , and parsing image captioning and machine translation . More interestingly, DecaNLP has a large set of tasks that are cast to question answering (QA), and learned by a single model. In this work, we focus more on conversational data, but in future works, we plan to include these QA tasks. We use the standard encoder-decoder architecture and avoid any task-specific designs; ), as we aim to build a generic conversation model for both chit-chat and task-oriented dialogues. More specifically, we use a Transformer for both encoder and decoder. Let us define the sequence of tokens in the dialogue history as D = {d 1, . . ., d m} and the dynamic memory content as a sequence of tokens M = {m 1, . . ., m z}. The latter can be the of a SQL query execution (e.g., table) or plain texts (e.g., persona description), depending on the task. The dialogue history D and the memory M are concatenated to obtain the final input denoted by We then denote Y = {y 1, . . ., y k} as the sequence of tokens that the model is expected to produce. Without loss of generality, Y can be both plain text and SQL-like queries. Hence, the model has to learn when to issue database queries and when to generate human-like responses. Finally, we define a binary skill vector V = {v 1, . . ., v r} that specifies the type of skills required to generate Y. This can be considered as a prior vector for learning to select the correct expert during the training 1. For example, in Table 1 the first response is of type SQL in the Hotel domain, thus the skill vector V will have v SQL = 1 and v Hotel = 1, while all the other skill/domains are set to zero 2. More importantly, we may set the vector V to have multiple ones to enforce the model to compose skills to achieve a semantic compositionality of different experts. To map the input sequence to the output sequence, we use a standard Transformer and denote the encoder and decoder as TRS enc and TRS dec, respectively. The input of a Transformer is the embedded representation of the input words; thus, we define a word embedding matrix E ∈ R d×|V | where d is the embedding size and |V | is the cardinality of the vocabulary. The input X, with its positional embedding (Appendix A2 for more information), are encoded as the following equation: where H ∈ R d model ×n, and E. Then the decoder receives the target sequence shifted by one Y:k−1 = {<SOS>, y 1, . . ., y k} as the input. Using teacher-forcing , the model is trained to produce the correct sequence Y. The output of the decoder is produced as follow: where O ∈ R d model ×k. Finally, a distribution over the vocabulary is generated for each token by an affine transformation W ∈ R d model ×|V | followed by a Softmax function. In addition, P (Y |X) is mixed with the encoder-decoder attention distribution to enable to copy token from the input sequence as in . The model is then trained to minimize a standard cross entropy loss function and at inference time to generate one token at the time in an auto-regressive manner . Hence, the training loss is defined as: The main idea is to produce a single set of parameters for decoder TRS dec by the weighted sum of r independently parameterized decoders. This process is similar to attention where the memories are the parameters and the query is the encoded representation. Let us define Θ = [θ 1, . . ., θ r] as the list of parameters for r decoders, since a TRS dec is represented by its parameters θ. Since each θ can be sized in the order of millions, we assign the corresponding key vectors to each θ, similar to key-value memory networks . Thus, we use a key matrix K ∈ R d model ×r and a Recurrent Neural Networks (RNN), in this instance a GRU , to produce the query vector by processing the encoder output H. The attention weights for each decoders' parameters is computed as follow: where q ∈ R d model and α ∈ R r is the attention vectors where each α i is the score corresponding to θ i. Hence, the new set of parameters is computed as follow: The combined set of parameters θ * are then used to initialize a new TRS dec, and Equation 2 will be applied to the input based on this. Equation 6 is similar to the gating function proposed 1 the vector V will be absent during the testing 2 With the assumption that at each index in V is assigned a semantic skill (e.g. SQL position i) in ), but the ing scoring vector α is applied directly to the parameter instead of the output representation of each decoder. This indeed makes the computation algorithmically faster since the forward computation is done only once and summing r elements of size |θ| is linear compared to forwarding r multiplications with the input. Interested readers may refer to Appendix A1 for the proof. Importantly, if we apply α to each of the output representation O i generated by the TRS i dec, we end up having a Transformer-based implementation of MoE 3. We call this model as Attention over Representation (AoR). Moreover, an additional loss term is used to supervise the attention vector α by using the prior knowledge vector V. Since multiple decoder parameters can be selected at the same time, we use a binary cross-entropy to train each α i. Thus a second loss is defined as: The final loss is the summation of L P (Y |X) and L V. Finally, in AoP, but in general in the MoE framework, stacking multiple layers (e.g., Transformer) leads to models with a large number of parameters, since multiple experts are repeated across layers. An elegant workaround is the Universal Transformer , which loops over an unique layer and, as shown by , holds similar or better performance than a multi-layer Transformer. In our experiment, we report a version of AoP that uses this architecture, which for instance does not add any further parameter to the model. To evaluate the performance of our model for different conversational skills, we propose to combine three publicly available datasets: MultiWOZ , Stanford Multi-domain Dialogue and Persona-Chat (MultiWOZ (MWOZ) is a human-to-human multi-domain goal-oriented dataset annotated with dialogue acts and states. In this dataset, there are seven domains (i.e., Taxi, Police, Restaurant, Hospital, Hotel, Attraction, Train) and two APIs interfaces: SQL and BOOK. The former is used to retrieve information about a certain domain and the latter is used to book restaurants, hotels, trains, and taxis. We refine this dataset to include SQL/BOOK queries and their outputs using the same annotations schema as . Hence, each response can either be plain text conversation with the user or SQL/BOOK queries, and the memory is dynamically populated with the from the queries as the generated response is based on such information. This transformation allows us to train end-to-end models that learns how and when to produce SQL queries, to retrieve knowledge from a dynamic memory, and to produce plain text response. A detailed explanation is reported in Appendix A4, together with some samples. Stanford Multi-domain Dialogue (SMD) is another human-to-human multi-domain goal-oriented dataset that is already designed for end-to-end training. There are three domains in this dataset (i.e., Point-of-Interest, Weather, Calendar). The difference between this dataset and MWOZ is that each dialogue is associated with a set of records relevant to the dialogues. The memory is fixed in this case so the model does not need to issue any API calls. However, retrieving the correct entities from the memory is more challenging as the model has to compare different alternatives among records. Persona-Chat is a multi-turn conversational dataset, in which two speakers are paired and different persona descriptions (4-5 sentences) are randomly assigned to each of them. For example, "I am an old man" and "I like to play football" are one of the possible persona descriptions provided to the system. Training models using this dataset in a more persona consistent and fluent conversation compared to other existing datasets . Currently, this dataset has become one of the standard benchmarks for chit-chat systems, thus, we include it in our evaluation. For all three datasets, we use the training/validation/test split provided by the authors and we keep all the real entities in input instead of using their delexicalized version as in . This makes the task more challenging, but at the same time more interesting since we force the model to produce real entities instead of generic and frequent placeholders. Goal-Oriented For both MWOZ and SMD, we follow the evaluation done by existing works;. We use BLEU 4 score to measure the response fluency and Entity F1-Score to evaluates the ability of the model to generate relevant entities from the dynamic memory. Since MWOZ also includes SQL and BOOK queries, we compute the exact match accuracy (i.e., ACC SQL and ACC BOOK) and BLEU score (i.e., BLEU SQL and BLEU BOOK). Furthermore, we also report the F1-score for each domain in both MWOZ and SMD. Chit-Chat We compare perplexity, BLEU score, F1-score , and Consistency score of the generate sentences with the human-generated prediction. The Consistency score is computed using a Natural Language Inference (NLI) model trained on dialogue NLI , a recently proposed corpus based on Persona dataset. We fine-tune a pre-trained BERT model using the dialogue DNLI corpus and achieve a test set accuracy of 88.43%, which is similar to the best-reported model in . The consistency score is defined as follow: where u is a generated utterance and p j is one sentence in the persona description. , the authors showed that by re-ranking the beam search hypothesis using the DNLI score (i.e., C score), they achieved a substantial improvement in dialogue consistency. Intuitively, having a higher consistency C score means having a more persona consistent dialogue response. In our experiments, we compare Sequence-to-Sequence (Seq2Seq) , Transformer (TRS), Mixture of Expert (MoE) and Attention over Representation (AoR) with our proposed Attention over Parameters (AoP). In all the models, we used the same copy-mechanism as in . In AoR instead of mixing the parameters as in Equation 7, we mix the output representation of each transformer decoder (i.e. Equation 2). For all AoP, AoR, and MoE, r = 13 is the number of decoders (experts): 2 skills of SQL and BOOK, 10 different domains for MWOZ+SMD, and 1 for Persona-Chat. Furthermore, we include also the following experiments: AoP that uses the gold attention vector V, which we refer as AoP w/ Oracle (or AoP + O); AoP trained by removing the L V from the optimization (AoP w/o L V); and as aforementioned, the Universal Transformer for both AoP (AoP + U) and the standard Transformer (TRS + U) (i.e., 6 hops). All detailed model description and the full set of hyper-parameters used in the experiments are reported in Appendix A5. Table 2 and Table 3 show the respectively evaluation in MWOZ+SMD and Persona-Chat datasets. From Table 2, we can identify four patterns. 1) AoP and AoR perform consistently better then other baselines which shows the effectiveness of combining parameters by using the correct prior V; 2) AoP performs consistently, but marginally, better than AoR, with the advantage of an algorithmic faster inference; 3) Using Oracle (AoP+O) gives the highest performance in all the measures, which shows the performance upper-bound for AoP. Hence, the performance gap when not using oracle attention is most likely due to the error in attention α (i.e., 2% error rate). Moreover, Table 2 shows that by removing L V (AoP w/o L V) the model performance decreases, which confirms that good inductive bias is important for learning how to select and combine different parameters (experts). Additionally, in Appendix A6, we report the per-domain F1-Score for SQL, BOOK and sentences, and Table 3 and Table 2 with the standard deviation among the three runs. Furthermore, from Table 3, we can notice that MoE has the lowest perplexity and F1-score, but AoP has the highest Consistency and BLUE score. Notice that the perplexity reported in is lower since the vocabulary used in their experiments is smaller. In general, the difference in performance among different models is marginal except for the Consistency score; thus, we can conclude that all the models can learn this skill reasonably well. Consistently with the previous , when L V is removed from the optimization, the models' performance decreases. Finally, in both Table 2 and Table 3, we report the obtained by using the Universal Transformer, for both AoP and the Transformer. By adding the layer recursion, both models are able to consistently improve all the evaluated measures, in both Persona-Chat and the Task-Oriented tasks. Especially AoP, which achieves better performance than Oracle (i.e. single layer) in the SQL accuracy, and a consistently better performance in the Persona-Chat evaluation. To demonstrate the effectiveness of our model in learning independent skills and composing them together, we manually trigger skills by modifying α and generate 14 different responses for the same input dialogue context. This experiment allows us to verify whether the model accurately captures the meaning of each skill and whether it can properly learn to compose the selected parameters (skills). Table 3 first shows the dialogue history along with the response of AoP on the top, and then different responses generated by modifying α (i.e., black cells correspond to 1 in the vector, while the whites are 0). By analyzing Table 3 5 we can notice that: • The model learns the correct semantics of each skill. For instance, the AoP response is of type SQL and Train, and by deactivating the SQL skill and activating other domain-skills, including Train, we can see that the responses are grammatical and they are coherent with the selected skill semantics. For instance, by just selecting Train, the generated answer becomes "what time would you like to leave?" which is coherent with the dialogue context since such information has not been yet provided. Interestingly, when Persona skill is selected, the generated response is conversational and also coherent with the dialogue, even though it is less fluent. • The model effectively learns how to compose multiple skills. For instance, when SQL or BOOK are triggered the response produces the correct SQL-syntax (e.g. "SELECT * FROM .." etc.). By also adding the corresponding domain-skill, the model generates the correct query format and attributes relative to the domain type (e.g. in SQL, Restaurant, the model queries with the relevant attribute food for restaurants). In this paper, we propose a novel way to train a single end-to-end dialogue model with multiple composable and interpretable skills. Unlike previous work, that mostly focused on the representationlevel mixing, our proposed approach, Attention over Parameters, learns how to softly combine independent sets of specialized parameters (i.e., making SQL-Query, conversing with consistent persona, etc.) into a single set of parameters. By doing so, we not only achieve compositionality and interpretability but also gain algorithmically faster inference speed. To train and evaluate our model, we organize a multi-domain task-oriented datasets into end-to-end trainable formats and combine it with a conversational dataset (i.e. Persona-Chat). Our model learns to consider each task and domain as a separate skill that can be composed with each other, or used independently, and we verify the effectiveness of the interpretability and compositionality with competitive experimental and thorough analysis. A.1 COMPUTATIONAL COST AOP Corollary A.0.1. The computation cost of Attention over Parameters (AoP) is always lower than Mixture Of Experts (MoE) as long as the processed sequence is longer than 1. Proof. Let f θ: R d → R n a generic function parametrized by θ. Without loss of generality, we define θ as a affine transformation W ∈ R d×n. Let X ∈ R t×d a generic input sequence of length t and d dimensional size. Let the set F = [f θ1, · · ·, f θr] be the set of r experts. Hence, the operation done by MoE are: 10) Thus the computational cost in term of operation is O(rtdn + rtn) since the cost of f θi (X) is O(tdn) and it is repeated r times, and the cost of summing the representation is O(rtn). On the other hand, the operation done by AoP are: in this case the computational cost in term of operation is O((r + t)dn) since the cost of summing the parameters is O(rdn) and the cost of f θ * is O(tdn). Hence, it is easy to verify that if t > 1 then: Furthermore, the assumption of using a simple affine transformation W is actually an optimal case. Indeed, assuming that the cost of parameters sum is equal to the number of operation is optimistic, for instance already by using attention the number of operations increases but the number of parameters remains constant. Since the model input may include structured data (e.g. DB records) we further define another embedding matrix for encoding the types and the segments as P ∈ R d×|S| where S is the set of positional tokens and |S| its cardinality. P is used to inform the model of the token types such as speaker information (e.g. Sys and Usr), the data-type for the memory content (e.g. Miles, Traffic etc.), and segment types like dialogue turn information and database record index . Figure 4 shows an example of the embedded representation of the input. Hence, we denote X T and X R as the type and segment tokens for each token in input X, respectively. Figure 5 shows the attention vector α over parameters for different generated sentences. In this figure, and by analyzing more examples 5, we can identify two patterns: • AoP learns to focus on the correct skills (i.e., SQL, BOOK) when API-calls are needed. From the first example in Figure 5, we can see that the activations in α are consistent with those in the correct attention vector P. There are also false positives, in which AoP puts too high weights on BOOK when the correct response is plain text that should request more information from the user (i.e., i can help you with that. when would you like to leave the hotel?). However, we can notice that this example is, in fact, "almost correct" as triggering a booking API call may also be considered a valid response. Meanwhile, the third example also fails to attend to the correct skill, but, in fact, generates a very fluent and relevant response. This is most likely because the answer is simple and generic. • The attention often focuses on multiple skills not directly relevant to the task. We observe this pattern especially when there are other skill-related entities mentioned in the context or the response. For example, in the second dialog example in Figure 5, we can notice that AoP not only accurately focuses on taxi domain, but also has non-negligible activations for restaurant and hotel. This is because the words "hotel" and "restaurant" are both mentioned in the dialogue context and the model has to produce two entities of the same type (i.e. finches bed and breakfast and ask). As mentioned in the main article, we convert MultiWOZ into an end-to-end trainable dataset. This requires to add sql-syntax queries when the system includes particular entities. To do so we leverage two annotations such as the state-tracker and the speech acts. The first is used to generate the a well-formed query, including key and attribute, the second instead to decide when to include the query. More details on the dialogue state-tracker slots and slots value, and the different speech acts can be found in . A query is create by the slots, and its values, that has been updated in the latest turn. The SQL query uses the following syntax: Similarly for the booking api BOOK the syntax is the following: In both cases the slot values are kept as real entities. More challenging is to decide when to issue such apis. Speech acts are used to decide by using the "INFORM-DOMAIN" and "RECOMMEND-DOMAIN" tag. Thus any response that include those speech tag will trigger an api if and only if: • there has been a change in the state-tracker from the previous turn • the produced query has never been issued before By a manual checking, this strategy to be effective. However, as reported by the speech act annotation includes some noise, which is reflected also into our dataset. The from the SQL query can be of more that 1K records with multiple attributes. Following we use the following strategy: • If no speech act INFORM or RECOMMEND and the number of records are more than 5, we use a special token in the memory < T M >. • If no speech act INFORM or RECOMMEND and the number of records are less or equal than 5, we put all the records in memory. • If any speech act INFORM or RECOMMEND, we filter the records to include based on the act value. Notice that this is a fair strategy, since all the ing record are correct possible answers and the annotators pick-up on of the record randomly . Notice that the answer of a booking call instead, is only one record containing the booking information (e.g. reference number, taxi plate etc.) or "Not Available" token in case the booking cannot made. We used a standard Transformer architecture with pre-trained Glove embedding . For the both Seq2Seq and MoE we use Adam optimizer with a learning rate of 1 × 10 −3, where instead for the Transformer we used a warm-up learning rate strategy as in. In both AoP and AoR we use an additional transformer layer on top the output of the model. Figure 6,7,8 shows the high level design MoE, AoR and AoP respectively. In all the model we used a batch size of 16, and we early stopped the model using the Validation set. All the experiments has been conducted using a single Nvidia 1080ti. We used a small grid-search for tuning each model. The selected hyper-parameters are reported in Table 4, and we run each experiment 3 times and report the mean and standard deviation of each . model consist of r feed-forward neural network (experts) which are embedded between two LSTM layers, a trainable gating network to select experts. Figure 7: Attention over Representation (AoR) consist of a transformer encoder which encode the source input and compute the attention over the skills. Then r transformer decoder layers computes r specialized representation and the output response is generated based on the weighted sum the representation. In the figure, we omitted the output layer. Figure 8: Attention over Parameters (AoP) consist of a transformer encoder which encode the source input and compute the attention over the skills. Then, r specialized transformer decoder layers and a dummy transformer decoder layer parameterized by the weighted sum of the r specialized transformer decoder layers parameters. In the figure, we omitted the output layer.

represent the short summary for finding the corresponding scientific article (long)

In this paper, we propose to learn a dialogue system that independently parameterizes different dialogue skills, and learns to select and combine each of them through Attention over Parameters (AoP).

Represent the short summary for finding the corresponding scientific article (long):

A modular approach consisting of a sentence selector module followed by the QA model can be made more robust to adversarial attacks in comparison to a QA model trained on full context.