Question: Represent the text to find a very short summary (1-2 sentences max):

Question: Teachers intentionally pick the most informative examples to show their students. However, if the teacher and student are neural networks, the examples that the teacher network learns to give, although effective at teaching the student, are typically uninterpretable. We show that training the student and teacher iteratively, rather than jointly, can produce interpretable teaching strategies. We evaluate interpretability by measuring the similarity of the teacher's emergent strategies to intuitive strategies in each domain and conducting human experiments to evaluate how effective the teacher's strategies are at teaching humans. We show that the teacher network learns to select or generate interpretable, pedagogical examples to teach rule-based, probabilistic, boolean, and hierarchical concepts. Human teachers give informative examples to help their students learn concepts faster and more accurately BID23 BID21 BID4. For example, suppose a teacher is trying to teach different types of animals to a student. To teach what a "dog" is they would not show the student only images of dalmatians. Instead, they would show different types of dogs, so the student generalizes the word "dog" to all types of dogs, rather than merely dalmatians. Teaching through examples can be seen as a form of communication between a teacher and a student. Recent work on learning emergent communication protocols in deep-learning based agents has been successful at solving a variety of tasks BID7 BID24 BID18 BID5 BID16. Unfortunately, the protocols learned by the agents are usually uninterpretable to humans, and thus at the moment have limited potential for communication with humans. We hypothesize that one reason the emergent protocols are uninterpretable is because the agents are typically optimized jointly. Consider how this would play out with a teacher network T that selects or generates examples to give to a student network S. If T and S are optimized jointly, then T and S essentially become an encoder and decoder that can learn any arbitrary encoding. T could encode "dog" through a picture of a giraffe and encode "siamese cat" through a picture of a hippo. The examples chosen by T, although effective at teaching S, are unintuitive since S does not learn in the way we expect. On the other hand, picking diverse dog images to communicate the concept of "dog" is an intuitive strategy because it is the effective way to teach given how we implicitly assume a student would interpret the examples. Thus, we believe that S having an interpretable learning strategy is key to the emergence of an interpretable teaching strategy. This raises the question of whether there is an alternative to jointly optimizing T and S, in which S maintains an interpretable learning strategy, and leads T to learn an interpretable teaching strategy. We would ideally like such an alternative to be domain-agnostic. Drawing on inspiration from the cognitive science work on rational pedagogy (see Section 2.1), we propose a simple change:1. Train S on random examples 2. Train T to pick examples for this fixed S Step 1, S learns an interpretable strategy that exploits a natural mapping between concepts and examples, which allows T to learn an interpretable teaching strategy in Step 2.1. Evaluating how similar T's strategy is to intuitive human-designed strategies (Section 4) 2. Evaluating how effective T's strategy is at teaching humans (Section 5)We find that, according to these metrics, T learns to give interpretable, pedagogical examples to teach rule-based, probabilistic, boolean, and hierarchical concepts. What does it mean to rationally teach and learn through examples? One suggestion is that a rational teacher chooses the examples that are most likely to make the student infer the correct concept. A rational student can then update their prior belief of the concept given the examples and the fact that the examples were chosen by a cooperative teacher. Shafto et al formalize this intuition in a recursive Bayesian model of human pedagogical reasoning BID21 BID22 BID23. In their model the probability a teacher selects an example e to teach a concept c is a soft maximization (with parameter α) over what the student's posterior probability of c will be. The student can then update their posterior accordingly. This leads to two recursive equations: DISPLAYFORM0 Note that in general there are many possible solutions to this set of dependent equations. A sufficient condition for a unique solution is an initial distribution for P teacher (e|c). Shafto et al suggest that a natural initial distribution for the teacher is a uniform distribution over examples consistent with the concept. They empirically show that the fixed point that from this initial distribution matches human teaching strategies. In our work, we initialize the teacher distribution in the way suggested by Shafto et al. We optimize in two steps: train the student on this initial distribution of examples optimize the teacher for this fixed student. This approach is analogous to doing one iteration of Equation 2 and then one iteration of Equation 1. We find that one iteration is sufficient for producing interpretable strategies. Teaching via examples can be seen as communication between a teacher to a student via examples. Much recent work has focused on learning emergent communication protocols in deep-learning based agents BID7 BID24. However, these emergent protocols tend to be uninterpretable. A number of techniques have been suggested to encourage interpretability, such as limiting symbol vocabulary size BID18, limiting memorization capabilities of the speaker, or introducing auxiliary tasks such as image labelling based on supervision data BID16.Despite these modifications, the protocols can still be difficult to interpret. Moreover, it is unclear how modifications like limiting vocabulary size apply when communication is in the form of examples because usually examples are already a fixed length (e.g coordinates in a plane) or constrained to be selected from a set of possible examples. So, there must be other reasons that humans come up with interpretable protocols in these settings, but neural networks do not. We suggest that one reason may be that these communication protocols are typically learned through joint optimization of all agents BID7 BID24 BID18 BID16, and evaluate how changing from a joint optimization to an iterative one can improve interpretability. Require: p(C): distribution over concepts DISPLAYFORM0 One problem studied in the machine teaching literature is finding a student-teacher pair such that the student can learn a set of concepts when given examples from the teacher BID12 BID3. However, it is difficult to formalize this problem in a way that avoids contrived solutions known as "coding tricks." Although the community has not agreed on a single definition of what a coding trick is, it refers to a solution in which the teacher and student simply "collude" on a pre-specified protocol for encoding the concept through examples. Many additional constraints to the problem have been proposed to try to rule out coding tricks. These additional constraints include requiring the student to be able to learn through any superset of the teacher's examples BID9, requiring the learned protocols to work for any ordering of the concepts or examples BID27, requiring the student to learn all concepts plus their images under primitive recursive operators BID19, and giving incompatible hypothesis spaces to the student and teacher BID1.The prior work has mainly been theoretically driven. The papers provide a definition for what it means to avoid collusion and then aim to find student-teacher pairs that provably satisfy the proposed definition. Our work takes a more experimental approach. We provide two criteria for interpretability and then empirically evaluate how modifying the optimization procedure affects these two criteria. We consider a set of possible concepts C and examples E. For example, C may be different animals like cats, dogs, parrots, etc and E may be images of those animals. The prior p(e|c) is a distribution over non-pedagogically selected examples of the concept. For example, if C is the set of all animals, then p(e|c) could be a uniform distribution over images of a given animal. A student S: E → C takes in a running sequence of K examples and at each step outputs a guessĉ for the concept the sequence of examples corresponds to. A teacher T: C × C → E takes in the target concept to teach and S's current guess of the concept and outputs the next example for the student at each step. When the set of examples is continuous T outputs the examples directly. When E is discrete we use the Gumbel-Softmax trick BID14 to have T generate a sample from E.The performance of both S and T is evaluated by a loss function L: C × C → R that takes in the true concept and S's output after K examples (although in some tasks we found it useful to sum the losses over all S's outputs). In our work, both S and T are modeled with deep recurrent neural networks parameterized by θ S and θ T, respectively. Recurrent memory allows the student and teacher to effectively operate over sequences of examples. T and S are illustrated graphically in FIG0.In the recent work on learning deep communication protocols, the standard way to optimize S and T would be to optimize them jointly, similar to the training procedure of an autoencoder (Algorithm 1). However, joint optimization allows S and T to form an arbitrary, uninterpretable encoding of Train student on random examples: DISPLAYFORM0 Train teacher best response to student: DISPLAYFORM1 the concept via examples. We compare joint optimization to an alternative approach we call a best response (BR) optimization (Algorithm 2), which iteratively trains S and T in two steps:1. Train S on concept examples e 1,... e K ∼ p(·|c) coming from prior example distribution. 2. Train T to select or generate examples for the fixed S from Step 1.The intuition behind separating the optimization into two steps is that if S learns an interpretable learning strategy in Step 1, then T will be forced to learn an interpretable teaching strategy in Step 2. 1 The reason we expect S to learn an "interpretable" strategy in Step 1 is that it allows S to learn a strategy that exploits the natural mapping between concepts and examples. For example, suppose the concept space is the set of all rectangles and p(e|c) is a uniform distribution over points within a rectangle (the task in Section 4.1). In Step 1, S learns to only guess rectangles that contain all the given examples. Because S expects examples to be within the rectangle, then in Step 2, T learns to only give examples that are within the rectangle, without explicitly being constrained to do so. So, T learns to picks the most informative examples that are still within the rectangle, which are the corners of the rectangle. We created a variety of tasks for evaluation that capture a range of different types of concepts (rulebased, probabilistic, boolean, and hierarchical concepts). Below we give a brief description of the tasks and why we chose them. The rest of the section provides further details on the tasks and the first interpretability criteria, while the next section addresses the second interpretability criteria. Rule-based concepts. We first aimed to replicate a common task in the rational pedagogy literature in cognitive science, known as the rectangle game BID21. In the variant of the rectangle game that we consider, there is a rectangle that is known to the teacher but unknown to the student. The student's goal is to infer the boundary of the rectangle from examples of points within the rectangle. The intuitive strategy that human teachers tend to use is to pick opposite corners of the rectangle BID22 BID23. We find that T learns to match this strategy. Probabilistic concepts. It is often difficult to define naturally-occurring concepts via rules. For example, it is unclear how to define what a bird is via logical rules. Moreover, some examples of a concept can seem more prototypical than others (e.g sparrow vs peacock) BID20, and this is not captured by simply modeling the concept as a set of rules that must be satisfied. An alternative approach models concept learning as estimating the probability density of the concept BID0 BID2 BID8 BID10. BID23 investigate teaching and learning unimodal distributions. But often a concept (e.g lamp) can have multiple subtypes (e.g. desk lamp and floor lamp). So, we investigate how T teaches a bimodal distribution. The bimodal distribution is parameterized as a mixture of two Gaussian distributions and S's goal is to learn the location of the modes. T learns the intuitive strategy of giving examples at the two modes. Boolean concepts. An object can have many properties, but only a few of them may be relevant for deciding whether the object belongs to a concept or not. For example, a circle is a circle whether it has a radius of 5 centimeters or 100 meters. The purpose of this task is to see what strategy T learns to quickly teach S which properties are relevant to a concept. The possible examples we consider are images that vary based on four properties: size (small, medium, large), color (red, blue, green), shape (square vs circle), and border (solid vs none). Only one to three of these properties define a concept. For example, if the concept is red circles, then red circles of any size or border fit the concept. T learns the intuitive strategy of picking two examples whose only common properties are the ones required by the concept, allowing S to learn that the other properties are not relevant for membership in the concept. Hierarchical concepts. Human-defined concepts are often hierarchical, e.g. animal taxonomies. Humans are sensitive to taxonomical structure when learning how to generalize to a concept from an example BID26. The purpose of this task is to test how T learns to teach when the concepts form a hierarchical structure. We create hierarchical concepts by pruning subtrees from Imagenet. T's goal is to teach S nodes from any level in the hierarchy, but can only give images from leaf nodes. T learns the intuitive strategy of picking two examples whose lowest common ancestor is the concept node, allowing S to generalize to the correct level in the hierarchy. A concept (rectangle) is encoded as a length four vector c ∈ [−10, 10] 4 of the minimum x, minimum y, maximum x, and maximum y of the rectangle. p(e|c) is a uniform distribution over points in the rectangle. Examples are two-dimensional vectors that encode the x and y coordinate of a point. The loss between the true concept c and S's output c is L(c,ĉ) = ||c −ĉ|| 2 2 and is only calculated on S's last output. S is first trained against ten examples generated from p(e|c). Then T is trained to teach S in two examples. T generates examples continuously as a two-dimensional vector. Figure 2 shows an example of T's choices and S's guess of the concept after each example given. Under both BR and joint optimization S is able to infer the concept in two examples. However, in joint optimization it is not clear how T's examples relate to the ground-truth rectangle (black) or what policy the student (orange) has for inferring the rectangle. On the other hand, in the BR case T outputs points close to opposite corners of the rectangle, and S expands its estimate of the rectangle to fit the examples the teacher gives. Figure 4 measures the distance between the random, best response (teacher), and joint strategy to the intuitive strategy of giving corners averaged over concepts. Specifically, let e = (e 1, e 2) be the two examples given and S(c) be the set of tuples of opposite corners of c. The distance measures how close these two examples are to a pair of opposite corners and is defined as d(e, c) = min s∈S(c) ||e 1 − s 1 || 2 + ||e 2 − s 2 || 2. T's examples are much closer to opposite corners than either the random or joint strategy. A concept is encoded as two-dimensional vector c = (µ 1, µ 2) ∈ 2 where µ 1 and µ 2 are the locations of the two modes and µ 1 < µ 2. p(e|c) = 0.5N (µ 1, 1) + 0.5N (µ 2, 1) is a mixture of two Gaussians. The loss between the true concept c and S's outputĉ is L(c,ĉ) = ||c −ĉ|| 2 2. S is first trained against five examples generated from p(e|c). Then T is trained to teach S in two examples. T generates examples continuously as a one-dimensional vector. T learns the intuitive strategy of giving the two modes as the examples. Figure 5 measures the distance to the intuitive strategy by the distance, ||e − c|| 2, between the examples, e, and the true modes, c. Both e and c are sorted when calculating the distance. T learns to match the intuitive strategy better than the random or joint strategy. Examples are images of size 25 x 25 x 3. Concepts are ten-dimensional binary vectors where each dimension represents a possible value of a property (size, color, shape, border). The value of one in the vector indicates that the relevant property (e.g. color) must take on that value (e.g. red) in order to be considered a part of the concept. p(e|c) is a uniform distribution over positive examples of the concept. The loss between the true concept c and S's outputĉ is L(c,ĉ) = ||c −ĉ|| 2 2. S is first trained on five examples generated from p(e|c). In both BR and joint optimization, we trained S with a curriculum starting with concepts defined by three properties, then two, and then one. T is trained to teach S with two examples. In this experiment, T selects an example from a discrete set of all images. We use the Gumbel-Softmax estimator BID14 to select discrete examples from final layer of T in a differentiable manner. T learns the intuitive strategy of picking two examples whose only common properties are the ones required by the concept, so that S can rule out the auxiliary properties. For example, Figure 7 shows T's examples for the concept of red. T selects a large red square with no border and then a small red circle with a border. The only property the two shapes have in common is red, so the concept must only consist of red. Indeed, 87% of T's examples only have the required properties in common, compared to 36% of random examples, and 0% of jointly trained examples (Figure 8). We create a set of hierarchical concepts by pruning a subtree from Imagenet. Each node in the subtree is a concept and is encoded as a one-hot vector. We randomly select 10 images of each leaf node. The possible examples for a leaf node are any of its ten images. The possible examples for an interior node are images from any of its descendant leaves. For example, in the hierarchy of apes shown in Figure 9, the possible examples for the "lesser apes" concept are images of siamangs or gibbons. We use a pretrained ResNet-50 model BID11 to embed each image into a 2048 length vector. p(e|c) is a uniform distribution over the possible examples for the concept. L(c,ĉ) is the softmax cross entropy loss between the true concept c and S's outputĉ. S is first trained on five examples generated from p(e|c). T then learns to teach S with two examples. As in 4.3, the final layer of T uses the Gumbel-Softmax estimator to sample an example image. T learns the intuitive strategy of picking examples from two leaf nodes such that the lowest common ancestor (LCA) of the leaf nodes is the concept node. This strategy encodes the intuition that to teach someone the concept "dog" you wouldn't only show them images of dalmations. Instead you would show examples of different types of dogs, so they generalize to a higher level in the taxonomy. For example, to teach what an ape is T could select an image of an orangutan and a siamang because the lowest common ancestor of the two is the ape concept (Figure 9). FIG0 shows T's correspondence to the intuitive strategy on the interior nodes of three example subtrees of Imagenet: apes, parrots, and felines. These subtrees have 16, 19, and 57 possible concepts respectively. T learns to follow the LCA strategy 100% of the time, whereas the highest the jointly trained strategy ever gets is 20%. In the previous section, we evaluated interpretability by measuring how similar T's strategy was to a qualitatively intuitive strategy for each task. In this section, we revisit two of the tasks and provide an additional measure of interpretability by evaluating how effective T's strategy is at teaching humans. We ran experiments to see how well T could teach humans the bimodal distributions task from Section 4.2. 60 subjects were recruited on Amazon Mehcanical Turk. They were tested on the ten concepts with modes in E = {4, 8, 12, 16, 20}. 30 subjects were shown two examples generated from p(e|c) for each concept and the other 30 subjects were shown two examples generated by T for each concept. The subjects were then given five test lines of lengths in E and asked to rate on a scale of one to five how likely they think the line is a part of the concept. For each concept there were two lines with very high probability of being in the concept and three lines with very low probability of being in the concept. A subject is said to have gotten the concept correct if they gave the high-probability lines a rating greater than three and the low-probability lines a rating less than or equal to three. The subjects given examples from the teacher had an average accuracy of 18%, compared to 8% with random examples. In addition, the teacher group had a much higher standard deviation than the random group, 19% compared to 6%. The maximum accuracy in the teacher group was 70%, but just 20% in the random group. The difference between groups was highly significant with p < 0.001, calculated using a likelihood-ratio test on an ordinary logit model as described in.Although the teacher group did better, neither group had a high mean accuracy. The task is difficult because a subject needs to get the entire distribution correct to be counted as a correct answer. But another possible reason for poor performance is people may have had the wrong hypothesis about the structure of concepts. It seems as though many subjects hypothesized that the structure of the concept space was unimodal, rather than bimodal, thus believing that lines with a length in between the two shown to them were very likely to be a part of the concept. An interesting open research question is how to ensure that the human has the correct model of the concept space. To evaluate human learning of boolean concepts (the task from Section 4.3), we sampled ten test concepts, five composed of one property and five composed of two properties. We recruited 80 subjects on Amazon Mechanical Turk and showed 40 of them two random positive examples of the ten concepts and the other 40 of them two examples chosen by the teacher. They were then asked to classify four new images as either a part of the concept or not. The four new images always had two positive examples and two negative examples for the concept. As shown in FIG0, the group that received examples from T performed better with a mean accuracy of 76%, compared to a mean accuracy of 71% for those that received random examples. This difference was highly significant with p < 0.001, calculated using the same procedure described in Section 5.1 from.What leads the protocols that humans learn to be so different from the protocols that deep learning models learn? One explanation is that humans have limitations that deep learning models do not. We investigated the impact of one limitation: humans cannot jointly optimize among themselves. We found that switching to an iterative optimization in which the student network is trained against examples coming from a non-pedagogical distribution and then the teacher network is trained against this fixed student leads to more interpretable teaching protocols. The intuition behind the approach is that leads the student to learn an interpretable learning strategy, which then constrains the teacher to learn an interpretable teaching strategy in.But this is just one of many possible limitations. For example, one reason we believe human students did not learn concepts as well as the student network (Section 5) is that humans had a different prior over concepts. In the probabilistic concepts task, humans seemed to believe that the lines came from a unimodal, rather than bimodal, distribution. In the boolean concepts task, humans tended to overemphasize color as a property. It is unrealistic to assume that a teacher and student have a perfectly matching prior over concepts or perfect models of each other. An important open question is which of these limitations are fundamental for the emergence of interpretable teaching protocols. While we carried out our experiments in the setting of teaching via examples, another direction for future work is investigating how an iterative optimization procedure works in more complex teaching settings (say teaching through demonstrations) and in communication tasks more broadly. Overall, we hope that our work presents a first step towards understanding the gap between the interpretability of machine agents and human agents.

We show that training a student and teacher network iteratively, rather than jointly, can produce emergent, interpretable teaching strategies.

Represent the TLDR for finding the corresponding scientific article.

We explore the problem of compositional generalization and propose a means for endowing neural network architectures with the ability to compose themselves to solve these problems.


Question: represent the text to find a very short summary (1-2 sentences max)

Question: Stochastic gradient descent (SGD), which trades off noisy gradient updates for computational efficiency, is the de-facto optimization algorithm to solve large-scale machine learning problems. SGD can make rapid learning progress by performing updates using subsampled training data, but the noisy updates also lead to slow asymptotic convergence. Several variance reduction algorithms, such as SVRG, introduce control variates to obtain a lower variance gradient estimate and faster convergence. Despite their appealing asymptotic guarantees, SVRG-like algorithms have not been widely adopted in deep learning. The traditional asymptotic analysis in stochastic optimization provides limited insight into training deep learning models under a fixed number of epochs. In this paper, we present a non-asymptotic analysis of SVRG under a noisy least squares regression problem. Our primary focus is to compare the exact loss of SVRG to that of SGD at each iteration t. We show that the learning dynamics of our regression model closely matches with that of neural networks on MNIST and CIFAR-10 for both the underparameterized and the overparameterized models. Our analysis and experimental suggest there is a trade-off between the computational cost and the convergence speed in underparametrized neural networks. SVRG outperforms SGD after a few epochs in this regime. However, SGD is shown to always outperform SVRG in the overparameterized regime. Many large-scale machine learning problems, especially in deep learning, are formulated as minimizing the sum of loss functions on millions of training examples . Computing exact gradient over the entire training set is intractable for these problems. Instead of using full batch gradients, the variants of stochastic gradient descent (SGD) (; ; ; ; ;) evaluate noisy gradient estimates from small mini-batches of randomly sampled training points at each iteration. The mini-batch size is often independent of the training set size, which allows SGD to immediately adapt the model parameters before going through the entire training set. Despite its simplicity, SGD works very well, even in the non-convex non-smooth deep learning problems . However, the optimization performance of the stochastic algorithm near local optima is significantly limited by the mini-batch sampling noise, controlled by the learning rate and the mini-batch size. The sampling variance and the slow convergence of SGD have been studied extensively in the past (; ;). To ensure convergence, machine learning practitioners have to either increase the mini-batch size or decrease the learning rate toward the end of the training . The minimum loss achieved in real dataset MNIST (a logistic regression model). Our theoretical prediction (a) matched with the training dynamics for real datasets, demonstrating tradeoffs between computational cost and convergence speed. The curves in red are SVRG and curves in blue are SGD. Different markers refer to different per-iteration computational cost, i.e., the number of backpropagation used per iteration on average. their strong theoretical guarantees, SVRG-like algorithms have seen limited success in training deep learning models . Traditional from stochastic optimization focus on the asymptotic analysis, but in practice, most of deep neural networks are only trained for hundreds of epochs due to the high computational cost. To address the gap between the asymptotic benefit of SVRG and the practical computational budget of training deep learning models, we provide a non-asymptotic study on the SVRG algorithms under a noisy least squares regression model. Although optimizing least squares regression is a basic problem, it has been shown to characterize the learning dynamics of many realistic deep learning models . Recent works suggest that neural network learning behaves very differently in the underparameterized regime vs the overparameterized regime ), characterized by whether the learnt model can achieve zero expected loss. We account for both training regimes in the analysis by assuming a linear target function and noisy labels. In the presence of label noise, the loss is lower bounded by the label variance. In the absence of the noise, the linear predictor can fit each training example perfectly. We summarize the main contributions as follows: • We show the exact expected loss of SVRG and SGD along an optimization trajectory as a function of iterations and computational cost. • Our non-asymptotic analysis provides an insightful comparison of SGD and SVRG by considering their computational cost and learning rate schedule. We discuss the trade-offs between the total computational cost, i.e. the total number of back-propagations performed, and convergence performance. • We consider two different training regimes with and without label noise. Under noisy labels, the analysis suggests SGD only outperforms SVRG under a mild total computational cost. However, SGD always exhibits a faster convergence compared to SVRG when there is no label noise. • Numerical experiments validate our theoretical predictions on both MNIST and CIFAR-10 using various neural network architectures. In particular, we found the comparison of the convergence speed of SGD to that of SVRG in underparameterized neural networks closely matches with our noisy least squares model prediction. Whereas, the effect of overparameterization is captured by the regression model without label noise. Stochastic variance reduction methods consider minimizing a finite-sum of a collection of functions using SGD. In case we use SGD to minimize these objective functions, the stochasticity comes from the randomness in sampling a function in each optimization step. Due to the induced noise, SGD can only converge using decaying step sizes with sub-linear convergence rate. Methods such as SAG , SVRG , and SAGA , are able to recover linear convergence rate of full-batch gradient descent with the asymptotic cost comparable to SGD. SAG and SAGA achieve this improvement at the substantial cost of storing the most recent gradient of each individual function. In contrast, SVRG spends extra computation at snapshot intervals by evaluating the full-batch gradient. Theoretical such as show that under certain smoothness conditions, we can use larger step sizes with stochastic variance reduction methods than is allowed for SGD and hence achieve even faster convergence. In situations where we know the smoothness constant of functions, there are on the optimal mini-batch size and the optimal step size given the inner loop size . Applying variance reduction methods in deep learning has been studied recently . The authors conjectured the ineffectiveness is caused by various elements commonly used in deep learning such as data augmentation, batch normalization and dropout. Such elements can potentially decrease the smoothness and make the stored gradients become stale quickly. The proposed solution is to either remove these elements or update the gradients more frequently than is practical. Dynamics of SGD and quadratic models Our main analysis tool is very closely related to recent work studying the dynamics of gradient-based stochastic methods. derived the dynamics of stochastic gradient descent with momentum on a noisy quadratic model , showing the problem of short horizon bias. In , the authors showed the same noisy quadratic model captures many of the essential characteristic of realistic neural networks training. Their noisy quadratic model successfully predicts the effectiveness of momentum, preconditioning and learning rate choices in training ResNets and Transformers. However, these previous quadratic models assume a constant variance in the gradient that is independent of the current parameters and the loss function. It makes them inadequate for analyzing the stochastic variance reduction methods, as SVRG can trivially achieve zero variance under the constant gradient noise. Instead, we adopted a noisy least-squares regression formulation by considering both the mini-batch sampling noise and the label noise. There are also recent works that derived the risk of SGD, for least-squares regression models using the bias-variance decomposition of the risk ). We use a similar decomposition in our analysis. In contrast to the asymptotic analysis in these works, we compare SGD to SVRG along the optimization trajectory for any finite-time horizon under limited computation cost, not just the convergence points of those algorithms. Underparameterization vs overparameterization. Many of the state-of-the-art deep learning models are overparameterized deep neural networks with more parameters than the number of training examples. Even though these models are able to overfit to the data, when trained using SGD, they generalize well . As suggested in recent work, underparameterized and overparameterized regimes have different behaviours; ). Given the infinite width and a proper weight initialization, the learning dynamics of a neural network can be well-approximated by a linear model via the neural tangent kernel (NTK) . In NTK regime, neural networks are known to achieve global convergence by memorizing every training example. On the other hand, previous convergence for SVRG have been obtained in stochastic convex optimization problems that are similar to that of an underparameterized model . Our proposed noisy least-squares regression analysis captures both the underparameterization and overparameterization behavior by considering the presence or the absence of the label noise. We will primarily focus on comparing the minibatch version of two methods, SGD and SVRG . Denote L i as the loss on i th data point. The SGD update is written as, is the minibatch gradient, t is the training iteration, and α (t) is the learning rate. The SVRG algorithm is an inner-outer loop algorithm proposed to reduce the variance of the gradient caused by the minibatch sampling. In the outer loop, for every T steps, we evaluate a large batch gradientḡ = 1 N N i ∇ θ (mT) L i, where N b, and m is the outer loop index, and we store the parameters θ (mT). In the inner loop, the update rule of the parameters is given by, whereĝ Note that in our analysis, the reference point is chosen to be the last iterate of previous outer loop θ (mT), recommended as a practical implementation of the algorithm by the original SVRG paper. We now define the noisy least squares regression model . In this setting, the input data is d-dimensional, and the output label is generated by a linear teacher model with additive noise, y. We assume WLOG θ * = 0. We also assume the data covariance matrix Σ is diagonal. This is an assumption adopted in many previous analysis and it is also a practical assumption as we often apply whitening to pre-process the training data. We would like to train a student model θ that minimizes the squared loss over the data distribution: At each iteration, the optimizer can query an arbitrary number of data points {x i, y i} i sampled from data distribution. The SGD method uses b data points to form a minibatch gradient: where SVRG on the other hand, queries for N data points every T steps to form a large batch gradient where X N and N are defined similarly. At each inner loop step, it further queries for another b data points, to form the update in Eq. 2. Lastly, note that the expected loss can be written as a function of the second moment of the iterate, Hence for the following analysis we mainly focus on deriving the dynamics of the second mo- When Σ is diagonal, the loss can further be reduced to Definition 1 (Formula for dynamics). We define the following functions and identities, The SGD update (Eq. 1) with the mini-batch gradient of of the noisy least squares model (Eq. 4) is, We substitute the update rule to derive the following dynamics for the second moment of the iterate: This dynamics equation can be understood intuitively as follows. The term 1 leads to an exponential shrinkage of the loss due to the gradient descent update. Since we are using a noisy gradient, the second term 2 represents the variance of stochastic gradient caused by the random input X b. The term 3 comes from the label noise. We show in the next theorem that when the second moment of the iterate approaches zero, 2 will also approach zero. However due to the presence of the label noise, the expected loss is lower bounded by 3. When Σ is diagonal, we further analyze and decompose C(M(θ)) as a function of m(θ) so as to derive the following dynamics and decay rate for SGD. Theorem 2 (SGD Dynamics and Decay Rate). Given the noisy linear regression objective function (Eq. 3), under the assumption that x ∼ N (0, Σ) with Σ diagonal and θ * = 0, we can express C(θ) as a function of m(θ): Then we derive following dynamics of expected second moment of θ: Under the update rule of SGD, R is the decay rate of the second moment of parameters between two iterations. And based on Theorem 2 the expected loss can be calculated by 3 A DILEMMA FOR SVRG By querying a large batch of datapoints X N every T steps, and a small minibatch X b at every step, the SVRG method forms the following update rule: To derive the dynamics of the second moment of the parameters following the SVRG update, we look at the dynamics of one round of inner loop updates, i.e., from θ (mT) to θ ((m+1)T ): Lemma 3. The dynamics of the second moment of the iterate following SVRG update rule is given by, The dynamics equation above is very illuminating as it explicitly manifests the weakness of SVRG. First notice that terms 1, 2, 3 reappear, contributed by the SGD update. The additional terms, 4 and 5, are due to the control variate. Observe that the variance reduction term 5 decays exponentially throughout the inner loop, with decay rate I − αΣ, i.e. P. We immediately notice that this is the same term that governs the decay rate of the term 1, and hence ing in a conflict between the two. Specifically, if we want to reduce the term 1 as fast as possible, we would prefer a small decay rate and a large learning rate, i.e. α → 1 λmax(Σ). But this will also make the boosts provided by the control variate diminish rapidly, leading to a poor variance reduction. The term 4 makes things even worse as it will maintain as a constant throughout the inner loop, contributing to an extra variance on top of the variance from standard SGD. On the other hand, if one chooses a small learning rate for the variance reduction to take effect, this inevitably will make the decay rate for term 1 smaller, ing in a slower convergence. Nevertheless, a good news for SVRG is that the label noise (term 3) is scaled by b N, which lets SVRG converge to a lower loss value than SGD -a strict advantage of SVRG compared to SGD. To summarize, the variance reduction from SVRG comes at a price of slower gradient descent shrinkage. In contrast, SVRG is able to converge to a lower loss value. This motivates the question, which algorithm to use given a certain computational cost? We hence performed a thorough investigation through numerical simulation as well as experiments on real datasets in Sec. 4. Similarly done for SGD, we decompose C(θ) as a function of m(θ) and derive the following decay rate for SVRG. Theorem 4 (SVRG Dynamics and Decay rate). Given the noisy linear regression objective function (Eq. 3), under the assumption that x ∼ N (0, Σ) with Σ diagonal and θ * = 0, the dynamics for SVRG in m(θ) is given by: In the absence of the label noise (i.e., σ y = 0), we observe that both SGD and SVRG enjoy linear convergence as a corollary of Theorem 2 and Theorem 4: Corollary 5. Without the label noise, the dynamics of the second moment following SGD is given by, and the dynamics of SVRG is given by, where λ is defined in Eq.. Note that similar have been shown in the past; ), where a general condition known as "interpolation regime" is used to show linear convergence of SGD. Specifically they assume that ∇L i (θ *) = 0 for all i, and our setting without label noise clearly also belongs to this regime. This setting also has practical implications, as one can treat training overparameterized neural networks as in interpolation regime. This motivates the investigation of the convergence rate of SGD and SVRG without label noise, and was also extensively studied in the experiments detailed as follows. In Sec. 3 we discussed a critical dilemma for SVRG that is facing a choice between effective variance reduction and faster gradient descent shrinkage. At the same time, it enjoys a strict advantage over SGD as it converges to a lower loss. We define the total computational cost as the total number of back-propagations performed. Similarly, per-iteration computational cost refers to the number of back-propagations performed per iteration. In this section, we study the question, which algorithm converges faster given certain total computational cost? We study this question for both the underparameterized and the overparameterized regimes. Our investigation consists of two parts. First, numerical simulations of the theoretical convergence rates (Sec. 4.1). Second, experiments on real datasets (Sec. 4.2). In both parts, we first fix the per-iteration computational cost. For SGD, the per-iteration computational budge is equal to the minibatch size. We picked three batch size {64, 128, 256}. Denote the batchsize of SGD as b, the equivalent batch size for SVRG is b = we also ran over a set of snapshot intervals T. After running over all sets of hyperparameters, we gather all training curves of all hyperparameters. We then summarize the performance for each algorithm by plotting the lower bound of all training curves, i.e. each point (l, t) on the curve showed the minimum loss l at time step t over all hyperparameters. We compared the two methods under different computational cost. Remarkably, we found in many cases phenomenon predicted by our theory matches with observations in practice. Our experiments suggested there is a trade-off between the computational cost and the convergence speed for underparameterized neural networks. SVRG outperformed SGD after a few epochs in this regime. Interestingly, in the case of overparameterized model, a setting that matches modern day neural networks training, SGD strictly dominated SVRG by showing a faster convergence throughout the entire training. We first performed numerical simulations of the dynamics derived in Theorem 2 for SGD and Theorem 4 for SVRG. We picked a data distribution, with data dimension d = 100, and the spectrum of Σ is given by an exponential decay schedule from 1 to 0.01. For both methods, we picked 50 learning rate from 1.5 to 0.01 using a exponential decay schedule. For SVRG, we further picked a set of snapshot intervals for each learning rate: {256, 128, 64}. We performed simulations in both underparameterized and overparameterized setting (namely with and without label noise), and plotted the lower bound curves over all hyperparameters at Figure 2. The x-axis represents the normalized total computational cost, denoting tbN −1, which is equivalent to the notion of an epoch in finite dataset setting. And the loss in Figure 2 does not contain bayes error (i.e. We have the following observations from our simulations. In the case with label noise, the plot demonstrated an explicit trade-off between computational cost and convergence speed. We observed a crossing point of between SGD and SVRG appear, indicating SGD achieved a faster convergence speed in the first phase of the training, but converged to a higher loss, for all per-iteration compute cost. Hence it shows that one can trade more compute cost for convergence speed by choosing SGD than SVRG, and vice versa. Interestingly, we found that the the per-iteration computational cost does not seem to affect the time crossing point takes place. For all these three costs, the crossing points in the plot are at around the same time: 5.5 epochs. In the case of no label noise, we observed both methods achieved linear convergence, while SGD achieved a much faster rate than SVRG, showing absolute dominance in this regime. In this section, we performed a similar investigation as in the last section, on two standard machine learning benchmark datasets: MNIST and CIFAR-10 . We present the from underparameterized setting first, followed by the overparameterized setting. We performed experiments with three batchsizes for SGD: {64, 128, 256}, and an equivalent batchsize for SVRG. For each batch size, we pick 8 learning rates varying from 0.3 to 0.001 following an exponential schedule. Additionally, we chose three snapshot intervals for every computational budget, searching over the best snapshot interval given the data. Hence for each per-iteration computational cost {64, 128, 256}, there are 24 groups of experiments for SVRG and 8 groups of experiments for SGD. for training on MNIST and CIFAR-10 with overparameterized models. In this setting we observed strict dominance of SGD over SVRG in convergence speed for all computational cost, matching our previous theoretical prediction. For MNIST, we trained two underparameterized model: 1. logistic regression 784 − 10 2. a underparameterized two layer MLP 784 − 10 − 10 where the hidden layer has 10 neurons. For CIFAR-10, we chose a underparameterized convolutional neural network model, which has only two 8-channel convolutional layers, and one 16-channel convolutional layer with one additional fully-connected layer. Filter size is 5. The lowest loss achieved over all hyperparameters for these models for each per-iteration computational cost are shown in Figure 3. From these experiments, we observe that on MNIST, the with underparameterized model were consistent with the dynamics simulation of noisy least squares regression model with label noise. First of all, SGD converged faster in the early phase, ing in a crossing point between SGD and SVRG. It showed a trade-offs between computational cost and convergence speed: before the crossing point, SGD converged faster than SVRG; after crossing point, SVRG attained a lower loss. In addition, in Fig 3a, all the crossing points of three costs matched at the same epoch (around 5), which was also consistent with the our findings with noisy least squares regression model. On CIFAR-10, SGD achieved slightly faster convergence in the early phase, but was surpassed by SVRG around 17 − 25 epochs, again showing a trade-off between compute and speed. Lastly, we compared SGD and SVRG on MNIST and CIFAR-10 using overparameterized models. For MNIST, we used a MLP with two hidden layers, each layer having 1024 neurons. For CIFAR-10, we chose a large convolutional network, which has one 64-channel convolutional layer, one 128-channel convolutional layer followed by one 3200 to 1000 fully connected layer and one 1000 to 10 fully connected layer. The lowest loss achieved over all hyperparameters for these models for each per-iteration computational cost are shown in Figure 4. For training on MNIST, both SGD and SVRG attained close to zero training loss. The were again consistent with our dynamics analysis on the noisy linear regression model without label noise. SGD has a strict advantage over SVRG, and achieved a much faster convergence rate than SVRG throughout the entire training. As for CIFAR-10, we stopped the training before either of the two got close to zero training loss due to lack of computing time. But we clearly see a trend of approaching to zero loss. Similarly, we also had the same observations as before, where SGD outperforms SVRG, confirms the limitation of variance reduction in the overparameterized regime. In this paper, we studied the convergence properties of SGD and SVRG in the underparameterized and overparameterized settings. We provided a non-asymptotic analysis of both algorithms. We then investigated the question about which algorithm to use under certain total computational cost. We performed numerical simulations of dynamics equations for both methods, as well as extensive experiments on the standard machine learning datasets, MNIST and CIFAR-10. Remarkably, we found in many cases phenomenon predicted by our theory matched with observations in practice. Our experiments suggested there is a trade-off between the computational cost and the convergence speed for underparameterized neural networks. SVRG outperformed SGD after the first few epochs in this regime. In the case of overparameterized model, a setting that matches with modern day neural networks training, SGD strictly dominated SVRG by showing a faster convergence for all computational cost. Lemma 6 (Gradient Covariance). Given the noisy linear regression objective function (Eq. 3), under the assumption that x ∼ N (0, Σ) with Σ diagonal and θ * = 0, we have Proof. In the following proof, we define the entry-wise p power on vector x as x •p. Under our assumption µ = 0, θ Eq. 12 is a from The Matrix Cookbook (See section 8.2.3 in). Then, for its main diagonal term, we have: Hence, for C M(θ (t) ), we have: which is the first of Theorem 2. Notice, this can be generalized to any square matrix A not only for E[θ (t) θ (t) ], i.e. for any square matrix A ∈ R d×d, with x ∼ N (0, Σ) and Σ diagonal, since we have For batch gradient where [N] b is the index set of X b. Theorem 2. Given the noisy linear regression objective function (Eq. 3), under the assumption that x ∼ N (0, Σ) with Σ diagonal and θ * = 0, we can express C(M(θ)) as a function of m(θ): Then we derive following dynamics of expected second moment of θ: Proof. Since, E[b] = 0, and b is independent with θ (t), X b, we have: and, Since X b is independent with θ (t), we have: Thus, For its diagonal term, we have: This formula can be written as: where D THE PROOF OF LEMMA 3 Lemma 3. The dynamics of the second moment of the iterate following SVRG update rule is given by, 4 variance due tog 5 Variance reduction from control variate Proof. For SVRG update rule Eq. 8, we have: Using the update rule of SVRG, we can get the outer product of parameters as: Likewise, since E[N] = 0 and N is independent with X b, X N and θ (t), we have the expectation of equation 46, equation 47 equal to 0. And same as SGD, we also have Then, we give a significant formula about the expectation of θ (mT +t) θ (mT), utilized to derive the expected term related to variance reduction amount. and N is independent with X N and θ (mT), the expectation of Eq. 53 is equal to 0. Therefore, which suggests the covariance betweenĝ (mT +t) andg (mT +t) is exponentially decayed. For every other term appearing in Eq. 41, we have the following . First, similar with SGD, we have the formula about gradient descent shrinkage as: Using Eq. 54, we have following for variance reduction term from control variate. We first take expectation over θ (mT +t) θ (mT) with Eq. 54 due to the independence among X b, X N and θ. For the forth term, which represents the variance ofg (mT +t), we consider the independence between X b and X N and get Thus, Under our definition, it can be expressed as: 4 variance due tog 5 Variance reduction from control variate E THE PROOF OF THEOREM 4 Theorem 4. Given the noisy linear regression objective function (Eq. 3), under the assumption that x ∼ N (0, Σ) with Σ diagonal and θ * = 0, the dynamics for SVRG in m(θ) is given by: Proof. Form lemma 3 and lemma 6, we can get: where Recursively expending the above formula from m(θ ((m+1)T ) ) to m(θ (mT) ), we can get the following : = R Rm(θ (mT +T −2) ) − QP T −2 m(θ (mT) ) + F m(θ (mT) ) + N −1 n In other word, Eq. 79 describe the dynamic of expected second moment of iterate between two nearby snapshots, m(θ ((m+1)T ) ) = λ(α, b, T, N, Σ)m(θ (mT) ) + (I − R T)(I − R) In our theoretical analysis (Section 3), we evaluate a large batch gradientḡ to control variance. That is because any number of data points can be directly sampled form the true distribution. But in order to compare the computational cost between SVRG and SGD, we set the number of data points used to calculateḡ as N, which is slightly different with the original SVRG's setup of full-batch gradient. Therefore, we evaluate the sensitivity of N to illustrate when N is beyond a threshold, it will cause little difference in convergence speed for SVRG. From figure 5a, we can tell N has little effect on the convergence speed of SVRG under the noisy least square model, but it determines the constant term of label noise in Eq. 9 which determines the level of final loss. Besides, we also compare large batch SGD to SVRG in Figure 5b

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Non-asymptotic analysis of SGD and SVRG, showing the strength of each algorithm in convergence speed and computational cost, in both under-parametrized and over-parametrized settings.

Represent the short summary for finding the corresponding scientific article (long):

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the text to find a very short summary (1-2 sentences max)!

Question: Non-autoregressive machine translation (NAT) systems predict a sequence of output tokens in parallel, achieving substantial improvements in generation speed compared to autoregressive models. Existing NAT models usually rely on the technique of knowledge distillation, which creates the training data from a pretrained autoregressive model for better performance. Knowledge distillation is empirically useful, leading to large gains in accuracy for NAT models, but the reason for this success has, as of yet, been unclear. In this paper, we first design systematic experiments to investigate why knowledge distillation is crucial to NAT training. We find that knowledge distillation can reduce the complexity of data sets and help NAT to model the variations in the output data. Furthermore, a strong correlation is observed between the capacity of an NAT model and the optimal complexity of the distilled data for the best translation quality. Based on these findings, we further propose several approaches that can alter the complexity of data sets to improve the performance of NAT models. We achieve the state-of-the-art performance for the NAT-based models, and close the gap with the autoregressive baseline on WMT14 En-De benchmark. Traditional neural machine translation (NMT) systems (; ;) generate sequences in an autoregressive fashion; each target token is predicted step-by-step by conditioning on the previous generated tokens in a monotonic (e.g. left-to-right) order. While such autoregressive translation (AT) models have proven successful, the sequential dependence of decisions precludes taking full advantage of parallelism afforded by modern hardware (e.g. GPUs) at inference time. On the other hand, there is a recent trend of non-autoregressive translation (NAT) models (;, trading the model's capacity for decoding efficiency by making it possible predict the whole sequence or multi-token chunks of the sequence simultaneously. Such a non-autoregressive factorization assumes that the output tokens are independent from each other. However, this assumption obviously does not hold in reality and as a NAT models generally perform worse than standard AT models. One key ingredient to reducing the performance degradation of NAT models that is used in almost all existing works ( ; ; , inter alia) is creation of training data through knowledge distillation . More precisely, sequence-level knowledge distillation -a special variant of the original approach -is applied during NAT model training by replacing the target side of training samples with the outputs from a pre-trained AT model trained on the same corpus with a roughly equal number of parameters. It is usually assumed that knowledge distillation's reduction of the "modes" (alternative translations for an input) in the training data is the key reason why distillation benefits NAT training. However, this intuition has not been rigorously tested, leading to three important open questions: • Exactly how does distillation reduce the "modes", and how we could we measure this reduction quantitatively? Why does this reduction consistently improve NAT models? • What is the relationship between the NAT model (student) and the AT model (teacher)? Are different varieties of distilled data better for different NAT models? • Due to distillation, the performance of NAT models is largely bounded by the choice of AT teacher. Is there a way to further close the performance gap with standard AT models? In this paper, we aim to answer the three questions above, improving understanding of knowledge distillation through empirical analysis over a variety of AT and NAT models. Specifically, our contributions are as follows: • We first visualize explicitly on a synthetic dataset how modes are reduced by distillation (§3.1). Inspired by the synthetic experiments, we further propose metrics for measuring complexity and faithfulness for a given training set. Specifically, our metrics are the conditional entropy and KL-divergence of word translation based on an external alignment tool, and we show these are correlated with NAT model performance (§3.2). • We conduct a systematic analysis (§4) over four AT teacher models and six NAT student models with various architectures on the standard WMT14 English-German translation benchmark. These experiments find a strong correlation between the capacity of an NAT model and the optimal dataset complexity for the best translation quality. • Inspired by these observations, we propose approaches to further adjust the complexity of the distilled data in order to match the model's capacity (§5). We also show that we can achieve the state-of-the-art performance for NAT and largely match the performance of the AT model. In order to model the joint probability of the output sequence y, NMT models usually generate each output token conditioned on the previous generated ones p(y|x) = T t=1 p(y t |y <t, x), known as the autoregressive factorization. To generate a translation from this model, one could predict one token at a time from left to right and greedily take arg max over each output probability distribution. Besides greedy decoding, beam search is often employed to generate better translations by considering a fixed number of hypotheses. In this work, we study non-autoregressive translation (NAT), a special subset of NMT models with an additional restriction (zeroth-order Markov assumption) upon the output predictions or a subset thereof. The simplest formulation of an NAT model independently factors the conditional distribution: p(y|x) = T t=1 p(y t |x). Standard NAT models adopt a similar architecture as the Transformer and make non-autoregressive predictions for the entire sequence with one forward pass of the decoder. However, because multiple translations are possible for a single input sentence (the so-called multi-modality problem;), vanilla NAT models can fail to capture the dependencies between output tokens and tend to make egregious mistakes such as outputting tokens repeatedly. To improve the model's ability to handle multi-modality, recent works have incorporated approaches including relaxing the fully non-autoregressive restriction and adopting K decoding passes (instead of just one) to iteratively refine the generated outputs;;; ); using latent variables (; ;) or structured information such as syntax trees to capture translation variation; training NAT models with objectives other than maximum likelihood; ) which ameliorates the effects of multi-modality. However, to achieve competitive performance with the autoregressive model, almost all existing NAT models rely on training using data distilled from a pre-trained AT model instead of the real parallel training set, as described below. Knowledge distillation was originally proposed for training a weaker student classifier on the targets predicted from a stronger teacher model. A typical approach is using the label probabilities produced by the teacher as "soft targets" q i = exp(z i /τ) j exp(z j /τ) for training the student model, where q i and z i are the probability and the logit of class i respectively and τ is the temperature. Prior work has shown the effectiveness of adopting knowledge distillation in adversarial defense , neural network compression , and fast inference of speech synthesis . In the context of sequence generation, extend knowledge distillation to the sentence level using "hard targets" from a pretrained large teacher model to train a small sequence generation model. More precisely, the teacher distribution q(t|x) is approximated by its mode: q(t|x) ≈ 1{t = arg max t∈T q(t|x)} with the following objectives: where t ∈ T is the space of possible target sequences. This can also be seen as a special case of the standard distillation over the sentence space when the temperature τ approaches 0, which is equivalent to taking the arg max over all feasible translations. While the "hard target"ŷ is the most likely translation predicted by the teacher, in practice we use beam search as an approximation. As mentioned earlier, almost all the existing literature trains NAT models using sequence-level knowledge distillation from a pre-trained AT model to achieve competitive performance. Particularly, it is common to train the teacher model as a standard Transformer with a roughly equal number of trainable parameters as the desired NAT model on the real data. In this paper, we will first study how this knowledge distillation process affects the behavior of NAT models. In this section, we start from an introductory example to illustrate how NAT models fail to capture the multi-modality of data. Then we propose a metric to assess the multi-modality of a data set and use that to test our hypothesis about the mechanism of knowledge distillation for NAT models. Dataset. We start by investigating NAT's difficulties in modeling multi-modality in output data using a synthetic setup where we explicitly include multiple modes in the training data. More specifically, we utilize three language pairs -English-German (En-De), English-French (En-Fr), and English-Spanish (En-Es) -from the Europarl parallel corpus. 2 We extract sentences that have aligned sentences for all languages, and create a multi-reference En-De/Es/Fr corpus. In this case every English input sentence always corresponds to target sentences in three different languages, which forms three explicit output modes. Notably, this is similar to the one-to-many translation setting in but in our case we do not have an explicit signal (e.g. target language tag) to tell the NMT model which target language to translate to. Models. We train both the AT and NAT models on this concatenated data set, then compare the distributions of translations with each other. We use the standard Transformer(base) model as the AT model, and a simplified version of as the NAT model where the decoder's inputs are monotonically copied from the encoder embeddings and a length predictor is learned to predict the target sentence length. Both models are trained for 300, 000 steps using maximum likelihood. After training, we use both models to translate the English sentences in the validation and test sets. Visualization of AT Outputs. The synthetic setup enables us to better understand and visualize the modes in the outputs more easily. First, we visualize the outputs from the AT model. For every translated sentence, we visualize the estimated probability distribution of language classes as a point in Fig. 1 (a). This probability is calculated as the average of the posterior probability of each token, and it is estimated based on the Bayes' law: Figure 1: Posterior distribution of language IDs for the outputs from different models. Each translation is represented as a point inside the simplex ∆ 2 = {(p de, p es, p fr)|p k ∈, p de +p es +p fr = 1} where p k is the estimated probability of being translated into language k ∈ (de, es, fr). We distinguish the language that has the largest probability with different colors. where l i denotes the language class i, and p(y t |l i) is the token frequency of y t in language l i. We assume p(l i) follows a uniform distribution. As shown in Fig. 1 (a), points of the AT outputs are clustered closely to each vertex of the simplex which indicates that the AT model prefers to generate the whole sequence in one language. This phenomenon verifies our assumption that decoding from the AT model (distillation) is essentially selecting "modes" over the real data. Visualization of NAT Outputs. We visualize outputs for the NAT model trained on the same data in Fig. 1 (b). In contrast to the AT , the NAT points are scattered broadly inside the simplex, indicating the NAT model fails to capture the mode of language types. Instead, it predicts tokens mixed with multiple languages, which corroborates our hypothesis about the NAT model. Next, we create two datasets that have fewer modes than the original dataset. First, we randomly select a single target sentence from one of the three languages for each source sentence. Second, we perform distillation, decoding from the AT model trained on the combined training set. As noted in the AT , distillation will also roughly be selecting a language mode, but we conjecture that this selection may be more systematic, selecting a particular language for a particular type of training sentence. As shown in Fig. 1 (c) (d), NAT models trained on both of these datasets are more likely to choose one mode (language) when generating translations, showing that training with reduced modes is essential for NAT model. Furthermore, points in Fig. 1 (d) are clearly clustered better than (c) indicating that modes selected by AT models are indeed likely more systematic and easy to capture than those generated by randomly assigning a language for each sentence. To better study why distillation is crucial for NAT models, in this section, we propose quantitative measures for analyzing the complexity and faithfulness of parallel data, two properties that we hypothesize are important for NAT training. Measure of Complexity. Inspired by the observations in the synthetic experiments, we propose to use a measure of translation uncertainty, specifically operationalized as conditional entropy, as the measurement of complexity C(d) for any given dataset d: Under review as a conference paper at ICLR 2020 where we use x and y to denote a word in the source and target vocabulary respectively. T x and T y denote the length of the source and target sentences. To make the computation tractable, we make two additional assumptions on the conditional distribution p(y|x): • Assumption 1: We assume the target tokens are independent given the source sentence. Then the conditional entropy of a sentence can be converted into the sum of entropy of target words conditioned on the source sentence x. • Assumption 2: We assume the distribution of p(y t |x) follows an alignment model 3 where y t is is generated from the word alignment distribution p(y t |Align(y t)). This makes it possible to simplify the conditional entropy to the sum of entropy of target words conditioned on the aligned source words. The corpus level complexity C(d) is then calculated by adding up the conditional entropy H(Y|X = x) of all sentences and averaging over all source tokens, denoted To illustrate that the proposed metric is a reasonable measure of complexity of a parallel corpus, in Tab. 1 we compute C(d) for the parallel data of different language pairs, the concatenated data set and the distilled data from the AT model described in §3.1. We observe that the conditional entropy of the distilled data is much smaller than that of the original concatenated data as well as the random-selection data mentioned above. Additionally, we find that the conditional entropy of En-Es and En-Fr are similar but that of En-De is relatively larger, which can also explain why the student NAT model prefers to predict the modes of Es or Fr more often than De as shown in Fig. 1(d). Measure of Faithfulness. C(d) reflects the level of multi-modality of a parallel corpus, and we have shown that a simpler data set is favorable to an NAT model. However, it is not fair to assess the data set only by its complexity, e.g. we can trivially construct a simple data set with no variations in the output, which obviously won't be useful for training. The other important measurement of the data set is its faithfulness to the real data distribution. To measure the faithfulness of a parallel corpus d, we use KL-divergence of the alignment distribution between the real parallel data set r and an altered parallel data set d, denoted F (d): In this section, we perform an extensive study over a variety of non-autoregressive (NAT) models trained from different autoregressive (AT) teacher models, to assess how knowledge distillation affects the performance of NAT models. Data. We use the data set commonly used by prior work as our evaluation benchmark: WMT14 English-German (En-De) 4. We use newstest2013 as the validation set for selecting the best model, and newstest2014 as the test set. We learn a byte-pair encoding vocabulary of 37,000 on the tokenized data. AT Models. We set up four Transformer models with different parameter sizes: Transformertiny/small/base/big denoted as tiny, small, base, big respectively. We build base and big models following settings described in , and reduce the model sizes for tiny, small to create weaker teacher models. Details of the model architectures can be found in Appendix A. All the models are trained using the Adam optimizer with the maximum number of steps set to 300, 000. After training, we use the ing AT models to decode the whole training set with beam size 5 and replace the real target sentences to create a new parallel corpus. NAT Models. We consider the following NAT models, from vanilla to state-of-the-art. All the models are using the Transformer as the basic backbone and are (re-)implemented based on Fairseq 5 except for FlowSeq. We briefly outline the methods and parameters here, and describe detailed settings in the Appendix A. • Vanilla NAT : Similarly to §3.1, we use a simplified version where the decoder's inputs are directly copied from the encoder without considering latent variables. • FlowSeq : FlowSeq adopts normalizing flows as the latent variables to model the mappings from source sentences to a latent space. • NAT with Iterative Refinement (iNAT, : iNAT extends the vanilla NAT by iteratively reading and refining the translation. The number of iterations is set to 10 for decoding. • Insertion Transformer : InsT adopts a similar architecture as iNAT while generating the sequence by parallel insertion operations. Here, we only consider InsT trained with uniform loss as described in the original paper. • MaskPredict : MaskT adopts a masked language model to progressively generate the sequence from an entirely masked input. The number of iterations is set to be 10. • Levenshtein Transformer : LevT uses similar architectures as in InsT and MaskT while generating based on both insertion and deletion operations. We experiment with a base and big LevT model (LevT and LevT-big in Tab. 2). We also summarize the parameter size, performance and relative decoding speed of the NAT models introduced in Tab Iters is number of passes used in decoding for output length n and hyperparameter k. Pass is relative time used for one pass of decoding. As mentioned earlier, we analyze each model by training from both the real and 4 distillation targets. We train the NAT models for the same number of steps as the AT models. For a fair comparison of the actual ability of each NAT-based model, we test all the models based on "greedy decoding" without any advanced search algorithms (e.g. length beam , noisy parallel decoding , or re-ranking from the teacher model ). Notably, the vanilla NAT and FlowSeq output translations with single forward pass, while the remaining models are based on the iterative refinement. We compare different dimensions of the data generated by the four AT models and the real data set in Fig. 3. First, Fig. 3 (a) shows that as the capacity of the AT model increases, the complexity C(d) of the distilled data increases, which indicates that the multi-modality increases as well. At the same time, we observe that F (d) defined in §3.2 also decreases, showing that the distilled data more faithfully represents the word-level translation distribution of the original data. Second, we plot the BLEU score of the distilled data w.r.t to the real data set in (b) and we observe that the BLEU score of the distilled data from a higher-capacity teacher model is higher, which is both intuitive and in agreement with the on KL divergence. We also investigate how the relative ordering of words in the source and target sentences is changed during distillation. We use the fuzzy reordering score proposed in. A larger fuzzy reordering score indicates the more monotonic alignments. As shown in Fig 3 (c), the distilled data has significantly less reordering compared to the real parallel sentences, and the distilled data from a weaker AT teacher is more monotonic than a stronger AT teacher. We also show a randomly sampled example in Fig. 2 where compared to the real translation, the AT distilled target is much more monotonically aligned to the source sentence. This has potential benefits in that these simpler reordering patterns may be easier to learn for NAT models, but also disadvantages in that it may prevent NAT models from learning complex reordering patterns. In §4.2, we have shown that decoding with an AT model reduces the conditional entropy of the parallel data set, which mitigates multi-modality in the output data. But does the decoding method of the AT model affect this change in the data set? We also investigate different decoding strategies when creating distilled data, using the base Transformer model as the teacher and the vanilla NAT model as the student. In Tab. 3, four decoding methods are presented: sampling, sampling within the top-10 candidates, beam search, and greedy decoding. With the same AT model, the performance of the NAT model differs widely depending on the decoding approach, where distillation with beam search in the best performance. We can see that beam search or greedy decoding can reduce the complexity of the real data the most while maintaining high faithfulness. In contrast, sampling based decoding methods less aggressively reduce the modes in the output sequence. This finding is in concert with , who demonstrate that because beam search approximately selects the most probable translation, it effectively reduces diversity in the output translations compared to sampling or the true distribution. We next examine the relationship between the NAT students and distilled training data from different AT models. In Fig. 4, we demonstrate for the NAT models listed in §4.1. We use the test set performance on real data as a simple metric to measure the capacity of the NAT model and arrange the subfigures in an increasing order of the performance (left-to-right, top-to-bottom Transformer LevT LevT-big Figure 4: The performance of NAT models of varying capacity trained on both the real and the distilled data from tiny, small, base and big AT models on WMT14-ENDE newstest 2014 test sets. smaller complexity as measured above in §4.2. The best performance of NAT models -from lower capacity ones to higher capacity ones -is achieved with distilled data of lower complexity to higher complexity, e.g. the vanilla NAT model performs best when using the distilled data from a small Transformer whereas the LevT achieves the best performance when training with the distilled data from a big Transformer. Third, and notably, by simply changing the distilled data set upon which the models are trained, we are able to significantly improve the state-of-the-art for models in a particular class. For example, FlowSeq increased to 22, by simply changing from the distilled data of Transformer(base) to Transformer(small). Finally, we find that by distilled from a big-AT model, LevT is able to close the gap with the Transformer (base) with a similar number of parameters. Both LevT and LevT-big achieve the state-of-the-art performance for NAT-based models. The previous section shows that the optimal complexity of the dataset is highly correlated with the capacity of the NAT model. In this section, we introduce three techniques that can be used to alter the distilled data to match the capacity of NAT model. Specifically, these techniques can be used to simplify the data further (BANs, MoE) for a lower-capacity student model or increase faithfulness of the data set (Interpolation) for a higher-capacity student model. Born-Again Networks. We apply Born-Again neworks (BANs) to create a simplified dataset for NAT models. BANs were originally proposed as a self-distillation technique that uses the output distribution of a trained model to train the original model. Starting from the real data, we repeatedly train new AT models with decoded sentences from the AT model at the previous iteration. This process is repeated for k times and yields k distilled data sets, upon which we perform NAT training and examine how the k born-again teachers affect the performance of NAT students. We conduct experiments using the vanilla NAT model (which achieved the best performance with distilled data from a small Transformer in §4.4) and the base Transformer as the AT model. As shown in Fig. 5, we can make the following observations: (i) The performance of the base AT model almost remains unchanged during the reborn iterations. (ii) The performance of the vanilla NAT model can be improved by 2 BLEU when using the distilled data from reborn iteration 6. (iii) As the reborn iterations continue, the complexity of the distilled data decreases and becomes constant eventually. Meanwhile, the quality of the distilled data compared to the real data decreases. Mixture-of-Experts. The mixture-of-expert model (MoE;) learns different experts for diverse machine translation, and different mixture components were shown to capture con- In Fig. 6, we observe that the performance of the best expert of MoE tends to decrease as the number of experts increases. However, the complexity (C(d)) and faithfulness (F (D)) of distilled data from different MoE models has a relatively large variance. Compared to using the distilled data from a plain base AT model, the performance of NAT model is improved by 1.21 BLEU when using the distilled data from the MoE model with the number of experts of 3 which produces the distilled data with least complexity. Table 4: Results w/ and w/o sequencelevel interpolation with LevT. Sequence-Level Interpolation. §4.4 shows stronger NAT models (e.g. MaskT, LevT) have the ability to learn from the dataset that is closer to the real data, and achieve better performance. We adopt the sequence-level interpolation proposed in as a natural way to create a better dataset. Different from distillation, interpolation picks the sentence with the highest sentence-level BLEU score w.r.t. the ground truth from K−best beam search hypotheses. In our experiments, we first run beam search using the base Transformer model with a beam size of 5 then select the sentences with the highest BLEU score from the top-3 candidates. Tab. 4 compares the performance of LevT trained with distilled data from the AT model with the standard distillation or interpolation. We observe that selection with BLEU score from the base AT model (base-inter) improves the performance of LevT ∼ 0.4 BLEU while the dataset complexity C(d) does not increase much. In this paper, we first systematically examine why knowledge distillation improves the performance of NAT models. We conducted extensive experiments with autoregressive teacher models of different capacity and a wide range of NAT models. Furthermore, we defined metrics that can quantitatively measure the complexity of a parallel data set. Empirically, we find that a higher-capacity NAT model requires a more complex distilled data to achieve better performance. Accordingly, we propose several techniques that can adjust the complexity of a data set to match the capacity of an NAT model for better performance. A EXPERIMENTAL DETAILS Model All the AT models are implemented based on the Transformer model using fairseq, and we basically follow the fairseq examples to train the transformers 6. Following the notation from , we list the basic parameters of all the AT model we used: Models Table 5: Basic hyper-parameters of architecture for AT models. Training For all experiments, we adopt the Adam optimizer using β 1 = 0.9, β 2 = 0.98, = 1e − 8. The learning rate is scheduled using inverse sqrt with a maximum learning rate 0.0005 and 4000 warmup steps. We set the label smoothing as 0.1. All the models are run on 8 GPUs for 300, 000 updates with an effective batch size of 32, 000 tokens. The best model is selected based on the validation loss except for FlowSeq which uses valid BLEU score. Decoding After training, we use beam-search with a fixed beam size 5 for all AT models to create the distilled dataset. No length penalty is used. Model Tab. 2 also lists all the NAT models we test in this work. In general, all the NAT models except FlowSeq and LevT-big adopts a similar architecture and hyper-parameters as the Transformerbase (see Tab. 5). LevT-big is a naive extension of the original LevT model with a comparable parameter setting as Transformer-big (Tab. 5). For FlowSeq, we use the base model (FlowSeq-base) described in . We re-implemented the vanilla NAT as a simplified version of where instead of modeling fertility as described in the original paper, we monotonically copy the encoder embeddings to the input of the decoder. All the models except InsT require the additional module to predict the length of the output sequence, or the number of placeholders to be inserted, which is implemented as a standard softmax classifier over the lengths of. For LevT, we also have a binary classifier to predict the deletion of the incorrect tokens. Training Similar to the AT models, all the NAT models are trained using the Adam optimizer with the same learning rate scheduler, in which the warmup steps are set to 10, 000. We train the FlowSeq model on 32 GPUs with a batch size as 2048 sentences, while all the other models are trained on 8 GPUs with an effective batch size of 64, 000 tokens. Note that, the batch sizes for training NAT is typically larger than the AT model to make the learning sufficient. There are also specialized training settings for each models: • iNAT: following the original paper, we train the iNAT model jointly with 4 iterations of refinement during training. For each iteration, the model has the 50% probability to learn as a denoising autoencoder, and the rest of the probability to learn from the model's own prediction. • InsT : in this work, we only consider training the Insertion Transformer (InsT) using the slot-loss based on the uniform loss function . That is, we assign equal probabilities to all the insertable tokens inside each slot. • MaskT : following the original paper, we train the model as a typical masked language model where the ratio of masked tokens is sampled from 0 ∼ 100%. • LevT : in this work, we only consider sequence generation tasks, which means the training of LevT is very similar to InsT. We use sentences with randomly deleted tokens to learn insertion, and learn deletion based on the model's own prediction. Decoding For a fair comparison over all the NAT models, we use greedy decoding for all the model without considering any advance decoding methods such as searching or re-ranking from a teacher model. For the vanilla NAT and FlowSeq, decoding is quite straight-forward and simply picking the arg max at every position. For iNAT and MaskT, we fix the decoding steps as 10. Both InsT and LevT decode in an adaptive number of iterations, and we set the maximum iterations for both models as 10. A special EOS penalty that penalizes generating too short sequences is tuned based on the validation set for both InsT and LevT. For all models, final are calculated as tokenized BLEU score. The detailed dataset split for WMT14 En-De is shown in Tab. 6. In Fig. 7, we also plot the histogram of the conditional entropy of each pair of sentences H(y|x) in the real parallel data and different distilled data sets from the big-AT, base-AT, small-AT and tiny-AT respectively. It shows that the distribution of the sentence-level conditional entropy differs widely. The mode of H(y|x) in the real data is the highest and follows by distilled data from the big-AT, base-AT, small-AT and tiny-AT. This observation aligns with the complexity value C(d) proposed in §3.2. WMT'14 En-De 4,500,966 3000 3003 37,009 In Figure 8, we also showed with different metrics together with BLEU scores considering BLEU score sometimes cannot capture the change of the system. We considered 5 additional metrics in our experiments: METEOR , RIBES , ChrF (Popović, 2015) TER , and BEER . Not surprisingly, we find that all the metrics are correlated with the original BLEU scores quite well showing the similar trend as discussed earlier. Bayesian decision theory is a fundamental statistical approach to the problem of pattern classification, which provides a principled rule of finding the optimal classification decision using probability and losses that accompany such decisions. In the problem of structured prediction, let x denote the input sequence and y denote the output label sequence. Let H denote all the possible hypothesis functions from the input to the output space: H = {h : X → Y}. Let r(y|x) denote the conditional risk on the input x, which is the expected loss of predicting y based on the posterior probabilities:, where L(y, y) is the loss function that penalizes predicting the true target y as y. The classification task aims to find a hypothesis function h that minimizes the overall risk R given by This is known as the Bayes risk. To minimize the overall risk, obviously we need to minimize the conditional risk for each input x. The Bayesian decision rule states that the global minimum of R(h) is achieved when the classifier make predictions that minimize each conditional risk given x and this gives the Bayes optimal classifier: Let us consider two loss functions defined in Eq. 5. First is the sequence-level loss L seq (y, y) = 1 − I(y = y), then in this case the Bayes classifier is:, which is the most probable output label sequence given the input sequence x. Second let us consider the token-level loss L tok (y, y) = T t=1 1 − I(y t = y t), i.e the sum of zero-one loss at each time step. We have: This suggests that the Bayes classifier finds the most probable label at each time step given the input sequence. To study how training data affects the performance of a weaker classifier, we construct a Hidden Markov Model (HMM) by sampling the parameters of the transition and emission probabilities uniformly within (0, a] and (0, b] respectively. A higher value of a and b indicates an HMM model with higher uncertainty. We refer this HMM as the "true HMM" as our real data generator. Next we consider a weaker classifier that uses a low-dimension bidirectional-LSTM (Bi-LSTM) to encode the input sequence and individual softmax functions at each time step to predict labels independently, which is referred as the "Bi-LSTM" classifier. Obviously, the Bi-LSTM classifier is not able to model the dependencies between output labels embedded in the HMM, and it is equivalent to a simplified non-autoregressive generation model. We generate the real training data D real = {(x 1, y 1), · · ·, (x N, y N)} of size N by sampling from the joint probability of the true HMM. Similarly we sample N test data points as the test data and N valid data points as the validation data. We evaluate the classifier's token-level accuracy tacc and sequence-level accuracy sacc on the test data respectively, where tacc =. These two metrics correspond to the token-level loss L tok and sequence-level loss L seq on each data point of the test data. First, we use h * seq (x) to generate the distillation labels y from the true HMM, which corresponds to applying the Viterbi decoding to each x i in D real. The training data set D seq is created with (x i, y i). Next, we use h * tok (x) to generate the distillation labelsŷ and create the training data D tok of (x i,ŷ i). To generateŷ, we apply the forward-backward algorithm to each x i in D real and obtain P (y t i |x i). We take arg max over the label space L:ŷ t i = arg max We use these three training data (D real, D tok, D seq) to train the Bi-LSTM classifier respectively. We repeat the experiment for 50 times by constructing 50 HMM models with different random seeds as the data generator. We find that when evaluating with the token-level accuracy tacc, models trained with D tok yields the best performance (Bi-LSTM trained with D tok win 97.6% runs); when evaluating with the sequence-level accuracy sacc, models trained with D seq yields the best performance (Bi-LSTM trained with D seq win 98.5% runs). This is because the Bi-LSTM classifier has difficulty modeling the true data distribution defined by an HMM. On the other hand, it is easier for the Bi-LSTM classifier to model the distributions of D seq and D tok. Data sets D seq and D tok define deterministic conditional distributions over the input data, which are much simpler than the real data distribution. By definition, D tok is created by the optimal Bayes classifier h * tok (x), this means that the Bi-LSTM classifier trained with D tok can better capture the distribution of P (y t |x) = max ut P (u t |x), which can generalize better to the test data when evaluated with the token-level accuracy. Similarly, Bi-LSTM trained with D seq performs better on the test data with the sequence-level metric. This corroborates our observation in machine translation task that NAT has difficulty in modeling the real conditional distribution of true sentence pairs. However, when using the distilled data translated from a pretrained autoregressive model with beam-search decoding, it performs better on the test set when evaluated with the BLEU score metric.

Represent the short summary for finding the corresponding scientific article (long)!

We systematically examine why knowledge distillation is crucial to the training of non-autoregressive translation (NAT) models, and propose methods to further improve the distilled data to best match the capacity of an NAT model.

We introduce and analyze the phenomenon of "hallucinations" in NMT, or spurious translations unrelated to source text, and propose methods to reduce its frequency.