Question: Represent the scientific article to find a short TLDR

Question: Dropout is a simple yet effective technique to improve generalization performance and prevent overfitting in deep neural networks (DNNs). In this paper, we discuss three novel observations about dropout to better understand the generalization of DNNs with rectified linear unit (ReLU) activations: 1) dropout is a smoothing technique that encourages each local linear model of a DNN to be trained on data points from nearby regions; 2) a constant dropout rate can in effective neural-deactivation rates that are significantly different for layers with different fractions of activated neurons; and 3) the rescaling factor of dropout causes an inconsistency to occur between the normalization during training and testing conditions when batch normalization is also used. The above leads to three simple but nontrivial improvements to dropout ing in our proposed method "Jumpout." Jumpout samples the dropout rate using a monotone decreasing distribution (such as the right part of a truncated Gaussian), so the local linear model at each data point is trained, with high probability, to work better for data points from nearby than from more distant regions. Instead of tuning a dropout rate for each layer and applying it to all samples, jumpout moreover adaptively normalizes the dropout rate at each layer and every training sample/batch, so the effective dropout rate applied to the activated neurons are kept the same. Moreover, we rescale the outputs of jumpout for a better trade-off that keeps both the variance and mean of neurons more consistent between training and test phases, which mitigates the incompatibility between dropout and batch normalization. Compared to the original dropout, jumpout shows significantly improved performance on CIFAR10, CIFAR100, Fashion- MNIST, STL10, SVHN, ImageNet-1k, etc., while introducing negligible additional memory and computation costs. Deep learning has achieved remarkable success on a variety of machine learning tasks BID15 BID14. Deep neural networks (DNN), however, are often able to fit the training data perfectly -this can in the overfitting problem, thereby weakening the generalization performance on unseen data. Dropout BID17 BID7 is a simple yet effective technique to mitigate such problems by randomly setting the activations of hidden neurons to 0, a strategy that reduces co-adaptation amongst neurons. Dropout applies to any layer in a DNN without causing significant additional computational overhead. Dropout, however, has several drawbacks. Firstly, dropout rates, constituting extra hyper-parameters at each layer, need to be tuned to get optimal performance. Too high a dropout rate can slow the convergence rate of the model, and often hurt final performance. Too low a rate yields few or no improvements on generalization performance. Ideally, dropout rates should be tuned separately for each layer and also during various training stages. In practice, to reduce computation, we often tune a single dropout rate and keep it constant for all dropout layers and throughout the training process. If we treat dropout as a type of perturbation on each training sample, it acts to generalize the DNN to noisy samples having that specific expected amount of perturbation (due to the fixed dropout rate) with high probability. The fixed rate rules out samples typical having less perturbation, i.e., those potentially more likely to be closer to the original samples and thus that are potentially more helpful to improve generalization. Also, when a constant dropout rate is applied to layers and samples having different fractions of activated neurons, the effective dropout rate (i.e., the proportion of the activated neurons that are deactivated by dropout) varies, which might in too much perturbation for some layers and samples and too little perturbation for others. Another deficiency of dropout lies in its incompatibility with batch normalization (BN) BID8 (more empirical evidence of this is shown in Section 3.3). As dropout randomly shuts down activated neurons, it needs to rescale the undropped neurons to match the original overall activation gain of the layer. Unfortunately, such rescaling breaks the consistency of the normalization parameters required between training and test phases 1 and may cause poor behavior when used with BN. Since BN, and its variants BID0 BID18 BID20, has become an almost indispensable component of modern DNN architectures to keep the training stable and to accelerate convergence, dropout itself often gets dropped out in the choice between these two non-complementary options and has recently become less popular. We propose three simple modifications to dropout in order to overcome the drawbacks mentioned above. These modifications lead to an improved version of dropout we call "jumpout." Our approach is motivated by three observations about how dropout in improved generalization performance for DNNs with rectified linear unit (ReLU) activations, which covers a frequently used class of DNNs. Firstly, we note that any DNN with ReLU is a piecewise linear function which applies different linear models to data points from different polyhedra defined by the ReLU activation patterns. Based on this observation, applying dropout to a training sample randomly changes its ReLU activation patterns and hence the underlying polyhedral structure and corresponding linear models. This means that each linear model is trained not only to produce correct predictions for data points in its associated polyhedron, but also is trained to work for data points in nearby polyhedra; what precisely "nearby" means depends on the dropout rate used. This partially explains why dropout improves generalization performance. The problem, however, is that with a fixed dropout rate, say p, and on a layer with n units, the typical number of units dropped out is np as that is the mode of a Binomial distribution with parameter p. It is relatively rare that either very few (closer to zero) or very many (closer to n) units are dropped out. Thus, with high probability, each linear model is smoothed to work on data points in polyhedra at a typical distance away. The probability of smoothing over closer distances is potentially much smaller, thus not achieving the goal of local smoothness. In jumpout, by contrast, p rather than being fixed is itself a random variable; we sample p from a distribution that is monotone decreasing (e.g., a truncated half-Gaussian). This achieves the property that Pr(i units dropping out) ≥ Pr(i+1 units dropping out) for all i ∈ {1, 2, . . ., n}. That is, a smaller dropout rate has a higher probability of being chosen. Hence, the probability of smoothing polyhedra to other points decreases as the points move farther away. Secondly, we notice that in dropout, the fraction of activated neurons in different layers, for different samples and different training stages, can be different. Although we are using the same dropout rate, since dropping out neurons that are already quiescent by ReLU changes nothing, the effective dropout rate, i.e., the fraction of the activated neurons that are dropped, can vary significantly. In jumpout, we adaptively normalize the dropout rate for each layer and each training sample/batch, so the effective neural-deactivation rate applied to the activated neurons are consistent over different layers and different samples as training proceeds. Lastly, we address the incompatibility problem between dropout and BN by rescaling the outputs of jumpout in order to keep the variance unchanged after the process of neural deactivation. Therefore, the BN layers learned in the training phase can be directly applied in the test phase without an inconsistency, and we can reap the benefits of both dropout and BN when training a DNN.In our implementation, similar to dropout, jumpout also randomly generates a 0/1 mask over the hidden neurons to drop activations. It does not require any extra training, can be easily implemented and incorporated into existing architectures with only a minor modification to dropout code. In our experiments on a broad range of benchmark datasets including CIFAR10, CIFAR100, Fashion-MNIST, SVHN, STL10 and ImageNet-1k, jumpout shows almost the same memory and computation costs as the original dropout, but significantly and consistently outperforms dropout on a variety of tasks, as we show below. Jumpout is not the first approach to address the fixed dropout rate problem. Indeed, recent work has proposed different methods to generate adaptive dropout rates. BID1 proposed "standout" to adaptively change the dropout rates for various layers and training stages. They utilized a binary belief network and trained it together with the original network to control the dropout rates. BID24 further extend the model so that adaptive dropout rates can be learned for different neurons or group of neurons. BID23 showed that the Rademacher complexity of a DNN is bounded by a function related to the dropout rate vectors, and they proposed to adaptively change dropout rates according to the Rademacher complexity of the network. In contrast to the above methods, jumpout does not rely on additional trained models: it adjusts the dropout rate solely based on the ReLU activation patterns. Moreover, jumpout introduces negligible computation and memory overhead relative to the original dropout methods, and can be easily incorporated into existing model architectures. BID19 showed that dropout has a Gaussian approximation called Gaussian dropout and proposed to optimize the Gaussian dropout directly to achieve faster convergence. The Gaussian dropout was also extended and studied from the perspective of variational methods. BID9 generalized Gaussian dropout and proposed variational dropout, where they connected the global uncertainty with the dropout rates so that dropout rates can be adaptive for every neuron. BID11 further extended variational dropout to reduce the variance of the gradient estimator and achieved sparse dropout rates. Other recent variants of dropout include Swapout BID16, which combines dropout with random skipping connections to generalize to different neural network architectures, and Fraternal Dropout BID25, which trains two identical DNNs using different dropout masks to produce the same outputs and tries to shrink the gap between the training and test phases of dropout. In this paper, we focus on changes to the original dropout that do not require any extra training/optimization costs or introduce more parameters to learn. Jumpout involves orthogonal and synergistic contributions to most of the above methods, and targets different problems of dropout. Indeed, jumpout can be applied along with most other previous variants of dropout. We study a feed-forward deep neural networks of the form: DISPLAYFORM0 where W j is the weight matrix for layer j, ψ j is the corresponding activation function (ReLU in this paper), x ∈ X is an input data point of d in dimensions andŷ(x) is the network's output prediction of d out dimensions, e.g., the logits before applying softmax. We denote the hidden nodes on layer j to be h j, i.e., h j = W j ψ j−1 (W j−1 ψ j−2 (. . . ψ 1 (W 1 x))), whose dimensionality is d j; they represent the nodes after applying the activation function ash j = ψ(h j).The above DNN formalization can generalize many DNN architectures used in practice. Clearly, Eqn. FORMULA0 can represent a fully-connected network of m layers. Note Eqn. covers the DNNs with bias terms at each layer since the bias terms can be written in the matrix multiplication as well by introducing dummy dimensions on the input data (append m 1's to input data). Moreover, the convolution operator is essentially a matrix multiplication, where every row of the matrix corresponds to applying a convolutional filter on a certain part of the input, and therefore the ing weight matrix is very sparse and has tied parameters, and typically has an enormous (compared to input size) number of rows. The average-pooling is a linear operator and therefore representable as a matrix multiplication, and max-pooling can be treated as an activation function. Finally, we can represent the residual network block by appending an identity matrix at the bottom of a weight matrix so that we can retain the input values, and add the retained input values later through another matrix operation. Therefore, we can also write a DNN with short-cut connections in the form of Eqn..For piecewise linear activation functions such as ReLU, the DNN in Eqn. FORMULA0 can be written as a piecewise linear function, i.e., the DNN in a region surrounding a given data point x is a linear model having the following form:ŷ DISPLAYFORM1 where W x j is the equivalent weight matrix after combining the ant activation pattern with W j. For instance, suppose we use ReLU activation ψ ReLU (z) = max(0, z); at every layer, we have an activation pattern for the input a j (x) ∈ {0, 1} dj, and a j (x)[p] = 0 indicates that ReLU sets the unit p to 0 or otherwise preserves the unit value. Then, ψ ReLU (W 1 x) = W x 1 x, where W x 1 is modified from W 1 by setting the rows, whose corresponding activation patterns are 0, to be all zero vectors. We can continue such a process to the deeper layers, and in the end we can eliminate all the ReLU functions and produce a linear model as shown in Eqn..In addition, the gradient DISPLAYFORM2 ∂x is the weight vector of the linear model. Note that the linear model in Eqn. 2 is specifically associated with the activation patterns {a j (x)} m j=1 on all layers for a data input x, which is equal to a set of linear constraints that defines a convex polyhedron containing Although the above analysis can be easily extended to DNNs with general piecewise linear activation functions, we focus on DNNs with ReLU activations in the rest of the paper for clarity. In addition to the piecewise linear property, ReLU units are cheap to compute, as is their gradient, and are widely applicable to many different tasks while achieving good performance BID5 BID22. In the following, we will study how dropout improves the generalization performance of a complicated DNN by considering how it generalizes each local linear model to its nearby convex polyhedra. This is easier to analyze and acts as the inspiration for our modifications to the original dropout. We will further elaborate the understandings of dropout based on the above insights of local linear models in the next section.3 THREE MODIFICATIONS TO DROPOUT LEAD TO JUMPOUT There have been multiple explanations for how dropout improves the performance of DNNs. Firstly, dropout prevents the co-adaptation of the neurons in the network, or in other words, encourages the independence or diversity amongst the neurons. Secondly, by randomly dropping a portion of neurons during training, we effectively train a large number of smaller networks, and during test/inference, the network prediction can be treated as an ensemble of the outputs from those smaller networks, and thus enjoys the advantages of using an ensemble such as variance reduction. Here we provide another perspective for understanding how dropout improves generalization performance by inspecting how it smooths each local linear model described in the previous section. As mentioned above, for a DNN with ReLUs, the input space is divided into convex polyhedra, and for any data point in every convex polyhedron of the final layer (a polyhedron that is not divided further into smaller regions), the DNN behaves exactly as a linear model. For large DNNs with thousands of neurons per layer, the number of convex polyhedra can be exponential in the number of neurons. Hence, there is a high chance that the training samples will be dispersedly situated amongst the different polyhedra, and every training data point is likely to be given its own distinct local linear model. Moreover, it is possible that two nearby polyhedra may correspond to arbitrarily different linear models, since they are the of consecutively multiplying a series of weight matrices FORMULA1 ), where each weight matrix W x j is W j with some rows setting to be all-zero according to the activation pattern a j (x) of a specific data point x. If the activation patterns of two polyhedra differ on some critical rows of the weight matrices, the ing linear models may differ a lot. Therefore, it is possible that the linear model of one polyhedron can only work for one or a few training data points strictly within the polyhedron, and may fail when applied to any nearby test data point (i.e., a lack of smoothness). This might make DNN fragile and perform unstably on new data, and thus weaken its generalization ability. DISPLAYFORM0 Given the problems of dropout mentioned in Section 1.1, we propose to sample a dropout rate from a truncated half-normal distribution (to get a positive value), which is the positive part of an ordinary Gaussian distribution with mean zero. In particular, we firstly sample p ∼ N (0, σ) from a Gaussian distribution, and then take the absolute value |p| as the dropout rate. We further truncate |p| so that |p| ∈ [p min, p max], where 0 ≤ p min < p max ≤ 1. These determine the lower and upper limits of the dropout rate and are used to ensure that the sampled probability does not get either too small, which makes jumpout ineffective, or too large, which may yield poor performance. Overall, this achieves a monotone decreasing probability of a given dropout rate as mentioned above. Other distributions (such as a Beta distribution) could also be used for this purpose, but we leave that to future work. We utilize the standard deviation σ as the hyper-parameter to control the amount of generalization enforcement. By using the above method, smaller dropout rates are sampled with higher probabilities so that a training sample will be more likely to contribute to the linear models of closer polyhedra. Therefore, such a Gaussian-based dropout rate distribution encourages the smoothness of the generalization performance of each local linear model, i.e., it will still perform well on points in closer polyhedra, but its effectiveness will diminish for a point farther away from the polyhedron it belongs to. The dropout rate for each layer is a hyper-parameter, and as stated above, it controls a form of smoothness amongst nearby local linear models. Ideally, the dropout rates of different layers should be tuned separately to improve network performance. In practice, it is computationally expensive or infeasible to tune so many hyper-parameters. One widely adopted approach is therefore to set the same drop rate for all layers and to tune one global dropout rate. Using a single global dropout rate is suboptimal because the proportion of active neurons (i.e., neurons with positive values) of each layer at each training stage and for each sample can be dramatically different (see FIG1). When applying the same dropout rate to different layers, different fractions of active neurons get deactivated, so the effective dropout rate applied to the active neurons varies significantly. Suppose the fraction of active neurons in layer j is q DISPLAYFORM0 Since dropping the inactive neurons has no effects (neurons with values ≤ 0 have already been set to 0 by ReLU), the effective dropout rate of every layer is p j q + j, where p j is the dropout rate of layer j. Thus, to better control the behavior of dropout for different layers and across various training stages, we normalize the dropout rate by q + j and use an actual dropout rate of p j = p j /q + j. By doing so, the hamming distance between the changed activation pattern and the original pattern is more consistent, and we can more precisely achieve the desirable level of smoothing encouragement by tuning the dropout rate as a single hyper-parameter. In standard dropout, if the dropout rate is p, we scale the neurons by 1/p during training and keeps the neuron values unchanged during the test/inference phase. The scaling factor 1/p keeps the mean of the neurons the same between training and test; this constitutes a primary reason for the incompatibility between dropout and batch normalization (BN) BID8. Specifically, though the mean of neurons is consistent, the variance can be dramatically different between the training and test phases, in which case the DNN might have unpredictable behavior as the BN layers cannot adapt to the change of variance from training to test condition. We consider one possible setting of combining dropout layers with BN layers where one linear computational layer (e.g., a fully-connected or a convolutional layer without activation function) is followed by a BN layer, then a ReLU activation layer, and then followed by a dropout layer. For layer j, without loss of generality, we may treat the value of a neuron i after ReLU, i.e.,h j [i] as a random variable with q + j probability of being 1 and 1 − q + j probability of being 0. If dropout is not applied,h j [i] then gets multiplied by certain entry in the weight matrix W j+1 [i, i], and contributes to the value of the i neuron of layer j +1. Since we consider any index i and i, we rename the following terms for simplicity: x j:=h j, w j:= W j+1 [i, i], y j:= h j+1 [i]. As neuron i of layer j + 1 (before ReLU) then gets fed into a BN layer, we will focus on the change of mean and variance as we add the dropout layer. Suppose we apply a dropout rate of p j, then DISPLAYFORM0 Hence, dropout changes both the scales of the mean and variance of neurons during training. Since the following BN's parameters are trained based on the scaled mean and variance, which however are not scaled by dropout during test/inference (because dropout is not used during testing), the trained BN is not consistent with the test phase. An easy fix of the inconsistency is to rescale the output y j to counteract dropout's on the scales of mean and variance. In order to recover the original scale of the mean, we should rescale the dropped neurons by (1 − p j) −1. However, the rescaling factor should be (1 − p j) −0.5 instead for recovering the scale of the variance if E(y j) is small and thus the second term of the variance can be ignored. Ideally, we can also take into account the value of E[w j], and scale the un-dropped nuerons by DISPLAYFORM1 However, computing information about w j, which is the weight matrix of the following layer, requires additional computation and memory cost. In addition, such a scaling factor is only correct for the variance of y j. To make the mean consistent, we should instead use (1 − p j) −1 (the original dropout scaling factor). No simple scaling method can resolve the shift in both mean and variance, as the mean rescaling (1 − p j) −1 does not solve the variance shift. −1, (1 − p) −0.5 and (1 − p) −0.75 as the dropout rescaling factor applied to y, when p = 0.2. The network is "CIFAR10(s)" (see Sec. 4). The left plot shows the empirical mean of y with dropout divided by the case without dropout (averaged over all layers), and the second plot shows the similar ratio for the variance. Ideally, both ratios should be close to 1. As shown in the plots, (1 − p) −0.75 gives nice trade-offs between the mean and variance rescaling. When the mean E(y j) is large in magnitude, so that the second term in the variance is comparable with the first term, in which case the variance is small, we should use the rescaling factor close to (1−p j) −1, which makes the mean exactly unchanged for training and test. In contrast, when the mean E(y j) is small in magnitude and close to 0, the second term in the variance is ignorable, and we should use (1 − p j) −0.5 as the rescaling factor, to make the variance unchanged. In practice, it is not efficient to compute E(y j) during training, so we propose to use a trade-off point (1 − p j) −0.75 between (1−p j) −1 and (1−p j) −0.5. In FIG4, we show that (1−p j) −0.75 makes both the mean and variance sufficiently consistent for the cases of using dropout and not using dropout. In FIG3, we compare the performance of the original dropout and dropout using our rescaling factor (1−p j) −0.75, when they DISPLAYFORM2 // Compute the fraction of activated neurons DISPLAYFORM3 // Sample a Gaussian dropout rate DISPLAYFORM4 // Normalize the dropout rate according to the fraction of activated neurons 4 Randomly generate a 0/1 mask z j for h j, with probability p j to be 0; // Sample the dropout mask 5 s j:= (1 − p) −0.75; // Compute the rescaling factor DISPLAYFORM5 // Rescale the outputs 7 return h j Algorithm 1: Jumpout layer for DNN with ReLU.are used with and without BN in a convolutional networks. It shows that using dropout with BN can potentially improve the performance, and larger dropout might in more improvement. However, using the original dropout with BN leads to a significant decrease in the accuracy once increasing the dropout rate over 0.15. In contrast, the performance of dropout using our rescaling keeps improving with increasing dropout rate (until reaching 0.25), and is the best among the four configurations. We combine the three modifications specifically designed to overcome the drawbacks of the original dropout in our proposed improved dropout, which we call "Jumpout" as shown in Alg. 1. Similar to the original dropout, jumpout essentially generates a 0/1 mask for the input neurons, and randomly drop a portion of the neurons based on the mask. Summarizing the novelty of jumpout, instead of using a fixed dropout rate as in the original dropout, jumpout samples from a monotone decreasing distribution as mentioned above to get a random dropout rate. Also, jumpout normalizes the dropout rate adaptively based on the number of active neurons, which enforces consistent regularization and generalization effects on different layers, across different training stages, and on different samples. Finally, jumpout further scales the outputs by (1 − p) −0.75, as opposed to (1 − p) −1 during training, in order to trade-off the mean and variance shifts and synergize well with batchnorm operations. Jumpout requires one main hyper-parameter σ to control the standard deviation of the half-normal distribution, and two auxiliary truncation hyperparameters (p min, p max). Though (p min, p max) can also be tunned, they serve to bound the samples from the half-normal distribution; in practice, we set p min = 0.01 and p max = 0.6, which work consistently well over all datasets and models we tried. Hence, jumpout has three hyperparameters, although we only tuned σ and achieved good performance, as can be seen below. Also, note that here we consider the input h j to be the features of layer j corresponding to one data point. For a mini-batch of data points, we can either estimate q + j separately for each single data point in the mini-batch or apply the average q + j over data points as the estimate for the mini-batch. In practice, we utilize the latter option as we find that it gives comparable performance to the first while using less computation and memory. Jumpout has almost the same memory cost as the original dropout, which is the additional 0/1 drop mask. For computation, jumpout requires counting the number of active neurons, which is insignificant compared to the other layers of a deep model, and sampling from the distribution, which is also insignificant compared to the other computation in DNN training. In this section, we apply dropout and jumpout to different popular DNN architectures and compare their performance on six benchmark datasets at different scales. In particular, these DNN architectures include a small CNN with four convolutional layers 2 applied to CIFAR10 BID10 ), WideResNet-28-10 (applied to CIFAR10 and CIFAR100 BID10), "pre-activation" version of ResNet-20 BID6 applied to Fashion-MNIST ("Fashion" in all tables) BID21, WideResNet-16-8 applied to SVHN BID12 and STL10, and ResNet-18 BID5 applied to ImageNet BID3 BID15. The information about the all the datasets can be found in TAB3 at Appendix. For all the experiments about CIFAR and Fashion-MNIST, we follow the standard settings, data preprocessing/augmentation, and hyperparameters used in an existing GitHub repository 3. On ImageNet, we starts from a pre-trained ResNet18 model 4, and train two copies of it with dropout and jumpout respectively for the same number of epochs. The reason for not starting from random initialized model weights is that training DNNs on ImageNet usually does not have overfitting problem if one follows the standard data augmentation methods used to train most modern models, but both dropout and jumpout are most effective in the case of overfitting. Therefore, we choose to start from the pre-trained model, on which training accuracy is relatively high (but still not overfit and very close to the test accuracy) 5. We summarize the experimental in TAB1 which shows that jumpout consistently outperforms dropout on all datasets and all the DNNs we tested. Moreover, for Fashion-MNIST and CIFAR10 on which the test accuracy is already > 95%, jumpout can still bring appreciable improvements. In addition, on CIFAR100 and ImageNet (on which a great number of DNNs and training methods are heavily tunned), jumpout achieves the improvement that can only be obtained by significantly increasing the model size in the past. These verify the effectiveness of jumpout and its advantage comparing to the original dropout. In addition, we conduct a thorough ablation study of all the possible combinations of the three proposed modifications, with reported in TAB0. It further verifies the effectiveness of each modification: 1) each modification improves the vanilla dropout; 2) adding any modification to another brings further improvements; and 3) applying the three modifications together (i.e., jumpout) achieves the best performance. We also provide the learning curves and convergence plots of dropout and jumpout equipped DNNs during training in FIG5. In all the figures, "Jumpout" applies adaptive dropout rate per minibatch. Jumpout exhibits substantial advantages over dropout in early learning stages, and reaches a reasonably good accuracy much faster. In the future, it may be possible to find a better learning rate schedule method specifically for jumpout, so it can reach the final performance earlier than dropout. −1, (1 − p) −0.5 and (1 − p) −0.75 as the dropout rescaling factor applied to y, when p = 0.1. The network is "CIFAR10(s)" (see Sec. 4). The left plot shows the empirical mean of y with dropout divided by the case without dropout (averaged over all layers), and the second plot shows the similar ratio for the variance. Ideally, both ratios should be close to 1. As shown in the plots, (1 − p) −0.75 gives nice trade-offs between the mean and variance rescaling.

represent the tldr for finding the corresponding scientific article!

Jumpout applies three simple yet effective modifications to dropout, based on novel understandings about the generalization performance of DNN with ReLU in local regions.

Represent the TLDR for finding the corresponding scientific article

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the article for finding a single-sentence TLDR!

Question: Concerns about interpretability, computational resources, and principled inductive priors have motivated efforts to engineer sparse neural models for NLP tasks. If sparsity is important for NLP, might well-trained neural models naturally become roughly sparse? Using the Taxi-Euclidean norm to measure sparsity, we find that frequent input words are associated with concentrated or sparse activations, while frequent target words are associated with dispersed activations but concentrated gradients. We find that gradients associated with function words are more concentrated than the gradients of content words, even controlling for word frequency. Researchers in NLP have long relied on engineering features to reflect the sparse structures underlying language. Modern deep learning methods promised to relegate this practice to history, but have not eliminated the interest in sparse modeling for NLP. Along with concerns about computational resources BID0 BID12 and interpretability BID10 BID21, human intuitions continue to motivate sparse representations of language. For example, some work applies assumptions of sparsity to model latent hard categories such as syntactic dependencies BID14 or phonemes BID1. BID13 found that a sparse attention mechanism outperformed dense methods on some NLP tasks; BID11 found sparsified versions of LMs that outperform dense originals. Attempts to engineer sparsity rest on an unstated assumption that it doesn't arise naturally when neural models are learned. Is this true?Using a simple measure of sparsity, we analyze how it arises in different layers of a neural language model in relation to word frequency. We show that the sparsity of a word representation increases with exposure to that word during training. We also find evidence of syntactic learning: gradient updates in backpropagation depend on whether a word's part of speech is open or closed class, even controlling for word frequency. Language model. Our LM is trained on a corpus of tokenized, lowercased English Wikipedia (70/10/20 train/dev/test split). To reduce the number of unique words (mostly names) in the corpus, we excluded any sentence with a word which appears fewer than 100 times. Those words which still appear fewer than 100 times after this filter are replaced with <UNK>. The ing training set is over 227 million tokens of around 19.5K types. We use a standard 2-layer LSTM LM trained with cross entropy loss for 50 epochs. The pipeline from input x t−1 at time step t − 1 to predicted output distributionx for time t is described in Figure 1, illustrating intermediate activations h e t, h 1 t, and h 2 t. At training time, the network observes x t and backpropagates the gradient updates h e t,h 1 t,h 2 t, andx t. The embeddings produced by the encoding layer are 200 units, and the recurrent layers have 200 hidden units each. The batch size is set to forty, the maximum sequence length to 35, and the dropout ratio to 0.2. The optimizer is standard SGD with clipped gradients at 2 = 0.25, where the learning rate begins at 20 and is quartered whenever loss fails to improve. Measuring sparsity. We measure the sparsity of a vector v using the reciprocal of the TaxicabEuclidean norm ratio BID17. This measurement has a long history as a measurement of sparsity in natural settings (Zibulevsky Figure 2: Average sparsity χ(h 2 t) over all training epochs (x-axis), for target words x t occurring more than 100k times in training. Target words are sorted from most frequent (bottom) to least frequent (top).and BID23 BID4 BID15 BID22 and is formally defined as χ(v) = v 2 / v 1. The relationship between sparsity and this ratio is illustrated in two dimensions in the image on the right, in which darker blue regions are more concentrated. The pink circle shows the area where 2 ≤ 1 while the yellow diamond depicts 1 ≤ 1. For sparse vectors 1, 0 or 0, 1, the norms are identical so χ is 1, its maximum. For a uniform vector like 1, 1, χ is at its smallest. In general, χ(v) is higher when most elements of v are close to 0; and lower when the elements are all similar in value. Sparsity is closely related to the behavior of a model: If only a few units hold most of the mass of a representation, the activation vector will be highly concentrated. If a neural network relies heavily on a small number of units in determining its predictions, the gradient will be highly concentrated. A highly concentrated gradient is mainly modifying a few specific pathways. For example, it might modify a neuron associated with particular inputs like parentheses BID5, or properties like sentiment BID16.Representations of Target Words. Our first experiments look at the relationship of sparsity to target word x t. Gradient updates triggered by the target are often used to identify units that are relevant to a prediction, and as shown in Figure 2, gradient sparsity increases with both the frequency of a word in the corpus and the overall training time. In other words, more exposure leads to sparser relevance. Because the sparsity ofh 2 increases with target word frequency, we measure not sparsity itself but the Pearson correlation, over all words w, between word frequency and mean χ(h) over representations h where w is the target: DISPLAYFORM0 Here FIG0 we confirm that concentrated gradients are not a of concentrated activations, as activation sparsity χ(h 2) is not correlated with target word frequency. The correlation is strong and increasing only for ρ ← (h 2). The sparse structure being applied is therefore particular to the gradient passed from the softmax to the top LSTM layer, related to how a word interacts with its context. The Role of Part of Speech. FIG1 shows that ρ ← (h 2) follows distinctly different trends for open POS classes 1 and closed classes 2. To associate words to POS, we tagged our training corpus with spacy 3; we associate a word to a POS only if the majority (at least 100) of its occurrences are tagged with that POS. We see that initially, frequent words from closed classes are highly concentrated, but soon stabilize, while frequent words from open classes continue to become more concentrated throughout training. Why?Closed class words clearly signal POS. But open classes contain many ambiguous words, like "report", which can be a noun or verb. Open classes also contain many more words in general. We posit that early in training, closed classes reliably signal syntactic structure, and are essential for shaping network structure. But open classes are essential for predicting specific words, so their importance in training continues to increase after part of speech tags are effectively learned. The high sparsity of function word gradient may be surprising when compared with findings that content words have a greater influence on outputs BID7. However, those findings were based on the impact on the vector representation of an entire sentence after omitting the word. BID6 found that content words have a longer window during which they are relevant, which may explain the of BID7. Neither of these studies controlled for word frequency in their analyses contrasting content and function words, but we believe this oversight is alleviated in our work by measuring correlations rather than raw magnitude. Because ρ ← (h 2) is higher when evaluated over more fre- Representations of Input Words. We next looked at the vector representations of each step in the word sequence as a representation of the input word x t−1 that produced that step. We measure the correlation with input word frequency: DISPLAYFORM1 Here FIG0 we find that the view across training sheds some light on the learning process. While the lower recurrent layer quickly learns sparse representations of common input words, ρ → (h 1) increases more slowly later in training and is eventually surpassed by ρ → (h e), while gradient sparsity never becomes significantly correlated with word frequency. studied the activations of feedforward networks in terms of the importance of individual units by erasing a particular dimension and measuring the difference in log likelihood of the target class. They found that importance is concentrated into a small number of units at the lowest layers in a neural network, and is more dispersed at higher layers. Our findings suggest that this effect may be a natural of the sparsity of the activations at lower layers. We relate the trajectory over training to the Information Bottleneck Hypothesis of BID20. This theory, connected to language model training by BID19, proposes that the earlier stages of training are dedicated to learning to effectively represent inputs, while later in training these representations are compressed and the optimizer removes input information extraneous to the task of predicting outputs. If extraneous information is encoded in specific units, this compression would lead to the observed effect, in which the first time the optimizer rescales the step size, it begins an upward trend in ρ → as extraneous units are mitigated. Why do common target words have such concentrated gradients with respect to the final LSTM layer? A tempting explanation is that the amount of information we have about common words offers high confidence and stabilizes most of the weights, leading to generally smaller gradients. If this were true, the denominator of sparsity, gradient 1, should be strongly anti-correlated with word frequency. In fact, it is only ever slightly anti-correlated (correlation > −.1). Furthermore, the sparsity of the softmax gradient χ(x) does not exhibit the strong correlation seen in χ(h 2), so sparsity at the LSTM gradient is not a direct effect of sparse logits. However, the model could still be "high confidence" in terms of how it assigns blame for error during common events, even if it is barely more confident overall in its predictions. According to this hypothesis, a few specialized neurons might be responsible for the handling of such words. Perhaps common words play a prototyping role that defines clusters of other words, and therefore have a larger impact on these clusters by acting as attractors within the representation space early on. Such a process would be similar to how humans acquire language by learning to use words like'dog' before similar but less prototypical words like'canine' BID18. As a possible mechanism for prototyping with individual units, BID2 found that some neurons in a translation system specialized in particular word forms, such as verb inflection or comparative and superlative adjectives. For example, a common comparative adjective like'better' might be used as a reliable signal to shape the handling of comparatives by triggering specialized units, while rarer words have representations that are more distributed according to a small collection of specific contexts. There may also be some other reason that common words interact more with specific substructures within the network. For example, it could be related to the use of context. Because rare words use more context than common words and content words use more context than function words BID6, the gradient associated with a common word would be focused on interactions with the most recent words. This would lead common word gradients to be more concentrated. It is possible that frequent words have sparse activations because frequency is learned as a feature and thus is counted by a few dimensions of proportional magnitude, as posited by. Understanding where natural sparsity emerges in dense networks could be a useful guide in deciding which layers we can apply sparsity constraints to without affecting model performance, for the purpose of interpretability or efficiency. It might also explain why certain techniques are effective: for example, in some applications, summing representations together works quite well BID3. We hypothesize that this occurs when the summed representations are sparse so there is often little overlap. Understanding sparsity could help identify cases where such simple ensembling approaches are likely to be effective. Future work may develop ways of manipulating the training regime, as in curriculum learning, to accelerate the concentration of common words or incorporating concentration into the training objective as a regularizer. We would also like to see how sparsity emerges in models designed for specific end tasks, and to see whether concentration is a useful measure for the information compression predicted by the Information Bottleneck.

Represent the TLDR for finding the corresponding scientific article

We study the natural emergence of sparsity in the activations and gradients for some layers of a dense LSTM language model, over the course of training.

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.


Question: represent the scientific article to find a short tldr

Question: The integration of a Knowledge Base (KB) into a neural dialogue agent is one of the key challenges in Conversational AI. Memory networks has proven to be effective to encode KB information into an external memory to thus generate more fluent and informed responses. Unfortunately, such memory becomes full of latent representations during training, so the most common strategy is to overwrite old memory entries randomly. In this paper, we question this approach and provide experimental evidence showing that conventional memory networks generate many redundant latent vectors ing in overfitting and the need for larger memories. We introduce memory dropout as an automatic technique that encourages diversity in the latent space by 1) Aging redundant memories to increase their probability of being overwritten during training 2) Sampling new memories that summarize the knowledge acquired by redundant memories. This technique allows us to incorporate Knowledge Bases to achieve state-of-the-art dialogue generation in the Stanford Multi-Turn Dialogue dataset. Considering the same architecture, its use provides an improvement of +2.2 BLEU points for the automatic generation of responses and an increase of +8.1% in the recognition of named entities. Given the large amount of dialogue data recorded in human-human or human-chatbot interactions, there is a great need for dialogue systems that infer automatic responses grounded to personal knowledge bases. This approach has the advantage of integrating semantic information that is fundamental to achieve dialogue understanding. We want to leverage the contextual information present in a KB (e.g., a calendar of events) to answer queries like What time is my dentist appointment?. This task is challenging because existing neural dialogue agents often assume that the dialogue history carries the information needed to provide an answer but struggle to interface with the structured data stored in a KB. This assumption prevents to have an end-to-end differentiable model to maintain the kind of contextual conversations that people desire. Memory networks has proven to be effective to encode KB information into an external memory to generate more fluent and informed responses. However, there is no much work in regularizing the latent representations stored in the external memory. Unlike the conventional dropout technique used to regularize deep neural networks , we propose memory dropout to attain the same goal (i.e., reduction of overfitting) but with different functionality and designed for memory networks. Given the long-term nature of memory networks, we do not immediately remove redundant memories with some probability as in the original dropout algorithm. Instead, we assign them the current maximum age increasing their probability of being overwritten by more recent latent representations in future training steps. Thus, in contrast to , our memory dropout is a delayed regularization mechanism. The main contributions of our work are the following: • We introduce a new regularization method called memory dropout designed for dealing with overfitting in Memory Augmented Neural Networks. To our best knowledge, ours is the first work on regularizing memory networks. • We build a neural dialogue agent that uses memory dropout to incorporate KB into an external memory for automatic response generation. Our show that this technique can generate more fluent and accurate responses: an improvement of +2.2 BLUE points and +8.1% Entity F1 score versus not using it in the Stanford Multi-Turn Dialogue dataset. Figure 1: Learning the (h, y) pair transitions the neighborhood of h (represented as an ellipse) to a new state in which a memory h is drawn as the distribution of positive memories. Small circles represent the uncertainty of using a particular memory to model h. In the new memory configuration, we age positive keys (now faded in grey) making them more likely of being overwritten by other training examples. This section describes the memory dropout neural model whose goal is to increase the diversity of latent representations stored in an external memory. For example, consider a neural encoder which receives an observation (x, y) and generates the latent representation h in a hidden layer. As illustrated in Figure 1, we want to incorporate a normalized h (i.e., h = 1) in the long-term memory M. The most similar memory entries to h form a neighborhood in which entries can be positive (+) or negative (-) depending on whether they share the same class label y or not. An external memory M augments the capacity of a neural encoder by preserving long-term latent representations. Figure 2 illustrates a memory network which consists of arrays K and V to store keys (latent representations) and values (class labels), respectively, as introduced in. To support our technique, we extend this definition with arrays A and S to store the age and the variance of each key, respectively. The final form of the memory module is as follows: Our main goal is to learn a mathematical space in which the margin between positive and negative memories is maximum while retaining the minimum number of positive keys. Core to this idea is the definition of a differentiable Gaussian Mixture Model parameterized by both the location and covariance matrix of each positive memory. Sampling from this distribution returns a new positive embedding h that characterizes its neighbors. So, given the embedding vector h, the collection of P positive keys } is a subpopulation of the memory keys that can be represented as a linear superposition of P Gaussian components aimed at providing a rich class of a density model in the form where each Gaussian is centered at a positive key µ p = k + p with covariance matrix Σ p = diag(s + p). Note that without the variances stored in the array S, the uncertainty of each key will be the same and extreme embedding vectors will dominate the likelihood probability. We shall view π = {π 1, ..., π P} in Equation 1 as a vector of probabilities that quantifies the mixing coefficients of the Gaussian components and whose values correspond to the similarity between h and the positive keys K Figure 2: A memory network consists of a neural encoder and an external memory that augments its capacity by preserving longer distinct versions of h during training. Some memory entries (in gray) are positive candidates to correctly answer to the embedding h. The conditional distribution of a new key k given a particular Gaussian is so sampling an index i over the P mixture components under the distribution π generates the new key k as a random variable from p(k |π p=i). As being sampled from the mixture model, k is representative of the subpopulation of positive memories. Then, we incorporate to the external memory the information encoded by the latent vector h to the new key k, reset its corresponding age, and compute its variance to account for the observed uncertainty in approximating h with k. Finally, the aging mechanism penalizes redundant keys indexed by i + as follows, We study the memory dropout neural model in a realistic scenario: As the external memory of a dialogue system that infers automatic responses grounded to a Knowledge Base (KB). Our goal is to leverage the contextual information present in a KB (e.g., a calendar of events) to answer queries like What time is my dentist appointment?. This task is challenging because existing neural dialogue agents often assume that the dialogue history carries the information needed to provide an answer but struggle to interface with the structured data stored in a KB. This assumption prevents to have an end-to-end differentiable model to maintain the kind of flexible conversations that people desire. Figure 3 illustrates our proposed architecture formed by a Sequence-to-Sequence model to represent the dialogue history and a Memory Augmented Neural Network (MANN) to encode the KB. To encode the KB, the addressable memory entries of a Memory Network allows us to generalize with fewer latent representations of the KB, even if they were present once during training,. Inspired by , we store the KB of a given dialogue by decomposing it from its tabular format to a collection of triplets that expresses the following relationship (subject, relation, object). For example, an entry in the KB representing a dentist appointment (event=dentist, date=the 19th, time=5pm, party=Mike) would be normalized into 12 different triplets: [(dentist, date, the 19th), (dentist, time, 5pm), (dentist, party, Mike), (Mike, time, 5pm), (dentist, date, the 19th), (dentist, time, 5pm), (dentist, party, Mike), (the 19th, event, dentist), (the 19th, party, Mike), (the 19th, time, 5pm), (5pm, event, dentist), (5pm, date, the 19th), (5pm, party, Mike), (Mike, event, dentist), (Mike, date, the 19th), (Mike, time, 5pm)]. Figure 3: Architecture of the neural dialogue model that incorporates a KB. Note the i th decoding step of the wordŷ i given the attention over the external memory which encodes KB triplets in its keys and that uses memory dropout for model regularization. Each triplet feeds the MANN with the following key-value format where φ emb is a trainable embedding function that maps input tokens to a fixed-dimensional vector. To encode the dialogue, we use an encoder-decoder network architecture with an LSTM encoder that outputs a context-sensitive hidden representation h enco based on the dialogue history and an LSTM decoder that generates the next responseŷ. At every timestep of decoding, the decoder predicts the i th token of the responseŷ by computing its corresponding hidden state h deco i applying the recurrence h. Instead of directly using the decoder output h deco to predict over the dialogue vocabulary as in , we combine h deco with the of querying the memory module. We compute this vector, h KB i, as the additive attention score between h deco and each key. This operation into the unnormalized probabilities of including its corresponding value in the prediction. More formally, where W 1, W 2, W vocab_dlg, and W vocab_KB are trainable parameters. Here, o i represents the concatenation of an extended vocabulary T formed by the logits over the dialogue tokens and the logits over the distinct values stored in the value array V. A Softmax induces a probability distribution over the extended vocabulary and we argmax to obtain the index of the predicted token. Naturally, the objective function is to minimize the cross entropy between the actual and generated responses: where N is the number of dialogues, L j is the number of turns in the j th dialogue, m is the length of the generated response, and y j,k i is the one-hot encoding of the i th word in the actual response. We evaluate our proposed method in the Stanford Multi-Turn Dialogue (SMTD) dataset. This dataset consists of 3,031 dialogues in the domain of an in-car assistant, which provides automatic responses grounded to a personalized KB, known only to the in-car assistant. The entries of the KB may contain information for satisfying a query formulated by the driver. There are three types of KBs: a schedule of events, the weekly weather forecast, and information for point-of-interest navigation. Table 1: Averaged and per-domain BLEU and Entity F1 scores for the SMTD dataset. We compare our approach Memory Augmented Neural Network with Memory Dropout (MANN+MD) with the following baseline models: • Seq2Seq+: An encoder-decoder architecture that maps between sequences with minimal assumptions on the sequence structure and attends to parts of the input to predict the target word. This approach only incorporates information from the dialogue history. • Key-Value Retrieval Network+ (KVRN): A memory network with not update memory operations, it also computes attention over the keys of each entry in the KB, and • Memory Augmented Neural Network (MANN): Our proposed model with no memory dropout mechanism. For all the experiments, we use a word embedding of size 256. The encoders and decoders are bidirectional LSTMs of 3-layers with state size of 256 for each direction. For the memory network models, the number of memory entries is 1,000. We train all models with Adam optimizer with a learning rate of 0.001 and initialized all weights from a uniform distribution in [−0.01, 0.01]. We also applied dropout with keep probability of 95.0% for input and output of recurrent neural networks. Finally, we split the dataset into partitions with ratios 0.8, 0.1, and 0.1 for generating the training, validation, and testing datasets, respectively. Evaluating dialogue systems is challenging because a trained model can generate free-form responses. We employ two metrics to quantify the performance of a model grounded to a knowledge base. • is a metric of fluency that looks at n-gram precisions for n = 1, 2, 3, 4 comparing between exact matches of words between responses provided by a model and human annotators. • Entity F1 measures the correct retrieval of entities expected to be present in a generated response. We found that memory dropout improves both dialogue fluency and accuracy in recognizing entities compared to other memory networks that did not use memory dropout. Table 1 reports averaged and per-domain on this dataset. Not attending to the KB seems to have an adverse in generating automatic responses. For example, the Seq2Seq+Attention model shows the lowest Entity F1 scores (30.0%), indicating its inability to infer responses while being agnostic to the KB. The memory network MANN attends to the KB to predict a response and achieves a BLEU score of 11.2 and an Entity F1 score of 50.3%. With memory dropout, the MANN+MD increases both BLEU and Entity F1 scores to 13.4 and 58.4%, respectively. Note that both MANN and MANN+MD share the same architecture and hyperparameters and only differ in the use of memory dropout. On the other hand, KVRN also attends to the KB and it is the best performing neural network that does not use memory dropout. This method achieves a BLEU score of 13.2 and Entity F1 score of 48.0%. Our approach outperforms KVRN by +10.4% in the Entity F1 score and provides slight positive advantage for the BLEU score +0.2 setting a new SOTA for this dataset. Interestingly, KVRN only outperforms the other models in the Scheduling Entity F1 domain with 62.9%. This can be because half of these dialogues are commands to execute a task and thus do not require of a KB (e.g., Set a meeting reminder at 3pm). The explicit penalization of redundant keys during training could explain the gains obtained by MANN+MD. In order to test this hypothesis, we study now the correlation of keys in Section 4.4. We found that keys in memory tend to become redundant as training progresses. To observe this effect, for each training step we compute the Pearson correlation between each pair of keys and average these values. Figure 4 compare the degree of linear correlations in the memory networks studied in this paper. Comparing the , we can see that initially all models show low correlations as keys are randomly initialized. In following steps, both MANN and KVRN show increasing correlation values indicating that more redundant keys are stored in memory over time. In contrast, MANN+MD shows low correlation values which do not increase at the same rate than the other methods and reach stable values around step 25,000. We conclude that using memory dropout explicitly encourages the overwriting of redundant keys which lead to diverse representations in the latent space. In order to test the advantage of using memory dropout for overfitting reduction, 1) We compare the Entity F1 scores for the MANN and MANN+MD models during training considering different neighborhood sizes, and 2) We disable traditional dropout for the inputs and outputs of the encoder and decoder in an attempt to isolate the contribution of memory dropout. Figure 5 shows two groups of behavior in each plot. During training, no using memory dropout (MANN) leads to obtain higher Entity F1 scores, a reasonable outcome considering that not regularizer is present and the external memory increases memory capacity. During testing, MANN shows lower Entity F1 scores. This is a sign of a model that overfits to its training dataset and have problems to generalize in the testing dataset. On the other hand, using memory dropout (MANN+MD) provides a more conservative performance during training but better Entity F1 scores during testing ing into an average improvement of 10%. Testing with different neighborhood sizes (from 5 to 15 elements) shows in general the same behavior: two groups of well-defined Entity F1 scores grouped by whether they use the memory dropout technique or not. To test the usefulness of large memories when encoding a KB with memory dropout, we compare the models that use an external memory and consider different memory sizes. The task is to compute the Entity F1 score during the automatic response generation. We found that a side-effect of using an external memory is the need for larger memories to accommodate the large number of redundant activations generated during training. As seen in the experiments of Section 4.4 (memory correlations) and Section 4.6 (memory size), using memory dropout leads to store diverse keys and therefore we can use smaller memories to obtain higher levels of accuracy in this dataset. Deep Neural Networks are models to solve classification tasks that involve non-linearly separable classes. Memory networks consider an external differentiable memory in which a neural encoder manages the hidden states of the model using attention to address similar content in memory. Some similar approaches have also studied the problem of few-shot learning as a way to remember infrequent patters, a problem that we also observed when training with small knowledge bases. Some representative examples include which also uses an external memory to extend the capacity of the entire architecture. Also, Neural Turing Machines (NTM)? are differentiable architectures allowing efficient training with gradient descend and showing important properties for associative recall for learning different sequential patterns. In this paper, we extend the key-value architecture introduced in because of its simplicity and also it has shown to be effective in learning small datasets in text and visual domains. Deep models have also been used to train in dialogue agents. Often they model the dialogue state considering belief tracking and generation components. More recent architectures consider the use of a knowledge base and an external memory to encode its content. Although the key-value architecture of this system allows for incorporating domainspecific knowledge with no need of dialogue state trackers, it overfits to the training dataset impacting the accuracy and fluent generation of responses. Our model contrasts with this work on designing a a memory augmented model that deals directly with overfitting and require smaller memory size, as shown in our experiments. The regularization of neural networks is also an important problem which have proven to be effective to control overfitting and generate sparse activations during training. proposes the regularization of state transitions in a recurrent neural network, but the notion of memory is still internal and individual memories cannot be addressed. The popular dropout technique works in hidden layers as a form of model averaging. In contrast to , our memory dropout is a delayed (age-sensitive) regularization mechanism that works at the level of memory entries and not individual activations. To the best of our knowledge, our is the first work that addresses the regularization of memory networks and prove its effectivity in a challenging task such as automatic dialogue response. Memory Dropout is a technique for improving memory augmented neural networks by breaking co-adaptating memories built during backpropagation. While conventional dropout works at the level of individual activations, our memory dropout deals with latent representations of the input. These arrays of activations are stored into an external memory module which resembles areas of the human brain that are content-addressable and sensitive to semantic information. Central to this technique is the idea that age and uncertainty play important roles to regularize the addressable keys of an external memory module that is persistent across training examples. By doing this, we obtain higher BLEU and Entity F1 scores when training a task-oriented dialogue agent that decodes an answer considering the entries of KB stored in the memory module.

Represent the short summary for finding the corresponding scientific article (long)

Conventional memory networks generate many redundant latent vectors resulting in overfitting and the need for larger memories. We introduce memory dropout as an automatic technique that encourages diversity in the latent space.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the text to find a very short summary (1-2 sentences max).

Question: made a connection between Nesterov's method and an ordinary differential equation (ODE). We show if a Hessian damping term is added to the ODE from , then Nesterov's method arises as a straightforward discretization of the modified ODE. Analogously, in the strongly convex case, a Hessian damping term is added to Polyak's ODE, which is then discretized to yield Nesterov's method for strongly convex functions. Despite the Hessian term, both second order ODEs can be represented as first order systems. Established Liapunov analysis is used to recover the accelerated rates of convergence in both continuous and discrete time. Moreover, the Liapunov analysis can be extended to the case of stochastic gradients which allows the full gradient case to be considered as a special case of the stochastic case. The is a unified approach to convex acceleration in both continuous and discrete time and in both the stochastic and full gradient cases. made a connection between Nesterov's method for a convex, L-smooth function, f, and the second order, ordinary differential equation (ODE) x + 3 tẋ + ∇f (x) = 0 (A-ODE) did not show that Nesterov's method arises as a discretization of (A-ODE). In order to obtain such a discretization, we consider the following ODE, which has an additional Hessian damping term with coefficient 1/ √ L. DISPLAYFORM0 Notice that (H-ODE) is a perturbation of (A-ODE), and the perturbation goes to zero as L → ∞. Similar ODEs have been studied by BID1, they have been shown to accelerate gradient descent in continuous time in.Next, we consider the case where f is also µ-strongly convex, and write C f:= L/µ for the condition number of f. Then Nesterov's method in the strongly convex case arises as discretization of the following second order ODË DISPLAYFORM1 (H-ODE-SC) is a perturbation of Polyak's ODE x + 2 √ µẋ + ∇f (x) = 0 which is accelerates gradient when f is quadratic see .In each case, both continuous and discrete, as well and convex and strongly convex, it is possible to provide a proof of the rate using a Liapunov function. These proofs are already established in the literature: we give citations below, and also provide proof in the Appendix. Moreover, the analysis for Nesterov's method in the full gradient can be extended to prove acceleration in the case of stochastic gradients. Acceleration of stochastic gradient descent has been established by and BID7, see also BID8. A direct acceleration method with a connection to Nestero'v method was done by BID0. Our analysis unifies the continuous time ODE with the algorithm, and includes full gradient acceleration as a special case. The analysis proceeds by first rewriting (H-ODE) (and (H-ODE-SC)) as first order systems involving ∇f, and then replacing the ∇f with g = ∇f + e. Both the continuous and discrete time methods achieve the accelerated rate of convergence, provided |e| goes to zero quickly enough. The condition on |e|, is given below in and -it is faster than the corresponding rate for stochastic gradient descent. When e = 0 we recover the full gradient case. The renewed interested in the continuous time approach began with the work of and was followed;. Continuous time analysis also appears in BID6, BID11, and BID10. However, continuous time approaches to optimization have been around for a long time. Polyak's method is related to successive over relaxation for linear equations which were initially used to accelerate solutions of linear partial differential equations . A continuous time interpretation of Newton's method can be found in or BID1. The mirror descent algorithm of has a continuous time interpretation BID5. The Liapunov approach for acceleration had already appeared in BID4 for FISTA.The question of when discretizations of dynamical systems also satisfy a Liapunov function has been studied in the context of stabilization in optimal control BID12. More generally, studies when a discretization of a dynamical system preserves a property such as energy dissipation. Despite the Hessian term, (H-ODE-SC) can be represented as the following first order system. Lemma 2.1. The second order ODE (H-ODE) is equivalent to the first order system DISPLAYFORM0 Proof. Solve for v in the first line of (1st-ODE) DISPLAYFORM1 Insert into the second line of (1st-ODE) DISPLAYFORM2 Simplify to obtain (H-ODE).The system (1st-ODE) can be discretized using the forward Euler method with a constant time step, h, to obtain Nesterov's method. Definition 2.2. Define y k as the following convex combination of x k and v k. DISPLAYFORM3 Let h > 0 be a given small time step/learning rate and let t k = h(k + 2). The forward Euler method for (1st-ODE) with gradients evaluated at y k is given by DISPLAYFORM4 Remark 2.3. The forward Euler method simply comes from replacingẋ with (x k+1 − x k)/h and similarly for v. Normally the velocity field is simply evaluated at x k, v k. The only thing different about (FE-C) from the standard forward Euler method is that ∇f is evaluated at y k instead of x k. However, this is still an explicit method. More general multistep methods and one leg methods in this context are discussed in.Recall the standard Nesterov's method from Nesterov (2013, Section 2.2) DISPLAYFORM5 Theorem 2.4. The discretization of (H-ODE) given by (FE-C) with h = 1/ √ L and t k = h(k+2) is equivalent to the standard Nesterov's method (Nest). Eliminate the variable v using to obtain (Nest). Now we consider µ-strongly convex, and L-smooth functions, f, and write C f:= L µ for the condition number. We first show that (H-ODE-SC) can be represented as a first order system. Lemma 2.5. The second order ODE (H-ODE-SC) is equivalent to the first order system DISPLAYFORM0 Proof. Solve for v in the first line of (1st-ODE-SC) DISPLAYFORM1 Insert into the second line of (1st-ODE-SC) DISPLAYFORM2 Simplify to obtain (H-ODE-SC).System (1st-ODE-SC) can be discretized using a forward Euler method with a constant time step h to obtain Nesterov's method. Let h > 0 be a small time step, and apply the forward Euler method for (1st-ODE-SC) evaluated at y k: DISPLAYFORM3 where, DISPLAYFORM4 Now we recall the usual Nesterov's method for strongly convex functions from Nesterov (2013, Section 2.2) DISPLAYFORM5 Theorem 2.6. The discretization of (H-ODE-SC) given by (FE-SC) with h = 1/ √ L is equivalent to the standard Nesterov's method (SC-Nest). Eliminate the variable v k using the definition of y k to obtain (SC-Nest).3 LIAPUNOV ANALYSIS 3.1 CONVEX CASE: CONTINUOUS AND DISCRETE TIME Definition 3.1. Define the continuous time Liapunov function DISPLAYFORM0 where E(t, x, v) in given by. In particular, for all t > 0, DISPLAYFORM1 Furthermore, let x k, v k be given by (FE-C). Then for all k ≥ 0, DISPLAYFORM2 In particular, if DISPLAYFORM3 then E k is decreasing. When equality holds in, DISPLAYFORM4 Most of the stated above are already known, but for completeness we refer the proofs in Appendix A. Since (FE-C) is equivalent to Nesterov's method, the rate is known. The proof of the rate using a Liapunov function can be found in BID4. Refer to? which shows that we can use the constant time step. The discrete Liapunov function was used in Su et al. FORMULA5; to prove a rate.3.2 STRONGLY CONVEX CASE: CONTINUOUS AND DISCRETE TIME Definition 3.3. Define the continuous time Liapunov function E(x, v) DISPLAYFORM5 Define the discrete time Liapunov function by DISPLAYFORM6 Proposition 3.4. Let (x, v) be the solution of (1st-ODE-SC), then DISPLAYFORM7 In particular, for all t > 0, DISPLAYFORM8, we have DISPLAYFORM9 In particular, for h = DISPLAYFORM10 The discrete Liapunov function E k was used to prove a rate in the strongly convex case by. The proof of can be found in Wilson et al. (2016, Theorem 6). For completeness we also provide the proof in Appendix E. In the appendix we present in continuous and discrete time for (non-accelerated) stochastic gradient descent. We also present in continuous time for the stochastic accelerated case in the Appendix. We present the in discrete time here. In this section we consider stochastic gradients, which we write as a gradient plus an error term DISPLAYFORM0 The stochastic gradient can be abstract, or it can error be a mini-batch gradient when f is a sum. Moreover, we can include the case where DISPLAYFORM1 corresponding to a correction by a snapshot of the full gradient at a snapshot location, which is updated every m iterations, as inJohnson &. The combination of gradient reduction and momentum was discussed in.In order to obtain the accelerated rate, our Liapuonov analysis requires that the |e i | be decreasing fast enough. This can also be accomplished in the minibatch setting by using larger minibatches. In this case, the rate of decrease of e i required gives a schedule for minibatch sizes. A similar was obtained in.When we replace gradients with the Forward Euler scheme (FE-C) becomes DISPLAYFORM2 where y k is given by, h is a constant time step, and t k:= h(k + 2). In Appendix C, we study the continuous version of (Sto-FE-C) and obtain a rate of convergence using a Liapunov function. Definition 4.1. Define the discrete stochastic Liapunov functionẼ k:= E k + I k, for k ≥ 0, where E k is given by and and, e −1:= 0 and for k ≥ 0, DISPLAYFORM3 Theorem 4.2. Assume that the sequence e k satisfies DISPLAYFORM4 We immediately have the following . DISPLAYFORM5 Remark 4.4. The assumption on e k is satisfied, for example, by a sequence of the form |e k | = 1/k α for any α > 2. By comparison for SGD, the corresponding condition is satisfied by such sequences with α > 1. Thus the norm of the noise needs to go to zero faster for accelerated SGD compared to regular SGD (see Appendix B) in order to obtain the rate. Remark 4.5. In Theorem 4.2, we focus on the maximum possible time step h = 1/ √ L. The is still true if we shrink the time step. In this case, I k can be defined using the tails h DISPLAYFORM6 *, e i−1, see. In this section, we consider that stochastic gradient, which we write as a gradient plus an error, as in section 4.1. In Appendix B.2, we study the Stochastic gradient descent and Appendix C.2 is devoted to the analysis of the continuous framework of Stochastic Accelerated method. The Forward Euler scheme (FE-SC) becomes DISPLAYFORM0 where e k is a given error and DISPLAYFORM1 Inspired by the continuous framework (Appendix C.2), we define a discrete Lyapunov function. Definition 4.6. DefineẼ k:= E k + I k, where E k is given by and DISPLAYFORM2 with the convention e −1 = 0.Then we obtain the following convergence for sequences generated by (Sto-FE-SC). Theorem 4.7. Let x k, v k be two sequences generated by the scheme (Sto-FE-SC) with initial con- DISPLAYFORM3 Then,Ẽ DISPLAYFORM4 In addition, sup i≥0 |v i − x * | ≤ M for a positive constant M and DISPLAYFORM5 We include the proof of Theorem 4.7 since this is new. Proof of Theorem 4.7. First we prove that DISPLAYFORM6 For the term I k, we obtain DISPLAYFORM7 Putting all together, we obtaiñ DISPLAYFORM8 And by definition of v k+1 − v k, we havẽ DISPLAYFORM9 We conclude, as in the convex case, applying discrete Gronwall Lemma and. DISPLAYFORM10 The proof is concluded by convexity, DISPLAYFORM11 Proof of Proposition 3.4. Using (1st-ODE-SC), we obtain DISPLAYFORM12 By strong convexity, we have DISPLAYFORM13 Let e: [0, +∞) → R d be a integrable function. Consider the gradient descenṫ DISPLAYFORM0 Then define the Lyapunov function,Ẽ, bỹ DISPLAYFORM1 where, DISPLAYFORM2 and, DISPLAYFORM3 Then the following holds. Proposition B.1. Let x be a solution of with initial condition x 0. Then, DISPLAYFORM4 Proof. For all t > 0, we have DISPLAYFORM5 Then, since f is convex, we obtain the first . We deduce thatẼ is decreasing. Arguing as along with the co-coercivity inequality, we prove that sup s≥0 |x(s) − x * | < +∞, sup s≥0 s|∇f (x(s))| < +∞ which concludes the proof. The discretization of FORMULA5 is DISPLAYFORM6 where e k = e(hk). DISPLAYFORM7 and, DISPLAYFORM8 Proposition B.2. Let x k be the sequence generated by with initial condition x 0. Assume that h satisfies, for all k ≥ 0, DISPLAYFORM9 Proof. By L-smoothness and convexity of f, we have DISPLAYFORM10 In addition, DISPLAYFORM11 when h satisfies. We conclude the proof with the same argument as Proposition B.1. Let us study the equationẋ DISPLAYFORM0 for an error function, e satisfying +∞ 0 e µs |e(s)| ds < +∞.This condition on the error function is classical. The case e = 0 is satisfied trivially and corresponds to the gradient descent ODE.We define the function DISPLAYFORM1 where, DISPLAYFORM2 Then we have the following . Proposition B.3. Let x be a solution of with initial data x 0 and suppose that e satisfies. DISPLAYFORM3 In addition, sup t≥0 |x − x * | < +∞ and DISPLAYFORM4 Therefore E(t, x(t)) is decreasing and then for all t > 0, DISPLAYFORM5 By Gronwall Lemma and FORMULA5, we deduce that sup t≥0 |x − x * | < +∞ and the proof is concluded. The discretization of is DISPLAYFORM6 where e k = e(hk). We define E k, for k ≥ 1, by DISPLAYFORM7 where, DISPLAYFORM8 with the notation e −1 = 0. DISPLAYFORM9 In addition, if the sequence (1 − hµ) −i |e i | is summable, sup i≥1 |x i − x * | < +∞ and we deduce, DISPLAYFORM10 Proof. First, as usual, we have DISPLAYFORM11 In addition, DISPLAYFORM12 Combining these two inequalities, DISPLAYFORM13 In order to conclude, we also need to establish that E k is bounded below. That follows from discrete Gronwall's inequality, as was already done in the continuous case in Proposition B.3. In this section, we consider that an error e(t) is made in the calculation of the gradient. We study the following perturbation of system (1st-ODE), DISPLAYFORM0 where e is a function satisfying DISPLAYFORM1 The corresponding ODE is DISPLAYFORM2 We follow the argument from Attouch et al. (2016, section 5) to define a Lyapunov function for this system. LetẼ be defined byẼ DISPLAYFORM3 where, DISPLAYFORM4 and DISPLAYFORM5 Lemma C.1. Let (x, v) be a solution of (Sto-1st-ODE) with initial condition (x, v) = (x 0, v 0) and suppose that e satisfies. Then DISPLAYFORM6 In addition, sup t≥0 |v(t) − x * | < +∞ and sup t≥0 |t∇f (x)| < +∞.Proof. Following the proof of Proposition 3.2, we have DISPLAYFORM7 In particular,Ẽ is decreasing and DISPLAYFORM8 Using the inequality of co-coercitivity, we obtain 1 2L DISPLAYFORM9 Using FORMULA5, we conclude applying Gronwall Lemma. Then we deduce Proposition C.2. Let (x, v) be a solution of (Sto-1st-ODE) with initial condition (x, v) = (x 0, v 0) and suppose that e satisfies. Then, DISPLAYFORM10 Define the perturbed system of (1st-ODE-SC) by DISPLAYFORM0 where e is a locally integrable function. Definition C.3. Define the continuous time Liapunov function E(x, v) DISPLAYFORM1 Define the perturbed Liapunov functionẼ, bỹ E(t, x, v):= E(x, v) + I(t, x, v), DISPLAYFORM2 Lemma C.5. Suppose f is bounded from below and s → e √ µs e(s) ∈ L 1. Let (x, v) be a solution of (Sto-1st-ODE-SC), then sup t≥0 |v(t) − x * | < +∞ and sup t≥0 |∇f (x)| < +∞.Proof. Same as Attouch et al. (2016, Lemma 5.2), using the fact that DISPLAYFORM3 ) is decreasing and Gronwall's inequality. Then, combining the two previous , we obtain: Corollary C.6. Suppose that s → e λs e(s) is a L 1 (0, +∞) function. Let (x, v) be a solution of (Sto-1st-ODE-SC) with initial condition (x, v) = (x 0, v 0). Then, DISPLAYFORM4 where, DISPLAYFORM5 Proof. By Proposition C.4 and Gronwall's Lemma, we havẽ DISPLAYFORM6 This is equivalent to DISPLAYFORM7 which concludes the proof with Lemma C.5. First, using the convexity and the L-smoothness of f, we obtain the following classical inequality (see or in the case e k = 0), DISPLAYFORM0 Then, we have DISPLAYFORM1 By defintion of v k+1, we have DISPLAYFORM2 In addition, DISPLAYFORM3 Combining these three previous inequalities, we obtaiñ DISPLAYFORM4, we deduce thatẼ k is decreasing. In particular, DISPLAYFORM5 and the discrete version of Gronwall Lemma gives the since (i+3)|e i | is a summable sequence due to.E PROOF OF PROPOSITION 3.4To simplify, we denote λ h = h √ µ 1+h √ µ. Note, however, since the gradients are evaluated at y k, not x k, the first step is to use strong convexity and L-smoothness to estimate the differences of E in terms of gradients evaluated at y k. Lemma E.1. Suppose that f is a µ-stgrongly convex and L-smooth function, then DISPLAYFORM6 Proof. First, we remark that DISPLAYFORM7 Since the first line of (1st-ODE-SC) can be rewritten as DISPLAYFORM8 we obtain.Proof of Proposition 3.4. Once FORMULA25 is established, since the expression on the right hand side is monotone in h, the largest choice of h is given by h = 1 √ L, which leads immediately to.In the proof we will estimate the linear term y k − x k, ∇f (y k) in terms of y k − x *, ∇f (y k) plus a correction which is controlled by the gap (the negative quadratic) in and the quadratic term in E.The second term in the Liapunov function gives, using 1-smoothness of the quadratic term in E. µ DISPLAYFORM9

We derive Nesterov's method arises as a straightforward discretization of an ODE different from the one in Su-Boyd-Candes and prove acceleration the stochastic case

represent the short summary for finding the corresponding scientific article (long).

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We propose learning to transfer learn (L2TL) to improve transfer learning on a target dataset by judicious extraction of information from a source dataset. L2TL considers joint optimization of vastly-shared weights between models for source and target tasks, and employs adaptive weights for scaling of constituent losses. The adaptation of the weights is based on reinforcement learning, guided with a performance metric on the target validation set. We demonstrate state-of-the-art performance of L2TL given fixed models, consistently outperforming fine-tuning baselines on various datasets. In the regimes of small-scale target datasets and significant label mismatch between source and target datasets, L2TL outperforms previous work by an even larger margin.

We propose learning to transfer learn (L2TL) to improve transfer learning on a target dataset by judicious extraction of information from a source dataset.

represent the short summary for finding the corresponding scientific article (long)

An empirical study that provides a novel perspective on few-shot learning, in which a fine-tuning method shows comparable accuracy to more complex state-of-the-art methods in several classification tasks.