Question: Represent the scientific article to find a short TLDR

Question: Deep neural networks (DNNs) typically have enough capacity to fit random data by brute force even when conventional data-dependent regularizations focusing on the geometry of the features are imposed. We find out that the reason for this is the inconsistency between the enforced geometry and the standard softmax cross entropy loss. To resolve this, we propose a new framework for data-dependent DNN regularization, the Geometrically-Regularized-Self-Validating neural Networks (GRSVNet). During training, the geometry enforced on one batch of features is simultaneously validated on a separate batch using a validation loss consistent with the geometry. We study a particular case of GRSVNet, the Orthogonal-Low-rank Embedding (OLE)-GRSVNet, which is capable of producing highly discriminative features residing in orthogonal low-rank subspaces. Numerical experiments show that OLE-GRSVNet outperforms DNNs with conventional regularization when trained on real data. More importantly, unlike conventional DNNs, OLE-GRSVNet refuses to memorize random data or random labels, suggesting it only learns intrinsic patterns by reducing the memorizing capacity of the baseline DNN. It remains an open question why DNNs, typically with far more model parameters than training samples, can achieve such small generalization error. Previous work used various complexity measures from statistical learning theory, such as VC dimension , Radamacher complexity BID1, and uniform stability BID2 BID10, to provide an upper bound for the generalization error, suggesting that the effective capacity of DNNs, possibly with some regularization techniques, is usually limited. However, the experiments by showed that, even with data-independent regularization, DNNs can perfectly fit the training data when the true labels are replaced by random labels, or when the training data are replaced by Gaussian noise. This suggests that DNNs with data-independent regularization have enough capacity to "memorize" the training data. This poses an interesting question for network regularization design: is there a way for DNNs to refuse to (over)fit training samples with random labels, while exhibiting better generalization power than conventional DNNs when trained with true labels? Such networks are very important because they will extract only intrinsic patterns from the training data instead of memorizing miscellaneous details. One would expect that data-dependent regularizations should be a better choice for reducing the memorizing capacity of DNNs. Such regularizations are typically enforced by penalizing the standard softmax cross entropy loss with an extra geometric loss which regularizes the feature geometry BID8; ). However, regularizing DNNs with an extra geometric loss has two disadvantages: First, the output of the softmax layer, usually viewed as a probability distribution, is typically inconsistent with the feature geometry enforced by the geometric loss. Therefore, the geometric loss typically has a small weight to avoid jeopardizing the minimization of the softmax loss. Second, we find that DNNs with such regularization can still perfectly (over)fit random training samples or random labels. The reason is that the geometric loss (because of its small weight) is ignored and only the softmax loss is minimized. This suggests that simply penalizing the softmax loss with a geometric loss is not sufficient to regularize DNNs. Instead, the softmax loss should be replaced by a validation loss that is consistent with the enforced geometry. More specifically, every training batch B is split into two sub-batches, the geometry batch B g and the validation batch B v. The geometric loss l g is imposed on the features of B g for them to exhibit a desired geometric structure. A semi-supervised learning algorithm based on the proposed feature geometry is then used to generate a predicted label distribution for the validation batch, which combined with the true labels defines a validation loss on B v. The total loss on the training batch B is then defined as the weighted sum l = l g + λl v. Because the predicted label distribution on B v is based on the enforced geometry, the geometric loss l g can no longer be neglected. Therefore, l g and l v will be minimized simultaneously, i.e., the geometry is correctly enforced (small l g) and it can be used to predict validation samples (small l v). We call such DNNs Geometrically-Regularized-Self-Validating neural Networks (GRSVNets). See FIG0 for a visual illustration of the network architecture. GRSVNet is a general architecture because every consistent geometry/validation pair can fit into this framework as long as the loss functions are differentiable. In this paper, we focus on a particular type of GRSVNet, the Orthogonal-Low-rank-Embedding-GRSVNet (OLE-GRSVNet). More specifically, we impose the OLE loss on the geometry batch to produce features residing in orthogonal subspaces, and we use the principal angles between the validation features and those subspaces to define a predicted label distribution on the validation batch. We prove that the loss function obtains its minimum if and only if the subspaces of different classes spanned by the features in the geometry batch are orthogonal, and the features in the validation batch reside perfectly in the subspaces corresponding to their labels (see FIG0). We show in our experiments that OLE-GRSVNet has better generalization performance when trained on real data, but it refuses to memorize the training samples when given random training data or random labels, which suggests that OLE-GRSVNet effectively learns intrinsic patterns. Our contributions can be summarized as follows:• We proposed a general framework, GRSVNet, to effectively impose data-dependent DNN regularization. The core idea is the self-validation of the enforced geometry with a consistent validation loss on a separate batch of features.• We study a particular case of GRSVNet, OLE-GRSVNet, that can produce highly discriminative features: samples from the same class belong to a low-rank subspace, and the subspaces for different classes are orthogonal.• OLE-GRSVNet achieves better generalization performance when compared to DNNs with conventional regularizers. And more importantly, unlike conventional DNNs, OLEGRSVNet refuses to fit the training data (i.e., with a training error close to random guess) when the training data or the training labels are randomly generated. This implies that OLE-GRSVNet never memorizes the training samples, only learns intrinsic patterns. Many data-dependent regularizations focusing on feature geometry have been proposed for deep learning BID8; ). The center loss produces compact clusters by minimizing the Euclidean distance between features and their class centers. LDMNet extracts features sampling a collection of low dimensional manifolds. The OLE loss BID8 ) increases inter-class separation and intra-class similarity by embedding inputs into orthogonal low-rank subspaces. However, as mentioned in Section 1, these regularizations are imposed by adding the geometric loss to the softmax loss, which, when viewed as a probability distribution, is typically not consistent with the desired geometry. Our proposed GRSVNet instead uses a validation loss based on the regularized geometry so that the predicted label distribution has a meaningful geometric interpretation. The way in which GRSVNets impose geometric loss and validation loss on two separate batches of features extracted with two identical baseline DNNs bears a certain resemblance to the siamese network architecture BID4 used extensively in metric learning BID3 BID6 BID7; ). The difference is, unlike contrastive loss BID6 and triplet loss in metric learning, the feature geometry is explicitly regularized in GRSVNets, and a representation of the geometry, e.g., basis of the low-rank subspace, can be later used directly for the classification of test data. Our work is also related to two recent papers (; BID0 addressing the memorization of DNNs. empirically showed that conventional DNNs, even with data-independent regularization, are fully capable of memorizing random labels or random data. BID0 argued that DNNs trained with stochastic gradient descent (SGD) tend to fit patterns first before memorizing miscellaneous details, suggesting that memorization of DNNs depends also on the data itself, and SGD with early stopping is a valid strategy in conventional DNN training. We demonstrate in our paper that when data-dependent regularization is imposed in accordance with the validation, GRSVNets will never memorize random labels or random data, and only extracts intrinsic patterns. An explanation of this phenomenon is provided in Section 4. DISPLAYFORM0 As pointed out in Section 1, the core idea of GRSVNet is to self-validate the geometry using a consistent validation loss. To contextualize this idea, we study a particular case, OLE-GRSVNet, where the regularized feature geometry is orthogonal low-rank subspaces, and the validation loss is defined by the principal angles between the validation features and the subspaces. The OLE loss was originally proposed by. Consider a K-way classification problem. DISPLAYFORM0 Let X c denote the submatrix of X formed by inputs of the c-th class. proposed to learn a linear transformation T: R d → R d that maps data from the same class X c into a low-rank subspace, while mapping the entire data X into a high-rank linear space. This is achieved by solving: DISPLAYFORM1 where · * is the matrix nuclear norm, which is a convex lower bound of the rank function on the unit ball in the operator norm . The norm constraint T 2 = 1 is imposed to avoid the trivial solution T = 0. It is proved by that the OLE loss is always nonnegative, and the global optimum value 0 is obtained if TX c ⊥TX c, ∀c = c. BID8 later used OLE loss as a data-dependent regularization for deep learning. Given a baseline DNN that maps a batch of inputs X into the features Z = Φ(X; θ), the OLE loss on Z is DISPLAYFORM2 The OLE loss is later combined with the standard softmax loss for training, and we will henceforth call such network "softmax+OLE." Softmax+OLE significantly improves the generalization performance, but it suffers from two problems because of the inconsistency between the softmax loss and the OLE loss: First, the learned features no longer exhibit the desired geometry of orthogonal low-rank subspaces. Second, as will be shown in Section 4, softmax+OLE is still capable of memorizing random data or random labels, i.e., it has not reduced the memorizing capacity of DNNs. We will now explain how to incorporate OLE loss into the GRSVNet framework. First, let us better understand the geometry enforced by the OLE loss by stating the following theorem. DISPLAYFORM0.e., the column spaces of Z c and Z c are orthogonal. The proof of Theorem 1, as well as those of the remaining theorems, is detailed in the Appendix. Note that Theorem 1, which ensures that the OLE loss is minimized if and only if features of different classes are orthogonal, is a much stronger than that by. We then need to define a validation loss l v that is consistent with the geometry enforced by l g. A natural choice would be the principal angles between the validation features and the subspaces spanned by {Z c} K c=1. Now we detail the architecture for OLE-GRSVNet. Given a baseline DNN, we split every training batch X ∈ R d×|B| into two sub-batches, the geometry batch X g ∈ R d×|Bg| and the validation batch X v ∈ R d×|Bv|, which are mapped by the same baseline DNN into features Z g = Φ(X g ; θ) and DISPLAYFORM1 ) is imposed on the geometry batch to ensure span(Z For any feature z = Φ(x; θ) ∈ Z v in the validation batch, its projection onto the subspace span(Z g c) is proj c (z) = U c U * c z. The cosine similarity between z and proj c (z) is then defined as the (unnormalized) probability of x belonging to class c, i.e., DISPLAYFORM2 where a small ε is chosen for numerical stability. The validation loss for x is then defined as the cross entropy between the predicted distributionŷ = (ŷ 1, . . .,ŷ K) T ∈ R K and the true label y ∈ {1, . . ., K}. More specifically, let Y v ∈ R 1×|Bv| andŶ v ∈ R K×|Bv| be the collection of true labels and predicted label distributions on the validation batch, then the validation loss is defined as DISPLAYFORM3 where δ y is the Dirac distribution at label y, and H(·, ·) is the cross entropy between two distributions. The empirical loss l on the training batch X is then defined as DISPLAYFORM4 See FIG0 for a visual illustration of the OLE-GRSVNet architecture. Because of the consistency between l g and l v, we have the following theorem: Theorem 2. For any λ > 0, and any geometry/validation splitting of X = [X g, X v] satisfying X v contains at least one sample for each class, the empirical loss function defined in is always nonnegative. l(X, Y) = 0 if and only if both of the following conditions hold true: Figure 2: Training and testing accuracy of different networks on the SVHN dataset with random labels or random data (Gaussian noise). Note that softmax, sotmax+wd, and softmax+OLE can all perfectly (over)fit the random training data or training data with random labels. However, OLE-GRSVNet refuses to fit the training data when there is no intrinsically learnable patterns.• The features of the geometry batch belonging to different classes are orthogonal, i.e., Moreover, if l < ∞, then rank(span(Z g c)) ≥ 1, ∀c, i.e., Φ(·; θ) does not trivially map data into 0. DISPLAYFORM5 Remark: The requirement that λ > 0 is crucial in Theorem 2, because otherwise the network can map every input into 0 and achieve the minimum. This is validated in our numerical experiments. Before delving into the implementation details of OLE-GRSVNet, we first present two toy experiments to illustrate our proposed framework. We use VGG-11 as the baseline architecture, and compare the performance of the following four DNNs: (a) The baseline network with a softmax classifier (softmax). (b) VGG-11 with weight decay (softmax+wd). (c) VGG-11 regularized by penalizing the softmax loss with the OLE loss (softmax+OLE) (d) OLE-GRSVNet. We first train these four DNNs on the Street View House Numbers (SVHN) dataset with the original data and labels without data augmentation. The test accuracy and the PCA embedding of the learned test features are shown in FIG0. OLE-GRSVNet has the highest test accuracy among the comparing DNNs. Moreover, because of the consistency between the geometric loss and the validation loss, the test features produced by OLE-GRSVNet are even more discriminative than softmax+OLE: features of the same class reside in a low-rank subspace, and different subspaces are (almost) orthogonal. Note that in FIG0, features of only four classes out of ten (though ideally it should be three) have nonzero 3D embedding (Theorem 2).Next, we train the same networks, without changing hyperparameters, on the SVHN dataset with either (a) randomly generated labels, or (b) random training data (Gaussian noise). We train the DNNs for 800 epochs to ensure their convergence, and the learning curves of training/testing accuracy are shown in Figure 2. Note that the baseline DNN, with either data-independent or conventional data-dependent regularization, can perfectly (over)fit the training data, while OLE-GRSVNet refuses to memorize the training data when there are no intrinsically learnable patterns. In another experiment, we generate three classes of one-dimensional data in R 10: the data points in the i-th class are i.i.d. samples from the Gaussian distribution with the standard deviation in the i-th coordinate 50 times larger than other coordinates. Each class has 500 data points, and we randomly shuffle the class labels after generation. We then train a multilayer perceptron (MLP) with 128 neurons in each layer for 2000 epochs to classify these low dimensional data with random labels. We found out that only three layers are needed to perfectly classify these data when using a softmax classifier. However, after incrementally adding more layers to the baseline MLP, we found out that OLE-GRSVNet still refuses to memorize the random labels even for 100-layer MLP. This further suggests that OLE-GRSVNet refuses to memorize training data by brute force when there is no intrinsic patterns in the data. A visual illustration of this experiment is shown in the Appendix. We provide an intuitive explanation for why OLE-GRSVNet can generalize very well when given true labeled data but refuses to memorize random data or random labels. By Theorem 2, we know that OLE-GRSVNet obtains its global minimum if and only if the features of every random training batch exhibit the same orthogonal low-rank-subspace structure. This essentially implies that OLEGRSVNet is implicitly conducting O(N |B|)-fold data augmentation, where N is the number of training data, and |B| is the batch size, while conventional data augmentation by the manipulation of the inputs, e.g., random cropping, flipping, etc., is typically O(N). This poses a very interesting question: Does it mean that OLE-GRSVNet can also memorize random data if the baseline DNN has exponentially many model parameters? Or is it because of the learning algorithm (SGD) that prevents OLE-GRSVNet from learning a decision boundary too complicated for classifying random data? Answering this question will be the focus of our future research. Most of the operations in the computational graph of OLE-GRSVNet FIG0 ) explained in Section 3 are basic matrix operations. The only two exceptions are the OLE loss (Z g → l g ((Z g))) and the SVD (Z g → (U 1, . . ., U K)). We hereby specify their forward and backward propagations. According to the definition of the OLE loss in, we only need to find a (sub)gradient of the nuclear norm to back-propagate the OLE loss. The characterization of the subdifferential of the nuclear norm is explained by. More specifically, assuming m ≥ n for simplicity, let U ∈ R m×m, Σ ∈ R m×n, V ∈ R n×n be the SVD of a rank-s matrix A. DISPLAYFORM0 ) ] be the partition of U, V, respectively, where U ∈ R m×s and V ∈ R n×s, then the subdifferential of the nuclear norm at A is: DISPLAYFORM1 where · 2 is the spectral norm. Note that to use, one needs to identify the rank-s column space of A, i.e., span(U ) to find a subgradient, which is not necessarily easy because of the existence of numerical error. BID8 intuitively truncated the numerical SVD with a small parameter chosen a priori to ensure the numerical stability. We show in the following theorem using the backward stability of SVD that such concern is, in theory, not necessary. DISPLAYFORM2 and δU 2, δV 2, δA 2 are all O(ε), where ε is the machine error. If rank(A) = s ≤ n, and the smallest singular value DISPLAYFORM3 However, in practice we did observe that using a small threshold (10 −6 in this work) to truncate the numerical SVD can speed up the convergence, especially in the first few epochs of training. With the help of Theorem 3, we can easily find a stable subgradient of the OLE loss in. Unlike the computation of the subgradient in Theorem 3, we have to threshold the singular vectors of Z g c, because the desired output U c should be an orthonormal basis of the low-rank subspace span(Z g c).In the forward propagation, we threshold the singular vectors U c such that the smallest singular value is at least 1/10 of the largest singular value. As for the backward propagation, one needs to know the Jacobian of SVD, which has been explained by BID9. Typically, for a matrix A ∈ R n×n, computing the Jacobian of the SVD of A involves solving a total of O(n 4) 2 × 2 linear systems. We have not implemented the backward propagation of SVD in this work because this involves technical implementation with CUDA API. In our current implementation, the node (U 1, . . ., U K) is detached from the computational graph during back propagation, i.e., the validation loss l v is only propagated back through the path l v →Ŷ v → Z v → θ. Our rational is this: The validation loss l v can be propagated back through two paths: DISPLAYFORM0 The first path will modify θ so that Z v c moves closer to U c, while the second path will move U c closer to Z v c. Cutting off the second path when computing the gradient might decrease the speed of convergence, but numerical experiments suggest that the training process is still well-behaved under such simplification. With such simplification, the only extra computation is the SVD of a mini-batch of features, which is negligible (<5%) when compared to the time of training the baseline network. In this section, we demonstrate the superiority of OLE-GRSVNet when compared to conventional DNNs in two aspects: (a) It has greater generalization power when trained on true data and true labels. (b) Unlike conventionally regularized DNNs, OLE-GRSVNet refuses to memorize the training samples when given random training data or random labels. We use similar experimental setup as in Section 4. The same four modifications to three baseline architectures (VGG-11,16,19 DISPLAYFORM0 The performance of the networks are tested on the following datasets:• MNIST. The MNIST dataset contains 28 × 28 grayscale images of digits from 0 to 9. There are 60,000 training samples and 10,000 testing samples. No data augmentation was used.• SVHN. The Street View House Numbers (SVHN) dataset contains 32 × 32 RGB images of digits from 0 to 9. The training and testing set contain 73,257 and 26,032 images respectively. No data augmentation was used.• CIFAR. This dataset contains 32 × 32 RGB images of ten classes, with 50,000 images for training and 10,000 images for testing. We use "CIFAR+" to denote experiments on CIFAR with data augmentation: 4 pixel padding, 32 × 32 random cropping and horizontal flipping. All networks are trained from scratch with the "Xavier" initialization BID5. SGD with Nesterov momentum 0.9 is used for the optimization, and the batch size is set to 200 (a 100/100 split for geometry/validation batch is used in OLE-GRSVNet). We set the initial learning rate to 0.01, and decrease it ten-fold at 50% and 75% of the total training epochs. For the experiments with true labels, all networks are trained for 100, 160 epochs for MNIST, SVHN, respectively. For CIFAR, we train the networks for 200, 300, 400 epochs for VGG-11, VGG16, VGG-19, respectively. In order to ensure the convergence of SGD, all networks are trained for 800 epochs for the experiments with random labels. The mean accuracy after five independent trials is reported. The weight decay parameter is always set to µ = 10 −4. The weight for the OLE loss in "softmax+OLE" is chosen according to BID8. More specifically, it is set to 0.5 for MNIST and SVHN, 0.5 for CIFAR with VGG-11 and VGG-16, and 0.25 for CIFAR with VGG-19. For OLE-GRSVNet, the parameter λ in is determined by cross-validation. More specifically, we set λ = 10 for MNIST, λ = 5 for SVHN and CIFAR with VGG-11 and VGG-16, and λ = 1 for CIFAR with VGG-19. Table 1 reports the performance of the networks trained on the original data with real or randomly generated labels. The numbers without parentheses are the percentage accuracies on the test data when networks are trained with real labels, and the numbers enclosed in parentheses are the accuracies on the training data when given random labels. Accuracies on the training data with real labels Table 1: Testing or training accuracies when trained on training data with real or random labels. The numbers without parentheses are the percentage accuracies on the testing data when networks are trained with real labels. The numbers enclosed in parentheses are the accuracies on the training data when networks are trained with random labels. The mean accuracy after five independent trials is reported. This suggests that OLE-GRSVNet outperforms conventional DNNs on the testing data when trained with real labels. Moreover, unlike conventional DNNs, OLE-GRSVNet refuses to memorize the training data when trained with random labels. (always 100%) and accuracies on the test data with random labels (always close to 10%) are omitted from the table. As we can see, similar to the experiment in Section 4, when trained with real labels, OLE-GRSVNet exhibits better generalization performance than the competing networks. But when trained with random labels, OLE-GRSVNet refuses to memorize the training samples like the other networks because there are no intrinsically learnable patterns. This is still the case even if we increase the number of training epochs to 2000.We point out that by combining different regularization and tuning the hyperparameters, the test error of conventional DNNs can indeed be reduced. For example, if we combine weight decay, conventional OLE regularization, batch normalization, data augmentation, and increase the learning rate from 0.01 to 0.1, the test accuracy of CIFAR can be pushed to 91.02%. However, this does not change the fact that such network can still perfectly memorize training samples when given random labels. This corroborates the claim by that conventional regularization appears to be more of a tuning parameter instead of playing an essential role in reducing network capacity. We proposed a general framework, GRSVNet, for data-dependent DNN regularization. The core idea is the self-validation of the enforced geometry on a separate batch using a validation loss consistent with the geometric loss, so that the predicted label distribution has a meaningful geometric interpretation. In particular, we study a special case of GRSVNet, OLE-GRSVNet, which is capable of producing highly discriminative features: samples from the same class belong to a low-rank subspace, and the subspaces for different classes are orthogonal. When trained on benchmark datasets with real labels, OLE-GRSVNet achieves better test accuracy when compared to DNNs with different regularizations sharing the same baseline architecture. More importantly, unlike conventional DNNs, OLE-GRSVNet refuses to memorize and overfit the training data when trained on random labels or random data. This suggests that OLE-GRSVNet effectively reduces the memorizing capacity of DNNs, and it only extracts intrinsically learnable patterns from the data. Although we provided some intuitive explanation as to why GRSVNet generalizes well on real data and refuses overfitting random data, there are still open questions to be answered. For example, what is the minimum representational capacity of the baseline DNN (i.e., number of layers and number of units) to make even GRSVNet trainable on random data? Or is it because of the learning algorithm (SGD) that prevents GRSVNet from learning a decision boundary that is too complicated for random samples? Moreover, we still have not answered why conventional DNNs, while fully capable of memorizing random data by brute force, typically find generalizable solutions on real data. These questions will be the focus of our future work. It suffices to prove the case when K = 2, as the case for larger K can be proved by induction. In order to simplify the notation, we restate the original theorem for K = 2:Theorem. Let A ∈ R N ×m and B ∈ R N ×n be matrices of the same row dimensions, and [A, B] ∈ R N ×(m+n) be the concatenation of A and B. We have DISPLAYFORM0 Moreover, the equality holds if and only if A * B = 0, i.e., the column spaces of A and B are orthogonal. Proof. The inequality and the sufficient condition for the equality to hold is easy to prove. More specifically, DISPLAYFORM1 Moreover, if A * B = 0, then DISPLAYFORM2 where |A| = (A * A) 1 2. Therefore, DISPLAYFORM3 Next, we show the necessary condition for the equality to hold, i.e., DISPLAYFORM4 DISPLAYFORM5 | be a symmetric positive semidefinite matrix. We DISPLAYFORM6 Let DISPLAYFORM7 be the orthonormal eigenvectors of |A|, |B|, respectively. Then DISPLAYFORM8 Similarly, DISPLAYFORM9 Suppose that [A, B] * = A * + B *, then DISPLAYFORM10 Therefore, both of the inequalities in this chain must be equalities, and the first one being equality only if G = 0. This combined with the last equation in FORMULA2 implies DISPLAYFORM11 APPENDIX B PROOF OF THEOREM 2Proof. First, l is defined in equation FORMULA8 as DISPLAYFORM12 The nonnegativity of l g (Z g) is guaranteed by Theorem 1. The validation loss l v (Y v,Ŷ v) is also nonnegative since it is the average (over the validation batch) of the cross entropy losses: DISPLAYFORM13 Therefore l = l g + λl v is also nonnegative. Next, for a given λ > 0, l(X, Y) obtains its minimum value zero if and only if both l g (Z g) and l v (Y v,Ŷ v) are zeros.• By Theorem 1, l g (Z g) = 0 if and only if span(Z g c)⊥ span(Z g c), ∀c = c.• According to, l v (Y v,Ŷ v) = 0 if and only ifŷ(x) = δ y, ∀x ∈ X v, i.e., for every x ∈ X v c, its feature z = Φ(x; θ) belongs to span(Z g c).At last, we want to prove that if λ > 0, and X v contains at least one sample for each class, then rank(span(Z g c)) ≥ 1 for any c ∈ {1, . . ., K}. If not, then there exists c ∈ {1, . . ., K} such that rank(span(Z g c)) = 0. Let x ∈ X v be a validation datum belonging to class y = c. The predicted probability of x belonging to class c is defined in: DISPLAYFORM14 Thus we have DISPLAYFORM15

Represent the TLDR for finding the corresponding scientific article.

we propose a new framework for data-dependent DNN regularization that can prevent DNNs from overfitting random data or random labels.

represent the short summary for finding the corresponding scientific article (long):

For instance, <<Towards Efficient Credit Assignment in Recurrent Networks without Backpropagation Through Time>>  to "A major drawback of backpropagation through time (BPTT) is the difficulty of learning long-term dependencies, coming from having to propagate credit information backwards through every single step of the forward computation. This makes BPTT both computationally impractical and biologically implausible. For this reason, full backpropagation through time is rarely used on long sequences, and truncated backpropagation through time is used as a heuristic. However, this usually leads to biased estimates of the gradient in which longer term dependencies are ignored. Addressing this issue, we propose an alternative algorithm, Sparse Attentive Backtracking, which might also be related to principles used by brains to learn long-term dependencies. Sparse Attentive Backtracking learns an attention mechanism over the hidden states of the past and selectively backpropagates through paths with high attention weights. This allows the model to learn long term dependencies while only backtracking for a small number of time steps, not just from the recent past but also from attended relevant past states. Recurrent Neural Networks (RNNs) are state-of-the-art for many machine learning sequence processing tasks. Examples where models based on RNNs shine include speech recognition BID21 BID3, image captioning BID32 BID29 BID17, machine translation BID1 BID26 BID18, and speech synthesis BID20. It is common practice to train these models using backpropagation through time (BPTT), wherein the network states are unrolled in time and gradients are backpropagated through the unrolled graph. Since the parameters of an RNN are shared across the different time steps, BPTT is more prone to vanishing and exploding gradients (; BID2 BID11 than equivalent deep feedforward networks with as many stages. This makes credit assignment particularly difficult for events that have occurred many time steps in the past, and thus makes it challenging in practice to capture long-term dependencies in the data (; BID2 . Having to wait for the end of the sequence in order to compute gradients is neither practical for machines nor animals when the dependencies extend over very long timescales. Training is slowed down considerably by long waiting times, as the rate of convergence crucially depends on how often parameters can be updated. In practice, proper long-term credit assignment in RNNs is very inconvenient, and it is common practice to employ truncated versions of BPTT for long sequences BID23 BID24 . In truncated BPTT (TBPTT), gradients are backpropagated only for a fixed and limited number of time steps and parameters are updated after each such subsequence. Truncation is often motivated by computational concerns: memory, computation time and the advantage of faster learning obtained when making more frequent updates of the parameters rather than having to wait for the end of the sequence. However, it makes capturing correlations across distant states even harder. Regular RNNs are parametric: their hidden state vector has a fixed size. We believe that this is a critical element in the classical analysis of the difficulty of learning long-term dependencies BID2. Indeed, the fixed state dimension becomes a bottleneck through which information has to flow, both forward and backward. We thus propose a semi-parametric RNN, where the next state is potentially conditioned on all the previous states of the RNN, making it possible-thanks to attention-to jump through any distance through time. We distinguish three types of states in our proposed semi-parametric RNN:• The fixed-size hidden state h (t), the conventional state of an RNN model at time t;• The monotonically-growing macrostate M = {m,..., m (s) }, the array of all past microstates, which plays the role of a random-access memory;• And the fixed-size microstate m (i), which is the ith hidden state (one of the h (t) ) that was chosen for inclusion within the macrostate M. There are as many hidden states as there are timesteps in the sequence being analyzed by the RNN. A subset of them will become microstates, and this subset is called the macrostate. The computation of the next hidden state h (t+1) is based on the whole macrostate M, in addition to the external input x (t). The macrostate being variable-length, we must devise a special mechanism to read from this ever-growing array. As a key component of our model, we propose to use an attention mechanism over the microstate elements of the macrostate. The attention mechanism in the above setting may be regarded as providing adaptive, dynamic skip connections: any past microstate can be linked, via a dynamic decision, to the current hidden state. Skip connections allow information to propagate over very long sequences. Such architectures should naturally make it easier to learn long-term dependencies. We name our algorithm sparse attentive backtracking (SAB). SAB is especially well-suited to sequences in which two parts of a task are closely related yet occur very far apart in time. Inference in SAB involves examining the macrostate and selecting some of its microstates. Ideally, SAB will not select all microstates, instead attending only to the most salient or relevant ones (e.g., emotionally loaded, in animals). The attention mechanism will select a number of relevant microstates to be incorporated into the hidden state. During training, local backpropagation of gradients happens in a short window of time around the selected microstates only. This allows for the updates to be asynchronous with respect to the time steps we attend to, and credit assignment takes place more globally in the proposed algorithm. With the proposed framework for SAB, we present the following contributions:• A principled way of doing sparse credit assignment, based on a semi-parametric RNN.• A novel way of mitigating exploding and vanishing gradients, based on reducing the number of steps that need to be backtracked through temporal skip connections.• Competitive compared to full backpropagation through time (BPTT), and much better as compared to Truncated Backpropagation through time, with significantly shorter truncation windows in our model. Mechanisms such as SAB may also be biologically plausible. Imagine having taken a wrong turn on a roadtrip and finding out about it several miles later. Our mental focus would most likely shift directly to the location in time and space where we had made the wrong decision, without replaying in reverse the detailed sequence of experienced traffic and landscape impressions. Neurophysiological findings support the existence of such attention mechanisms and their involvement in credit assignment and learning in biological systems. In particular, hippocampal recordings in rats indicate that brief sequences of prior experience are replayed both in the awake resting state and during sleep, both of which conditions are linked to memory consolidation and learning BID7 BID6 BID8. Moreover, it has been observed that these replay events are modulated by the reward an animal does or does not receive at the end of a task in the sense that they are more pronounced in the presence of a reward signal and less pronounced or absent in the absence of a reward signal BID0. Thus, the mental look back into the past seems to occur exactly when credit assignment is to be performed.2 RELATED WORK 2.1 TRUNCATED BACKPROPAGATION THROUGH TIME When training on very long sequences, full backpropagation through time becomes computationally expensive and considerably slows down training by forcing the learner to wait for the end of each (possibly very long sequence) before making a parameter update. A common heuristic is to backpropagate the loss of a particular time step through only a limited number of time steps, and hence truncate the backpropagation computation graph BID30. While truncated backpropagation through time is heavily used in practice, its inability to perform credit assignment over longer sequences is a limiting factor for this algorithm, ing in failure cases even in simple tasks, such as the Copying Memory and Adding task in BID12. The Decoupled Neural Interfaces method BID14 replaces full backpropagation through time with synthetic gradients, which are essentially small networks, mapping the hidden unit values of each layer to an estimator of the gradient of the final loss with respect to that layer. While training the synthetic gradient module requires backpropagation, each layer can make approximate gradient updates for its own parameters in an asynchronous way by using its synthetic gradient module. Thus, the network learns how to do credit assignment for a particular layer from a few examples of the gradients from backpropagation, reducing the total number of times that backpropagation needs to be performed. Online credit assignment in RNNs without backtracking remains an open research problem. One approach BID22 attempts to solve this problem by estimating gradients using an approximation to forward mode automatic differentiation instead of backpropagation. Forward mode automatic differentiation allows for computing unbiased gradient estimates in an online fashion, however it normally requires storage of the gradient of the current hidden state values with respect to the parameters, which is O(N 3) where N is the number of hidden units. The Unbiased Online Recurrent Optimization (UORO) ) method gets around this by updating a rank-1 approximation to this gradient tensor, which is shown to keep the estimate of the gradient unbiased, but potentially at the risk of increasing the variance of the gradient estimator. Neural architectures such as Residual Networks BID9 and Dense Networks BID13 ) allow information to skip over convolutional processing blocks of an underlying convolutional network architecture. In the case of Residual Networks identity connections are used to skip over convolutional processing blocks and this information is recombined using addition. This construction provably mitigates the vanishing gradient problem by allowing the gradient at any given layer to be bounded. Densely-connected convolutional networks alleviate the vanishing gradient problem by allowing a direct path from any point in the network to the output. In contrast, here we propose and explore what one might regard as a form of dynamic skip connection, modulated by an attention mechanism. We now introduce the idea of Sparse Attentive Backtracking (SAB). Classical RNN models such as those based on LSTMs or GRUs only use the previous hidden state in the computation of the next one, and therefore struggle with extremely long-range dependencies. SAB sidesteps this limitation by additionally allowing the model to select and use (a subset of) any of the past microstates in the computation of the next hidden state. In doing so the model may potentially reference microstates computed arbitrarily long ago in time. Since the classic RNN models do not support such operations on their past, we make a few architectural additions. On the forward pass of a training step, a mechanism is introduced that selects microstates from the macrostate, summarizes them, then incorporates this summary into the next hidden state. The hidden state may or may not become a microstate. On the backward pass, the gradient is allowed to flow not only through the (truncated) master chain linking consecutive hidden states, but also to the microstates which are selected in the forward pass. In the forward pass, the microstate selection process can be denser or sparser, and the summarization and incorporation can be more or less sophisticated. In the backward pass, the gating of gradient flow from a hidden state to its ancestor microstates can also be denser or sparser, although it can be no denser than the forward pass was. t{−8,−7,−6,−5,−4,−3,−2,−1} are evaluated, first by broadcasting the current provisional hidden stateĥ (t) against the macrostate (which, in the presented case of k att = 1, consists of all past hidden states), concatenating, then passing the to an MLP. The attention weights are then run through the sparsifier which selects the k top = 3 attention weights, while the others are zeroed out. Black arrows show the microstates corresponding to the non-zero sparse attention weights {ã DISPLAYFORM0}, being weighted, summed, then incorporated intoĥ (t) to compute the current final hidden state h (t).For instance, it is possible for the forward pass to be dense, incorporating a summary of all microstates, but for the backward pass to be sparse, only allowing gradient flow to some of the microstate contributors to the hidden state (Dense Forward, Sparse Backward). Another possibility is for the forward pass to be sparse, making only a few, hard, microstate selections for the summary. In this case, the backward pass will necessarily also be sparse, since few microstates will have contributed to the hidden state, and therefore to the loss (Sparse Forward, Sparse Backward).Noteworthy is that not all hidden states need be eligible to become microstates. In practice, we have found that restricting the pool of eligible hidden states to only every k att'th one still works well, while reducing both memory and computation expense. Such an increase in the granularity of microstate selection can also improve performance, by preventing the model from attending exclusively to the most recent hidden states and temporally spreading microstates out from each other. The SAB algorithm is widely applicable, and is compatible with numerous RNN architectures, including vanilla, GRU and LSTM models. However, since it necessarily requires altering the hiddento-hidden transition function substantially, it's currently incompatible with the accelerated RNN kernels offered by e.g. NVIDIA on its GPU devices through cuDNN library BID5.For vanilla and GRU-inspired RNN architectures, SAB's selection and incorporation mechanisms operate over the (hidden) state. For the LSTM architecture, which we adopt for our experiments, they operate over the hidden state but not the cell state. The microstate selection mechanism determines which microstate subset of the macrostate will be selected for summarization on the forward pass of the RNN, and which subset of that subset will receive gradient on the backward pass during training. This makes it the core of the attention mechanism of a SAB implementation. While the selection mechanism may use hard-coded attention heuristics, there is no reason why the microstate selection mechanism could not itself be a (deep) neural network trained alongside the RNN model over which it operates. In the models we use here, the selection mechanism is chosen to be a 1-hidden-layer Linear-TanhLinear MLP that computes a scalar attention weight a i for each eligible microstate vector m (i), and a sparsifier that masks out all but the k top greatest attention weights, producing the sparse attention weightsã i. We empirically demonstrate that even this simple mechanism learns to focus on past time steps relevant to the current one, thus successfully performing credit assignment. The use of a higher complexity model here would be an interesting avenue for future research. The selected microstates must be somehow combined into a fixed-size summary for incorporation into the next hidden state. While many options exist for doing so, we choose to simply perform a summation of the microstates, weighted by their sparsified attention weightã i. Lastly, the summary must be incorporated into the hidden state. Again, multiple options exist, such as addition (as done in ResNets) or concatenation (as done in DenseNets).For our purposes we choose to simply sum the summary into the provisional hidden state output h (t) computed by the LSTM cell to produce the final hidden state h (t) that will be conditioned upon in the next timestep. We now give the equations for the specific SAB-augmented LSTM model we use in our experiments. At time t, the underlying LSTM receives a vector of hidden states h (t−1), a vector of cell states c (t−1) and an input x (t), and computes a provisional hidden state vectorĥ (t) that also serves as a provisional output. We next use an attention mechanism that is similar to BID1, but modified to produce sparse discrete attention decisions. First, the provisional hidden state vectorĥ (t) is concatenated to each microstate vector m (i). Then, an MLP maps each such concatenated vector to an attention weight a (t)i representing the salience of the microstate i at the current time t. This can be expressed as: a DISPLAYFORM0 DISPLAYFORM1 where the weights matrices w 1, w 2 and w 3 are learned parameters. Following this, we apply a piece-wise linear function that sparsifies the attention while making discrete decisions. (This is different from typical attention mechanisms that normalize attention weights using a Softmax function BID1, whose output is never sparse). Let a (t) ktop be the k top th greatest-valued attention weight at time t; then the sparsified attention weights are computed asã DISPLAYFORM2 This has the effect of zeroing all attention weights less than a DISPLAYFORM3 ktop, thus masking out all but the k top most salient microstates in M. The few selected microstates receive gradient information, while no gradient flows to the rest. A summary vector s (t) is then obtained using a weighted sum over the macrostate, employing the sparse attention weights: DISPLAYFORM4 Given that this sum is very sparse, the summary operation is very fast. To incorporate the summary into the final hidden state at timestep t, we simply sum the summary and the provisional hidden state: DISPLAYFORM5 Lastly, to compute the output at the time step t, we concatenate h (t) and the sparse attention weights a (t), then apply an affine output transform to compute the output. This can be equivalently expressed as: DISPLAYFORM6 where the weights matrices V 1 and V 2 and bias vector b are learned parameters. In summary, for a given time step t, a hidden state h (i) selected by the hard-attention mechanism has two paths contributing to the hidden states h (t) in the forward pass. One path is the regular sequential forward path in an RNN; the other path is through the dynamic skip connections in the attention mechanism. When we perform backpropagation through the skip connections, gradient only flows from h (t) to microstates m (i) selected by the attention mechanism (those for which DISPLAYFORM7 In the preparation of this work, it was discovered that the attention mechanism absolutely must include a non-linearity in the computation of the raw attention weights a (t) i. Our failure to do so in an early iteration of the work ed in a catastrophic cancellation in the subsequent sparsification of the weights toã (t) i. This is because in, a rectified difference between a (t) i is computed to zero all but the k top greatest attention weights. Subtraction is linear; And since our earlier attention mechanism was linear as well, it could be separated into two parts, a first half to which only the microstate m (i) contributed and a second half to which only the hidden state h (t) contributed. This second half of the contribution is catastrophically cancelled in the difference a DISPLAYFORM0 ktop, because it was computed from the same h (t) for both, and therefore equal. We now report and discuss the of an empirical study that analyses the performance of SAB using five different tasks. We first study synthetic tasks-the copying and adding problems designed to measure models' abilities to learn long-term dependenciesmeant to confirm that SAB can successfully perform credit assignment for events that have occurred many time steps in the past. We then study more realistic tasks and larger datasets. Baselines We compare the quantitative performance of our model against two LSTM baselines BID12. The first is trained with backpropagation through time (BPTT) and the second is trained using truncated backpropagation through time (TBTPP). Both methods are trained using teacher forcing BID31. We also used gradient clipping (that is, we clip the gradients to 1 to avoid exploding gradients). Hyperparameters that are task-specific are discussed in the tasks' respective subsections, other hyperparameters that are also used by SAB and that we set to the same value are discussed below. Compared to standard RNNs, our model has two additional hyperparameters:• k top, the number of most-salient microstates to select at each time step for passing gradients in the backward pass• k att, the granularity of attention. Every k att th hidden state is chosen to be a microstate. The special case k att = 1 corresponds to choosing all hidden states to be microstates as well. In addition, we also study the impact of the TBPTT truncation length, which we denote as k trunc. This determines how many timesteps backwards to propagate gradients through in the backward pass. This effect of this hyperparameter will also be studies for the LSTM with TBTPP baseline. For all experiments we used a learning rate of 0.001 with the Adam BID15 optimizer unless otherwise stated. For SAB, we attend to every second hidden states, i.e. k att =2, unless otherwise stated. Our main findings are:1. SAB performs almost optimally and significantly outperforms both full backpropagtion through time (BPTT), and truncated backpropagation through time (TBPTT) on the synthetic copying task.2. For the synthetic adding, two language modelling task (using PennTree Bank and Text8), and permuted sequential MNIST classification tasks, SAB reaches the performance of BPTT and outperforms TBPTT. In addition, for the adding task, SAB outperforms TBPTT using much shorter truncation lengths. The copying memory task tests the model's ability to memorize salient information for long time periods. We follow the setup of the copying memory problem from BID12. In details, the network is given a sequence of T + 20 inputs consisting of: a) 10 (randomly generated) digits (digits 1 to 8) followed by; b) T blank inputs followed by; c) a special end-ofsequence character followed by; d) 10 additional blank inputs. After the end-of-sequence character the network must output a copy of the initial 10 digits. TAB1, and 3 report both accuracy and cross-entropy (CE) of the models' predictions on unseen sequences. We note that SAB is able to learn this copy task almost perfectly for all sequence-lengths T. Further, SAB outperforms all baselines. This is particularly noticeable for longer sequences, for example, when T is 300 the best baseline achieves 35.9% accuracy versus SAB's 98.9%.To better understand the learning process of SAB, we visualized the attention weights while learning the copying task (T = 200, k trunc = 10, k top = 10). FIG5 (appendix) shows the attention weights (averaged over a single mini-batch) at three different learning stages of training, all within the first epoch. We note that the attention quickly (and correctly) focuses on the first ten timesteps which contain the input digits. Furthermore, we experimented with LSTM with self-attention trained using full BPTT. The setup is very similar to unidirectional LSTM with self-attention BID16. Due to GPU memory constraints, we are only able to run this experiment up to small sequence lengths. For T=200 and T = 300, we could see that SAB performs comparably to LSTM with full self-attention trained with full BPTT. The adding task requires the model to sum two specific entries in a sequence of T (input) entries BID12. In the spirit of the copying task, larger values of T will require the model to keep track of longer-term dependencies. The exact setup is as follows. Each example in the task consists of 2 input vectors of length T. The first, is a vector of uniformly generated values between 0 and 1. The second vector encodes binary a mask that indicates which 2 entries in the Table 1: Test accuracy and cross-entropy loss performance on the copying task with sequence lengths of T = 100. Models that use TBPTT cannot solve this task while SAB and BPTT can both achieve optimal performance. Table 3: Test accuracy and cross-entropy loss performance on copying task with sequence lengths of T = 300. On these long sequences SAB's performance can still be very close to optimal. first input to sum (it consists of T − 2 zeros and 2 ones). The mask is randomly generated with the constraint that masked-in entries must be from different halves of the first input vector. TAB2 report the cross-entropy (CE) of the model's predictions on unseen sequences. When T = 200, SAB's performance is similar to the best performance of both baselines. With even longer sequences (T = 400), SAB outperforms the TBPTT but is outperformed by BPTT. We evaluate our model on language modelling task using the Penn TreeBank dataset BID19. Our LSTM baselines use 1000 hidden units and a learning rate of 0.002. We used nonoverlapping sequences of 100 in the batches of 32. We trained SAB for 100 epochs. We evaluate the performance of our model using the bits-per-character (BPC) metric. As shown in Table 6, we perform slightly worse than BPTT, but better than TBPTT. Table 4: Performance on unseen sequences of the T = 200 adding task. We note that all methods have configurations that allow them to perform near optimally. Table 6: BPC evaluation on the validation set of the character-level PTB (lower is better). This dataset is derived from the text of Wikipedia and consists of a sequence of a total of 100M characters (non-alphabetical and non-space characters were removed). We follow the setup of; use the first 90M characters for training, the next 5M for validation and the final 5M characters for testing. We train on non-overlapping sequences of length 180. Due to computational constraints, all baselines use 1000 hidden units. We trained all models using a batch size of 64. We trained SAB for a maximum of 30 epochs. We have not done any hyperparameter search for our model as it's computationally expensive. TAB3: Bit-per-character (BPC) Results on the validation and test set for Text8 (lower is better). Our last task is a sequential version of the MNIST classification dataset. The task involves predicting the label of the image after being given the image pixel by pixel (pixels are processed in a fixed random order.). All models use an LSTM with 128 hidden units. The prediction is produced by passing the final hidden state of the network into a softmax. We used a learning rate of 0.001. We trained our model for about 100 epochs, and did early stopping based on the validation set. An interesting direction for future development of the Sparse Attentive Backtracking method from the machine learning standpoint would be improving the computational efficiency when the sequences in question are very long. Since the Sparse Attentive Backtracking method uses selfattention on every step, the memory requirement grows linearly in the length of the sequence and computing the attention mechanism requires computing a scalar between the current hidden states and all previous hidden states (to determine where to attend). It might be possible to reduce the memory requirement by using a hierarchical model as done by BID4, and then recomputing the states for the lower levels of the hierarchy only when our attention mechanism looks at the corresponding higher level of the hierarchy. It might also be possible to reduce the computational cost of the attention mechanism by considering a maximum inner product search algorithm BID25, instead of naively computing the inner product with all hidden states values in the past. Improving the modeling of long-term dependencies is a central challenge in sequence modeling, and the exact gradient computation by BPTT is not biologically plausible as well as inconvenient computationally for realistic applications. Because of this, the most widely used algorithm for training recurrent neural networks on long sequences is truncated backpropagation through time, which is known to produced biased estimates of the gradient, focusing on short-term dependencies. We have proposed Sparse Attentive Backtracking, a new biologically motivated algorithm which aims to combine the strengths of full backpropagation through time and truncated backpropagation through time. It does so by only backpropagating gradients through paths selected by its attention mechanism. This allows the RNN to learn long-term dependencies, as with full backpropagation through time, while still allowing it to only backtrack for a few steps, as with truncated backpropagation through time, thus making it possible to update weights as frequently as needed rather than having to wait for the end of very long sequences. The time complexity of the forward pass of both training and inference in SAB is O(tn 2), with t the number of timesteps and n the size of the hidden state, although our current implementation scales as O(t 2 n 2). The space complexity of the forward pass of training is unchanged at O(tn), but the space complexity of inference in SAB is now O(tn) rather than O(n). However, the time cost of the backward pass of training cost is very difficult to formulate. Hidden states depend on a sparse subset of past microstates, but each of those past microstates may itself depend on several other, even earlier microstates. The web of active connections is, therefore, akin to a directed acyclic graph, and it is quite possible in the worst case for a backpropagation starting at the last hidden state to touch all past microstates several times. However, if the number of microstates truly relevant to a task is low, the attention mechanism will repeatedly focus on them to the exclusion of all others, and pathological runtimes will not be encountered. Our method approximates the true gradient but in a sense it's no different than the kind of approximation made with truncated gradient, except that instead of truncating to the last k trunc time steps, we truncate to one skip-step in the past, which can be arbitrarily far in the past. This provides a way of combating exploding and vanishing gradient problems by learning long-term dependencies. To verify the fact, we ran our model on all the datasets (Text8, Pixel-By-Pixel MNIST, char level PTB) with and without gradient clipping. We empirically found, that we need to use gradient clipping only for text8 dataset, for all the other datasets we observed little or no difference with gradient clipping. We visualize how the attention weights changes during training for the Copying Memory Task in section 4.1. The attention weights are averaged over the batch. The salient information in a copying task are in the first 10 steps. The figure shows how the attention learns to move towards and concentrate on the beginning of the sequence as training procedes. Note these all happened with the first epoch of training, such that the model learns in a reasonable amount of time."

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: End-to-end automatic speech recognition (ASR) commonly transcribes audio signals into sequences of characters while its performance is evaluated by measuring the word-error rate (WER). This suggests that predicting sequences of words directly may be helpful instead. However, training with word-level supervision can be more difficult due to the sparsity of examples per label class. In this paper we analyze an end-to-end ASR model that combines a word-and-character representation in a multi-task learning (MTL) framework. We show that it improves on the WER and study how the word-level model can benefit from character-level supervision by analyzing the learned inductive preference bias of each model component empirically. We find that by adding character-level supervision, the MTL model interpolates between recognizing more frequent words (preferred by the word-level model) and shorter words (preferred by the character-level model). End-to-end automatic speech recognition (ASR) allows for learning a direct mapping from audio signals to character outputs. Usually, a language model re-scores the predicted transcripts during inference to correct spelling mistakes BID16. If we map the audio input directly to words, we can use a simpler decoding mechanism and reduce the prediction time. Unfortunately, word-level models can only be trained on known words. Out-of-vocabulary (OOV) words have to be mapped to an unknown token. Furthermore, decomposing transcripts into sequences of words decreases the available number of examples per label class. These shortcomings make it difficult to train on the word-level BID2.Recent works have shown that multi-task learning (MTL) BID8 on the word-and character-level can improve the word-error rate (WER) of common end-to-end speech recognition architectures BID2 BID3 BID18 BID21 BID22 BID24 BID29. MTL can be interpreted as learning an inductive bias with favorable generalization properties BID6. In this work we aim at characterizing the nature of this inductive bias in word-character-level MTL models by analyzing the distribution of words that they recognize. Thereby, we seek to shed light on the learning process and possibly inform the design of better models. We will focus on connectionist temporal classification (CTC) BID15. However, the analysis can also prove beneficial to other modeling paradigms, such as RNN Transducers BID14 or Encoder-Decoder models, e.g., BID5 BID9.Contributions. We show that, contrary to earlier negative BID2 BID27, it is in fact possible to train a word-level model from scratch on a relatively small dataset and that its performance can be further improved by adding character-level supervision. Through an empirical analysis we show that the ing MTL model combines the preference biases of word-and character-level models. We hypothesize that this can partially explain why word-character MTL improves on only using a single decomposition, such as phonemes, characters or words. Several works have explored using words instead of characters or phonemes as outputs of the end-toend ASR model BID2 BID27. Soltau et al. BID27 found that in order to solve the problem of observing only few labels per word, they needed to use a large dataset of 120, 000 hours to train a word-level model directly. Accordingly, Audhkhasi et al. BID2 reported difficulty to train a model on words from scratch and instead fine-tuned a pre-trained character-level model after replacing the last dense layer with a word embedding. MTL enables a straightforward joint training procedure to integrate transcript information on multiple levels of granularity. Treating word-and character-level transcription as two distinct tasks allows for combining their losses in a parallel BID21 BID22 BID28 BID29 or hierarchical structure BID13 BID20 BID24. Augmenting the commonly-used CTC loss with an attention mechanism can help with aligning the predictions on both character-and word-level BID3 BID12 BID22. All these MTL methods improve a standard CTC baseline. Finding the right granularity of the word decomposition is in itself a difficult problem. While Li et al. BID22 used different fixed decompositions of words, sub-words and characters, it is also possible to optimize over alignments and decompositions jointly BID23. Orthogonal to these works different authors have explored how to minimize WER directly by computing approximate gradients BID25 BID32.When and why does MTL work? Earlier theoretical work argued that the auxiliary task provides a favorable inductive bias to the main task BID6. Within natural language processing on text several works verified empirically that this inductive bias is favorable if there is a certain notion of relatedness between the tasks BID4 BID7 BID26. Here, we investigate how to characterize the inductive bias learned via MTL for speech recognition. The CTC loss is defined as follows BID15: DISPLAYFORM0 where x is the audio input, commonly a spectrogram, and π is a path that corresponds to the groundtruth transcript z. The squashing function B maps a path π to the output z by first merging repetitions and then deleting so-called blank tokens. The gradient of the CTC loss can be computed efficiently using a modified forward-backward algorithm. Typically, π t is a categorical random variable over the corresponding output alphabet A = {a, b, c, ...,}. Here, is the blank token which encodes the empty string. This output representation enables the model to be able to transcribe any word possible without a specified alignment. Character-level CTC models are often supplemented by an external language model that can significantly improve the accuracy of the ASR. This is because these models still make spelling mistakes despite being trained on large amounts of data BID0.By using an alphabet of words one can ensure that there are no misspellings. The alphabet could contain, for example, the most common words found in the training set. This has the advantage that any word is guaranteed to be spelled correctly and that costly re-scoring on a character-level is avoided. However, by using a word-level decoding, we can no longer predict rare or new words. In this case the model has to be content with outputting an unknown token. Another challenge when using a word-level model is label sparsity. While we will observe many examples of a single character, there will be fewer for a single word, making overfitting more likely. We aim at counter-acting these shortcomings by making use of character-level information during training, similar to Audhkhasi et al. BID2.In this work we combine word-and character-level models via an MTL loss and denote this a word-character-level model. We treat each output-level prediction as a separate task and form a linear combination of the losses. The MTL loss is then defined as DISPLAYFORM1 where λ ≥ 0 defines a hyperparameter to weight the influence of the character-level CTC loss L char against the word-level CTC loss L word. In our experiments we set it to 1, giving equal contribution to both loss terms, but other choices may improve the performance. Alternatively, one could try to estimate this weight based on the uncertainty BID17 or gradient norm BID10 of each loss term. We experimented with these approaches, but did not observe any significant improvement in performance over the equally-weighted loss. We trained our models using a convolutional architecture which is based on Wav2Letter BID11. Details can be found in the appendix. Compared to recurrent neural networks, convolutional neural networks avoid iterative computation over time and suffer less from the vanishing/exploding gradient problem. They achieve comparable performance in terms of WER BID11 BID31. We performed all experiments on read news articles from the Wall Street Journal (WSJ) BID30. This dataset has relatively little noise and allows us to focus on the influence of word frequency and word length. We used the si284 subset for training, and dev92 for validation. For the character-level model we used 32 different characters which include the space-character and a blank token. To define the output alphabet for the word-level model, we included all words that appeared at least 5 times in the training set in addition to a blank and an unknown token. This corresponds to an alphabet of 9411 units with an OOV rate of 9 % on the training set, and 10 % on the validation set, which represents a lower bound for the achievable WER of a word-level model. For the MTL model we let word-and character-level model share every layer but the last. To decode the output on the character-and word-level, we used greedy decoding. In order to get rid of unknown tokens in our prediction, we employed the following heuristic BID21: For each unknown token predicted on the word-level, we substituted the corresponding word on the character-level that was defined at the same time step. To compare our we also trained word-and character-only models. For optimization we used the Adam-optimizer BID19 with a learning rate of 5e−4 and a batch size of 16 to fit the whole model into the memory of one GPU. We applied batch normalization and dropout. For the input data, we transformed each utterance into spectrograms over 20 ms with a shift of 10 ms using 40 log-mel coefficients, standardized per spectrogram. We ran each experiment for 100 epochs, corresponding to 233, 838 updates. MTL performance. The of our experiments can be found in FIG0. It shows the learning curve for the word-and character-level components by measuring the WER on the validation set. The dashed line shows the achieved WER using a character-level model without joint word-level training. We observe that MTL converges faster and to a lower WER of 23 %, which is 5 percentage points lower than the character-level component of the MTL network, or the single-task character-level baseline. Using a beam search decoder with a lexicon constraint on the character-level model reduces the WER from 28 % to a WER of 24 %, which is still higher than our MTL error. This shows that MTL performs favorably even without a language model. A word-level-only model achieved the same performance as the character-level baseline on this dataset. Contrary to the findings of Audhkhasi et al. BID2, this shows that it is indeed possible to train a word-level model from scratch, even without a large amount of training data. While the combined decoding only gives an improvement of 0.7 percentage points in terms of WER, it eliminates unknown-token predictions which might make transcripts more readable. Characterizing the inductive bias. Arpit et al. BID1 have shown that a neural network trained with stochastic gradient descent learns easier examples first. We argue that we can characterize the preference bias of our model and learning algorithm by showing which examples are easy to classify in the particular representation that each of the models is learning. Since ASR models are usually evaluated in terms of WER, we consider which words each model is learning. To this end we chose a relatively clean dataset and considered the attributes frequency and length to describe a word. We trained each model for 4 epochs and recorded the distribution of the recognized words during training. Since we are not given a perfect alignment between speech and ground-truth transcript, we define a word as being recognized if it is both present in the greedy prediction on the validation set and the corresponding ground-truth transcript. FIG1 shows how the distribution of recognized words changes during training. We see that the word-level model is biased towards recognizing the most common words and slowly learns less frequent words over time. This makes sense since more weight is given to the corresponding examples. While the same effect is present in the character-level model, it covers the complete support of the word frequency distribution in the same number of steps. On the other hand for the length distribution, we see that the word-level model covers all words independent of its length within the beginning of training. The character-level model focuses strongly on shorter words before it covers the whole range of the word length distribution. If we compare the learning dynamics of both models, we find that each model learns words with different characteristics more easily. If we take a look at the MTL model, we see that it combines both biases and arrives at learning a distribution that is much more uniform across both word frequency and word length. We hypothesize that putting more emphasis on the tail of each of these distributions combines the strengths of the two models and makes them perform better, especially in distributions that follow a power law such as word frequency rank. In contrast to earlier studies in the literature, we found that, even on a relatively small dataset, training on a word-level can be feasible. Furthermore, we found that combining a word-level model with character-level supervision in MTL can improve noticeably. To gain a better understanding of this, we characterized the inductive bias of word-character MTL in ASR by comparing the distributions of recognized words at the beginning of training. We found that adding character-level supervision to a word-level interpolates between recognizing more frequent words (preferred by the word-level model) and shorter words (preferred by the character-level model). This effect could be even more pronounced on harder datasets than WSJ, such as medical communication data where many long words are infrequent, but very important. Further analysis of word distributions in terms of pitch, noise and acoustic variability could provide additional insight.

Multi-task learning improves word-and-character-level speech recognition by interpolating the preference biases of its components: frequency- and word length-preference.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

A fully connected architecture is used to produce word embeddings from character representations, outperforms traditional embeddings and provides insight into sparsity and dropout.