Question: Represent the scientific article to find a short TLDR

Question: In the context of optimization, a gradient of a neural network indicates the amount a specific weight should change with respect to the loss. Therefore, small gradients indicate a good value of the weight that requires no change and can be kept frozen during training. This paper provides an experimental study on the importance of a neural network weights, and to which extent do they need to be updated. We wish to show that starting from the third epoch, freezing weights which have no informative gradient and are less likely to be changed during training, in a very slight drop in the overall accuracy (and in sometimes better). We experiment on the MNIST, CIFAR10 and Flickr8k datasets using several architectures (VGG19, ResNet-110 and DenseNet-121). On CIFAR10, we show that freezing 80% of the VGG19 network parameters from the third epoch onwards in 0.24% drop in accuracy, while freezing 50% of Resnet-110 parameters in 0.9% drop in accuracy and finally freezing 70% of Densnet-121 parameters in 0.57% drop in accuracy. Furthermore, to experiemnt with real-life applications, we train an image captioning model with attention mechanism on the Flickr8k dataset using LSTM networks, freezing 60% of the parameters from the third epoch onwards, ing in a better BLEU-4 score than the fully trained model. Our source code can be found in the appendix. The immense success of deep neural networks we are witnessing since the deep learning revolution occurred is surprising. A large variety of vision and language applications ranging from image classification, object detection, image synthesis, image super-resolution, image captioning, language modeling.... etc. has proved that neural networks possess a powerful capability of learning very complex data. However, training these networks to perform as expected is very time-consuming and requires powerful graphical processing units (GPUs). A recently published open-source project by NVIDIA 1 claimed that training a generative adversarial network (GAN) took more than 6 days on 8 Tesla V100 GPUs. However, we argue that a lot of parameters involved during training are important for update only for the first few epochs (in our experiments, the first two epochs only), and can be frozen for the rest of the training epochs. The backpropagation algorithm is the base algorithm used to optimize deep neural networks. For each weight, a gradient is computed with respect to the loss which indicates the amount a weight should change. Large gradients correspond to a large change that will occur in the weight, while small ones (near to zero) indicate that the weight is nearly optimized and does not need much change. In particular, if a gradient for a particular weight is zero or close to zero, this means that it has either reached its optimal solution, or it is stuck at a saddle point. The former means that the weight has a good value and is less likely to change throughout the training and can be kept frozen. In this paper, we wish to show the redundancy of weights in a neural network that have no influence and can be kept frozen during training. In particular, we demonstrate that fully training a model with all its weights is required for the first two epochs only. To justify this, we propose an experimental technique named Partial Backpropagation, which freezes weights that have gradients very near to zero and are less likely to change, with the rest of the weights trained normally. This induces a very slight drop in accuracy (and no harm in accuracy for lesser freezing). An overview of our experimental technque is shown in Figure 1. Note that in Figure 1(b), the red weights are frozen and not removed or zeroed out. We can further visualize the histogram of gradients across the network layers to have a better understanding of their distributions. In Figure 2, we visualize the distribution of gradients from several layers in a VGG19 convolutional network . In particular, we visualize the gradients of layers 3, 7, 10 and 13 after training for 2 epochs. We can see a large number of gradients with values very near to zero, suggesting that a lot of weights in these layers have already been optimized and are less likely to change throughout the training. We discuss two topics closely (but not exactly) related to our work, namely weight pruning and progressive freezing of hidden layers. The reduction of the heavy inference cost of deep networks in low-resource settings is the primary purpose of weight pruning. Neural Networks usually contain a large number of parameters that make them hard to bring into effective action on embedded devices such as mobile phones. The hardship here happens due to both the network size and evaluation time. Significant redundant parameters could be reduced ing in a compressed network with the unnecessary computation being alleviated while maintaining the same accuracy. In short, the main objective of weight pruning techniques is the reduction of the energy and storage needed to process inference on deep neural networks. Hence, they can be deployed on embedded devices. learn only the significant connections to reduce storage. A neural network is first trained to learn the significant connections. Subsequently, the insignificant connections are pruned out and the remaining weights are then fine-tuned. In particular, after the initial training phase, all weights lower than a certain threshold are eliminated. This in a sparse network which is then re-trained so the remaining weights can compensate for the removed ones. The phases of pruning and retraining is an iterative process. After many iterations, the minimum number of weights could be found. In , two neurons with highly correlated activations are merged into one instead of removing a neuron, which keeps the signals inside the network as close as possible to the original one. If the highest correlated neurons are not fully correlated, merging them into one neuron could possibly alter the accuracy factor. proposed a dense-sparse-dense training flow. Other works also focus on pruning convolutional filters for efficient inference. In , a new criterion based on Taylor expansion which approximates the change in the cost function induced by pruning network parameters is proposed. Similarly, proposed an acceleration method for convolutional networks where different filters which have a small influence on the output accuracy are pruned. By pruning whole filters in the network with their corresponding feature maps, the computation cost is greatly minimized. Moreover, enforces channel-level sparsity in the network to reduce the model size of a CNN, which decreases the run-time memory and lower the number of computing operations with no harm on the accuracy factor. presented a systematic weight pruning framework of DNNs using the alternating direction method of multipliers (ADMM). A recenetly proposed netowrk, namely slimmable neural network presents a simple and general method to train a single neural network which is capable of being exucted at different widths. Rather than training an individual network with different width configurations, a shared network with switchable batch normalization could be trained. During runtime, the network can change its width on the fly according to the resource limitation. Other works on compressing deep neural networks include and . However, an important aspect of weight pruning models is that deep neural networks are mainly trained for the objective of reducing computation at testing time for efficient implementation on hardware systems, which is an iterative training process and requires a lot of procedures and algorithms to be involved during training, which in turn increases the training time. In contrast to our work, we wish to demonstrate the number of redundant parameters, which if frozen and not zeroed out, can have a very slight (or no) influence on the overall accuracy. Since the two objectives are different, we cannot directly compare weight pruning methods with our experimental technique. In , the authors proposed training each layer for a set portion of the training schedule, progressively "freezing out" layers and excluding them from the backward pass. This technique is motivated by the fact that in many deep networks, initial layers consume most of the cost, but have the fewest parameters and tend to converge to reasonable simple structures, implying that they do not require as much update as the later layers where most of the parameters are populated. This concept aims to reduce training time, however works on freezing complete layers which may contain informative gradients, and corresponding weights avoid getting updated. The study reports training time improvements with loss/no loss in accuracy, however does not report the number of parameters frozen in the overall network. Moreover, the study "progressively" freezes some layers in the network, implying that important weights at specific times in the training schedule experience an update. Therefore, it is also hard to directly compare Freezout with our experimental technique. As mentioned in Section 1, weights with gradients very near to zero imply that the particular weight is nearly optimized and is less likely to change throughout the training and can therefore be kept frozen. Our experimental technique freezes weights that have gradients very near to zero from the third epoch onwards, while training rest of the weights normally. All weights with gradients below a particular threshold are frozen. In this section, we first describe how we calculate the threshold of gradients and then elaborate on the proposed experimental technique. We start by extracting the non-zero gradients of the neural network. After training, a slight amount of weights may posses zero gradients, which are not considered when performing thresholding. Given the gradients of each layer l i where i indicates the layer number, we have: consists of all the gradients in the layer i. For convenience, we divide each gradient by the corresponding learning rate α to obtain the original gradient (non-scaled by the learning rate), and take its absolute value dE(y) dw. This is because we are interested in the magnitude of the gradient, and not its direction. We then plot the kernel density estimation (KDE) of G, given by the equation: where K is the kernel function and h is the kernel bandwidth. Fromf (x), we find the region which has a high likelihood of the gradients observation, obtaining the new range of gradients G n. The new range G n is then divided into b sub-ranges and the mean of each sub-range b is obtained to form, where each gradient in G n is assigned to its corresponding mean value. We set b to 300. We then apply Otsu thresholding technique to find the optimal threshold value. We re-write the mathematical formulation of otsu's method operating on the mean values in G d. The within-class variance is calculated as: where, where t is the mean value and represents the threshold at each of the 300 steps, (i. After calculating the within-class variance of all values in G d, we select the mean value from G d with the least within-class variance as our final threshold value. We first start by training the network with full backpropagation for 2 epochs. It is necessary that the network first understands the data it is dealing with and performs weights update. After the second epoch, the network is considered to have known what weights are optimized and what weights further need optimization. We then perform full backpropagation on the first batch of the third epoch, however before updating the weights, we perform the gradient mask generation. Given the Jacobian matrix of the gradients with respect to the loss F at the first batch of the third epoch: we divide each gradient by the corresponding learning rate α to obtain the original non-scaled gradient, and take its absolute value dE(y) dw, since we are interested in the magnitude of the gradient and not its direction. Ω is the number of epochs to train for with full backpropagation at the beginning before starting partial backpropagation. As discussed earlier, we set Ω = 2. For each layer i = 1: n, we create a mask m i that has the same shape of the gradient matrix J. We then set the corresponding mask value for each gradient in the layer i to 0 if the gradient is smaller than the threshold t, and 1 otherwise. Thus, we have a list of masks for each layer: m = (m 1, m 2, . . . .. m n) where n is the number of layers. We then perform weight update only on the weights corresponding to an entry of 1 in the corresponding mask m i. For the rest of the batches in the third epoch and all the remaining epochs, we use the same mask m. Gradients and weight updates are only calculated for entries that correspond to 1 in the mask m i. If the weight corresponds to an entry of 0 in the mask, its gradient is set to 0 and thus no weight update will happen to it (frozen weight). Therefore, the mask is a one-time calculation that depends on one batch in the third epoch, which makes it critical to have a reasonable batch size that contains all possible non-linear data combinations. In our experiments, we set the batch size to 100. Note that generating a mask requires one full iteration (one batch) of backpropagation. We have also experimented with changing the mask every several epochs (changing the mask each 35 epochs), and found out that it leads to the same performance as using the same mask for all epochs. Moreover, we experimented with performing full backpropagation at the last epoch in order to fine-tune the network so that the frozen weights could possibly change if needed. However, we found that the accuracy maintains as it is, implying that the frozen weights with least gradients are already optimized from the first two epochs, which validates our methodology. We start our experiments on the CIFAR-10 dataset using several CNN architectures. The CIFAR-10 dataset consists of 60000 32x32 colour images with 10 classes and 6000 images per class. There are 50,000 training images and 10,000 test images. Figure 3 shows the training and validation plots for both full backpropagation (FB) and experimental technique (PB). We use four different architectures: VGG19 , ResNet-110 , DenseNet-121 and LeNet-5 blocks with a growth rate of 12. Finally, for LeNet-5, the number of channels are. Table 1 demonstrates the obtained, while Table 2 demonstrates the average performance over 3 runs. The lowest freezing percentage is witnessed by residual networks. This is expected since these type of networks only learn the residual information and thus a lot of redundancy is already removed. Moreover, we find that VGG19 experiences 9.4% parameters with zero gradients by default (without applying our experimental technique), while LeNet-5 experiences 31.8% by default. Moreover, to demonstrate best performance of our methodology (i.e. highest freezing percentage with lowest accuracy drop), we slightly tweak the original threshold value obtained as discussed in section 3.1, and report the tweaked threshold values in Table 1. It is worth noting that when training the networks, we avoid using any regularization techniques. In particular, we don't use data augmentation, neither weight decay or dropout. We focus on delivering our experiments under normal settings. Models are all trained for 110 epochs. We start with a learning rate of 0.001 and decay the learning rate by a factor of 0.1 at epoch 40 and 80. We train all networks with AMSGrad which takes the maximum between the current and the previous exponentially weighted average of gradients. In order to experiment on complex real-life applications, we trained an image captioning model with visual attention mechanism as proposed in on the Flickr8k dataset using our experimental technique (PB). Flickr8k is a dataset that contains 8,000 images with up to 5 captions for each image. We divide the dataset into 6,000 training images and 1,000 validation and testing images, each. In our setup, we use the 5 captions for each image (if all 5 are present) and randomly sample from any present caption (5-present captions) times if the 5 captions are not present, ing in 30,000 training captions and 5,000 validation and testing captions, each. We Table 2: Average Performance over 3 runs on CIFAR10 and MNIST datasets for 4 different types of CNNs with the threshold, validation accuracy for full backpropagation (FB), validation accuracy for our experimental technique (PB) and the freezing ratio in parameters which is calculated as: (total gradients -non-zero gradients / total gradients) × 100 resize all images to 256 × 256 pixels. We use a single layer LSTM with a hidden size of 512. The batch size is set 60. We use the Adam optimizer with an initial learning rate of 5e-4 and anneal the learning rate by a factor of 0.8 once the BLEU-4 score shows no improvement for 3 consecutive epochs. The word embedding and attention size is set to 512. We train for a maximum of 15 epochs with early stopping if the validation BLEU-4 score has not improved for 10 consecutive epochs. When sampling, we use a beam size of 3. We use BLEU with up to 4 grams (BLEU-1, BLEU-2, BLEU-3, BLEU-4) as our evaluation metric. We experiment on the soft attention variant of. The best BLEU-4 score obtained when training with full backpropagation is 0.099. When applying our experimental technique (PB), using a threshold of 0.05, we obtain 61.81% redundant parameters which are frozen from the third epoch onwards. Under this setting, we obtain a higher BLEU-4 score of 0.101 than the fully trained model. The freezing ratio is calculated as: (total gradients -non-zero gradients / total gradients) × 100. Figure 4 shows some generated captions. For evaluation on all BLEU scores, see Figure 5. on the Flickr8k dataset using experimental technique (PB). The predicted word and attention weights for each timestep are shown for 4 generated captions of different lengths Figure 5: BLEU-n scores (where n is the number of grams) reported on the the Flickr8k dataset for image captioning with attention mechanism using both full backpropagation (FB) and our experimental technique (PB). We provided an experimental study on the importance of a neural network weights, and to which extent do they need to be updated. Through our experiments, we emphasized the number of redundant parameters that carry no informative gradient, which if frozen from the third epoch onwards, slightly effect (and in sometimes do not) the overall accuracy of the model. To prove our concern, we ran experiments on the MNIST and CIFAR10 datasets using several CNN architectures (VGG19, ResNet-110 and DenseNet-121), as well as the Flick8k dataset using an image captioning architecture composed of LSTM networks with attention mechanism. Our experiments successfully prove the concern of this paper. We provide our base code in this appendix. We use PyTorch to implement our experiments. We modified publicly available codes for the architectures we used.

represent the tldr for finding the corresponding scientific article

Examples:

"We have proposed using the recent GrOWL regularizer for simultaneous parameter sparsity and tying in DNN learning. " == "Deep neural networks (DNNs) usually contain millions, maybe billions, of parameters/weights, making both storage and computation very expensive. This has motivated a large body of work to reduce the complexity of the neural network by using sparsity-inducing regularizers. Another well-known approach for controlling the complexity of DNNs is parameter sharing/tying, where certain sets of weights are forced to share a common value. Some forms of weight sharing are hard-wired to express certain in- variances, with a notable example being the shift-invariance of convolutional layers. However, there may be other groups of weights that may be tied together during the learning process, thus further re- ducing the complexity of the network. In this paper, we adopt a recently proposed sparsity-inducing regularizer, named GrOWL (group ordered weighted l1), which encourages sparsity and, simulta- neously, learns which groups of parameters should share a common value. GrOWL has been proven effective in linear regression, being able to identify and cope with strongly correlated covariates. Unlike standard sparsity-inducing regularizers (e.g., l1 a.k.a. Lasso), GrOWL not only eliminates unimportant neurons by setting all the corresponding weights to zero, but also explicitly identifies strongly correlated neurons by tying the corresponding weights to a common value. This ability of GrOWL motivates the following two-stage procedure: (i) use GrOWL regularization in the training process to simultaneously identify significant neurons and groups of parameter that should be tied together; (ii) retrain the network, enforcing the structure that was unveiled in the previous phase, i.e., keeping only the significant neurons and enforcing the learned tying structure. We evaluate the proposed approach on several benchmark datasets, showing that it can dramatically compress the network with slight or even no loss on generalization performance. Deep neural networks (DNNs) have recently revolutionized machine learning by dramatically advancing the state-of-the-art in several applications, ranging from speech and image recognition to playing video games BID20. A typical DNN consists of a sequence of concatenated layers, potentially involving millions or billions of parameters; by using very large training sets, DNNs are able to learn extremely complex non-linear mappings, features, and dependencies. A large amount of research has focused on the use of regularization in DNN learning BID20, as a means of reducing the generalization error. It has been shown that the parametrization of many DNNs is very redundant, with a large fraction of the parameters being predictable from the remaining ones, with no accuracy loss BID14. Several regularization methods have been proposed to tackle the potential over-fitting due to this redundancy. Arguably, the earliest and simplest choice is the classical 2 norm, known as weight decay in the early neural networks literature, and as ridge regression in statistics. In the past two decades, sparsity-inducing regularization based on the 1 norm (often known as Lasso) BID35, and variants thereof, became standard tools in statistics and machine learning, including in deep learning BID20. Recently, BID32 used group-Lasso (a variant of Lasso that assumes that parameters are organized in groups and encourages sparsity at the group level BID37) in deep learning. One of the effects of Lasso or group-Lasso regularization in learning a DNN is that many of the parameters may become exactly zero, thus reducing the amount of memory needed to store the model, and lowering the computational cost of applying it. Figure 1: A DNN is first trained with GrOWL regularization to simultaneously identify the sparse but significant connectivities and the correlated cluster information of the selected features. We then retrain the neural network only in terms of the selected connectivities while enforcing parameter sharing within each cluster. It has been pointed out by several authors that a major drawback of Lasso (or group-Lasso) regularization is that in the presence of groups of highly correlated covariates/features, it tends to select only one or an arbitrary convex combination of features from each group BID6 BID7 BID17 BID28 BID42. Moreover, the learning process tends to be unstable, in the sense that subsets of parameters that end up being selected may change dramatically with minor changes in the data or algorithmic procedure. In DNNs, it is almost unavoidable to encounter correlated features, not only due to the high dimensionality of the input to each layer, but also because neurons tend to co-adapt, yielding strongly correlated features that are passed as input to the subsequent layer BID34.In this work, we propose using, as a regularizer for learning DNNs, the group version of the ordered weighted 1 (OWL) norm BID17, termed group-OWL (GrOWL), which was recently proposed by BID28. In a linear regression context, GrOWL regularization has been shown to avoid the above mentioned deficiency of group-Lasso regularization. In addition to being a sparsity-inducing regularizer, GrOWL is able to explicitly identify groups of correlated features and set the corresponding parameters/weights to be very close or exactly equal to each other, thus taking advantage of correlated features, rather than being negatively affected by them. In deep learning parlance, this corresponds to adaptive parameter sharing/tying, where instead of having to define a priori which sets of parameters are forced to share a common value, these sets are learned during the training process. We exploit this ability of GrOWL regularization to encourage parameter sparsity and group-clustering in a two-stage procedure depicted in Fig. 1: we first use GrOWL to identify the significant parameters/weights of the network and, simultaneously, the correlated cluster information of the selected features; then, we retrain the network only in terms of the selected features, while enforcing the weights within the same cluster to share a common value. The experiments reported below confirm that using GrOWL regularization in learning DNNs encourages sparsity and also yields parameter sharing, by forcing groups of weights to share a common absolute value. We test the proposed approach on two benchmark datasets, MNIST and CIFAR-10, comparing it with weight decay and group-Lasso regularization, and exploring the accuracy-memory trade-off. Our indicate that GrOWL is able to reduce the number of free parameters in the network without degrading the accuracy, as compared to other approaches. In order to relieve the burden on both required memory and data for training and storing DNNs, a substantial amount of work has focused on reducing the number of free parameters to be estimated, namely by enforcing weight sharing. The classical instance of sharing is found in the convolutional layers of DNNs BID20. In fact, weight-sharing as a simplifying technique for NNs can be traced back to more than 30 years ago BID24 BID30.Recently, there has been a surge of interest in compressing the description of DNNs, with the aim of reducing their storage and communication costs. Various methods have been proposed to approximate or quantize the learned weights after the training process. BID15 have shown that, in some cases, it is possible to replace the original weight matrix with a low-rank approximation. Alternatively, BID1 propose retraining the network layer by layer, keeping the layer inputs and outputs close to the originally trained model, while seeking a sparse transform matrix, whereas BID19 propose using vector quantization to compress the parameters of DNNs. Network pruning is another relevant line of work. In early work, BID25 and BID22 use the information provided by the Hessian of the loss function to remove less important weights; however, this requires expensive computation of second order derivatives. Recently, BID21 reduce the number of parameters by up to an order of magnitude by alternating between learning the parameters and removing those below a certain threshold. propose to prune filters, which seeks sparsity with respect to neurons, rather than connections; that approach relieves the burden on requiring sparse libraries or special hardware to deploy the network. All those methods either require multiple training/retraining iterations or a careful choice of thresholds. There is a large body of work on sparsity-inducing regularization in deep learning. For example, BID12 exploit 1 and 0 regularization to encourage weight sparsity; however, the sparsity level achieved is typically modest, making that approach not competitive for DNN compression. Group-Lasso has also been used in training DNNs; it allows seeking sparsity in terms of neurons BID32 BID2 BID41 BID27 or other structures, e.g., filters, channels, filter shapes, and layer depth BID36. However, as mentioned above, both Lasso and group-Lasso can fail in the presence of strongly correlated features (as illustrated in Section 4, with both synthetic data and real data. A recent stream of work has focused on using further parameter sharing in convolutional DNNs. By tying weights in an appropriate way, BID16 obtain a convolutional DNN with rotation invariance. On the task of analyzing positions in the game Go, BID10 showed improved performance by constraining features to be invariant to reflections along the x-axis, y-axis, and diagonal-axis. Finally, BID9 used a hash function to randomly group the weights such that those in a hash bucket share the same value. In contrast, with GrOWL regularization, we aim to learn weight sharing from the data itself, rather than specifying it a priori. Dropout-type methods have been proposed to fight over-fitting and are very popular, arguably due to their simplicity of implementation BID34 . Dropout has been shown to effectively reduce over-fitting and prevent different neurons from co-adapting. Decorrelation is another popular technique in deep learning pipelines BID4 BID11 BID29 ; unlike sparsity-inducing regularizers, these methods try to make full use of the model's capacity by decorrelating the neurons. Although dropout and decorrelation can reduce over-fitting, they do not compress the network, hence do not address the issue of high memory cost. It should also be mentioned that our proposal can be seen as complementary to dropout and decorrelation: whereas dropout and decorrelation can reduce co-adaption of nodes during training, GrOWL regularization copes with co-adaptation by tying together the weights associated to co-adapted nodes. We start by recalling the definition of the group-OWL (GrOWL) regularizer and very briefly reviewing some of its relevant properties BID28. Definition 1. Given a matrix W ∈ R n×m, let w [i]· denote the row of W with the i-th largest 2 norm. Let λ ∈ R n +, with 0 < λ 1 ≥ λ 2 ≥ · · · ≥ λ n ≥ 0. The GrOWL regularizer (which is a norm) DISPLAYFORM0 This is a group version of the OWL regularizer BID17, also known as WSL1 (weighted sorted 1 BID40) and SLOPE BID5, where the groups are the rows of its matrix argument. It is clear that GrOWL includes group-Lasso as a special case when λ 1 = λ n. As a regularizer for multiple/multi-task linear regression, each row of W contains the regression coefficients of a given feature, for the m tasks. It has been shown that by adding the GrOWL regularizer to a standard squared-error loss function, the ing estimate of W has the following property: rows associated with highly correlated covariates are very close or even exactly equal to each other BID28. In the linear case, GrOWL encourages correlated features to form predictive clusters corresponding to the groups of rows that are nearly or exactly equal. The rationale underlying this paper is that when used as a regularizer for DNN learning, GrOWL will induce both sparsity and parameters tying, as illustrated in Fig. 2 and explained below in detail. A typical feed-forward DNN with L layers can be treated as a function f of the following form: DISPLAYFORM0 L denotes the set of parameters of the network, and each f i is a componentwise nonlinear activation function, with the rectified linear unit (ReLU), the sigmoid, and the hyperbolic tangent being common choices for this function BID20. DISPLAYFORM1, DNN learning may be formalized as an optimization problem, DISPLAYFORM2 where L y,ŷ is the loss incurred when the DNN predictsŷ for y, and R is a regularizer. Here, we adopt as regularizer a sum of GrOWL penalties, each for each layer of the neural network, i.e., DISPLAYFORM3 where N l denotes the number of neurons in the l-th layer and 0 < λ DISPLAYFORM4.., b L, the biases are not regularized, as is common practice. As indicated in Eq., the number of groups in each GrOWL regularizer is the number of neurons in the previous layer, i.e., λ (l) ∈ R N l−1. In other words, we treat the weights associated with each input feature as a group. For fully connected layers, where W l ∈ R N l−1 ×N l, each group is a row of the weight matrix. In convolutional layers, where W l ∈ R Fw×F h ×N l−1 ×N l, with F w and F h denoting the width and height, respectively, of each filter, we first reshape W l to a 2-dimensional array, i.e., DISPLAYFORM5, and then apply GrOWL on the reshaped matrix. That is, if the l-th layer is convolutional, then DISPLAYFORM6 Each row of W 2D l represents the operation on an input channel. The rationale to apply the GrOWL regularizer to each row of the reshaped weight matrix is that GrOWL can select the relevant features of the network, while encouraging the coefficient rows of each layer associated with strongly correlated features from the previous layer to be nearly or exactly equal, as depicted in Fig. 2. The goal is to significantly reduce the complexity by: (i) pruning unimportant neurons of the previous layer that correspond to zero rows of the (reshaped) weight matrix of the current layer; (ii) grouping the rows associated with highly correlated features of the previous layer, thus encouraging the coefficient rows in each of these groups to be very close to each other. As a consequence, in the retraining process, we can further compress the neural network by enforcing the parameters within each neuron that belong to the same cluster to share same values. In the work of BID2, each group is predefined as the set of parameters associated to a neuron, and group-Lasso regularization is applied to seek group sparsity, which corresponds to zeroing out redundant neurons of each layer. In contrast, we treat the filters corresponding Figure 2: GrOWL's regularization effect on DNNs. Fully connected layers (Left): for layer l, GrOWL clusters the input features from the previous layer, l − 1, into different groups, e.g., blue and green. Within each neuron of layer l, the weights associated with the input features from the same cluster (input arrows marked with the same color) share the same parameter value. The neurons in layer l − 1 corresponding to zero-valued rows of W l have zero input to layer l, hence get removed automatically. Convolutional layers (right): each group (row) is predefined as the filters associated with the same input channel; parameter sharing is enforced among the filters within each neuron that corresponds with the same cluster (marked as blue with different effects) of input channels.to the same input channel as a group, and GrOWL is applied to prune the redundant groups and thus remove the associated unimportant neurons of the previous layer, while grouping associated parameters of the current layer that correspond with highly correlated input features to different clusters. Moreover, as shown in Section 4, group-Lasso can fail at selecting all relevant features of previous layers, and for the selected ones the corresponding coefficient groups are quite dissimilar from each other, making it impossible to further compress the DNN by enforcing parameter tying. To solve, we use a proximal gradient algorithm BID3, which has the following general form: at the t-th iteration, the parameter estimates are updated according to DISPLAYFORM0 where, for some convex function Q, prox Q denotes its proximity operator (or simply "prox") BID3, defined as prox DISPLAYFORM1 2 denotes the sum of the squares of the differences between the corresponding components of ν and ξ, regardless of their organization (here, a collection of matrices and vectors).Since R(θ), as defined in FORMULA4, is separable across the weight matrices of different layers and zero for b 1,..., b L, the corresponding prox is also separable, thus DISPLAYFORM2 DISPLAYFORM3 It was shown by BID28 that the prox of GrOWL can be computed as follows. For some matrix V ∈ R N ×M, let U = prox Ω λ (V), and v i and u i denote the corresponding i-th rows. Then, DISPLAYFORM4 DISPLAYFORM5 For vectors in R N (in which case GrOWL coincides with OWL), prox Ω λ (l) can be computed with O(n log n) cost, where the core computation is the socalled pool adjacent violators algorithm (PAVA BID13)) for isotonic regression. We provide one of the existing algorithms in Appendix A; for details, the reader is referred to the work of BID5 and BID39. In this paper, we apply the proximal gradient algorithm per epoch, which generally performs better. The training method is summarized in Algorithm 1. GrOWL is a family of regularizers, with different variants obtained by choosing different weight sequences λ 1,..., λ n. In this paper, we propose the following choice: DISPLAYFORM6 where p ∈ {1, ...n} is a parameter. The first p weights follow a linear decay, while the remaining ones are all equal to Λ 1. Notice that, if p = n, the above setting is equivalent to OSCAR BID6. Roughly speaking, Λ 1 controls the sparsifying strength of the regularizer, while Λ 2 controls the clustering property (correlation identification ability) of GrOWL BID28. Moreover, by setting the weights to a common constant beyond index p means that clustering is only encouraged among the p largest coefficients, i.e., only among relevant coefficient groups. Finding adequate choices for p, Λ 1, and Λ 2 is crucial for jointly selecting the relevant features and identifying the underlying correlations. In practice, we find that with properly chosen p, GrOWL is able to find more correlations than OSCAR. We explore different choices of p in Section 4.1. After the initial training phase, at each layer l, rows of W l that corresponds to highly correlated outputs of layer l − 1 have been made similar or even exactly equal. To further compress the DNN, we force rows that are close to each other to be identical. We first group the rows into different clusters 1 according to the pairwise similarity metric DISPLAYFORM0 where W l,i and W l,j denote the i-th and j-th rows of W l, respectively. With the cluster information obtained by using GrOWL, we enforce parameter sharing for the rows that belong to a same cluster by replacing their values with the averages (centroid) of the rows in that cluster. In the subsequent retraining process, let Gk denote the k-th cluster of the l-th layer, then centroid g DISPLAYFORM1 We assess the performance of the proposed method on two benchmark datasets: MNIST and CIFAR-10. We consider two different networks and compare GrOWL with group-Lasso and weight decay, in terms of the compression vs accuracy trade-off. For fair comparison, the training-retraining pipeline is used with the different regularizers. After the initial training phase, the rows that are close to each other are clustered together and forced to share common values in the retraining phase. We implement all models using Tensorflow BID0. We evaluate the effect of the different regularizers using the following quantities: sparsity = (#zero params)/(# total params), compression rate = (# total params)/(# unique params), and parameter sharing = (# nonzero params)/(# unique params). First, we consider a synthetic data matrix X with block-diagonal covariance matrix Σ, where each block corresponds to a cluster of correlated features, and there is a gap g between two blocks. Within each cluster, the covariance between two features X i and X j is cov(X i, X j) = 0.96 |i−j|, while features from different clusters are generated independently of each other. We set n = 784, K = 10, block size 50, and gap g = 28. We generate 10000 training and 1000 testing examples. FORMULA14 ).We train a NN with a single fully-connected layer of 300 hidden units. FIG0 the first 25000 entries of the sorted pairwise similarity matrices (Eq 10) obtained by applying GrOWL with different p (Eq 9) values. By setting the weights beyond index p to a common constant implies that clustering is only encouraged among the p largest coefficients, i.e., relevant coefficient groups; however, FIG0 shows that, with properly chosen p, GrOWL yields more parameter tying than OSCAR (p = n). On the other hand, smaller p values allow using large Λ 2, encouraging parameter tying among relatively loose correlations. In practice, we find that for p around the target fraction of nonzero parameters leads to good performance in general. The intuition is that we only need to identify correlations among the selected important features. FIG0 shows that weight decay (denoted as 2) also pushes parameters together, though the parameter-tying effect is not as clear as that of GrOWL. As has been observed in the literature BID6, weight decay often achieves better generalization than sparsity-inducing regularizers. It achieves this via parameter shrinkage, especially in the highly correlated region, but it does not yield sparse models. In the following section, we explore the compression performance of GrOWL by comparing it with both group-Lasso and weight decay. We also explore how to further improve the accuracy vs compression trade-off by using sparsity-inducing regularization together with weight decay. For each case, the baseline performance is provided as the best performance obtained by running the original neural network (without compression) after sweeping the hyper-parameter on the weight decay regularizer over a range of values. The MNIST dataset contains centered images of handwritten digits, of size 28×28 pixels FIG2 shows the (784 × 784) correlation matrix of the dataset (the margins are zero due to the redundant of the images). We use a network with a single fully connected layer of 300 hidden units. The network is trained for 300 epochs and then retrained for an additional 100 epochs, both with momentum. The initial learning rate is set to 0.001, for both training and retraining, and is reduced by a factor of 0.96 every 10 epochs. We set p = 0.5, and Λ 1, Λ 2 are selected by grid search. Pairwise similarities (see Eq. FORMULA15) between the rows of the weight matrices learned with different regularizers are shown in FIG2. As we can see, GrOWL (+ 2) identifies more correlations than group-Lasso (+ 2), and the similarity patterns in FIG2 are very close to that of the data FIG2 ). On the other hand, weight decay also identifies correlations between parameter rows, but it does not induce sparsity. Moreover, as shown in Table 1 FORMULA15 ) of the parameter rows obtained by training the neural network with GrOWL, GrOWL+ 2, group-Lasso, group-Lasso+ 2 and weight decay. Table 1: Sparsity, parameter sharing, and compression rate on MNIST. Baseline model is trained with weight decay and we do not enforce parameter sharing for baseline model. We train each model for 5 times and report the average values together with their standard deviations. Sparsity Parameter Sharing Compression ratio Accuracy none 0.0 ± 0% 1.0 ± 0 1.0 ± 0 98.3 ± 0.1% weight decay 0.0 ± 0% 1.6 ± 0 1.6 ± 0 98.4 ± 0.0% group-Lasso 87.6 ± 0.1% 1.9 ± 0.1 15.8 ± 1.0 98.1 ± 0.1% group-Lasso+ 2 93.2 ± 0.4%1.6 ± 0.1 23.7 ± 2.1 98.0 ± 0.1% GrOWL 80.4 ± 1.0% 3.2 ± 0.1 16.7 ± 1.3 98.1 ± 0.1% GrOWL+ 2 83.6 ± 0.5% 3.9 ± 0.1 24.1 ± 0.8 98.1 ± 0.1%The compression vs accuracy trade-off of the different regularizers is summarized in Table 1, where we see that applying 2 regularization together with group-Lasso or GrOWL leads to a higher compression ratio, with negligible effect on the accuracy. Table 1 also shows that, even with lower sparsity after the initial training phase, GrOWL (+ 2) compresses the network more than group-Lasso (+ 2), due to the significant amount of correlation it identifies; this also implies that group-Lasso only selects a subset of the correlated features, while GrOWL selects all of them. On the other hand, group-Lasso suffers from randomly selecting a subset of correlated features; this effect is illustrated in FIG4, which plots the indices of nonzero rows, showing that GrOWL (+ 2) stably selects relevant features while group-Lasso (+ 2) does not. The mean ratios of changed indices 2 are 11.09%, 0.59%, 32.07%, and 0.62% for group-Lasso, GrOWL, group-Lasso+ 2, and GrOWL+ 2, respectively. To evaluate the proposed method on large DNNs, we consider a VGG-like BID33 architecture proposed by BID38 on the CIFAR-10 dataset. The network architecture is summarized in Appendix C; comparing with the original VGG of BID33, their fully connected layers are replaced with two much smaller ones. A batch normalization layer is added after each convolutional layer and the first fully connected layer. Unlike BID38, we don't use dropout. We first train the network under different regularizers for 150 epochs, then retrain it for another 50 epochs, using the learning rate decay scheme described by He et al. 2 The mean ratio of changed indices is defined as: Table 2: Sparsity (S1) and Parameter Sharing (S2) of VGG-16 on CIFAR-10. Layers marked by * are regularized. We report the averaged over 5 runs. Weight Decay group-Lasso group-Lasso + 2 GrOWL GrOWL + 2 (S1, S2) (S1, S2) (S1, S2) (S1, S2) (S1, S2) conv1 0%, 1.: the initial rates for the training and retraining phases are set to 0.01 and 0.001, respectively; the learning rate is multiplied by 0.1 every 60 epochs of the training phase, and every 20 epochs of the retraining phase. For GrOWL (+ 2), we set p = 0.1 n (see Eq. FORMULA14) for all layers, where n denotes the number of rows of the (reshaped) weight matrices of each layer. The are summarized in Table 2. For all of the regularizers, we use the affinity propagation algorithm (with preference value 3 set to 0.8) to cluster the rows at the end of initial training process. Our experiments showed that it is hard to encourage parameter tying in the first 7 convolutional layers; this may be because the filters of these first 7 convolutional layers have comparatively large feature maps (from 32 × 32 to 8 × 8), which are only loosely correlated. We illustrate this reasoning in Fig. 6, showing the cosine similarity between the vectorized output channels of layers 1, 6, 10, and 11, at the end of the training phase; it can be seen that the outputs of layers 10 and 11 have many more significant similarities than that of layer 6. Although the output channels of layer 1 also Figure 6: Output channel cosine similarity histogram obtained with different regularizers. Labels: GO:GrOWL, GOL:GrOWL+ 2, GL:group-Lasso, GLL:group-Lasso+ 2, WD:weight decay.have certain similarities, as seen in Table 2, neither GrOWL (+ 2) nor weight decay tends to tie the associated weights. This may mean that the network is maintaining the diversity of the inputs in the first few convolutional layers. Although GrOWL and weight decay both encourage parameter tying in layers 9-13, weight decay does it with less intensity and does not yield a sparse model, thus it cannot significantly compress the network. propose to prune small weights after the initial training phase with weight decay, then retrain the reduced network; however, this type of method only achieves compression 4 ratios around 3. As mentioned by, layers 3-7 can be very sensitive to pruning; however, both GrOWL (+ 2) and group-Lasso (+ 2) effectively compress them, with minor accuracy loss. On the other hand, similar to what we observed by running the simple fully-connected network on MNIST, the accuracy-memory trade-off improves significantly by applying GrOWL or group-Lasso together with 2. However, Table 2 also shows that the trade-off achieved by GrOWL (+ 2) and group-Lasso (+ 2) are almost the same. We suspect that this is caused by the fact that CIFAR-10 is simple enough that one could still expect a good performance after strong network compression. We believe this gap in the compression vs accuracy trade-off can be further increased in larger networks on more complex datasets. We leave this question for future research. We have proposed using the recent GrOWL regularizer for simultaneous parameter sparsity and tying in DNN learning. By leveraging on GrOWL's capability of simultaneously pruning redundant parameters and tying parameters associated with highly correlated features, we achieve significant reduction of model complexity, with a slight or even no loss in generalization accuracy. We evaluate the proposed method on both a fully connected neural network and a deep convolutional neural network. The show that GrOWL can compress large DNNs by factors ranging from 11.4 to 14.5, with negligible loss on accuracy. The correlation patterns identified by GrOWL are close to those of the input features to each layer. This may be important to reveal the structure of the features, contributing to the interpretability of deep learning models. On the other hand, by automatically tying together the parameters corresponding to highly correlated features, GrOWL alleviates the negative effect of strong correlations that might be induced by the noisy input or the co-adaption tendency of DNNs. The gap in the accuracy vs memory trade-off obtained by applying GrOWL and group-Lasso decreases as we move to large DNNs. Although we suspect this can be caused by running a much larger network on a simple dataset, it motivates us to explore different ways to apply GrOWL to compress neural networks. One possible approach is to apply GrOWL within each neuron by predefining each 2D convolutional filter as a group (instead all 2D convolutional filters corresponding to the same input features). By doing so, we encourage parameter sharing among much smaller units, which in turn would further improve the diversity vs parameter sharing trade-off. We leave this for future work. Various methods have been proposed to compute the proximal mapping of OWL (ProxOWL). It has been proven that the computation complexity of these methods is O(n log n) which is just slightly worse than the soft thresholding method for solving 1 norm regularization. In this paper, we use Algorithm 2 that was originally proposed in BID5.Unlike k-means or agglomerative algorithm, Affinity Propagation does not require the number of clusters as an input. We deem this as a desired property for enforcing parameter sharing in neural network compression because it's impossible to have the exact number of clusters as a prior information. In practice, the input preference of Affinity Propagation determines how likely each sample will be chosen as an exemplar and its value will influence the number of clusters created. APPENDIX C VGG-16 ON CIFAR-10 Table 4: VGG: Clustering rows over different preference values for running the affinity propagation algorithm (Algorithm 3). For each experiment, we report clustering accuracy (A), compression rate (C), and parameter sharing (S) of layers 9-14. For each regularizer, we use different preference values to run Algorithm 3 to cluster the rows at the end of initial training process. Then we retrain the neural network correspondingly. The are reported as the averages over 5 training and retraining runs. Preference Value 0.6 0.7 0.8 0.9 (A, C, S) (A, C, S) (A, C, S) (A, C, S) GrOWL 92.2%, 13.6, 3.5 92.2%, 12.5, 2.6 92.2%, 11.4, 2.1 92.2%, 10.9, 1."

An experimental paper that proves the amount of redundant weights that can be freezed from the third epoch only, with only a very slight drop in accuracy.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.


Question: Represent the article for finding a single-sentence TLDR

Question: Like humans, deep networks learn better when samples are organized and introduced in a meaningful order or curriculum. While conventional approaches to curriculum learning emphasize the difficulty of samples as the core incremental strategy, it forces networks to learn from small subsets of data while introducing pre-computation overheads. In this work, we propose Learning with Incremental Labels and Adaptive Compensation (LILAC), which introduces a novel approach to curriculum learning. LILAC emphasizes incrementally learning labels instead of incrementally learning difficult samples. It works in two distinct phases: first, in the incremental label introduction phase, we unmask ground-truth labels in fixed increments during training, to improve the starting point from which networks learn. In the adaptive compensation phase, we compensate for failed predictions by adaptively altering the target vector to a smoother distribution. We evaluate LILAC against the closest comparable methods in batch and curriculum learning and label smoothing, across three standard image benchmarks, CIFAR-10, CIFAR-100, and STL-10. We show that our method outperforms batch learning with higher mean recognition accuracy as well as lower standard deviation in performance consistently across all benchmarks. We further extend LILAC to state-of-the-art performance across CIFAR-10 using simple data augmentation while exhibiting label order invariance among other important properties. Deep networks have seen rich applications in high-dimensional problems characterized by a large number of labels and a high volume of samples. However, successfully training deep networks to solve problems under such conditions is mystifyingly hard . The go-to solution in most cases is Stochastic Gradient Descent with mini-batches (simple batch learning) and its derivatives. While offering a standardized solution, simple batch learning often fails to find solutions that are simultaneously stable, highly generalizable and scalable to large systems (; ; ;). This is a by-product of how mini-batches are constructed. For example, the uniform prior assumption over datasets emphasizes equal contributions from each data point regardless of the underlying distribution; small batch sizes help achieve more generalizable solutions, but do not scale as well to vast computational resources as large mini-batches. It is hard to construct a solution that is a perfect compromise between all cases. Two lines of work, curriculum learning and label smoothing, offer alternative strategies to improve learning in deep networks. Curriculum learning, inspired by strategies used for humans , works by gradually increasing the conceptual difficulty of samples used to train deep networks;; ). This has been shown to improve performance on corrupted and small datasets . More recently, deep networks have been used to categorize samples and variations on the pace with which these samples were shown to deep networks were analyzed in-depth . To the best of our knowledge, previous works assumed that samples cover a broad spectrum of difficulty and hence need to be categorized and presented in a specific order. This introduces computational overheads e.g. pre-computing the relative difficulty of samples, and also reduces the effective amount of data from which a model can learn in early epochs. Further, curriculum learning approaches have not been shown to compete with simple training strategies at the top end of performance in image benchmarks. A complementary approach to obtaining generalizable solutions is to avoid over-fitting or getting stuck in local minima. In this regard, label smoothing offers an important solution that is invariant to the underlying architecture. Early works like replace ground-truth labels with noise while uses other models' outputs to prevent over-fitting. This idea was extended in to an iterative method which uses logits obtained from previously trained versions of the same deep network. use local distributional smoothness, based on the robustness of a model's distribution around a data point, to regularize outcomes, penalized highly confident outputs directly. Closest in spirit to our work is the label smoothing method defined in , which offers an alternative target distribution for all training samples with no extra data augmentation. In general, label smoothing is applied to all examples regardless of how it affects the network's understanding of them. Further, in methods which use other models to provide logits/labels, often the parent network used to provide those labels is trained using an alternate objective function or needs to be fully re-trained on the current dataset, both of which introduce additional computation. In this work, we propose LILAC, Learning with Incremental Labels and Adaptive Compensation, which emphasizes a label-based curriculum and adaptive compensation, to improve upon previous methods and obtain highly accurate and stable solutions. LILAC is conceived as a method to learn strong embeddings by using the recursive training strategy of incremental learning alongside the use of unlabelled/wrongly-labelled data as hard negative examples. It works in two key phases, 1) incremental label introduction and 2) adaptive compensation. In the first phase, we incrementally introduce groups of labels in the training process. Data, corresponding to labels not yet introduced to the model, use a single fake label selected from within the dataset. Once a network has been trained for a fixed number of epochs with this setup, an additional set of ground-truth labels is introduced to the network and the training process continues. In recursively revealing labels, LILAC allows the model sufficient time to develop a strong understanding of each class by contrasting against a large and diverse set of negative examples. Once all ground-truth labels are revealed the adaptive compensation phase of training is initiated. This phase mirrors conventional batch learning, except we adaptively replace the target one-hot vector of incorrectly classified samples with a softer distribution. Thus, we avoid adjusting labels across the entire dataset, like previous methods, while elevating the stability and average performance of the model. Further, instead of being pre-computed by an alternative model, these softer distributions are generated on-the-fly from the outputs of the model being trained. We apply LILAC to three standard image benchmarks and compare its performance to the strongest known baselines. While incremental and continual learning work on evolving data distributions with the addition of memory constraints ( and derivative works), knowledge distillation and similar works) or other requirements, this work is a departure into using negative mining and focused training to improve learning on a fully available dataset. In incremental/continual learning works, often the amount of data used to retrain the network is small compared to the original dataset while in LILAC we fully use the entire dataset, distinguished by Seen and Unseen labels. Thus, it avoids data deficient learning. Further, works like;; emphasize the importance of hard negative mining, both in size and diversity, in improving learning. Although the original formulation of negative mining was based on imbalanced data, recent object detection works have highlighted its importance in contrasting and improving learning in neural networks. To summarize, our main contributions in LILAC are as follows, • we introduce a new take on curriculum learning by incrementally learning labels as opposed to samples, • our method adaptively compensates incorrectly labelled samples by softening their target distribution which improves performance and removes external computational overheads, • we improve average recognition accuracy and decrease the standard deviation of performance across several image classification benchmarks compared to batch learning, a property not shared by other curriculum learning and label smoothing methods. In LILAC, our main focus is to induce better learning in deep networks. Instead of the conventional curriculum learning approach of ranking samples, we consider all samples equally beneficial. Early on, we focus on learning labels in fixed increments (Section 2.1). Once the network has had a chance to learn all the labels, we shift to regularizing the network to prevent over-fitting by providing a softer distribution as the target vector for previously misclassified samples (Section 2.2). An overview of the entire algorithm discussed is available in the appendix as Algorithm 1. In the incremental phase, we initially replace the ground-truth labels of several class using a constant held-out label. Gradually, over the course of several fixed intervals of training we reveal the true label. Within a fixed interval of training, we keep constant two sets of data, "Seen", whose groundtruth labels are known and "Unseen", whose labels are replaced by a fake value. When training, Illustration of the evolution of data partitions in the incremental label introduction phase for a four label dataset. In the first incremental step, only one label is used for training while the remaining data use label 4. A short period of training is performed with this fixed setup, where data from U is uniformly sampled to match the number of samples from S, in every mini-batch. The final incremental step depicted is equivalent to batch learning since all the labels are available to the network. Once all the ground-truth labels are revealed we begin the adaptive compensation phase described in Sec. 2.2. mini-batches are uniformly sampled from the entire training set, but the instances from "Unseen" classes use the held-out label. By the end of the final interval, we reveal all ground-truth labels. We now describe the incremental phase in more detail. At the beginning of the incremental label introduction phase, we virtually partition data into two mutually exclusive sets, S: Seen and U: Unseen, as shown in Fig. 1. Data samples in S use their ground-truth labels as target values while those in U use a designated unseen label, which is held constant throughout the entire training process. LILAC assumes a random ordering of labels, Or(M), where M denotes the total number of labels in the dataset. Within this ordering, the number of labels and corresponding data initially placed in S is defined by the variable b. The remaining labels, M − b, are initially placed in U and incrementally revealed in intervals of m labels, a hyper-parameter defined by the user. Training in the incremental phase happens at fixed intervals of E epochs each. Within a fixed interval, the virtual data partition is held constant. Every mini-batch of data is sampled uniformly from the entire original dataset and within each mini-batch, labels are obtained based on their placement in S or U. Then the number of samples from U is reduced or augmented, using a uniform prior, to match the number of samples from S. This is done to ensure no unfair skew in predictions towards U since all data points use the same designated label. Finally, the curated mini-batches of data are used to train the neural network. At the end of each fixed interval, we reveal another set of m groundtruth labels and move samples of those classes from U to S after which the entire data curation and training process is repeated for the next interval. Once all the ground-truth labels are available to the deep network, we begin the adaptive compensation phase of training. The main idea behind adaptive compensation is, if the network is unable to correctly predict a sample's label even after allowing sufficient training time, then we alter the target vector to a less peaked distribution. Compared to learning one-hot vectors, this softer distribution can more easily be learned by the network. Unlike prior methods we adaptively modify the target vector only for incorrectly classified samples on-the-fly. In this phase, the network is trained for a small number of epochs using standard batch learning. Let T be the total number of training epochs in the incremental phase and batch learning. During the adaptive compensation phase, we start at epoch e, where e > T. For a mini-batch of samples in epoch e, predictions from the model at e − 1 are used to determine the final target vector used in the objective function; specifically, we soften the target vector for an instance iff it was misclassified by the model at the end of epoch e − 1. The final target vector for the ith instance at epoch e, t e,i, is computed based on the model φ e−1 using Equation 1. Here, (x i, y i) denote a training sample and its corresponding ground-truth label for sample index i while δ yi represents the corresponding one-hot vector. 1 is a vector of M dimensions with all entries as 1 and is a scaling hyper-parameter. Datasets We use three datasets, CIFAR-10, CIFAR-100 and STL-10 , to evaluate our method and validate our claims. CIFAR-10 and CIFAR-100 are 10 and 100 class variants of the popular image benchmark CIFAR. Each of these contains 50,000 images in the training set and 10,000 images in the testing set. STL-10 is a 10 class subset of ImageNet with 500 and 800 samples per class for training and testing subsets, respectively. Metrics The common metric used to evaluate the performance of all the learning algorithms is average recognition accuracy(%) and standard deviation across 5 trials. We also report consistency, which is a binary metric that indicates whether the training strategy in higher average performance and lower standard deviation compared to standard batch learning across all datasets. Experimental Setup For CIFAR-10/100, we use ResNet18 as the architectural backbone for all methods; for STL-10, we use ResNet34. In each interval of LILAC's incremental phase, we train the model for 10 epochs each for CIFAR-10/100, and 5 epochs each for STL-10. During these incremental steps, we use a learning rate of 0.1, 0.01 and 0.1 for CIFAR-10, CIFAR-100, and STL-10 respectively. The standard batch learning settings used across all datasets are listed in the appendix. These settings reflect the setup used in LILAC once the incremental portion of training is complete and the algorithm moves into the adaptive compensation phase. Within this phase epochs 175, 525 and 120 are used as thresholds (epoch T) for CIFAR-10, 100 and STL-10 respectively. • Stochastic gradient descent with mini-batches is the baseline against which all methods are compared. • Curriculum learning ) forms a family of related works which aim to help models learn faster and optimize to a better minimum. Following the methodology proposed in this work we artificially create a subset of the dataset called "Simple", by selecting data that is within a value of 1.1 as predicted by a linear one-vs-all SVR model trained to regress to the ground-truth label. The deep network is trained on the "Simple" dataset for a fixed period of time that mirrors the total number of epochs of the incremental phase of LILAC after which the entire dataset is used to train the network. • Label Smoothing is the closest relevant work to use label smoothing as a form of regularization without extra data augmentation. This non-invasive baseline is used as a measure of the importance of regularization and for its ability to boost performance. • Dynamic Batch Size (DBS) is a custom baseline used to highlight the importance of variable batch size in training neural networks. DBS randomly copies data available within a mini-batch to mimic variable batch size. Further, all ground-truth labels are available to this model throughout the training process. • Random Augmentation (RA) is a custom baseline used to highlight the importance of virtual data partitions in LILAC. Its implementation closely follows LILAC but excludes the adaptive compensation phase. The main difference between LILAC and RA is that RA uses data from a one randomly chosen class, in U, within a mini-batch while data from all classes in U is used in LILAC to equalize the number of samples from S and U. Table 1 clearly illustrates improvements in average recognition accuracy, decrease in standard deviation and consistency when compared to batch learning. While certain setups have the highest 96.23 Fractional Max-pooling 96.53 Densely Connected Convolutional Networks 96.54 Drop-Activation 96.55 Shake-Drop 96.59 Shake-Drop + LILAC (ours) 96.79 performance on specific datasets (e.g., Label Smoothing on CIFAR-10/100), they are not consistent across all datasets and do not find more stable solutions than LILAC (std. of 0.216 compared to 0.127 from LILAC) LILAC is able to achieve superior performance without unnecessary overheads such as computing sample difficulty or irreversibly altering the ground-truth distribution across all samples. A key takeaway from DBS is the relative drop in standard deviation combined with higher average performance when compared to baselines like fixed curriculum and label smoothing. RA serves to highlight the importance of harvesting data from all classes in U simultaneously, for "negative" samples. The variety of data to learn from provides a boost in performance and standard deviation across the board in LILAC w/o AC as opposed to RA. DBS and RA underline the importance of variable batch size and data partitioning in the makeup of LILAC. We further extend LILAC to train the base pyramidnet with shake-drop regularization (p = 1.0) . From Table 2 we clearly see that LILAC can be extended to provide the highest performance on CIFAR-10 given a standard preprocessing setup. To provide a fair comparison we highlight top performing methods with standard preprocessing setups that avoid partial inputs (at the node or image level) since LILAC was developed with fully available inputs in mind. Across all these learning schemes, LILAC is the only one to consistently increase classification accuracy and decrease the standard deviation across all datasets compared to batch learning. Fig. 2 illustrates the evolution of the embedding across the span of the incremental phase. This space has more degrees of separation when compared to an equivalent epoch of training with batch learning where all the labels are available. Table 3 provides a breakdown of the contribution of each phase of LILAC and how they combine to elevate the final performance. Here, in LILAC w/o AC we replace the entire AC phase with simple batch learning while in Batch + AC we include adaptive compensation with adjusted thresholds. The first half of this table compares the impact of incre- Figure 2: Side-by-side comparison between the representation spaces learned by LILAC and batch learning. Through the entire incremental label introduction phase, the representation space evolves to being more well spaced. Images in column 4 and 5 show comparable points in training space when all labels are available to the deep network being trained. These images support our claim that the deep network starts at a better initialization than standard batch learning, whose effect is carried throughout the training process. mentally introducing labels to a deep network against standard batch learning. We clearly observe that performances across Rows 1 and 2 fall within the indicated standard deviation of each other. However, from Fig. 2 we know that LILAC start from a qualitatively better solution. Combining these , we conclude that the emphasis on a lengthy secondary batch learning phase erodes overall performance. The second half of Table 3 shows the impact of adding adaptive compensation on batch learning and LILAC. When added to standard batch learning there isn't a clear and conclusive indicator of improvement across all benchmarks in both average performance and standard deviation. However, in combination with the incremental label introduction phase of LILAC, adaptive compensation improves average performance as well as decreases standard deviation, indicating an improved stability and consistency. Given the similarity in learning between the batch setup and LILAC, when all labels have been introduced, we show that the embedding space learned by incrementally introducing labels (Fig. 2) is distinct from standard batch learning and is more amenable to AC. Through previous experiments we have established the general applicability of LILAC while contrasting its contributions to that of standard batch learning. In this section we dive deeper to reveal some characteristics of LILAC that further supplement the claim of general applicability. Specifically, we characterize the impact of label ordering, smoothness of alternate target vector distribution and injection of larger groups of labels in the incremental phase. Ordering of Labels Throughout the standard experiments, we assume labels are used in the ascending order of value. When this is modified to a random ordering or in ascending order of diffi- Table 4 suggest that there is no explicit benefit or pattern. Other than the extra impact of continually fluctuating label orders across trials, there isn't a large gap in performance. Thus, we claim LILAC is relatively invariant to the order of label introduction. During adaptive compensation, = 0.5 is used in the alternate target vector for samples with failed predictions throughout all experiments in Sections 3.1 and 3.2. When extended to a variety of values, we observe that most variations of the peak performance still fall within the standard deviation range for each dataset. However, the peak average performance values usually occur between 0.7 to 0.5. While LILAC was designed to allow the introduction of multiple labels in a single incremental step, throughout the experiments in Sections 3.1 and 3.2 only 1 label was introduced per step to allow thorough learning while eliminating the chance of conflicting decision boundaries from multiple labels. Revealing multiple labels instead of 1 label per incremental step has a negative impact on the overall performance of the model. Table 4 shows that adding large groups of labels force lower performance, which is in line with our hypothesis that revealing fewer labels per incremental step makes the embedding space more amenable to adaptive compensation. In this work, we proposed LILAC which rethinks curriculum learning based on incrementally learning labels instead of samples. This approach helps kick-start the learning process from a substantially better starting point while making the learned embedding space amenable to adaptive negative logit compensation. Both these techniques combine well in LILAC to show the highest performance on CIFAR-10 for simple data augmentations while easily outperforming batch and curriculum learning and label smoothing on comparable network architectures. The next step in unlocking the full potential of this setup is to extend this setup to include a confidence measure on the predictions of network so that it can handle the effects of dropout or partial inputs. In further expanding LILAC's ability to handle partial inputs, we aim to explore its effect on standard incremental learning (memory constrained) while also extending it applicability to more complex neural network architectures. A LILAC: ALGORITHM Table 8: The table captures the effect of varying the number of epochs used for the fixed training intervals in the incremental label introduction phase. Across CIFAR-10 there is an obvious peak after which the mean value decreases. However, in STL-10 there seems to be a consistent increase, with the assumption of minor noise. Finally, in CIFAR-100 there isn't a clear pattern. From the in Table 8, we observe that the choice of E is dependent on the dataset. There isn't an explicit pattern that can be used to select the value of E without trial runs. Further, the available run-time is an important constraint when select E from a range of values since both m and E affect it.

Represent the short summary for finding the corresponding scientific article (long)

A novel approach to curriculum learning by incrementally learning labels and adaptively smoothing labels for mis-classified samples which boost average performance and decreases standard deviation.

We introduce a type of neural network that is structurally resistant to adversarial attacks, even when trained on unaugmented training sets.  The resistance is due to the stability of network units wrt input perturbations.