Question: Represent the scientific article to find a short TLDR

Question: It is fundamental and challenging to train robust and accurate Deep Neural Networks (DNNs) when semantically abnormal examples exist. Although great progress has been made, there is still one crucial research question which is not thoroughly explored yet: What training examples should be focused and how much more should they be emphasised to achieve robust learning? In this work, we study this question and propose gradient rescaling (GR) to solve it. GR modifies the magnitude of logit vector’s gradient to emphasise on relatively easier training data points when noise becomes more severe, which functions as explicit emphasis regularisation to improve the generalisation performance of DNNs. Apart from regularisation, we connect GR to examples weighting and designing robust loss functions. We empirically demonstrate that GR is highly anomaly-robust and outperforms the state-of-the-art by a large margin, e.g., increasing 7% on CIFAR100 with 40% noisy labels. It is also significantly superior to standard regularisers in both clean and abnormal settings. Furthermore, we present comprehensive ablation studies to explore the behaviours of GR under different cases, which is informative for applying GR in real-world scenarios. DNNs have been successfully applied in diverse applications (; ;). However, their success is heavily reliant on the quality of training data, especially accurate semantic labels for learning supervision. Unfortunately, on the one hand, maintaining the quality of semantic labels as the scale of training data increases is expensive and almost impossible when the scale becomes excessively large. On the other hand, it has been demonstrated that DNNs are capable of memorising the whole training data even when all training labels are random . Therefore, DNNs struggle to discern meaningful data patterns and ignore semantically abnormal examples 1 simultaneously . Consequently, it becomes an inevitable demand for DNNs to hold robustness when training data contains anomalies (; ; ; ; ; ; ;). Recently, great progress has been made towards robustness against anomalies when training DNNs . There are three appealing perspectives in terms of their simplicity and effectiveness: 1) Examples weighting. For example, knowledge distilling from auxiliary models is popular for heuristically designing weighting schemes. However, it is challenging to select and train reliable auxiliary models in practice (; ; ; ; b). 2) Robust loss functions (; ; ; b); 3) Explicit regularisation techniques (; . Although designing robust losses or explicit regularisation is easier and more flexible in practice, the performance is not the optimal yet. 1 One training example is composed of an input and its corresponding label. A semantically abnormal example means the input is semantically unrelated to its label, which may come from corrupted input or label. For example, in Figure 3 in the supplementary material: 1) Out-of-distribution anomalies: An image may contain only or an object which does not belong to any training class; 2) In-distribution anomalies: An image of class a may be annotated to class b or an image may contain more than one semantic object. Regarding examples weighting, there is a core research question which is not well answered yet: What training examples should be focused on and how large the emphasis spread should be? In this work, we present a thorough study of this practical question under different settings. For better analysis, we propose two basic and necessary concepts: emphasis focus and spread with explicit definition in Sec. 3.2. They are conceptually introduced as follows: Emphasis focus. It is a common practice to focus on harder instances when training DNNs . When a dataset is clean, it achieves faster convergence and better performance to emphasise on harder examples because they own larger gradient magnitude, which means more information and a larger update step for model's parameters. However, when severe noise exists, as demonstrated in , DNNs learn simple meaningful patterns first before memorising abnormal ones. In other words, anomalies are harder to fit and own larger gradient magnitude in the later stage. Consequently, if we use the default sample weighting in categorical cross entropy (CCE) where harder samples obtain higher weights, anomalies tend to be fitted well especially when a network has large enough capacity. That is why we need to move the emphasis focus towards relatively easier ones, which serves as emphasis regularisation. Emphasis spread. We term the weighting variance of training examples emphasis spread. The key concept is that we should not treat all examples equally, neither should we let only a few be emphasised and contribute to the training. Therefore, when emphasis focus changes, the emphasis spread should be adjusted accordingly. We integrate emphasis focus and spread into a unified example weighting framework. Emphasis focus defines what training examples own higher weights while emphasis spread indicates how large variance over their weights. Specifically, we propose gradient rescaling (GR), which modifies the magnitude of logit vector's gradient. The logit vector is the output of the last fully connected (FC) layer of a network. We remark that we do not design the weighting scheme heuristically from scratch. Instead, it is naturally motivated by the gradient analysis of several loss functions. Interestingly, GR can be naturally connected to examples weighting, robust losses, explicit regularisation: 1) The gradient magnitude of logit vector can be regarded as weight assignment that is built-in in loss functions (; ; b). Therefore, rescaling the gradient magnitude equals to adjusting the weights of examples; 2) A specific loss function owns a fixed gradient derivation. Adjusting the gradient can be treated as a more direct and flexible way of modifying optimisation objectives; 3) Instead of focusing on harder examples 2 by default, we can adjust emphasis focus to relative easier ones when noise is severe. GR serves as emphasis regularisation and is different from standard regularisers, e.g., L2 weight decay constraints on weight parameters and Dropout samples neural units randomly ; GR is simple yet effective. We demonstrate its effectiveness on diverse computer vision tasks using different net architectures: 1) Image classification with clean training data; 2) Image classification with synthetic symmetric label noise, which is more challenging than asymmetric noise evaluated by (; ; 3) Image classification with real-world unknown anomalies, which may contain open-set noise, e.g., images with only , or outliers, etc.; 4) Video person re-identification, a video retrieval task containing diverse anomalies. Beyond, we show that GR is notably better than other standard regularisers, e.g., L2 weight decay and dropout. Besides, to comprehensively understand GR's behaviours, we present extensive ablation studies. Main contribution. Intuitively and principally, we claim that two basic factors, emphasis focus and spread, should be babysat simultaneously when it comes to examples weighting. To the best of our knowledge, we are the first to thoroughly study and analyse them together in a unified framework. Aside from examples weighting, robust losses minimisation and explicit regularisation techniques, there are another two main perspectives for training robust and accurate DNNs when anomalies exist: 2 An example's difficulty can be indicated by its loss (; ;), gradient magnitude , or input-to-label relevance score . The input-to-label relevance score means the probability of an input belonging to its labelled class predicted by a current model. The difficulty of an example may change as the model learns. In summary, higher difficulty, larger loss, larger gradient magnitude, and lower input-to-label relevance score are equal concepts. 1) Robust training strategies (; ; ;); 2) Noise-aware modelling, and alternative label and parameter optimisation are popular when only label noise exists. Some methods focus on noise-aware modelling for correcting noisy labels or empirical losses (; ; ; ; ; ; ; a). However, it is non-trivial and time-consuming to learn a noise-aware model, which also requires prior extra information or some specific assumptions. For example, Masking (a) is assisted by human cognition to speculate the noise structure of noise-aware matrix while (; ; ;) exploit an extra clean dataset, which is a hyper-factor and hard to control in practice. Some other algorithms iteratively train the model and infer latent true labels ). Those methods have made great progress on label noise. But they are not directly applicable to unknown diverse semantic anomalies, which covers both out-of-distribution and in-distribution cases. We note that proposed some theorems showing that empirical risk minimization is robust when the loss function is symmetric and the noise type is label noise. However, they are not applicable for deep learning under arbitrary unknown noise: 1) We remark that we target at the problem of diverse or arbitrary abnormal examples, where an input may be out-of-distribution, i.e., not belonging to any training class. As a , the symmetric losses custom-designed for label noise are not applicable. 2) GR is independent of empirical loss expressions as presented in Table 1. Therefore, one specific loss is merely an indicator of how far we are away from a specific minimisation objective. It has no impact on the robustness of the learning process since it has no direct influence on the gradient back-propagation. Similar to the prior work of rethinking generalisation , we need to rethink robust training under diverse anomalies, where the robustness theorems conditioned on symmetric losses and label noise are not directly applicable. Notation. We are given N training examples, where (x i, y i) denotes i−th sample with input x i ∈ R D and label y i ∈ {1, 2, ..., C}. C is the number of classes. Let's consider a deep neural network z composed of an embedding network f (·): Generally, the linear classifier is the last FC layer which produces the final output of z, i.e., logit vector z ∈ R C. To obtain probabilities of a sample belonging to different classes, logit vector is normalised by a softmax function: p(j|x i) is the probability of x i belonging to class j. A sample's input-to-label relevance score is defined by In what follows, we will uncover the sample weighting in popular losses: CCE, Mean Absolute Error (MAE) and Generalised Cross Entropy (GCE) . CCE. The CCE loss with respect to (x i, y i), and its gradient with respect to z ij are defined as: Therefore, we have || Here we choose L1 norm to measure the magnitude of gradient because of its simpler statistics and computation. Since we back-propagate ∂L CCE /z i to update the model's parameters, an example's gradient magnitude determines how much impact it has, i.e., its weight w In CCE, more difficult examples with smaller p i get higher weight. When it comes to MAE, the loss of (x i, y i) and gradient with respect to z im are: (a) GR, CCE, MAE, GCE. We show 3 settings of GR: (β = 2, λ = 0), (β = 8, λ = 0.5) and (β = 12, λ = 1). Their corresponding emphasis focuses are 0, 0∼0.5 and 0.5. (b) GR when fixing λ = 0.5 (emphasis focus is within 0∼0.5) or λ = 2 (emphasis focus is within 0.5∼1). (c) GR when fixing β = 8. When λ increases, the emphasis focus moves towards 1 and emphasis spread drops. Figure 1: A sample's weight w i along with its input-to-label relevance score p i. GR is a unified sample reweighting framework from the perspective of gradient rescaling, where the emphasis focus and spread can be adjusted by choosing proper λ and β in practice. Better viewed in colour. Therefore, w In MAE, those images whose input-to-label relevance scores are 0.5 become the emphasis focus. GCE. In GCE, the loss calculation of (x i, y i) and gradient with respect to logit vector z i are: where In this case, the emphasis focus can be adjusted from 0 to 0.5 when q ranges from 0 to 1. However, in their practice , instead of using this naive version, a truncated one is applied: The loss of an example with p i ≤ 0.5 is constant so that its gradient is zero, which means it is dropped and does not contribute to the training. The main drawback is that at the initial stage, the model is not well learned so that the predicted p i of most samples are smaller than 0.5. To address it, alternative convex search is exploited for iterative data pruning and parameters optimisation, making it quite complex and less appealing in practice. The derivation details of Eq.,, are presented in Section B of the supplementary material. A loss function provides supervision information by its derivative with respect to a network's output. Therefore, there are two perspectives for improving the supervision information: 1) Modifying the loss format to improve its corresponding derivative; 2) Manipulating the gradient straightforwardly. In this work, we choose to control the gradient, which is more direct and flexible. According to Eq.,,, the gradients of CCE, MAE and GCE share the same direction. Our proposal GR unifies them from the gradient perspective. Being independent of loss formulas, a sample's gradient is rescaled linearly so that its weight is w GR i: where λ, β are hyper-parameters for controlling the emphasis focus and spread, respectively. By choosing a larger λ when more anomalies exist, GR regularises examples weighting by moving emphasis focus toward relatively easier training data points, thus embracing noise-robustness. For clarification, we explicitly define the emphasis focus and spread over training examples: Definition 1 (Emphasis Focus ψ). The emphasis focus refers to those examples that own the largest weight. Since an example's weight is determined by its input-to-label relevance score p i, for simplicity, we define the emphasis focus to be an input-to-label score to which the largest weight is assigned, i.e., ψ = arg max Definition 2 (Emphasis Spread σ). The emphasis spread is the weight variance over all training instances in a mini-batch, i.e., σ = E((w, where E(·) denotes the expectation value of a variable. With these definitions, we differentiate GR with other methods in Table 1. We show the sample weighting curves of GR with different settings in Figure 1. As shown in Figure 1c, the emphasis spread declines as λ increases. Therefore, we choose larger β values when λ is larger in Sec. 4.2.1. Principally, transformation g could be designed as any monotonically increasing function. Because the non-linear exponential mapping can change the overall weights' variance and relative weights between any two examples, we choose g(·) = exp(·), which works well in our practice. By integral, the exact loss format is an error function (non-elementary). We summarise several existing cases as follows (the ellipsis refers to other potential options which can be explored in the future): Let's regard a deep network z as a black box, which produces C logits. C is the class number. Then during gradient back-propagation, an example's impact on the update of z is determined by its gradient w.r.t. the logit vector. The impact can be decomposed into two factors, i.e., gradient direction and magnitude. To reduce the impact of a noisy sample, we can either reduce its gradient magnitude or amend its gradient direction. In this work, inspired by the analysis of CCE, MAE and GCE, which only differ in the gradient magnitude while perform quite differently, leading to a natural motivation that gradient magnitude matters. That is why we explore rescaling the gradient magnitude as illustrated in Figure 1. It is worth studying amending gradient directions in the future. Datasets. We test on CIFAR-10 and CIFAR-100 , which contain 10 and 100 classes, respectively. In CIFAR-10, the training data contains 5k images per class while the test set includes 1k images per class. In CIFAR-100, there are 500 images per class for training and 100 images per class for testing. Implementation details. On CIFAR-10, following , we adopt ResNet-20 and ResNet-56 as backbones so that we can compare fairly with their reported . On CIFAR-100, we follow D2L to choose ResNet-44 and compare with its reported . We also use an SGD optimiser with momentum 0.9 and weight decay 10 −4. The learning rate is initialised with 0.1, and multiplied with 0.1 every 5k iterations. We apply the standard data augmentation as in (; : The original images are padded with 4 pixels on every side, followed by a random crop of 32 × 32 and horizontal flip. The batch size is 128. Table 2 : Classification accuracies (%) of CCE, and GR on clean CIFAR-10 and CIFAR-100. λ = 0 means the emphasis focus is 0 where we fix β = 2. β = 0 means all examples are treated equally. Backbone CCE GR (λ = 0) GR (β = 0) Results. Our purpose is to show GR can achieve competitive performance with CCE under clean data to demonstrate its general applicability. As reported in D2L, all noise-tolerant proposals (; perform similarly with CCE when training labels are clean. Therefore we do not present other related competitors here. Our reimplemented are shown in Table 2 . For reference, the reported in on CIFAR-10 with CCE are 91.3% for ResNet-20 and 93.0% for ResNet-56. In D2L, the on CIFAR-100 with ResNet-44 is 68.2%. Our reimplemented performance of CCE is only slightly different. For GR, we observe the best performance when emphasis focus is 0, i.e., λ = 0. Furthermore, it is insensitive to a wide range of emphasis spreads according to our observations in Figure 5 in the supplementary material. Treating training examples equally. As shown in Table 2, we obtain competitive performance by treating all training examples equally when β = 0. This is quite interesting and motivates us that sample differentiation and reweighting work much better only when noise exists. Symmetric noise generation. Given a probability r, the original label of an image is changed to one of the other class labels uniformly following (; . r denotes the noise rate. Symmetric label noise generally exists in large-scale real-world applications where the dataset scale is so large that label quality is hard to guarantee. It is also demonstrated in that it is more challenging than asymmetric noisy labels ), which assume that label errors only exist within a predefined set of similar classes. All augmented training examples share the same label as the original one. To understand GR well empirically, we explore the behaviours of GR on CIFAR-10 with r = 20%, 40%, 60%, 80%, respectively. We use ResNet-56 which has larger capacity than ResNet-20. Design choices. We mainly analyse the impact of different emphasis focuses for different noise rates. We explore 5 emphasis focuses by setting β = 0 or different λ: 1) None: β = 0. There is no emphasis focus since all examples are treated equally; 2) 0: λ = 0; 3) 0∼0.5: λ = 0.5; 4) 0.5: λ = 1; 5) 0.5∼1: λ = 2. We remark that when λ is larger, the emphasis focus is higher, leading to relatively easier training data points are emphasised. As shown in Figure 1, when emphasis focus changes, emphasis spread changes accordingly. Therefore, to set a proper spread for each emphasis focus, we try 4 emphasis spread and choose the best one 3 to compare the impact of emphasis focus. Results analysis. We show the in Table 3. The intact training set serves as a validation set and we observe that its accuracy is always consistent with the final test accuracy. This motivates us that we can choose our model's hyper-parameters β, λ via a validation set in practice. We display the training dynamics in Figure 2. We summarise our observations as follows: Fitting and generalisation. We observe that CCE always achieves the best accuracy on corrupted training sets, which indicates that CCE has a strong data fitting ability even if there is severe noise . As a , CCE has much worse final test accuracy than most models. Emphasising on harder examples. When there exist abnormal training examples, we obtain the worst final test accuracy if emphasis focus is 0, i.e., CCE and GR with λ = 0. This unveils that in applications where we have to learn from noisy training data, it will hurt the model's generalisation dramatically if we use CCE or simply focus on harder training data points. Emphasis focus. When noise rate is 0, 20%, 40%, 60%, and 80%, we obtain the best final test accuracy when λ = 0, λ = 0.5, λ = 1, λ = 2, and λ = 2, respectively. This demonstrates that when noise rate is higher, we can improve a model's robustness by moving emphasis focus towards relatively less difficult examples with a larger λ, which is informative in practice. Emphasis spread. As displayed in Table 3 and Figures 7-10 in the supplementary material, emphasis spread also matters a lot when fixing emphasis focus, i.e., fixing λ. For example in Table 3, when λ = 0, although focusing on harder examples similarly with CCE, GR can outperform CCE by modifying the emphasis spread. As shown in Figures 7-10, some models even collapse and cannot converge if the emphasis spread is not rational. Implementation details. We follow the same settings as MentorNet to compare fairly with its reported . Optimiser and data augmentation are described in Section 4.1. Competitors. FullModel is the standard CCE trained using L2 weight decay and dropout . Forgetting searches the dropout parameter in the range of (0.2-0.9). Self-paced , Focal Loss , and MentorNet are representatives of example reweighting algorithms. Reed Soft ) is a weaklysupervised learning method. All methods use GoogLeNet V1. Results. We compare the under different noise rates in Table 4. GR with fixed hyperparameters β = 8, λ = 0.5 outperforms the state-of-the-art GCE by a large margin, especially when label noise becomes severe. Better can be expected when optimising the hyper-parameters for each case. We remark that FullModel (naive CCE) was trained with L2 weight decay and dropout. However, GR's regularization effect is much better in both clean and noisy cases. Figure 6 in the supplementary material. We have two key observations: 1) When noise rate increases, better generalisation is obtained with higher emphasis focus, i.e., focusing on relatively easier examples; 2) Both overfitting and underfitting lead to bad generalisation. For example,'CCE: 0' fits training data much better than the others while'GR: None' generally fits it unstably or a lot worse. Better viewed in colour. Implementation details. Most baselines have been reimplemented in with the same settings. Therefore, for direct comparison, we follow exactly their experimental configurations and use ResNet-44 . Optimiser and data augmentation are described in Section 4.1. We repeat training and evaluation 5 times where different random seeds are used for generating noisy labels and model's initialisation. The mean test accuracy and standard deviation are reported. Competitors. We compare with D2L, GCE , and other baselines reimplemented in D2L: 1) Standard CCE; 2) Forward uses a noise-transition matrix to multiply the network's predictions for label correction; 3) Backward applies the noise-transition matrix to multiply the CCE losses for loss correction; 4) Bootstrapping trains models with new labels generated by a convex combination of the original ones and their predictions. The convex combination can be soft (Boot-soft) or hard (Boot-hard); 5) D2L achieves noise-robustness from a novel perspective of restricting the dimensionality expansion of learned subspaces during training and is the state-of-the-art; 6) Since GCE outperforms MAE , we only reimplement GCE for comparison; 7) SL (c) boosts CCE symmetrically with a noise-robust counterpart, i.e., reverse cross entropy. Results. We compare the of GR and other algorithms in Table 5. GR outperforms other competitors by a large margin, especially when label noise is severe, e.g., r = 40% and 60%. More importantly, we highlight that GR is much simpler without any extra information. Compared with Forward and Backward, GR does not need any prior knowledge about the noise-transition matrix. Bootstrapping targets at label correction and is time-consuming. D2L estimates the local intrinsic dimensionality every b mini-batches and checks the turning point for dimensionality expansion every e epochs. However, b and e are difficult to choose and iterative monitoring is time-consuming. Dataset. Clothing 1M contains 1 million images. It is an industrial-level dataset and its noise structure is agnostic. According to , around 61.54% training labels are reliable, i.e., the noise rate is about 38.46%. There are 14 classes from several online shopping websites. In addition, there are 50k, 14k, and 10k images with clean labels for training, validation, Table 5: The accuracies (%) of GR and recent approaches on CIFAR-100. The of fixed parameters (β = 8, λ = 0.5) are shown in the second last column. With a little effort for optimising β and λ, the and corresponding parameters are presented in the last column. The trend is consistent with Table 3: When r raises, we can increase β, λ for better robustness. The increasing scale is much smaller. This is because CIFAR-100 has 100 classes so that its distribution of p i (input-to-label relevance score) is different from CIFAR-10 after softmax normalisation. and testing, respectively. Here, we follow and compare with existing methods that only learn from noisy training data since we would like to avoid exploiting auxiliary information. Implementation details. We train ResNet-50 and follow exactly the same settings as : 1) Initialisation: ResNet-50 is initialised by publicly available model pretrained on ImageNet ; 2) Optimisation: A SGD optimiser with a momentum of 0.9 and a weight decay of 10 −3 is applied. The learning rate starts at 10 −3 and is divided by 10 after 5 epochs. Training terminates at 10 epochs; 3) Standard data augmentation: We first resize a raw input image to 256 × 256, and then crop it randomly at 224 × 224 followed by random horizontal flipping. The batch size is 64 due to memory limitation. Since the noise rate is around 38.46%, we simply set λ = 1, β = 16 following Table 3 when noise rate is 40%. Competitors. We compare with other noise-robust algorithms that have been evaluated on Clothing 1M with similar settings: 1) Standard CCE ; 2) Since Forward outperforms Backward on Clothing 1M , we only present the of Forward; 3) S-adaptation applies an additional softmax layer to estimate the noise-transition matrix ; 4) Masking is a human-assisted approach that conveys human cognition to speculate the structure of the noise-transition matrix (a). 5) Label optimisation learns latent true labels and model's parameters iteratively. Two regularisation terms are added for label optimisation and adjusted in practice. Results. The are compared in Table 6. Under real-world agnostic noise, GR also outperforms the state-of-the-art. It is worth mentioning that the burden of noise-transition matrix estimation in Forward and S-adaptation is heavy due to alternative optimisation steps, and such estimation is non-trivial without big enough data. Masking exploits human cognition of a structure prior and reduces the burden of estimation, nonetheless its performance is not competitive. Similarly, Label Optimisation requires alternative optimisation steps and is time-consuming. Dataset and evaluation settings. MARS contains 20,715 videos of 1,261 persons . There are 1,067,516 frames in total. Because person videos are collected by tracking and detection algorithms, abnormal examples exist as shown in Figure 3 in the supplementary material. We remark that there are some anomalies containing only or an out-of-distribution person. Exact noise type and rate are unknown. Following standard settings, we use 8,298 videos of 625 persons for training and 12,180 videos of the other 636 persons for testing. We report the cumulated matching characteristics (CMC) and mean average precision (mAP) . and (c), respectively. CCE* and GCE* are our reproduced using the Caffe framework (Implementation details. Following (; a), we train GoogleNet V2 and treat a video as an image set, which means we use only appearance information without exploiting latent temporal information. A video's representation is simply the average fusion of its frames' representations. The learning rate starts from 0.01 and is divided by 2 every 10k iterations. We stop training at 50k iterations. We apply an SGD optimiser with a weight decay of 0.0005 and a momentum of 0.9. The batch size is 180. We use standard data augmentation: a 227 × 227 crop is randomly sampled and flipped after resizing an original image to 256 × 256. Training settings are the same for each method. We implement GCE with its reported best settings. At testing, following (a; ;), we first L 2 normalise videos' features and then calculate the cosine similarity between every two of them. Results. The are displayed in Table 7. Although DRSA and CAE exploit extra temporal information by incorporating attention mechanisms, GR is superior to them in terms of both effectiveness and simplicity. OSM+CAA (a) is the only comparable method. However, OSM+CAA combines CCE and weighted contrastive loss to address anomalies, thus being more complex than GR. In addition, we highlight that one query may have multiple matching instances in the MARS benchmark. Consequently, mAP is a more reliable and accurate performance assessment. GR is the best in terms of mAP. In Table 8, we compare our proposed regulariser GR with other standard ones, i.e., L2 weight decay and Dropout . We set the dropout rate to 0.2 and L2 weight decay rate to 10 −4. For GR, as mentioned in Section 4.2.3, we fix β = 8, λ = 0.5. Interestingly, Dropout+L2 achieves 52.8% accuracy, which is even better than the state-of-the-art in Table 5, i.e., D2L with 52.0% accuracy. However, GR is better than those standard regularisers and their combinations significantly. GR works best when it is together with L2 weight decay. Table 8: Results of GR and other standard regularisers on CIFAR-100. We set r = 40%, i.e., the label noise is severe but not belongs to the majority. We train ResNet-44. We report the average test accuracy and standard deviation (%) over 5 trials. Baseline means CCE without regularisation. In this work, we present three main contributions: 1) We analyse and answer a core research question: What training examples should be focused on and how large the emphasis spread should be? 2) We uncover and analyse that two basic factors, emphasis focus and spread, should be babysat simultaneously when it comes to examples weighting. Consequently, we propose a simple yet effective gradient rescaling framework serving as emphasis regularisation. 3) Extensive experiments on different tasks using different network architectures are reported for better understanding and demonstration of GR's effectiveness, which are also valuable for applying GR in practice. . Out-of-distribution anomalies: 1) The first image in the 3rd row contains only and no semantic information at all. 2) The 2nd first image or the last one in the 3rd row may contain a person that does not belong to any person in the training set. In-distribution anomalies: 1) Some images of deer class are wrongly annotated to horse class. 2) We cannot decide the object of interest without any prior when an image contains more than one object, e.g., some images contain two persons in the 2nd row. For left and right sides of Eq., we calculate their derivatives w.r.t. z ij simultaneously. If j = y i, In summary, the derivation of softmax layer is: B.2 DERIVATION OF CCE According to Eq., we have Therefore, we obtain (the parameters are omitted for brevity), B.3 DERIVATION OF MAE According to Eq., we have Therefore, we obtain According to Eq., we have Therefore, we obtain B.5 DERIVATIVES W.R.T. LOGITS z i The calculation is based on Eq. and Eq.. If j = y i, we have: If j = y i, it becomes: In summary, ∂L CCE /∂z i can be represented as: otherwise (j = y i): In summary, ∂L MAE /∂z i is: The calculation is based on Eq. and Eq.. If j = y i, we have: If j = y i, it becomes: In summary, ∂L GCE /∂z i can be represented as: C SMALL-SCALE FINE-GRAINED VISUAL CATEGORISATION OF VEHICLES How does GR perform on small datasets, for example, the number of data points is no more than 5,000? We have tested GR on CIFAR-10 and CIFAR-100 in the main paper. However, both of them contain a training set of 50,000 images. For this question, we answer it from different perspectives as follows: 1. The problem of label noise we study on CIFAR-10 and CIFAR-100 in Section 4.2 is of similar scale. For example: • In Table 4, when noise rate is 80% on CIFAR-10, the number of clean training examples is around 50, 000 × 20% = 5, 000 × 2. Therefore, this clean set is only two times as large as 5,000. Beyond, the learning process may be interrupted by other noisy data points. • In Table 5, when noise rate is 60% on CIFAR-100, the number of clean training data points is about 50, 000 × 40% = 5, 000 × 4, i.e., four times as large as 5,000. 2. We compare GR with other standard regularisers on a small-scale fine-grained visual categorisation problem in Table 9. Vehicles-10 Dataset. In CIFAR-100 , there are 20 coarse classes, including vehicles 1 and 2. Vehicles 1 contains 5 fine classes: bicycle, bus, motorcycle, pickup truck, and train. Vehicles 2 includes another 5 fine classes: lawn-mower, rocket, streetcar, tank, and tractor. We build a small-scale vehicles classification dataset composed of these 10 vehicles from CIFAR-100. Specifically, the training set contains 500 images per vehicle class while the testing set has 100 images per class. Therefore, the number of training data points is 5,000 in total. We evaluate on CIFAR-100, whose 100 classes are grouped into 20 coarse classes. Every coarse class has 5 fine classes. Within each coarse class, an image's label is flipped to one of the other four labels uniformly with a probability r. r represents the noise rate. We set r = 0.2. The are displayed in Table 10. When GR is used, the performance is better than its counterparts without GR. The are shown in Table 11. Proposal: Gradient rescaling incorporates emphasis focus (centre/focal point) and emphasis spread, and serves as explicit regularisation in terms of sample reweighting/emphasis. Finding: When noise rate is higher, we can improve a model's robustness by moving emphasis focus towards relatively less difficult examples. The more detailed on CIFAR-100 are shown in Table 12, which is the supplementary of Table 5 in the main text. Table 12: Exploration of GR with different emphasis focuses (centres) and spreads on CIFAR-100 when r = 20%, 40%, 60%, respectively. This table presents detailed information of optimising λ, β mentioned in Table 5 in the paper. Specifically, for each λ, we try 5 β values from {2, 4, 6, 8, 10} and select the best one as the final of the λ. We report the mean test accuracy over 5 repetitions. Our key finding is demonstrated again: When r raises, we can increase β, λ for better robustness. The increasing scale is much smaller than CIFAR-10. This is because CIFAR-100 has 100 classes so that its distribution of p i (input-to-label relevance score) is different from CIFAR-10 after softmax normalisation. Figure 2 in the paper. We have two key observations: 1) When noise rate increases, better generalisation is obtained with higher emphasis focus, i.e., focusing on relatively easier examples; 2) Both overfitting and underfitting lead to bad generalisation. For example,'CCE: 0' fits training data much better than the others while'GR: None' generally fits it unstably or a lot worse. Better viewed in colour.. From left to right, the of four emphasis focuses 0, 0∼0.5, 0.5, 0.5∼1 with different emphasis spreads are displayed in each column respectively. When λ is larger, β should be larger as displayed in Figure 1c in the paper. Specifically: 1) when λ = 0: we tried β = 0.5, 1, 2, 4; 2) when λ = 0.5: we tried β = 4, 8, 12, 16; 3) when λ = 1: we tried β = 8, 12, 16, 20; 4) when λ = 2: we tried β = 12, 16, 20, 24.

Represent the short summary for finding the corresponding scientific article (long)

ROBUST DISCRIMINATIVE REPRESENTATION LEARNING VIA GRADIENT RESCALING: AN EMPHASIS REGULARISATION PERSPECTIVE

Represent the TLDR for finding the corresponding scientific article

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.


Question: Represent this text

Question: Generative Adversarial Networks (GANs) have achieved remarkable in the task of generating realistic natural images. In most applications, GAN models share two aspects in common. On the one hand, GANs training involves solving a challenging saddle point optimization problem, interpreted as an adversarial game between a generator and a discriminator functions. On the other hand, the generator and the discriminator are parametrized in terms of deep convolutional neural networks. The goal of this paper is to disentangle the contribution of these two factors to the success of GANs. In particular, we introduce Generative Latent Optimization (GLO), a framework to train deep convolutional generators without using discriminators, thus avoiding the instability of adversarial optimization problems. Throughout a variety of experiments, we show that GLO enjoys many of the desirable properties of GANs: learning from large data, synthesizing visually-appealing samples, interpolating meaningfully between samples, and performing linear arithmetic with noise vectors. Generative Adversarial Networks (GANs) BID15 are a powerful framework to learn generative models of natural images. GANs learn these generative models by setting up an adversarial game between two learning machines. On the one hand, a generator plays to transform noise vectors into fake samples, which resemble real samples drawn from a distribution of natural images. On the other hand, a discriminator plays to distinguish between real and fake samples. During training, the generator and the discriminator learn in turns. First, the discriminator learns to assign high scores to real samples, and low scores to fake samples. Then, the generator learns to increase the scores of fake samples, as to fool the discriminator. After proper training, the generator is able to produce realistic natural images from noise vectors. Recently, GANs have been used to produce high-quality images resembling handwritten digits, human faces, and house interiors BID36. Furthermore, GANs exhibit three strong signs of generalization. First, the generator translates linear interpolations in the noise space into semantic interpolations in the image space. In other words, a linear interpolation in the noise space will generate a smooth interpolation of visually-appealing images. Second, the generator allows linear arithmetic in the noise space. Similarly to word embeddings BID31, linear arithmetic indicates that the generator organizes the noise space to disentangle the nonlinear factors of variation of natural images into linear statistics. Third, the generator is able to to synthesize new images that resemble those of the data distribution. This allows for applications such as image in-painting BID18 and super-resolution BID26.Despite their success, training and evaluating GANs is notoriously difficult. The adversarial optimization problem implemented by GANs is sensitive to random initialization, architectural choices, and hyper-parameter settings. In many cases, a fair amount of human care is necessary to find the correct configuration to train a GAN in a particular dataset. It is common to observe generators with similar architectures and hyper-parameters to exhibit dramatically different behaviors. Even when properly trained, the ing generator may synthesize samples that resemble only a few localized regions (or modes) of the data distribution BID14. While several advances have been made to stabilize the training of GANs BID37, this task remains more art than science. The difficulty of training GANs is aggravated by the challenges in their evaluation: since evaluating the likelihood of a GAN with respect to the data is an intractable problem, the current gold standard to evaluate the quality of GANs is to eyeball the samples produced by the generator. The evaluation of discriminators is also difficult, since their visual features do not always transfer well to supervised tasks BID12 BID13. Finally, the application of GANs to non-image data has been relatively limited. Research question To model natural images with GANs, the generator and discriminator are commonly parametrized as deep Convolutional Networks (convnets) BID24. Therefore, it is reasonable to hypothesize that the reasons for the success of GANs in modeling natural images come from two complementary sources: (A1) Leveraging the powerful inductive bias of deep convnets. (A2) The adversarial training protocol. This work attempts to disentangle the factors of success (A1) and (A2) in GAN models. Specifically, we propose and study one algorithm that relies on (A1) and avoids (A2), but still obtains competitive when compared to a GAN. We investigate the importance of the inductive bias of convnets by removing the adversarial training protocol of GANs (Section 2). Our approach, called Generative Latent Optimization (GLO), maps one learnable noise vector to each of the images in our dataset by minimizing a simple reconstruction loss. Since we are predicting images from learnable noise, GLO borrows inspiration from recent methods to predict learnable noise from images BID3. Alternatively, one can understand GLO as an auto-encoder where the latent representation is not produced by a parametric encoder, but learned freely in a non-parametric manner. In contrast to GANs, we track of the correspondence between each learned noise vector and the image that it represents. Hence, the goal of GLO is to find a meaningful organization of the noise vectors, such that they can be mapped to their target images. To turn GLO into a generative model, we observe that it suffices to learn a simple probability distribution on the learned noise vectors. In our experiments (Section 3), we show that GLO inherits many of the appealing features of GANs, while enjoying a much simpler training protocol. In particular, we study the efficacy of GLO to compress and decompress a dataset of images (Section 3.3.1), generate new samples (Section 3.3.2), perform linear interpolations and extrapolations in the noise space (Section 3.3.3), and perform linear arithmetics (Section 3.3.5). Our experiments provide quantitative and qualitative comparisons to Principal Component Analysis (PCA), Variational Autoencoders (VAE) and GANs. We focus on the CelebA and LSUN-Bedroom datasets. We conclude our exposition in Section 5. First, we consider a large set of images {x 1, . . ., x N}, where each image x i ∈ X has dimensions 3 × w×h. Second, we initialize a set of d-dimensional random vectors {z 1, . . ., z N}, where z i ∈ Z ⊆ R d for all i = 1,... N. Third, we pair the dataset of images with the random vectors, obtaining the dataset {(z 1, x 1),..., (z N, x N)}. Finally, we jointly learn the parameters θ in Θ of a generator g θ: Z → X and the optimal noise vector z i for each image x i, by solving: DISPLAYFORM0 In the previous,: X × X is a loss function measuring the reconstruction error from g(z i) to x i. We call this model Generative Latent Optimization (GLO).Learnable z i. In contrast to autoencoders BID6, which assume a parametric model f: X → Z, usually referred to as the encoder, to compute the vector z from samples x, and minimize the reconstruction loss (g(f (x)), x), in GLO we jointly optimize the inputs z 1,..., z N and the model parameter θ. Since the vector z is a free parameter, our model can recover all the solutions that could be found by an autoencoder, and reach some others. In a nutshell, GLO can be viewed as an "encoder-less" autoencoder, or as a "discriminator-less" GAN. Figure 1: Plot of the cumulative sum of the singular values of the optimal Z * matrix. We observe that the proposed GLO model has a better conditioned covariance matrix and therefore better fills the latent space. Choice of Z. The representation space Z should encapsulate all of our prior knowledge about the data {x 1, . . ., x N}. Since we are interested in matching the properties of GANs, we make similar choices to them when it comes to the representation space Z. The most common choices of the representation space for GANs are either the uniform distribution on the hypercube DISPLAYFORM1 In previous literature, Gaussian distributions lead to more stable GAN training BID36, we will take this choice to design our representation space. In GLO, the random vectors z are learnable and can therefore attain any value during the training process. To avoid extremely large values, we normalize our learnable noise vectors z at all times, to lay on the unit 2 sphere. Choice of loss function. On the one hand, the squared-loss function 2 (x, x) = x − x 2 2 is a simple choice, but leads to blurry (average) reconstructions of natural images. On the other hand, GANs use a convnet (the discriminator) as loss function. Since the early layers of convnets focus on edges, the samples from a GAN are sharper. Therefore, our experiments provide quantitative and qualitative comparisons between the 2 loss and the Laplacian pyramid Lap 1 loss DISPLAYFORM2 where L j (x) is the j-th level of the Laplacian pyramid representation of x BID27. Therefore, the Lap 1 loss weights the details at fine scales more heavily. In order to low-frequency content such as color information, we will use a weighted combination of the Lap 1 and the 2 costs. Optimization. For any choice of differentiable generator, the objective is differentiable with respect to z, and θ. Therefore, we will learn z and θ by Stochastic Gradient Descent (SGD). The gradient of with respect to z can be obtained by backpropagating the gradients through the generator function BID4. We project each z back to the representation space Z after each update. To have noise vectors laying on the unit 2 sphere, we project z after each update by dividing its value by max(z 2, 1). We initialize the z by either sampling them from a gaussian distribution or by taking the whitened PCA of the raw image pixels. We organized our experiments as follows. First, Section 3.1 describes the generative models that we compare, along with their implementation details. Section 3.2 reviews the image datasets used in our experiments. Section 3.3 discusses the of our experiments, including the compression of datasets (Section 3.3.1), the generation (Section 3.3.2) and interpolation (Section 3.3.3) of samples, and the of arithmetic operations with noise vectors (Section 3.3.5). First, PCA BID33, equivalent to a linear autoencoder BID1, where we retain the top 256 principal components. Second, DCGAN BID36. Since GANs do not come with a mechanism (inverse generator) to retrieve the random vector g −1 (x) associated with an image x, we estimate this random vector by 1) instantiating a random vector z 0, and 2) computing updates DISPLAYFORM0 by backpropagation until convergence, where is either the 2 or the Lap 1 loss. Our experiments measuring reconstruction error are disadvantageous to GANs, since these are models that are not trained to minimize this metric. We use the Adam optimizer BID20 with the parameters from BID36 Table 1: Reconstruction errors in MSE. We consider methods using both MSE and Lap 1 loss. We also specify the initialization method between random and PCA.Third, VAE BID21. We train a VAE with the same encoder and decoder architectures as DCGAN. We train it with the default hyper-parameters for 25 epochs. Third, GLO (proposed model). We will train a GLO model where the generator follows the same architecture as the generator in DCGAN. We use Stochastic Gradient Descent (SGD) to optimize both θ and z, setting the learning rate for θ at 1 and the learning rate of z at 10. After each udpdate, the noise vectors z are projected to the unit 2 sphere. In the sequel, we initialize the random vectors of GLO using a Gaussian distribution (for the CelebA dataset) or the top d principal components (for the LSUN dataset). We evaluate all models on two datasets of natural images. Unless specified otherwise, we use the prescribed training splits to train our generative models. All the images are rescaled to have three channels, center-cropped, and normalized to pixel values in [−1, +1].First, CelebA BID29 is a set of 202, 599 portraits of celebrities. We use the aligned and cropped version, scaled to 128 × 128 pixels. Second, LSUN BID44 ) is a set of millions of images of scenes belonging to different scene categories. Following the tradition in GAN papers BID36, we use the 3, 033, 042 images belonging to the bedroom category. We resize the images to 64 × 64 pixels. We compare the methods described on Section 3.1 when applied to the datasets described on Section 3.2. In particular, we evaluate the performance of the methods in the tasks of compressing a dataset, generating new samples, performing sample interpolation, and doing sample arithmetic. We start by measuring the reconstruction error in terms of the mean-squared loss 2 (x, x) = x−x 2 2 and the Lap 1 loss Table 1 shows the reconstruction error of all models and datasets for the 2. This gives a rough idea about the coverage of each model over the dataset. Figure 1 show the quantity of the representation space explained as a function of the number of eigenvectors used to reconstruct it. GLO trained from a random initialization is more aggressive about using the full representation space to spread the information around while PCA or autoencoders tend to concentrate the information in a few directions. For completeness, we computed image reconstructions for the various models on a held-out set of images. To this end we use face images from deep funneled images from Labeled Faces in the Wild BID17. In order to make the images similar to those found in CelebA we crop the images so as to align the location of eyes. The reconstructions of a random sample of images are presented in Fig. 10. Figure 4 shows samples from the each of the models on the CelebA dataset, and Figure 5 shows the same fro the LSUN dataset. In the case of GLO, we fit a Gaussian distribution with full covariance to the representation space Z, and sample from such distribution to generate new samples. We can see that the samples are visually appealing even when placing such a simple probabilistic model on the representation space. We leave more careful modeling of Z for future work. The latent space can be explored by decomposing the covariance matrix of the latent vectors and moving along the eigenvectors associated with the largest eigenvalues from an image. The ing image transformation often contains information about attributes that varies in the dataset. Figure 8 show some examples of image deformation along the principal axes. The image in the middle is the original image. Moving in either direction along an axis produces the images on its left and its right. We see that the main axes seem to contain information about standard face attributes. For example, the 4th component seems to be capturing information about facial expression while the 9th one seems to be capturing information about the age. In absence of supervision, some directions make several attributes move simultaneously, for example smiling seems correlated with the hair color. These correlations are artifacts of the CelebA dataset distribution. In the spirit of BID36, we showcase the effect of simple arithmetic operations in the noise space of the various models. More precisely, we average the noise vector of three images of men wearing sunglasses, remove the average noise vector of three images of men not wearing sunglasses, and add the average noise vector of three images of women not wearing sunglasses. The ing image resembles a woman wearing sunglasses glasses, as shown in Figure 9. Generative Adversarial Networks. GANs were introduced by BID15, and refined in multiple recent works BID11 BID36 BID47 BID37. As described in Section 1, GANs construct a generative model of a probability distribution P by setting up an adversarial game between a generator g and a discriminator d: DISPLAYFORM0 In practice, most of the applications of GANs concern modeling distributions of natural images. In these cases, both the generator g and the discriminator d are parametrized as deep convnets BID24. Among the multiple architectural variations explored in the literature, the most prominent is the Deep Convolutional Generative Adversarial Network (DCGAN) BID36. Therefore, in this paper we will use the specification of the generator function of the DCGAN to construct the generator of GLO across all of our experiments. Autoencoders. In their simplest form, an Auto-Encoder (AE) is a pair of neural networks, formed by an encoder f: X → Z and a decoder g: Z → X. The role of an autoencoder is the compress the data {x 1, . . ., x N} into the representation {z 1, . . ., z N} using the encoder f (x i), and decompress it using the decoder g(f (x i)). Therefore, autoencoders minimize E x∼P (g(f (x)), x), where: X ×X is a simple loss function, such as the mean squared error. There is a vast literature on autoencoders, spanning three decades from their conception BID6 BID1, renaissance BID16, and recent probabilistic extensions BID43 BID21.Several works have combined GANs with AEs. For instance, BID47 replace the discriminator of a GAN by an AE, and BID42 replace the decoder of an AE by a generator of a GAN. Similar to GLO, these works suggest that the combination of standard pipelines can lead to good generative models. In this work we attempt one step further, to explore if learning a generator alone is possible. Inverting generators. Several works attempt at recovering the latent representation of an image with respect to a generator. In particular, BID28; BID48 show that it is possible to recover z from a generated sample. Similarly, BID10 show that it is possible to learn the inverse transformation of a generator. These works are similar to BID45, where the gradients of a particular feature of a convnet are back-propagated to the pixel space in order to visualize what that feature stands for. From a theoretical perspective, BID7 explore the theoretical conditions for a network to be invertible. All of these inverting efforts are instances of the pre-image problem, BID22. BID4 have recently showed that it is possible to recover from a trained generator with compressed sensing. Similar to our work, they use a 2 loss and backpropagate the gradient to the low rank distribution. However, they do not train the generator simultaneously. Jointly learning the representation and training the generator allows us to extend their findings. BID38 also use generative models to compress images.1st eigenvector 2nd eigenvector 4th eigenvector 9th eigenvector Figure 8: Illustration of the variation around principal components of the GLO latent space on the CelebA 128 × 128 dataset. The original image is in the middle and we move along a eigenvector in both directions. We illustrate this process with the first 2 components as well as some later ones. Several works have used an optimization of a latent representation for the express purpose of generating realistic images, e.g. BID34 BID32. In these works, the total loss function optimized to generate is trained separately from the optimization of the latent representation (in the former, the loss is based on a complex wavelet transform, and in the latter, on separately trained autoencoders and classification convolutional networks). In this work we train the latent representations and the generator together from scratch; and show that at test time we may sample new latent z either with simple parametric distributions or by interpolation in the latent space. Learning representations. Arguably, the problem of learning representations from data in an unsupervised manner is one of the long-standing problems in machine learning BID2 BID25. One of the earliest algorithms used to achieve is goal is Principal Component Analysis, or PCA BID33 BID19. For instance, PCA has been used to learn low-dimensional representations of human faces BID41, or to produce a hierarchy of features BID8. The nonlinear extension of PCA is an autoencoder BID1, which is in turn one of the most extended algorithms to learn low-dimensional representations from data. Similar algorithms learn low-dimensional representations of data with certain structure. For instance, in sparse coding BID0 BID30, the representation of one image is the linear combination of a very few elements from a dictionary of features. More recently, BID46 realized the capability of deep neural networks to map large collections of images to noise vectors, and exploited a similar procedure to learn visual features unsupervisedly. Similarly to us, BID3 allow the noise vectors z to move in order to better learn the mapping from images to noise vectors. The proposed GLO is the analogous to these works, in the opposite direction: learn a map from noise vectors to images. Finally, the idea of mapping between images and noise to learn generative models is a well known technique BID9 BID23 BID39 BID5.Nuisance Variables One might consider the generator parameters the variables of interest, and Z to be "nuisance variables". There is a classical literature on dealing with nuisance parameters while estimating the parameters of interest, including optimization methods as we have used BID40. In this framing, it may be better to marginalize over the nuisance variables, but for the models and data we use this is intractable. PCA VAE with Lap 1 GLO with Lap 1 Figure 9: Illustration of feature arithmetic on CelebA. We show that by taking the average hidden representation of row a, substracting the one of row b and adding the one of row c, we obtain a coherent image. We show such interpolations with PCA, VAE and GLO.Speech generation Optimizing a latent representation of a generative model has a long history in speech BID35, both for fitting single examples in the context of fitting a generative model, and in the context of speaker adaptation. The experimental presented in this work suggest that, in the image domain, we can recover many of the properties of GAN models by using convnets trained with simple reconstruction losses. While this does not invalidate the promise of GANs as generic models of uncertainty or as methods for building generative models, our suggest that, in order to more fully test the adversarial construction, research needs to move beyond images and convnets. On the other hand, practitioners who care only about generating images for a particular application, and find that the parameterized discriminator does improve their can use reconstruction losses in their model searches, alleviating some of the instability of GAN training. While the visual quality of the are promising, especially on the CelebA dataset, they are not yet to the level of the obtained by GANs on the LSUN bedrooms. This suggest several research directions: one possibility, suggested by 3, is that being able to cover the entire dataset is too onerous a task if all that is required is to generate a few nice samples. In that figure we see that GANs have trouble reconstructing randomly chosen images at the same level of fidelity as their generations. However, GANs can produce good images after a single pass through the data with SGD. In future work we hope to better understand the tension between these two observations. There are many possibilities for improving the quality of GLO samples beyond understanding the effects of coverage. For example other loss functions (e.g. a VGG metric, as in BID32), model architectures (here we stayed close to DCGAN for ease of comparison), and more sophisticated sampling methods after training the model all may improve the visual quality of the samples. There is also much work to be done in adding structure to the Z space. Because the methods here keep track of the correspondence between samples and their representatives, and because the Z space is free, we hope to be able to organize the Z in interesting ways as we train.

Are GANs successful because of adversarial training or the use of ConvNets? We show a ConvNet generator trained with a simple reconstruction loss and learnable noise vectors leads many of the desirable properties of a  GAN.

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.