title: Webly Supervised Image Classification with Self-Contained Confidence
authors: Jingkang Yang Litong Feng Weirong Chen Xiaopeng Yan Huabin Zheng Ping Luo Wayne Zhang
text: 
  
  
  
  headings
  701  
  
   
  
  
  
  ECCV-20 submission ID  
  ECCV-20 submission ID  
  
  Paper ID 
  
  
  
  
  
  Webly Supervised Image Classification with Self-Contained Confidence
  
  
  
  
  
  J. Yang et al.
  
  
  
  
  
  
  
  
  
  SenseTime Research
  
  {yangjingkang,fenglitong,chenweirong,yanxiaopeng,
  zhenghuabin,wayne.zhang}@sensetime.com
  
  Rice University, Houston, TX, USA
  
  The Chinese University of Hong Kong, Hong Kong SAR, China
  
  The University of Hong Kong, Hong Kong SAR, China
  
  
  
  
  Webly Supervised Image Classification with 
   Self-Contained Confidence
      Jingkang Yang1,2Work done during an internship at SenseTime EIG Research. Litong Feng1 Weirong Chen1,3⋆ Xiaopeng Yan1 
  
  Huabin Zheng1 Ping Luo4 Wayne Zhang10000-0002-8415-1062
  
        ;  
  ====================================================================================================================================================================================
  
  
  This paper focuses on webly supervised learning (WSL), where datasets are built by crawling samples from the Internet and directly using search queries as web labels. 
  Although WSL benefits from fast and low-cost data collection, noises in web labels hinder better performance of the image classification model. 
  To alleviate this problem, in recent works, self-label supervised loss ℒ_s is utilized together with webly supervised loss ℒ_w. ℒ_s relies on pseudo labels predicted by the model itself. 
  Since the correctness of the web label or pseudo label is usually on a case-by-case basis for each web sample, it is desirable to adjust the balance between ℒ_s and ℒ_w on sample level. 
  Inspired by the ability of Deep Neural Networks (DNNs) in confidence prediction, we introduce Self-Contained Confidence (SCC) by adapting model uncertainty for WSL setting, and use it to sample-wisely balance ℒ_s and ℒ_w. 
  Therefore, a simple yet effective WSL framework is proposed. A series of SCC-friendly regularization approaches are investigated, among which the proposed graph-enhanced mixup is the most effective method to provide high-quality confidence to enhance our framework. The proposed WSL framework has achieved the state-of-the-art results on two large-scale WSL datasets, WebVision-1000 and Food101-N. Code is available at https://github.com/bigvideoresearch/SCChttps://github.com/bigvideoresearch/SCC.
  
  
  
  
  
  § INTRODUCTION
  
  
  Large-scale human-labeled data plays a vital role in deep learning-based applications such as image classification <cit.>, scene recognition <cit.>, face recognition <cit.>, etc. However, high-quality human annotations require significant cost in labor and time. Webly supervised learning (WSL), therefore, has attracted more attention recently as a cost-effective approach for developing learning systems from abundant web data. Generally, search queries fed into image crawlers are directly used as web labels for crawled images, which also introduce label noise due to semantic ambiguity and search engine bias. How to deal with these unreliable and noisy web labels becomes a key task in WSL.
  
  
  
  A straight-forward approach of WSL is to treat web labels as ground truth and all web samples are directly used to train DNNs <cit.>. 
  Some previous methods <cit.> require additional clean subsets to learn a guidance model to judge the correctness of web labels and adopt a sample reweighting strategy for robust training of DNNs.
  CurriculumNet <cit.> avoids extra clean set by leveraging density assumption that samples from the high-density region are more reliable, and trains the model in a curriculum learning manner.
  As all the above works only use webly supervised loss ℒ_w, recent works attempt to combine self-label supervised loss ℒ_s with ℒ_w <cit.>. ℒ_s comes from the predictions of the model itself in a fashion of self-distillation <cit.> or prototype-based rectification <cit.>. 
  
  Although it is promising to utilize ℒ_s together with ℒ_w, we argue that the ratio balancing ℒ_w and ℒ_s should not be a constant across the entire dataset as in previous works <cit.>. The correctness of web labels varies on a case-by-case basis, due to various causes of real-world label noise. Motivated by this observation, we design a framework that adaptively balances ℒ_w and ℒ_s on sample level.  
  
  Inspired by the uncertainty prediction ability of DNNs <cit.>, we use DNN's prediction confidence, termed as self-contained confidence (SCC), to achieve a sample-wise balance between ℒ_w and ℒ_s. 
  Model uncertainty shows how unsure the model considers its correctness on its own prediction, which is revealed by DNN's soft label output. 
  When the model is trained with binary cross entropy (BCE) loss, the model uncertainty can be estimated independently across all categories.
  Here, we regard model uncertainty corresponding to the category of the sample's web label as SCC, reflecting the likelihood of web label correctness from the model's scope <cit.>.
  Fig. <ref> vividly shows a strong positive correlation between SCC and the correctness of web labels. This association is further confirmed by Expected Calibrated Error (ECE) plot <cit.>, who groups samples with SCC scores within an interval and calculates their average web label correctness rate using a human-annotated verification set from Food-101N <cit.>. According to Fig. <ref>, samples who lie in higher SCC intervals generally have larger probabilities of correct web labels.
  
  With SCC as an effective indicator of web label correctness, a generic SCC-based WSL framework is proposed. Intuitively, with the help of SCC, our framework enforces a webly supervised loss ℒ_w if a web label is considered reliable, and a self-label supervised loss ℒ_s otherwise. The self-label supervised loss utilizes the soft label predicted by a model pretrained on the WSL dataset as a self-supervised target. SCC, which is also extracted from the pretrained model, balances the ratio between ℒ_w and ℒ_s for each web sample. Our SCC is emphasized as `self-contained', as no extra guidance model or labeled clean dataset is needed. 
  Following the uncertainty calibration approaches <cit.>, we also investigate the relationship between statistical metrics (e.g. ECE metric) and image classification accuracy.
  
  Our contributions are summarized as follows:
  
  	
    * A generic noise-robust WSL framework that does not require a human-verified clean dataset is proposed, novelly featured by sample-level confidence from the perspective of model uncertainty.
  	
    * Based on our framework, we further design a graph-enhanced mixup method that stands out among a series of SCC-friendly regularization methods to achieve better classification performance.
      
    * We empirically conclude that under our framework, the statistical metrics of SCC are positively correlated with final classification accuracy, and self-label supervision is superior to consistency regularization for WSL tasks.
  	
    * The proposed framework achieves state-of-the-art results on two large-scale realistic WSL datasets,  WebVision-1000 and Food-101N.
  
  
  
  
  
  § RELATED WORK
  
  
  
  
   §.§ Webly Supervised Learning
  
  Learning with noisy labels can be divided into two categories of problems according to sources of label noise, i.e., synthetic or realistic. For synthetic label noise, some works estimate a noisy channel (e.g., a transition matrix) to model the label noise <cit.>. However, the designed or estimated channels might not stay effective in the real-world scenario.
  WSL lies in the realistic noisy label problem. Seminal WSL works attempted to leverage a subset of human-verified samples, referred as `clean set'. 
  MentorNet <cit.> learns a dynamic curriculum from the clean set for the sample-reweighting scheme, making the StudentNet only focus on probably correct samples. CleanNet <cit.> transfers knowledge of label noise learned from a clean set with partial categories towards all categories, and adjust sample weights accordingly to alleviate the impact of noisy labels. In contrast to `clean set' prior, CurriculumNet <cit.> assumes that samples with correct labels usually locate at high-density regions in visual feature space and designs a three-stage training strategy to train the model with data stratified by cleanness-levels.
  
  Self labeling is another solution to purify noisy labels by replacing unreliable web labels with predictions by a model. Joint Optimization <cit.> uses DNN's predictions as self labels, and Self-Learning <cit.> generates self labels by prototype voting and combines web labels and pseudo-labels using constant ratio. Compared to them, we balance self labels and web labels using sample-wise confidence, which relies on our observation that DNNs are capable of perceiving noisy labels with self-contained confidences. Self labels and confidences are unified in a single pretrained model in our approach. Table <ref> clarifies the differences between other WSL solutions and ours.
  
  
  	
  
  
   §.§ Semi-Supervised Learning
  
  Semi-supervised learning (SSL) utilizes a small fraction of labeled data and a large unlabeled data set altogether <cit.>. 
  Solutions to SSL are basically within two main categories. 
  One uses consistency regularization to ensure the model robustness by forcing networks producing identical predictions upon inputs with different augmentations, which is used in MixMatch <cit.> and UDA <cit.>. 
  Another uses pseudo-labeling in the representative methods of Billion Scale <cit.> and data distillation <cit.>, which firstly trains models on the clean labeled set and then provides pseudo-labels for unlabeled data. 
  
  The differences between WSL and SSL settings lead to key differences between our method and SSL methods. First, the self-label supervision in our method has a close connection with pseudo-labeling. However, our method utilizes all samples with both web labels and self labels, and SSL methods utilize a subset of unlabeled data with pseudo-labels only. The model for self-labeling in our method is learned from the entire noisy dataset, and the model for pseudo-labeling in SSL methods is trained on a small `clean' labeled set. Second, our self-label supervised loss has a similar form to consistency regularization. However, consistency regularization may be less powerful to correct the bias caused by label noise than cleaning the labels with self-labeling explicitly.
  Details are discussed in Sec. <ref>.
  	
  
  
   §.§ Model Uncertainty
  
  Model uncertainty refers to the level of distrust that the model considers its own prediction, which is vital for real-world applications. For classification tasks, the calculation is as simple as leveraging the highest score of the softmax output. To quantify the quality of model uncertainty, expected calibration error (ECE) is one widely used metric that claims an accurate uncertainty should align model predictions with classification accuracy <cit.>. For instance, if a network predicts a group of samples with a probability of 0.6, we expect exactly 60% samples of this group are classified correctly.
  
  Following this path, several methods were proposed to improve the quality of uncertainty. Post-hoc calibration such as temperature scaling is one family of methods, which optimizes the mapping of produced uncertainty on the verification set <cit.>. However, such data-dependent rescaling methods cannot improve confidence quality fundamentally. Some other works explored within-training strategies that can provide high-quality model uncertainty, such as label smoothing <cit.>, dropout <cit.>, mixup <cit.>, Bayesian models <cit.>, etc. AugMix <cit.> is directly designed to improve uncertainty estimates through a data augmentation approach. However, few research works utilized the model confidence to architect model training.
  
  In our work, model uncertainty is adapted for web label confidence estimation. Instead of using the maximum of the model's output probabilities, we pick the value on the exact web label from the probability distribution, which estimates the correctness of the sample's web label.[Web label confidence and self-contained confidence are used interchangeably throughout the paper.] Metrics such as ECE can also be adapted, i.e., web label confidence is considered well-calibrated if a model predicts all samples in a group with web label confidences of 0.6, 60% samples in this group have correct web labels. Being aware that the extraction of web label confidence requires the probability of each class to be calculated independently, binary cross-entropy (BCE) rather than softmax cross-entropy loss is used for training the network. 
  
  
  
  
  
  § PROPOSED METHOD
  
  In this section, after a formal description of WSL task, we introduce two loss functions and the proposed framework with highlighted SCC. Our framework is compatible with various regularization methods. Especially, we propose graph-based aggregation (GBA) to enhance SCC for network training. The diagram of our framework is shown in Fig. <ref>.
  	
  
  	
  
  
   §.§ Webly Supervised Learning: Problem Statement and Notations
  
  
  Webly Supervised Learning (WSL) aims at training an optimal deep neural network ℳ_θ from a dataset 𝒟={(x_1, y^*_1),…,(x_N, y^*_N)} collected from the Internet.
  x_i denotes the i-th sample in the dataset, and the one-hot web label y^*_i is the one-hot encoding of the web label ω_i (referring to ω_i-th category). The web label ω_i is obtained from the search query of crawling the image x_i. Consider the massive noise in retrieved images from a search engine, ω_i or y_i^* might not reflect the correct category that x_i belongs to. Therefore, suppressing the noise in unreliable web labels becomes the main challenge in WSL.
  
  For convenience, we use symbols x, y^*, ω directly to represent an arbitrary sample, its one-hot web label and its web label, respectively. For the multi-label problem, y_(j) denotes sample's label on j-th class.
  p(y|x,θ) denotes the label prediction of sample x by the model ℳ_θ.
  	
  
  
   §.§ Webly Supervised Loss and Self-Label Supervised Loss
  
  
  Webly supervised loss and self-label supervised loss are two widely adopted loss functions in WSL <cit.>. 
  Webly supervised loss utilizes web labels as supervision information, and self-label supervised loss <cit.> utilizes predictions of a pretrained model instead. Formally, we define them as follows.
  
  For webly supervised loss, given x' augmented from x with web label ω, the loss function can be expressed as
  
      ℒ_w=-[log(p(y_(ω)|x',θ))
      	+∑_j∈𝒮\ωlog(1-p(y_(j)|x',θ))].
  
  Notice that webly supervised loss is in the form of binary cross-entropy (BCE) loss, because a webly-crawled image probably has multi-label semantics.
  
  For self-label supervised loss, we use the prediction of the pretrained model ℳ_θ_0, which is trained directly on the original web label dataset. As the predictions on samples will be used for finetuning ℳ_θ_0 itself, We call them self labels.  Therefore, with self label p(y|x,θ_0) from model ℳ_θ_0, the self-label supervised loss is 
  
      ℒ_s = -∑_j∈𝒮[p(y_(j)|x,θ_0)log(p(y_(j)|x',θ)) 
      + (1-p(y_(j)|x,θ_0))log(1-p(y_(j)|x',θ))
      ],
  
  where y_(j) represents the prediction for j-th class in label set 𝒮 for multi-class classification problem.
  
  A similar loss to self-label supervised loss is consistency loss <cit.>, which provides an auxiliary regularization by enforcing a model to output similar predictions on different augmented counterparts of the same image. Consistency loss is proven to be effective on a large number of unlabeled images for semi-supervised learning. In WSL, however, as the quality of self labels can be guaranteed by feeding a pretrained model with weak augmented images, we found that the high-quality self-supervised loss is more effective than auxiliary consistency loss. An experimental comparison will be shown in Sec. <ref>.
  
  
  
   §.§ Self-Contained Confidence
  
  
  It is desirable to adaptively balance webly supervised loss and self-label supervised loss on sample level. Intuitively, we should trust webly supervised loss more on samples with reliable web labels, while self-label supervised loss would dominate the total loss confronting incorrect web labels. 
  
  In our method, model ℳ_θ_0 provides only self labels, but also the reliability of web labels. Notice that with BCE loss, model ℳ_θ_0 predicts the probability that x belongs to class i as p(y_(i)|x,θ_0). Specially, we focus on the model prediction on the one-hot web label y^* whose category index is ω, denoted as p(y_(ω)|x, θ_0). 
  Therefore, the only trainable parameter θ would be updated by minimizing the final loss 
  
      ℒ=c×ℒ_w+(1-c)×ℒ_s,   where  c=p(y_(ω)|x, θ_0).
  
  The confidence c is named as self-contained confidence (SCC), as it is self contained in the pretrained model and requires no extra data or knowledge.
  
  
  
   §.§ Graph-Based Aggregation
  
  
  A key component in the proposed method is the pretrained model ℳ_θ_0 for estimating both SCC and self labels. As the model is trained on noisy web labels, we employ
  mixup <cit.>, which is known as an effective regularization to make DNNs less prone to over-confident predictions and predicted scores of DNNs better calibrated to the actual confidence of a correct prediction <cit.>.
  
  In addition, we propose a graph-based aggregation (GBA) method to further boost the confidence quality and classification performance. GBA does a smoothing operation on a visual similarity graph spanned by image features. By viewing every image as a node, a k-nearest-neighbor (k-NN) graph is firstly constructed based on features located before fc layer of pretrained model ℳ_θ_0. Cosine similarity of features is computed across every pair in the neighborhood as edge weight. Hereby, an undirected k-NN graph with weighted adjacent matrix 𝐀 is obtained. Let 𝐏 denote a matrix of self labels, and the corrected self labels after GBA are denoted as
  
      𝐏̂=𝐃^-1/2(λ𝐈+𝐀)   𝐃^-1/2𝐏,
  
  where D(i,i)=λ+∑_j=1^NA(i,j). λ controls the portion of original self labels in the post-GBA self labels. SCC will also be extracted from 𝐏̂.
  GBA is a post-processing step with graph filtering <cit.> and complementary to other methods such as mixup. We evaluate several potential methods and conclude mixup + GBA leads to the optimal performance in Sec. <ref>. 
  	
  
  
  
  § EXPERIMENTS
  
  In this section, we firstly introduce three public WSL datasets. Then, we investigate several SCC-friendly methods, among which GBA-enhanced mixup stands out as the best one in both statistical metrics and classification accuracy. More ablation studies demonstrate the effectiveness of both sample-wise adaptive loss and self-label supervision. Finally, we show that the proposed method reaches the state-of-the-art on the public WSL datasets. We leave the exploration of robustness of our framework and formal algorithm in the Appendix.
  
  
  
   §.§ Datasets and Configurations
  
  
  WebVision-1000 <cit.> contains 2.4M noisy-labeled training images crawled from Flickr and Google, with keywords from 1000 class-labels in ILSVRC-2012 <cit.>.
  The estimated web label accuracy is 48% <cit.>. 
  The ILSVRC-2012 validation set is also utilized along with WebVision-1000's own validation set.
  
  WebVision-500 is a quarter-sized version of WebVision-1000 for evaluation and ablation study in low cost without losing generalization. We randomly sample one-half categories with one-half samples in the training set, and keep the full validation set of the selected 500 categories. This dataset is used for our ablation study in Sec. <ref>, <ref>, <ref>. 
  
  Food-101N <cit.> is another web dataset with 310k images classified into 101 food categories. Images are crawled from Google, Yelp, etc. We evaluate our model on the test set of Food-101 <cit.>, Food-101N's clean dataset counterpart. 60k human verification labels are provided, indicating the correctness of web labels. The estimated label accuracy is around 80%.
  
  Configuration details.
  ResNet50 is selected as our CNN model in all experiments <cit.>. 
  For more efficient training on WebVision, a minor-revised ResNet50-D is utilized <cit.>. Food101N uses standard ResNet50 for a fair comparison.
  We use the following settings that completely refer to <cit.>.
  
  Batch size is set as 256 and mini-batch size as 32. We use the standard SGD with the momentum of 0.9 and weight decay of 10^-4. A warm-start linearly reaches the initial learning rate (LR) in the first 10 epochs. The remained epochs are ruled by a cosine learning rate scheduler. A simple class reweighting is performed to deal with class imbalance. The initial LR is 0.1 with total L epochs for pretrained models. The main model has initial LR of 0.05 with identical epoch numbers. L=120 for WebVision-500 and Food101N, L=150 for WebVision-1000.
  	
  
  
   §.§ Exploring Optimal Regularization Method
  
  
  In this section, we experiment with seven different confidence-friendly regularization methods for ℳ_θ under our framework. We conclude that GBA-enhanced mixup (mixup+GBA) is the most efficient one for the best performance. However, as the main contribution of our work is the simple yet effective noise-robust pipeline with SCC, regularization is not a necessary part of our model.
  
  
      
  
  Besides the standard setting with BCE loss, which is denoted as `Vanilla', we introduce the following  regularization methods for model ℳ_θ.
  
  Label Smoothing prevents over-confidence problems by adding a small value of ϵ on the zero-values in one-hot encoding labels <cit.>. We use ϵ=0.1.
  
  Entropy Regularizer discourages over-confident model prediction by adding a penalizing term to standard loss functions <cit.>. Regularizer weight is set as 0.1.
  
  MC Dropout is selected as the representation of Bayesian methods. It approximates Bayesian inference by randomness in dropout operation <cit.>. Dropout rate p is set 0.5. When testing, we infer 50 times and average the predictions.
  
  Mixup is a simple but effective pre-processing method that convexly combines every pair of two sampled images and labels <cit.>. 
  <cit.> proves its strong uncertainty calibration capability beyond its label smoothing effects.
  
  AugMix is another data augmentation method with consistency loss, which produces well-calibrated model uncertainty <cit.>.
  
  Ensemble utilizes several models with identical tasks to boost the ultimate performance <cit.>. With E vanilla models with different random initializations, we average their predictions on every sample.
  
  Graph-based Aggregation (GBA) is introduced in Sec. <ref>. We use k=10 and λ=0.5 as hyper-parameters.
  	
  
  
  Result Analysis.
  Table <ref> reports the results. S1 is short for the pretraining stage for ℳ_θ_0, S2 for the finetuning stage using our framework. Generally, good performance in S1 favors S2. Mixup and Ensemble are the two most effective regularizers. As mentioned in Sec. <ref>, Mixup smooths discriminative spaces and ensemble averages models’ biases. The advantages of these two methods are combined in GBA design, as a graph smoothing operator for neighbor predictions, which is proven effective empirically. Improvement from GBA is weaker on mixup compared to vanilla since mixup offers the same effect of smoothing space with GBA. However, mixup+GBA still reaches the optimal result besides the costly ensemble method.
  
  
  
   §.§ Understanding Self-Contained Confidence
  
  
  As SCC plays a critical role in our framework, we explore an interesting question: 
  how great the SCC quality affects the final accuracy reported in the previous section? We also show the relationship between three statistical metrics adapted from uncertainty theories and our accuracy-based metric. 
  
  For statistical metrics, we manually create a verification set 𝒱={v_1, …, v_n} for WebVision-500 by annotating whether the web label is correct on n=12500 samples, with 50 randomly sampled cases from 250 random classes. 
  
  
  
  To evaluate the quality of SCC, The following metrics are utilized.
  
  Second-stage Accuracy on Vanilla (SAV). 
  To empirically evaluate different SCCs, we use an identical vanilla pretrained model for self-labeling and finetuning under our framework. Therefore, the accuracy of second-stage finetuned model is only determined by the quality of SCC. Note that the models for producing SCC are different and with different regularization methods.
  	
  Mean Square Error (MSE). Verification set 𝒱 can be considered as a set of ground-truth confidence since it values 1 with the correct web label and values 0 when incorrect. Thus, MSE estimates the squared difference between the given confidence and the ground-truth, which is defined as 
  
      MSE = 1/n∑_i=1^n(v_i - c_i)^2.
  
  
  Expected Calibration Error (ECE). Calibration error is originally used to evaluate the model interpretability on their predictions <cit.>, while we slightly adapt it for confidence quality evaluation. 
  Formally, in the verification set 𝒱, for all samples whose confidences fall into (m-1/M,m/M] form the m-th bin, where average confidence conf(B_m)=1/|B_m|∑_i∈ B_mc_i and the average web-label reliability rel(B_m)=1/|B_m|∑_i∈ B_mv_i are calculated. Thus, ECE is defined as
  
      ECE = ∑_m=1^M|B_m|/n|rel(B_m) - conf(B_m)|.
  
  
  
  
  
  Over-Confidence Error (OCE). Samples with high SCC but incorrect web labels are especially harmful to our framework, since introducing the wrong web label is much worse than using self labels. OCE evaluates the level of over-confidence by punishing more on higher-confident bins with low reliability, defined as
  
      OCE = ∑_m=1^M|B_m|/n[conf(B_m)×max{conf(B_m)-rel(B_m), 0}].
  
  In this work, we calculate ECE and OCE with M=100. For visualization in Fig. <ref>, we use M=10. Fig.<ref> uses M=100.
  
  Result Analysis.
  Best metric performance is reached by either mixup or Vanilla+GBA, while the ensemble also produces a good result. Fig. <ref> visualizes ECE diagrams, where GBA and mixup look more calibrated than any other model. A similar result is shown in Tabel <ref> Column 2-4. Column 5-8 presents the metric of SAV which shows GBA provides good quality confidence that favors our proposed framework. According to our exploration of SCC, we conclude the following insights:
  (1) SCC can reflect the reliability of the web label according to Fig. <ref>;
  (2) SCC plays a key role in our pipeline through adaptively balancing two losses on the sample level since empirical metric SAV is generally proportional to the statistical metric ECE.
      
  
  
   §.§ Ablation Study
  
  
  
  
  
  
  On Self-Contained Confidence.
  To show the necessity of sample-wise SCC, we follow the settings of Table <ref> and replace SCC with constant confidence values.
  Fig. <ref> shows that any constant confidence is unable to surpass 77.17% WebVision Top-1 accuracy reached by Vanilla+GBA (marked as dashed line).
  
  On Self-Label Supervised Loss.
  We demonstrate the superiority of self-label supervised loss over consistency loss <cit.>. 
  Consistency loss is trained in an end-to-end fashion since it does not require a pretrained model ℳ_θ_0, 
  whereas our self-label supervised loss expects a two-stage approach with static self labels and SCC.
  For fairness, we make comparisons using the same backbone with mixup regularization. Fig. <ref> shows the model with our loss reaches better performance than consistency loss. An interesting observation is that a performance drop exists at the beginning of S2 in our method. 
  Since S1 is trained with web labels, the model may memorize label noise and result in suboptimal performance. Thus, a large LR is required to destruct the noise-affected S1 model, causing a sudden performance drop with S2. Such a two-stage approach is adopted, because we find the end-to-end approach unsuitable for our method: in the early stage, inaccurate pseudo labels and SCC mislead the model, and in the late stage, the model finally obtains reliable SCC, however, small LR cannot correct the accumulated errors.
  
  
  
  
  
  
   §.§ Real-world Experiments
  
  
  
  WebVision-1000.
  Table <ref> reports experiments on WebVision-1000 using both vanilla and mixup models.
  With the vanilla model, using our pipeline, a 0.3% improvement is achieved for WebVision top-1 accuracy, and 0.7% increase on ImageNet top-1/5. 
  GBA can further improve the performance of every metric.
  When enabling mixup operation (α=0.2), although on top of a high-accuracy pretrained model, our method can still improve both ImageNet top-1 and top-5 accuracy by 1.9%. The WebVision top-5 accuracy is improved by 0.7%. The WebVision top-1 accuracy is improved a little. More improvements on ImageNet prove a good generalization ability of the proposed method. The larger improvement than vanilla may attribute to the higher SCC quality achieved by mixup. GBA advances an average 0.3% extra improvement on every metric.
  
  We also show the superiority of our method over state-of-the-art methods. Note that both MentorNet <cit.> and CleanNet  <cit.> use extra human-verified datasets to train a guidance network first, and MentorNet <cit.> chooses a backbone of InceptionResNetV2 stronger than ResNet50-D. Multimodal image classification <cit.> uses ImageNet data for training visual embedding and a query-image pairs dataset for training phrase generation. Stronger InceptionV3 is also selected as the backbone. Although with these disadvantages, our ResNet50-D still works the best among all.
  
  Food-101N.
  According to Table <ref>, we significantly advance the state-of-the-art model without any usage of human annotations.
  For vanilla model, the second stage of our method pushes 0.8% higher accuracy than the first stage, and the usage of GBA even double the improvement.
  For the mixup model (α=0.5), the second stage increases a higher 1.4% accuracy as mixup provides better SCC and self labels than vanilla, but the advance of GBA is deducted due to the overlapping effects of mixup and GBA.
  Rather than our normally used ResNet50-D, we use standard ResNet50 here for fair comparisons with others. While all the other methods (except <cit.>) train Food-101N from ImageNet pretrained model, we train our model from scratch and still reach optimal performance.
  
  
  
  
  
  
  § CONCLUSION
  
  We propose a generic noise-robust framework featured by sample-level confidence balancing webly supervised loss and self-label supervised loss. Our framework is compatible with model regularization methods, among which our proposed mixup+GBA is the most effective. 
  
  Here we recall two main takeaway messages from our extensive experiments:
  (1) Reliability of the web label can be reflected by SCC (ref. Fig.<ref>), and empirical metric SAV is generally proportional to the statistical metrics like ECE (ref. Table <ref>).
  (2) Our framework is in favor of high-quality confidence provided by the pretrained model, and mixup and ensemble are the two most effective regularizers (ref. Fig.<ref>&<ref>). Considering that mixup smooths discriminative spaces and ensemble averages models’ biases, both advantages are combined in GBA design, as a graph smoothing operator for neighbor predictions.
  
  We also leave a valuable discussion in the appendix for readers of interests, which basically shows: although the performance is largely dependent on the quality of SCC, the framework still works on Food101N even with a weak DNN backbone.
  
  Acknowledgement. 
  The work described in this paper was partially supported by Innovation and Technology Commission of the Hong Kong Special Administrative Region, China (Enterprise Support Scheme under the Innovation and Technology Fund B/E030/18).
  	
  
  
  
  
  
  splncs04
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  	
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  