Matrix Smoothing: A Regularization for DNN with Transition Matrix under Noisy Labels
Xianbin Lv Dongxian Wu Shu-Tao Xia
cs.CV cs.LG


𝐱
ŁL



Xianbin Lv^1, 2, ∗, Dongxian Wu^1, 2, ∗, Shu-Tao Xia^1, 2,
^∗Equal contribution.
^Corresponding author: Shu-Tao Xia

This work is supported in part by the National Key Research and Development Program of China under Grant 2018YFB1800204, the National Natural Science Foundation of China under Grant 61771273, the R&D Program of Shenzhen under Grant JCYJ20180508152204044, and the research fund of PCL Future Regional Network Facilities for Large-scale Experiments and Applications (PCL2018KP001).

^1Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China

^2PCL Research Center of Networks and Communications, Peng Cheng Laboratory, Shenzhen, China


lv.xianbin@foxmail.com; wu-dx16@mails.tsinghua.edu.cn; xiast@sz.tsinghua.edu.cn

Matrix Smoothing: A Regularization for DNN with Transition Matrix 
under Noisy Labels
    Jerónimo R. Maze
    December 30, 2023
=====================================================================================




Training deep neural networks (DNNs) in the presence of noisy labels is an important and challenging task. Probabilistic modeling, which consists of a classifier and a transition matrix, depicts the transformation from true labels to noisy labels and is a promising approach. However, recent probabilistic methods directly apply transition matrix to DNN, neglect DNN's susceptibility to overfitting, and achieve unsatisfactory performance, especially under the uniform noise. In this paper, inspired by label smoothing, we proposed a novel method, in which a smoothed transition matrix is used for updating DNN, to restrict the overfitting of DNN in probabilistic modeling. Our method is termed Matrix Smoothing. We also empirically demonstrate that our method not only improves the robustness of probabilistic modeling significantly, but also even obtains a better estimation of the transition matrix.



noisy labels, deep learning, robustness





§ INTRODUCTION


Deep neural networks (DNNs) have achieved great success on many tasks, e.g.  speech recognition <cit.>, computer vision <cit.> and natural language processing <cit.>. To obtain the satisfactory performance, DNNs in supervised learning rely on massive training samples with high-quality annotations such as ImageNet <cit.>. However, labeling large-scale datasets is a costly and error-prone procedure. Therefore, training DNNs in the presence of noisy labels has become a task of great importance in practice.




Recently, several studies <cit.> focused on the dynamics of DNN learning with noisy labels. Zhang et al. <cit.> argued that DNNs first memorize the training data with clean labels and subsequently memorize data with noisy labels, which leads to severe overfitting. To alleviate this problem, numerous approaches <cit.> have been proposed to train DNNs with noisy labels.
Since depicting the transformation from true labels to noisy labels, probabilistic modeling is a promising method in robust training <cit.> and crowdsourcing <cit.>.
The typical probabilistic modeling consists of a classifier,  which predicts a probability distribution over a set of classes, and a transition matrix, which is usually unknown and requires estimation.
Patrini et al. <cit.> proposed to estimate the transition matrix through a DNN which is trained under label noise. Bekker and Goldberger <cit.> derived a learning scheme based on the EM algorithm, which estimated the matrix in every M-step. Goldberger and Ben-Reuven <cit.> introduced another softmax layer to the network, so as to learn the matrix and the model simultaneously. Sukhbaatar et al. <cit.> emphasized the necessity to add a regularization to the ill-posed matrix estimation. In conclusion, recently published studies only focus on the estimation of the unknown transition matrix, and just regard DNN as the classifier in modeling.



However, DNNs are prone to overfit noisy labels, and probabilistic modeling only resolves this problem partially. For example, even if the accurate transition matrix is given, DNN still suffers from severe overfitting as shown in Figure <ref>(green curve), especially after learning rate decays. Unlike the previous research, this paper pays attention to a better method to update parameters of DNN in probabilistic modeling.
Inspired by label smoothing <cit.> in standard DNN training, we propose a novel technique called Matrix Smoothing. It substitutes the original transition matrix with a smoothed one for updating DNN, and keeps the matrix estimation method unchanged. Further, we provide a label-correction perspective to understand the mechanism of matrix smoothing. Finally, comprehensive experiments are conducted to demonstrate the efficiency of our method in different variants of probabilistic modeling.




§ MATRIX SMOOTHING






 §.§ Preliminaries

Assume we want to train a multi-class DNN p(y |x, w) where x is the input vector, w is the parameter-set of DNN and y is the clean label that we cannot directly observe. Instead, only noisy labels are observed and denoted by ỹ. Given the transformation from true labels to noisy labels, the probability of the noisy label ỹ is

    p(ỹ|x, w) = ∑_i=1^c p(ỹ| y=i,x)p(y=i |x, w),

where c is the number of classes. In a common assumption, the noise is conditionally independent of inputs given the true labels so that

    p(ỹ=j | y=i, x) = p(ỹ=j | y=i) = T_ij,

where T is the transition matrix for noise modeling. In general, this label noise is defined to be class dependent. The noise is uniform with rate η, if T_ij = 1 - η for i = j, and T_ij = η/c-1 for i ≠ j.

In practice, we are only given the inputs x_1, ⋯, x_N and their corresponding noisy labels ỹ_1, ⋯, ỹ_n. During training, the loss for DNN combined with transition matrix T is 

    ℒ_T = 1/N∑_n=1^N-log∑_i^c p(ỹ| y=i) p(y=i|x_n,w).

The model is illustrated in Figure<ref>. After training, DNN becomes modestly robust to noisy labels as shown in Figure <ref>. However, this method still achieves unsatisfactory performance in the clean test set, especially under training with the uniform noise.







 §.§ Label Smoothing


Label smoothing has been widely utilized in many tasks, including image classification, language translation, and speech recognition <cit.>. It replaces the hard targets with soft targets that are a weighted average of the hard targets and the uniform distribution over labels as illustrated in Figure <ref>, and improves generalization and learning speed of DNNs. Recently, Jenni demonstrated label smoothing also slightly improves robustness of DNNs to noisy labels<cit.>. Unfortunately, it is incompatible with probabilistic modeling and degrades the performance on the clean test set.




 §.§ Our Method




Inspired by label smoothing in standard DNN training, we introduce a novel technique to the probabilistic modeling. Assuming the true transition matrix T is given, we always update parameters of DNN through T directly in the conventional probabilistic modeling. Instead, we substitute the original transition matrix T with a smoothed one S, that is,

    S_ij = T_ij^β/∑_l T_il^β,

where β∈ (0, 1]. Figure <ref> exhibits an example of matrix smoothing with a 3 × 3 matrix and β=0.5.




  §.§.§ The Smoothing Methods


Actually, there are several methods to smooth the matrix. For example, we could utilize a weighted average of the original transition matrix and a special matrix whose all elements are same, that is,


    S = γ T + (1 - γ) 1/c1,

where γ∈ (0, 1] and 1 is a matrix whose elements are all equal to 1. This method is similar to label smoothing. Since it is a linear weighted average of the original matrix and a matrix of the uniform distribution, we denote it as “the linear method”. However, if there are zero elements in the transition matrix, e.g. the first row in Figure <ref>, the second smoothing method introduces incorrect information and misleads DNN to believe the impossible noise. By contrast, our method does not change elements in the matrix if they are equal to zero.

Another method is to introduce a hyper-parameter γ and the smoothed matrix S becomes

    S_ij = exp( b_ij / γ)/∑_l exp(b_il / γ),

where γ∈ [1, ∞) and b_ij = log(T_ij). This formula is related to model distilling <cit.>, and  γ is the hyper-parameter of the temperature. And we call this “the temperature method”. However, this method cannot deal with the situation in which any element is zero, since log 0 = - ∞.

These methods have the same effect for training DNN with a suitable selection of hyper-parameters under the uniform noise. We will compare these three smoothing methods under the asymmetric noise in Section <ref>.



  §.§.§ Combination with Matrix Estimation


However, the transition matrix is not always accessible in practice. An accurate estimation of the transition matrix is of significance in realistic scenarios. There are several proposed estimation methods including direct estimation and joint learning with DNN. Fortunately, it is easy to combine matrix smoothing with these matrix estimation methods.

Patrini et al. <cit.> proposed to train DNN in a standard manner with noisy labels, estimate the transition matrix from the trained DNN and train another DNN with the estimated transition matrix. Since the transition matrix is fixed in the second round of training, we could apply matrix smoothing directly. Goldberger and Ben-Reuven <cit.> introduced another softmax layer to the network for the transition matrix, so as to learn the matrix and DNN simultaneously. Sukhbaatar <cit.> has a similar training strategy. For these joint learning strategies, we could decouple the learning steps: first, we update DNN with matrix smoothing when the matrix is fixed; second, we utilize the original method to update the matrix. In conclusion, matrix smoothing is compatible with these matrix estimation methods.





§ UNDERSTANDING MATRIX SMOOTHING



 §.§ A Label-Correction Perspective

Matrix smoothing is a simple yet effective method to restrict DNN from overfitting in probabilistic modeling. Here, we provide an explanation of its mechanism. To explore the mechanism of matrix smoothing, it is convenient to rewrite the probabilistic modeling into a label-correction form. Here, we consider the loss of one sample, i.e. (x,ỹ = j), and the result is

    ∂/∂w( - log∑_i=1^c T_ij p(y=i |x, w) ) 
    
    =    - ∑_i=1^c T_ij/∑_k T_kj p(y=k|x, w)∂/∂wp(y=i|x, w)
    
    =    -∑_i=1^c T_ij p(y=i|x, w)/∑_k T_kjp(y=k|x, w)∂/∂wlog p(y=i|x, w)
    
    =    -∑_i=1^c p(y=i|ỹ=j,x, w) ∂/∂wlog p(y=i|x, w),

where

    p(y=i|ỹ=j,x, w)    = p(y=i, ỹ=j |x, w)/p(ỹ=j |x, w)
       = T_ij p(y=i|x, w)/∑_k T_kjp(y=k|x, w)

is the posteriori probability for the clean label, in which a noisy label is given and the output probability of DNN is regarded as the priori probability. Training DNN in probabilistic modeling with noisy labels is equivalent to training standard DNN with modified labels as Eqn.(<ref>)




 §.§ The Mechanism of Matrix Smoothing


For simplicity, we only consider the situation with the uniform noise[The situation with the uniform noise is more important than that with the asymmetric noise since the former is more difficult and the probabilistic modeling performs poorly with the uniform noise.]. The gradient of the loss with matrix smoothing has a similar formula as Eqn.(<ref>), except the posteriori item, which becomes,

    q(y |ỹ,x, w) = α p(y, ỹ|x, w) + (1 - α) 1/c p(y |x,w)/α p(ỹ|x, w) + (1 - α)1/c,

where

    α = (1-η)^β/(1-η)^β + (c-1)(η/c-1)^β - 1/c/1 - η - 1/c.

When β=0, we have α = 0, which makes modified labels become the output probability of DNN. If β=1, we have α=1 and modified labels degenerate to the posteriori probability as Eqn.(<ref>). When 0<α<1, modified labels are a combination of output probability and the original posterior probability. Since DNN behaves robustly to noisy labels and achieves satisfactory performance at the early stage during training as indicated in <cit.>, it is reasonable to rely more on its outputs at the early training stage.



§ EXPERIMENT



In this section, we comprehensively verify the effectiveness of our proposed method, Matrix Smoothing, on synthetic noisy datasets.



 §.§ Detailed Comparison of Smoothing Methods




  §.§.§ Comparison with Label Smoothing



We first demonstrate the effectiveness of matrix smoothing, in comparison with another regularization technique, label smoothing<cit.>. As indicated in <cit.>, poor matrix estimation degrades robustness. To eliminate the influence of estimation and just study the performance limits, we assume that the transition matrix T is given here.

Datasets.
We conduct experiments on noisy CIFAR-10. 
Noisy labels are generated by introducing the uniform noise, in which the label of a given training sample is flipped uniformly to one of the other classes with probability η. The uniform noise has been verified to be more challenging than the asymmetric noise as demonstrated in <cit.>

Baselines. 
Forward method <cit.> with a given transition matrix T is regarded as the basic model. We compare the robustness of the original Forward method(FD), Forward with label smoothing<cit.> (FD+LS) and Forward with the proposed Matrix Smoothing (FD+MS).

Experimental setup.
We utilize PreActResNet18<cit.> as the basic classifier in probabilistic modeling, which is a powerful DNN with accuracy 94.18%(74.41%) on the test set after trained on clean CIFAR-10(CIFAR-100). We use a batch size of 128, a weight decay of 10^-4, and SGD with the momentum of 0.9. We train the network until 150 epochs with initial learning rate of 0.1, and decrease the learning rate by dividing it by 10 after 80 and 120 epochs.

As shown in Table <ref>, even though the accurate transition matrix is given, Forward still suffers severe overfitting, especially in a large noise rate. The label smoothing is incompatible with probabilistic modeling based on the transition matrix, despite the fact that it improves robustness slightly in standard DNN training with noisy labels<cit.>. The proposed matrix smoothing improves the robustness of Forward remarkably by 5% ∼ 20%. This result indicates the effectiveness of our method as a regularization for training DNN with a transition matrix.







  §.§.§ Comparison with Other Matrix Smoothing Methods


Since there are several methods to smooth matrix as shown in Section <ref>, we compare these methods under the asymmetric noise[These methods have the same effect under the uniform label noise with suitable hyper-parameters. Thus, we compare them under the asymmetric noise instead.] here.

Datasets & Experimental setup & Baselines. We compare these methods on CIFAR-10 under the asymmetric noise and use the same experimental setup (architecture, prepossessing, training setting) as Section <ref>. Forward method(FD) is chosen as the baseline. For fairness, we select the best hyper-parameters for every method: for our chosen method(Ours), the parameter β is 0.5; for “the linear method”(L) that uses all-one matrix to smooth transition matrix, the parameter γ is 0.8; for “the temperature method”(T), the parameter of the temperature γ is 1.1.

The results are demonstrated in Table <ref>. Our method achieves the best performance under all the asymmetric noise rates. Although the hyper-parameters have been selected, the other methods is still unsatisfactory, which is compatible with the discussion in Section <ref>, i.e., the other methods would introduce incorrect information for probabilistic modeling.



 §.§ Matrix Smoothing with An Unknown Matrix


The transition matrix is not always accessible in practice, and the estimation of the transition matrix is important in realistic scenarios. In this section, we combine several matrix estimation methods with matrix smoothing, to demonstrate that our method not only prevents DNN from overfitting, but also improves the estimation quality. 

Datasets & Experimental setup. For comprehensive comparison, we utilize CIFAR-10 and CIFAR-100 with the uniform noise. We use the same experimental setup (architecture, prepossessing, training setting) as Section <ref>.







Baselines.
Two different estimation methods for T are chosen as our baselines. (1) Estimation from a pre-trained model: In <cit.>, a standard DNN is trained on noisy dataset, and transition matrix is estimated from some “perfect samples”. We denote this as FD_est.(2) Adaption Layer (AL)<cit.>: The transition matrix T is regarded as an extra fully connected layer and train with DNN simultaneously. We initialize T using an identity matrix[For numerical stability, the actual T is an identity matrix added by an extremely small uniform noise.] as shown in <cit.>. In the first 15 epochs, we freeze the transition matrix and only update DNN as a warmup. Then we update the matrix and DNN simultaneously. To illustrate the superiority of Forward combined with Matrix Smoothing, Cross Entropy(CE) and Generalized Cross Entropy(GCE) Loss are included as extra baselines. We exclude FD_est on CIFAR-100, since it has bad performance.
 
Hyper-parameter selection.
We select suitable β of matrix smoothing for different methods. First, we split the original noisy training set (50k images in CIFAR-10/CIFAR-100) into a new training set(40k) and a validation set(10k). We train models in 40k training set and compare hyper-parameter in the noisy 10k validation set. As the hyper-parameter is fixed, we retrain models in the total training set(50k) from scratch. As a result, β=0.5 for FD+MS, β=0.8 for AL+MS, and β=0.1 for FD_est+MS.




  §.§.§ Robustness against Noisy Labels


As demonstrated in Table <ref>, unsatisfying estimation of T degrades the performance of probabilistic modeling by 5% ∼ 23% (comparing FD_est/AL with FD), which restricts its practical application. Further, even though T is known, the robustness of Forward is still limited in comparison with GCE, a state-of-art robust loss. Fortunately, matrix smoothing solves these problems effectively. FD+MS has similar or even better results than GCE in CIFAR-10, and FD_est+MS and AL+MS have similar performance to FD+MS in CIFAR-10. As shown in Table <ref>, GCE has best performance on CIFAR-100 since it was proposed given the situation with the uniform noise. FD is worse than GCE significantly. Combined with MS, FD has much better performance and its results are close to GCE, which also indicates the effectiveness of MS in CIFAR-100. In conclusion, the improvements brought by matrix smoothing in probabilistic modeling is significant and consistent, no matter the matrix is known or estimated.



  §.§.§ Estimation of Transition Matrix


In Adaption Layer, transition matrix and DNN are updated simultaneously, and their qualities affect the updating of each other. Previously published studies only focused on the impact on DNN from matrix estimation and introduced regularization to matrix learning. Instead, our method keeps matrix learning unchanged and utilizes the smoothed matrix when updating DNN. In this section, we demonstrate the matrix estimation has satisfying performance as long as DNN is trained of high quality. 

To demonstrate the estimation quality of AL and AL+MS, we visualize the elements in the main diagonal of the learned transition matrix with 0.4 uniform noise in Figure <ref>. The true value of these elements is 0.6. For original AL, the model has unsatisfactory performance as the learned transition matrix tends to identity matrix and DNN overfits to noisy labels severely. By contrast, elements in main diagonal converge to the true value (0.6) in ours(AL+MS), which indicates that we could obtain a satisfied estimation of transition matrix just through updating DNN in a better way.




§ CONCLUSION


In this paper, we demonstrate that DNN with a transition matrix in probabilistic modeling still suffers from severe overfitting. Inspired by label smoothing, we propose a new method called Matrix Smoothing, which replaces the original matrix with a smoothed one when updating parameters of DNN. Further, we provide a label-correction perspective for probabilistic modeling to explain the mechanism of matrix smoothing, and verify its effectiveness through comprehensive experiments. Our results in this paper remind researchers to consider not only the estimation of transition matrix in probabilistic modeling, but also the parameter updating for DNN.

IEEEbib


