Robust Loss Functions under Label Noise for Deep Neural Networks
Aritra Ghosh Himanshu Kumar P. S. Sastry
stat.ML cs.LG

Decay mechanisms of superflow of Bose-Einstein condensates in ring traps
    Ippei Danshita
    December 30, 2023
========================================================================


In many applications of classifier learning, training data suffers from label noise. Deep networks are learned using huge training data where the problem of noisy labels is particularly relevant. The current techniques proposed for learning deep networks under label noise focus on modifying the network architecture and on algorithms for estimating true labels from noisy labels. An alternate approach would be to look for loss functions that are inherently noise-tolerant. For binary classification there exist theoretical results on loss functions that are robust to label noise. 
In this paper, we provide some sufficient conditions on a loss function so that risk minimization under that loss function would be inherently tolerant to label noise for multiclass classification problems. These results generalize the existing results on noise-tolerant loss functions for binary classification.  We study some of the widely used loss functions in deep networks  and show that the loss function based on mean absolute value of error is inherently robust to label noise. Thus standard back propagation is enough to learn the true classifier even under label noise. Through experiments, we illustrate the robustness of risk minimization with such loss functions for learning neural networks.




§ INTRODUCTION

Recently, deep neural networks have exhibited very impressive performance in many classification problems. However, in all such cases 
one needs very large training data. Labelling the training data patterns and ensuring correctness of the labels thus becomes a serious challenge in many applications of deep neural networks. 

When the class labels in the training data are noisy (i.e., may be incorrect) then it is referred to as label noise. Human labelling errors, measurement errors, subjective biases of labellers are among the reasons for label noise. In many large scale classification problems, labelled data is often obtained through crowd sourcing or by automatically using information on the web. This is another main reason for unreliability of labels in the training data. 

Robust learning of classifiers in the presence of label noise has been investigated from many viewpoints. In this paper, we study it in the framework of risk minimization which is a popular method for classifier learning. For example, Bayes classifier minimizes risk under 0–1 loss function. The standard backpropagation-based learning of neural networks is also risk minimization under different loss functions (such as squared error or cross entropy). The robustness of risk minimization depends on the loss function used. We call a loss function noise-tolerant if 
minimizer of risk (under that loss function) with noisy labels would be same as that with noise-free labels.

In this paper we present some novel analytical results on noise-tolerance of loss functions in a multiclass classification scenario. We derive sufficient conditions on a loss function so that it would be noise-tolerant for different types of label noise. 
We then examine some of the popular loss functions used for learning neural networks and show that loss function based on mean-absolute error (MAE) satisfies our sufficient conditions. Empirical investigations are presented to compare robustness of learning neural networks under label noise using different loss functions based on mean-absolute error, mean-square error and categorical cross entropy. The empirical results well demonstrate the utility of the theoretical results presented here. 





 §.§ Related Work


Learning in presence of label noise is a long standing problem in machine learning.  
A detailed survey is available in <cit.>.


There are many approaches to learning under label noise.
Data cleaning approaches rely on finding points which are corrupted by label noise. 
Once these points are identified, they can be either filtered out or their labels suitably altered. 
Several heuristics have been used to guess such noisy points <cit.>.
There have also been attempts at (heuristically) modifying existing learning algorithms to make them robust 
<cit.>.


Another prominent approach is to treat the (unknown) true labels as  hidden variables and to estimate a generative or discriminative model. <cit.> proposed such a method, based on maximum-likelihood estimation of the model using the EM algorithm, in the context of fisher linear discriminant classifier. Similar methods have been proposed to make logistic regression robust to label noise <cit.>. Recently many algorithms based on such ideas are proposed in deep learning literature to mitigate the adverse effect of label noise. <cit.> use a generative model for label corruption,  estimated through an approximate EM algorithm, and show its effectiveness in a binary classification problem. <cit.> proposed a modified architecture of a neural network to learn with noisy labels by effectively estimating label corruption probabilities.
Motivated by similar ideas,   a method to estimate a generative model incorporating label corruption is proposed in <cit.>. 
This method is applicable for multiclass classification and can handle fairly general cases of label noise. Methods based on  bootstrapping are also proposed for learning deep networks under label noise <cit.>. While all these methods are seen to deliver good performance, they do not guarantee, in any probabilistic sense, robustness to label noise. 

All the above methods focus on changing the learning algorithm so that one can estimate the true labels of the training examples and thus be able to learn under label noise. As opposed to this, one can also look for methods that are inherently noise tolerant. Such algorithms treat noisy data and noise-free data the same way but achieve noise robustness due to properties of the algorithm. Such methods have been mostly investigated in the framework of risk minimization. 

Robustness of risk minimization depends on the loss function.
For binary classification, it is shown that 0–1 loss is robust to symmetric or uniform  label  noise while most of the standard convex loss functions are not robust <cit.>. The problem of learning from positive and unlabeled data can be cast as learning under label noise and it is seen that none of the common convex surrogate losses are good for this problem <cit.>. Unhinged loss, a convex loss (which is not convex potential), is robust to symmetric label noise <cit.>.  <cit.> proposed a method for robust risk minimization through an implicit estimation of noise probabilities. In a similar spirit, <cit.> proposed a method of estimating Type 1 and Type 2 error rates of any specific classifier under the noise-free distribution given only the noisy  training data. Recently, a general sufficient condition on a loss function is derived so that risk minimization is robust to label noise  <cit.>. It is shown that the 0–1 loss, ramp loss and sigmoidal loss satisfy this condition. 

All the above are for the case of binary classification. Recently <cit.> proposed a robust risk minimization approach for learning neural networks for multiclass classification by estimating label corruption probabilities.  In our work here we also investigate robustness of risk minimization in the context of multiclass classification. We provide analytical results on conditions under which risk minimization is robust to different types of label noise. Our results generalize the existing results for 2-class problems. The currently known noise-tolerant loss functions (such as 0–1 loss or ramp loss) are not commonly used while learning neural networks. In this paper, we examine some common loss functions for learning neural networks for multiclass classification and show that the one based on mean absolute error is noise-tolerant. Through empirical studies we demonstrate the relevance of our theoretical results.   





§ PRELIMINARIES AND PROBLEM STATEMENT


In this section we introduce some notation and 
define the notion of noise tolerance of a loss function.  



 §.§ Risk Minimization


Let ⊂^d be the feature space from which
the examples are drawn and let =[k] = {1,⋯,k} be the class labels.
In a typical classifier learning problem, we are given training data,
S={(_1,y__1),…, (_N,y__N)}∈ (×)^N, 
drawn iid according to an unknown distribution, 𝒟,  over ×. We represent a classifier as 
h() = ∘ f() where f: →,  ⊆^k.
Here, h (which predicts the class label given f()) maps  to . Even though the final classification decision on a feature vector  is ∘ f(), we use the notation of calling f itself as the classifier. 


A loss function is a map L: ×→^+. Given any loss function, L, and 
a classifier, f, we define the L-risk of f by 

    R_L(f) =  _[ L(f(), y_)] =  _, y_ [ L(f(), y_)]

where, as a notation throughout this paper, the  denotes expectation and its subscript indicates the random variables or the distribution with respect to which the expectation is taken. 
Under risk minimization framework, the objective is to learn a classifier, f, which is a global minimizer of R_L. Note that the L-risk, R_L, depends on L, the loss function. When L happens to be the 0–1 loss, R_L would be the usual Bayes risk. 



 §.§ Noise Tolerance of Loss Functions



When there is label noise, the learner does not have access to the clean training data (represented by S above).
The noisy training data available to the learner is S_η = {(_n,ŷ__n),n=1,⋯,N} where,

    ŷ__n = 
    y__n   with probability (1-η__n)
    
    j,   j ∈ [k],   j≠ y__n   with probability  η̅__n j

Note that, for all , conditioned on y_=i, we have ∑_j ≠ iη̅_ j = η_.


In general, for any , its true label (that is, label under distribution 𝒟) is denoted by the random variable  y_ while the 
noise corrupted label is denoted by ŷ_. We use 𝒟_η to denote the 
joint probability distribution of  and ŷ_.

The noise is termed symmetric or uniform if η_=η, η̅_ j=η/k-1,  ∀ j≠ y_, ∀, where η is a constant. 

Noise is said to be class-conditional or asymmetric if the dependence of η_ on  is only through y_ and similarly for η̅_ j. 
In this case, with a little abuse of notation, we write η_=η_y_,  η̅_ j=η̅_y_j. Thus, for example, η̅_ij would be the probability that a class-i pattern would have label as class-j when the label is corrupted. 

In general, when noise rate η_ as well as η̅_ j is a function of , 
it is termed as non-uniform noise. A simple special case is when η̅_ j=η_/k-1,  ∀ j≠ y_. We define it as simple non-uniform noise. Furthermore, when η_ is fixed for each class, we call it simple class conditional noise.

The L-risk, R_L(f), given by eq.<ref> is for the  noise-free case.
Let f^* be the global minimizer (over the chosen function class) of R_L(f). 
When there is label noise, the data is drawn according to distribution  𝒟_η. 
Then L-risk of a classifier f under noisy data is

    R_L^η(f)= __η[L(f(),ŷ_)] =  _, ŷ_[L(f(),ŷ_)]

(We use _, y_(_, ŷ_) and _(__η) interchangeably). Let f^*_η be the global minimizer (over the chosen function class) of R_L^η(f). 
Risk minimization under loss function L, is said to be noise-tolerant if <cit.>

    _𝒟[∘ f^*()=y_]=_𝒟
    [∘ f^*_η()=y_]

Risk minimization under a given loss function is  
noise tolerant if the f^*_η has the same probability of misclassification as
that of f^*  on the noise free data.
When the above is satisfied we also say that the loss function L is noise-tolerant. For this, it is sufficient if f^*=f_η^*. 


§ THEORETICAL RESULTS

We call a loss function L symmetric if it  satisfies, for some constant C, 

    ∑_i=1^k L(f(),i)=C,  ∀∈, ∀ f.

In the following, we prove distribution independent sufficient conditions for loss function to be robust under different kinds of label noises. In Theorem 1, we prove sufficiency results for symmetric label noise, followed by simple non-uniform noise and class-conditional noise in Theorems 2, 3.

		
	In a multi-class classification problem, let loss function L satisfy Eq <ref>.
	Then L is noise tolerant under symmetric or uniform label noise if η < k-1/k.
 

	Recall that for any f, 
	
    R_L(f)=_, y_ L(f(),y_)

	For uniform noise, we have, for any f,[In the following, _y_ | _ŷ_ | , y_ etc. denote expectation with respect to the corresponding conditional distributions. Note that __ y_ | =_, y_=_.]
	
    R_L^η(f)   =  _, ŷ_ L(f(), ŷ_)
       =  __y_ | _ŷ_ | , y_ L(f(), ŷ_) 
       = __y_ | [ (1-η) L(f(),y_) + η/k-1∑_i≠ y_ L(f(),i) ] 
        =  (1 - η) R_L(f) +  η/k-1 (C - R_L(f))
       = C η/k-1 + (1-η k/k-1) R_L(f).

	Thus, for any f,
	
    R_L^η(f^*)-R_L^η(f)=(1-η k/k-1)(R_L(f^*)-R_L(f)) ≤ 0

	because 
η < k-1/k  and f^* is a minimizer of R_L.
	This proves f^* is also minimizer of risk under uniform noise.

Remark 1 Theorem <ref> shows that symmetric losses are robust to uniform label noise. This does not depend on the data distribution. The only condition is that noise rate is less than k-1/k which is not restrictive. 
	This theorem (along with the next one) generalizes the existing results for the 2-class case <cit.>.


	
	Suppose loss L satisfies Eq <ref>. If R_L(f^*)=0, then L is also noise tolerant under simple non uniform noise when η_ < k-1/k, ∀.

	If R_L(f^*) = ρ >0 then, under simple non-uniform noise, R_L(f^*_η) is upper bounded by ρ/(1-kη_max/k-1), where η_max is maximum noise rate over ∈. (Recall that f^* is minimizer of R_L and f^*_η is minimizer of R^η_L). 
	


	Under simple non-uniform noise, for any f, 
	
    R_L^η(f) =    __y_ | _ŷ_ | , y_ L(f(), ŷ_)
     
            =    _[(1- η_) L(f(),y_)  
    	+ ∑_i≠ y_η_ L(f(), i)/k-1] 
    
    	=    _ (1- η_) L(f(),y_)+ _η_/k-1(C- L(f(),y_))
    
    	=    _ C'η_ +_((1-kη_/k-1)L(f(),y_))

	where C'= C/k-1. Hence we have
	
    R_L^η(f^*)-R_L^η(f)= _{ (1-kη_/k-1)(L(f^*(),y_)-L(f(),y_))}

	Since R_L(f^*) = 0 and L is non-negative by definition,
we have L(f^*(),y_)=0,  ∀. 
	In addition, since (1-kη_/k-1) > 0, we have R_L^η(f^*)-R_L^η(f) ≤ 0, for any f. Thus minimizer of noise free case is also a minimizer of noisy case. This completes proof of first part of theorem.
	
	For the second part of the theorem, we have, 
	
    R_L^η(f^∗_η)-R_L^η(f^∗) ≤ 0 
    ⇒_(1-kη_/k-1) (L(f^*_η(),y_)-
    	L(f^∗(),y_))≤ 0 
    ⇒min_η_ (1-kη_/k-1)_ L(f^*_η(),y_)≤_L(f^∗(),y_)
    ⇒ R_L(f^∗_η)≤ρ/(1-kη_max/k-1)

	where _L(f^∗(),y_) = R_L(f^*) = ρ. 
Note that, in the above, we used  η_<k-1/k, and hence  0 < (1-kη_/k-1) ≤ 1,  ∀. This completes the proof.

	Remark 2 Theorem <ref> establishes a sufficient condition for risk minimization to be robust to simple non uniform label noise. The condition needs R_L(f^*) to be zero. If L is the 0–1 loss then R_L is the Bayes risk and then the sufficient condition is that the classes are separable (under noise-free case). However, even if the classes are separable, for a general loss function (e.g., sigmoidal loss), R_L(f^*) may not be zero. The second part of Theorem <ref> gives a bound on R_L(f_η^*) in such cases. This part is useful even when classes are not separable and the optimal Bayes risk is non-zero. In case of high noise rate the bound might be loose, but one should note that, this is a distribution independent bound.
In the binary classification case, 	
	if data is separable, robustness can be achieved even though minimum value of L-risk is not zero if the loss function is `sufficiently steep' <cit.>. It appears possible to prove a similar result in multiclass case also. 
	
	

	
	Suppose L satisfies Eq <ref> and 0≤ L(f(),i)≤ C/(k-1),∀ i∈ [k]. If R_L(f^*)=0, then, L is noise tolerant under class conditional noise when η̅_ij<(1-η_i), ∀ j≠ i, ∀ i,j∈ [k].	


	For class-conditional noise, we have
	
	
    R_L^η(f)=    _ (1- η_y_) L(f(),y_)+  _∑_i≠ y_η̅_y_ i L(f(),i) 
    
    	=     _ (1-η_y_)(C-∑_i≠ y_xL(f(),i))_∑_i≠ y_xη̅_y_ i L(f(x),i) 
    
    	=    C_ (1-η_y_)-_∑_i≠ y_(1-η_y_-η̅_y_ i) L(f(),i)
	
	Since f^∗_η is the minimizer of R_L^η, we have R_L^η(f_η^∗)-R_L^η(f^∗)≤  0 and hence from Eq.(<ref>) we have
	
	
    _∑_i≠ y_(1-η_y_-η̅_y_ i)(L(f^∗(),i)-L(f^∗_η(),i)) ≤ 0

	
	Since we are given R_L(f^∗)=0, we have L(f^∗(),y_) = 0. Given the condition on L in the theorem, this implies L(f^∗(),i)=C/(k-1),  i≠ y_. As per the assumption on noise in the theorem,  (1-η_y_-η̅_y_ i)>0. Also, L has to satisfy L(f^∗_η(),i)≤ C/(k-1), ∀ i. Thus for Eq.(<ref>) to hold, it must be the case that L(f^∗_η(),i)=C/(k-1),  ∀ i ≠ y_ which, by symmetry of L, implies L(f^∗_η(),y_) = 0. 
	Thus minimizer of true risk is also a minimizer of risk under noisy data. This completes the proof.

	 Remark 3 Note that 1-η_y_>η̅_y_ i implies η_y_<(k-1)/k. Thus the condition on noise rates for this theorem is more strict.  For i≠ j, η̅_ij is the probability that a feature vector of class-i is labelled as class-j. If we set η̅_ii = 1- η_i which is the probability of a feature vector of class-i having correct label, then the condition is that the matrix [η̅_ij] of label noise probabilities should be diagonal dominant. The condition on the loss function in the theorem is satisfied by some of the symmetric losses such as 0–1 loss and MAE loss. The condition that R_L(f^*)=0 is restrictive.   However, experimentally, even though minimum risk might not be 0, symmetric losses show good robustness even under class-conditional noise.






 §.§ Some Loss Functions for Neural Networks

We assume standard neural network architecture with softmax output layer. 
If input to network is , then input to softmax layer is f(). 
Softmax layer  computes:

    u_i=exp(f()_i)/∑_j=1^k exp(f()_j),   i∈ [k]

where  f()_i represents i^ component of f().
We have ∑_i=1^ku_i=1.  We define =[u_1,⋯,u_k]. 
The label for the training patterns is in `one-of-K' representation. If the class of  is j then y_ is given as  where
e_ji=1 if i= j, otherwise 0.
We can now define some popular loss functions  namely,  categorical cross entropy (CCE), Mean square error (MSE) and Mean absolute error (MAE) as below.


    L (f(),) =∑_i=1^k e_jilog1/u_i =log1/u_j   CCE
    
    ||-u||_1
    =2-2u_j   MAE
    
    ||-u||_2^2
    = ||u||_2^2+1-2u_j   MSE

For these loss functions, we have

    ∑_i=1^k L(f(), ) = ∑_i=1^klog1/u_i   CCE
    ∑_i=1^k (2-2u_i)=2k-2   MAE
    
    k||u||_2^2+k-2   MSE

Thus, among these, only MAE satisfies symmetry condition given by Eq.(<ref>). 
While MSE does not satisfy Eq.(<ref>), one can show, using 1/k≤ ||u||_2^2≤1, that
k-1≤∑_i^k L_mse(f(x),i)≤ 2k-2.
This boundedness makes it more robust than an unbounded loss such as CCE. 

Informally, a loss function is said to be classification calibrated if a classifier having low enough risk under that loss would also have low risk under 0–1 loss. 
Logistic loss and exponential loss in multi-class settings have been proved to be classification calibrated <cit.>. One can show that, MAE, MSE losses are also classification calibrated. 



§ CLASSIFICATION CALIBRATED LOSSES

For the sake of completeness, we show that  loss functions considered in the main paper are classification calibrated. We follow the notation and convention defined in <cit.>. We denote L_ψ and R_ψ as the conditional ψ risk and conditional ψ regret. 
In multi-class case, a surrogate loss function ψ:[k]×→,  ⊆^k, is called 0-1 classification calibrated if, ∀∈Δ^k,

    inf_∈:pred()∉_t∈ [k]L_0-1(,t)L_ψ(,)>   inf_u ∈ L_ψ(,)
    inf_∈: R_0-1(,pred())>0R_ψ(,)>    0

Logistic loss and exponential loss in multi-class settings have been proved to be classification calibrated. <cit.>.
We show MAE and MSE are also classification calibrated. Note that, ∑_i=1^k u_i=1 for both these losses. 

We define S_={∈: R_0-1(,pred()>0)}. It is easy to see that S_={∈:_i u_i≠_i p_i }.
For MAE ,

    inf_∈ L_ψ(,)=inf_∈∑_i=1^k |p_i-u_i|

For MSE,

    inf_∈ L_ψ(,)=inf_∈∑_i=1^k (p_i-u_i)^2

For both these cases, unique minimizer is p_i=u_i. The S_ is set is non-empty unless p_i=1/k  ∀ k.
Thus the conditional ψ regret is always positive on the set S_. This completes the proof.


 §.§ Consistency under Symmetric Label Noise

We showed that risk minimization with symmetric losses is robust to label noise. Since there is only a finite training set, one can only minimize Empirical Risk. 
We now prove consistency of empirical risk minimization under label noise.

	
	Consider empirical risk minimization (ERM) under symmetric label noise over a given function class of finite VC dimension.  If the loss L used for ERM is robust to label noise (i.e., satisfies eq. <ref>), then the error rate of minimizer of empirical risk with noisy samples converges uniformly  to the error rate of the minimizer of risk under noise-free distribution. 


	We denote by er_[g] the error rate (i.e., 0-1 risk) of classifier g in the noise-free case.  Under noise we denote it as er__η[g]. 
	Let ĝ^∗ (ĝ^∗_η) be the minimizer of empirical risk over n noise-free (noisy) samples. Let g^∗ (g^∗_η) be the minimizer of risk. Since VC bounds are distribution independent, we have,
	
		er_[ĝ^∗]≤    er_(g^∗)+ϵ(n, vc)

		er__η[ĝ_η^∗]≤    er__η(g_η^∗)+ϵ(n, vc) 
	

	The term ϵ(n,vc) or simply ϵ goes to 0 with vc/n, where vc is the VC dimension of the function class  <cit.>.
	
	Under symmetric label noise η, we derived how error rate changes. (Note that 0-1 loss is symmetric). Thus,
	
    er__η=er_(1-kη/k-1)+c'η

	Then we have the following,
	
	   er__η[ĝ^∗_η]-er__η[g^∗_η]
	=(er_[ĝ^∗_η]-er_[g_η^∗])(1-kη/k-1)≤ϵ

	   ⇒  	er_[ĝ^∗_η]-er_[g^∗_η]=
	er_[ĝ^∗_η]-er_[g^∗]≤ϵ/1-kη/k-1	
	
	where we have used er_[g^∗_η]=er_[g^∗] which follows because L is robust to label noise.
	This completes the proof.	






	

§ EMPIRICAL RESULTS

	
	In this section we illustrate the robustness of symmetric loss functions. We present results with two image data sets and four text data sets. In each case we learn a neural network classifier using the CCE, MSE and MAE loss functions. We add symmetric or class conditional noise with different noise rates to the training set. For learning, we minimize the empirical risk, with different loss functions, using stochastic gradient descent through backpropagation <cit.>. The learnt networks are tested on noise-free test sets.

	

	

 §.§ Experimental Setting

	The specific image and text data sets used are shown in Table <ref>. In the table, for each data set, we mention size of training and test sets (n_tr,n_te), number of classes (c) and input dimension (d). Since some are image data while others are text data and feature space dimensions are all different, we have used different network architectures for each data set. These are also specified in Table <ref>. All networks used Rectified Linear Unit (ReLU) in the hidden layers and have softmax layer at the output with the size of the layer being the number of classes. All networks are trained through backpropagation with momentum term and weight decay. We have also used dropout regularization and the dropout rates are also shown in Table <ref>. 

The results reported are averages over six runs. Label noise is added in the training set by changing the label of each example independently. For symmetric noise, we fix η and randomly change the label of each example. For class conditional noise, for each experiment, we generate a fixed label noise probability matrix, [η̅_ij], randomly and use that to decide the new labels. We ensure that the matrix is diagonal dominant as needed for our theoretical results. 

				

		
	

 §.§ Results and Discussion

	
	In figure <ref>,  we compare the robustness of MAE and CCE losses on MNIST image data set and RCV1 text data set. We have used symmetric label noise with η = 0,   0.4,   0.8. The figure  shows the evolution of training and test accuracies of the network with number of epochs of training. As the graphs in Fig. <ref>(a)–(c) show, MAE loss is highly robust to symmetric label noise. The test accuracy achieved with MAE even under 80% noise is close to that with zero noise. On noise-free data, the accuracy achieved with CCE loss is a little bit higher than that with MAE. However, even at 40% noise, the test accuracy with CCE loss drops sharply. Similar trend can be seen on the RCV1 data. (See Fig. <ref>(e)–(g)). Here, eventhough the drop in accuracy of CCE loss is not as sharp, it is clearly seen that MAE is much more robust. 
	In Fig. <ref> (d) and (h) we show results under class-conditional noise (CC).  
In these problems we have no idea whether the minimum risk is zero. However, as can be seen from the figure, the symmetric MAE loss exhibits a good level of robustness under class conditional noise also. 
	
	Table <ref>, shows average test accuracy  and standard deviation (over six runs)  of the learnt networks for different noise rates. 
	We show results for noise rate η =0.0, 0.3, 0.6. (For the 2-class Imdb dataset,  noise rate used is η =0.0, 0.15, 0.3). We also show results for class conditional noise. As can be seen from the table, MAE exhibits  good robustness. When the accuracy of MAE at 0% noise is high (e.g., MNIST and RCV1), it drops very little even under 60% noise rate. However, when accuracy achieved at 0% noise is poor (showing perhaps that the optimal risk is large or that number of examples is inadequate), the accuracy drops with noise. However, in all cases the drop in accuracy with MAE is much less than that with CCE. This is in accordance with our results on robustness of symmetric losses. We also see that robustness achieved by MSE (which is a bounded loss though it is not symmetric) is in between that of MAE and CCE. As can be seen from the last column of the table, MAE exhibits fair amount of robustness under class conditional noise also. 		
	We observed that higher dropout rates reduce sensitivity of CCE to label noise. This is expected as dropout works like regularizer <cit.>. Interestingly, even without dropout, in many datasets MAE showed good robustness under label noise. On MNIST data with zero dropout, MAE retained almost same accuracy. We did not include these results  in the table because it is customary to have high dropout rate. 
	


§ CONCLUSION

In this paper, we derived some theoretical results on robustness of loss functions in multi-class classification. Such robust loss functions are useful because we can learn a good classifier (without any change in the algorithm or network architecture) even when training set labels are noisy. While we discussed these in the context of learning neural networks, our theoretical results are general and apply to any multi-class classifier learning through risk minimization. For learning neural networks, we showed that the commonly used CCE loss is sensitive to label noise while MAE loss is robust. We presented extensive empirical results to illustrate this. However, training a network under MAE loss would be slow because the gradient can quickly saturate while training. On the other hand, training under CCE is fast. Thus, designing better optimization methods for MAE is an interesting problem for future work. This would allow one to really exploit the robustness properties of MAE (and other such symmetric losses) proved here. 


