Towards Understanding Why Label Smoothing Degrades Selective Classification and How to Fix It
Uncertainty Estimation;Selective Classification;Label Smoothing
Yes
No ethics review needed.
**Q1:** As observed by the authors, LS consistently leads to degraded SC performance, even if it may improve accuracy. What do authors think about the connection between NLS and SC, where NLS refers to Negative Label Smoothing introduced in R1.

**Q2:** In Line 132, it would be beneficial to include the definition of Kronecker delta.

**References:**

R1: To smooth or not? when label smoothing meets noisy labels. ICML 2022.
**Originality**

The paper addresses a unique gap by investigating LS's unintended effect on SC. The logit-level gradient analysis provides a fresh perspective on why LS interferes with SC, helping bridge theoretical understanding with observed results.

**Quality** 

The experiments are well-designed, involving diverse datasets (e.g., ImageNet, Cityscapes) and model architectures (e.g., ResNet-50, ViT) to thoroughly validate the findings. The analysis at both empirical and gradient levels strengthens the rigor, making the results compelling.

**Clarity** 

The paper is well-organized, with clear visuals that illustrate LS’s effect on SC. Figures showing SC degradation and the effects of logit normalization make complex points more accessible.
This paper investigates the impact of label smoothing (LS) on selective classification (SC), showing that while LS is a popular regularization technique for improving classification accuracy, it degrades SC performance. The authors empirically confirm this degradation across various models and tasks, then analyze the cause at a gradient level, finding that LS suppresses the highest logit differently based on prediction correctness. They propose post-hoc logit normalization as a solution, showing it effectively recovers SC performance degraded by LS.
My only concern is about the novelty of the contributions.

**Novelty concern**

* As admitted by the authors (Line 212), some of the core conclusions in the main paper are based on previous empirical observations (```for a single value of alpha LS degrades SC for CNN-based image classification```). And the introduced ```broader investigation``` draws the same conclusion as the literature.

* As specified by the authors (Line 416), another main contribution of the paper (```Logit Nomarlisation Improves the SC of LS-Trained Models```) also follows from the literature that ```logit normalization can improve the SC performance of many (but not all) pretrained models```.
Yes
No ethics review needed.
- The experimental analysis is primarily focused on image classification and segmentation tasks, using two specific datasets. Could these findings generalize to other domains, such as text or tabular data?
- In Figure 3, the degradation impact of LS on SC decreases at high coverage. Is it possible to identify a threshold at which the effect of LS on SC begins to diminish?
- The paper provides a well-rounded analysis, both theoretical and experimental, demonstrating how LS degrades SC. By analyzing logit-level gradients, it addresses why LS has this adverse effect on SC, filling an important gap in understanding.
- Visualizations in the paper effectively illustrate the results, providing an intuitive understanding of how LS impacts SC performance.
- The findings have practical impact. By showing that LS leads to consistent degradation in SC, the paper suggests that it may partially explain the findings of strong classifiers surprisingly underperform on SC. They further shows that logit normalization can recover the degradation. This could be useful especially in the high-risk tasks like medical or robotics tasks.
This paper shows how label smoothing (LS) can negatively impact selective classification (SC) by combining empirical analysis on large-scale tasks with theoretical insights on logit-level gradients. The authors show that the degradation in SC performance worsens with increased LS strength, and they propose post-hoc logit normalization as an effective method for recovering SC performance lost due to LS.
The experimental analysis is limited to image classification and segmentation tasks, using only two specific datasets. This raises questions about the generalizability of the findings to other domains, such as text or tabular data, where label smoothing and selective classification may behave differently. Expanding the analysis to include diverse data types would strengthen the claim that label smoothing consistently degrades SC performance across various domains.
Yes
No ethics review needed.
N.A
1. I enjoy reading the paper. The presentation and organization look great.
2. This work provided both empirical and theoretic analysis to answer a research problem and considerably contributed to the field.
This paper explains why logit nomalisation improves SC performance for models trained with label smoothing.
1. While the authors provided detailed experiments to demonstrate the properties of LS. I think it is also important to quantitatively compare the LS+logit normalisation with other existing solutions under the experimental setting used in this work to demonstrate the effectiveness of such a combination. 

2. In figure 7(a), it is hard to find the "CE logit norm".
Yes
No ethics review needed.
All my confusions have been listed under the weaknesses section. I welcome the authors to provide clarifications on points 1, 3, and 4.
Originality: the paper provides a very detailed and theoretic explanation for the unresolved issue of why label smoothing degrades selective classification.
Quality & Clarity: their explanations appear to be very reasonable. The theoretical gradient descent expressions align with the actual experimental results, making their work seem like a solid theoretical piece rather than some far-fetched guess.
Significance: The authors' explanation addresses a research gap and is likely to be valuable and insightful for future studies in selective classification.
Label smoothing (LS) is a popular regularization technique that improves test accuracy, yet previous research has shown that the LS models achieve degraded performance of selective classification (SC)
By comparing the formulation of Cross Entropy (CE) model and LS model, this work provides an explicit explanation on why this performance degradation is happening.
Building on the identified reasons for this degradation, the authors further discuss the impact of logit normalization and why it significantly improves SC performance of LS models but not that of CE models.
The primary contribution of this paper is the theoretical clarification of a previously unresolved issue. The experimental results are consistent with the theoretical findings presented in the paper.
While the authors present most of the logic and theory in an understandable and clear manner, unfortunately, the confusion in notation and concepts keeps me oscillating between being confused and understanding. The authors probably need to clarify some logic that they consider obvious but which I find uncertain. These points include:

1. Line 318: I would like to know more clearly the connection between the regularization in Equation 13 and the regularization gradient in Equation 14. Because, in my view, regularization refers to -\alpha/K, which is an item in L_LS, but the regularization gradient includes \alpha \bar{\pi}_k - \alpha/K, which is the result of L_LS - L_CE. The relationship between these two needs to be explicitly identified when they have similar names.
2. Line 319: There is a typo after the second equals sign in the formula.
3. Line 353: 'This directly impacts softmax-based U such as MSP', but I am not very clear on how. In fact, starting from Part 4.2, I completely lost track of the concepts of uncertainty, various expressions of pi, regularization, and P_error. This makes me feel like I understand but I don't.
4. Line 371: Again, due to the confusion on the concepts, 'v_max is more strongly suppressed for lower P_error' is not intuitive to me.
5. The caption for Figure 4 is confusing. Which one is the left, and which one is the right?
6. The caption for Figure 5 is confusing. What is the purpose of Figure 3? What does it illustrate? It is not clear.
@inproceedings{
anonymous2024towards,
title={Towards Understanding Why Label Smoothing Degrades Selective Classification and How to Fix It},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=6oWFn6fY4A},
note={under review}
}
Label smoothing (LS) is a popular regularisation method for training neural networks as it is effective in improving test accuracy and is simple to implement. ''Hard'' one-hot labels are ''smoothed'' by uniformly distributing probability mass to other classes, reducing overfitting. Prior work has shown that in some cases *LS can degrade selective classification (SC)* -- where the aim is to reject misclassifications using a model's uncertainty. In this work, we first demonstrate empirically across an extended range of large-scale tasks and architectures that LS *consistently* degrades SC. 
We then address a gap in existing knowledge, providing an *explanation* for this behaviour by analysing logit-level gradients: LS degrades the uncertainty rank ordering of correct vs incorrect predictions by regularising the max logit *more* when a prediction is likely to be correct, and *less* when it is likely to be wrong.
    This elucidates previously reported experimental results where strong classifiers underperform in SC.
    We then demonstrate the empirical effectiveness of post-hoc *logit normalisation* for recovering lost SC performance caused by LS. Furthermore, linking back to our gradient analysis, we again provide an explanation for why such normalisation is effective.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Uncertainty Estimation Selective Classification Label Smoothing
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/bbd494238c64778e70846d4d5621010857768d32.pdf
other topics in machine learning (i.e., none of the above)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/b3a4dd93942fe9ad0097483877176e3466b402a4.zip
Towards Understanding Why Label Smoothing Degrades Selective Classification and How to Fix It
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission