Towards Better Understanding Open-set Noise in Learning with Noisy Labels
Open-set noise;Noisy labels
Yes
No ethics review needed.
- For Figure 2 (c) (d), Authors state in the paper that "the presence of open-set noise degrades OOD detection performance, whereas, conversely, the presence of closed-set noise could even improve OOD detection performance.". However, from my observation, the closed-set noise does not improve detection performance from the Figure. 

- In Section 3.3.2, Is it possible to assume the same pattern and noise ratio of close-set noise and then study how different open-set noise compare to each other?

- For Figure 2, is it drawn when model converges for each case?
- The paper studies a detailed examination of open-set noise in the context of learning with noisy labels, an area that has been largely overlooked in previous research.

- It offers a robust theoretical framework to analyze the effects of open-set noise and supports these findings with empirical evidence through the creation and testing on synthetic datasets.

- The paper provides a careful look at different modes of open-set noise, comparing 'easy' and 'hard' open-set noise scenarios, which is crucial for understanding how various types of noise affect model performance.

- It introduces the use of out-of-distribution (OOD) detection as a complementary evaluation metric to traditional accuracy measures, enhancing the assessment of model performance in the presence of open-set noise.

- The paper underscores the significance of open-set noise in real-world datasets and demonstrates the practical impact of its findings on existing learning methods, highlighting the need for more research in this area.
This paper addresses the challenge of open-set noise in learning with noisy labels (LNL), a problem where the true labels of noisy samples may not belong to any known category. The authors refine the LNL problem by accounting for open-set noise and theoretically analyze its impact compared to closed-set noise. They construct two open-set noisy datasets and introduce a novel open-set test set for the WebVision dataset to empirically validate their findings. The results indicate that open-set noise has distinct characteristics and a lesser negative impact on model performance compared to closed-set noise, highlighting the need for further exploration into model evaluation under such conditions. The paper also examines an entropy-based open-set noise detection mechanism and proposes additional out-of-distribution detection tasks for model evaluation.
- The paper examines a few existing learning with noisy labels (LNL) methods on the synthetic datasets, outlined in Appendix E.1. However, it does not explore a wide range of existing methods such as robust losses in LNL, which could limit the comprehensiveness of the comparison and the conclusions drawn about the state-of-the-art in handling open-set noise. 

- For Section 3.3.2, Authors exclude the effect of closed-set noise (Cx = 0) and only focus on open-set noise which could limits the findings in real-world scenarios. For example, It is not clear how different open-set noise with same close-set label noise.

- Experiments on the main paper are not thorough enough. I suggest add more LNL methods and discusses how different methods behave and align with the theorems in the main paper.
Yes
No ethics review needed.
No
This paper analyzes the learning with noisy labels problem by explicitly accounting for the presence of open-set noise.
This paper analyzes the learning with noisy labels problem by explicitly accounting for the presence of open-set noise. However, only theoretical analysis and dataset construction are not sufficient to be published in ICLR.
Only theoretical analysis and dataset construction are not sufficient to be published in ICLR.
Yes
No ethics review needed.
What does this sentence mean: obtaining a model that perfectly fits the data distribution is often challenging; here, we consider training a single-layer linear classifier upon a frozen pretrained encoder. I think a single-layer linear classifier may even be worse to fit the data distribution.
1.The paper is easy to follow and the conclusions are easy to understand.

2.The mathematical analysis is adequate and there are experimental results to support these insights.

3.The experimental figures are clear and adequate on the constructed dataset.
In this paper, the authors theoretically analyze and compare the impacts of open-set and closed-set noise, as well as the differences between various open-set noise modes. Additionally, they examine a common open-set noise detection mechanism based on
prediction entropy. Moreover, to validate their insights, they construct two open-set noisy datasets and a open-set test-set for evaluation.
1.The way of constructing benchmark is quite similar to existing benchmarks in LNL. Thus it may be not suitable to list it as a contribution.

2.Though the author introduces a hard open-set noise (seems like a combination of feature-dependent noise and open-set noise), but it seems that the author does not design a method to tackle such kind of noise.

3.Some of the conclusions may be naive and simple.  For example, " it may be effective only for ‘easy’ open-set noise.". Because entropy-based methods generally fail to detect close-set feature-dependent noise as well.


There are some minor issues that will not affect the rating:
1." the concept of complete noise transition matrix" should be " the concept of a complete noise transition matrix".

2. "namely fitted case and overfitted case" should be "namely the fitted case and overfitted case".

3.It would be better if the contributions are more compact. There are six contributions now.
We theoretically and empirically analyze and validate the impact of open-set noise.
@inproceedings{
anonymous2024towards,
title={Towards Better Understanding Open-set Noise in Learning with Noisy Labels},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=EzB0n8aRqI},
note={under review}
}
To reduce reliance on labeled data, learning with noisy labels (LNL) has garnered increasing attention. However, most existing works primarily assume that noisy datasets are dominated by closed-set noise, where the true labels of noisy samples come from another known category, thereby overlooking the widespread presence of open-set noise—where the true labels may not belong to any known category.
In this paper, we refine the LNL problem by explicitly accounting for the presence of open-set noise. We theoretically analyze and compare the impacts of open-set and closed-set noise, as well as the differences between various open-set noise modes. Additionally, we examine a common open-set noise detection mechanism based on prediction entropy. To empirically validate our theoretical insights, we construct two open-set noisy datasets—CIFAR100-O and ImageNet-O—and introduce a novel open-set test set for the widely used real-world noisy dataset, WebVision. Our findings indicate that open-set noise exhibits distinct qualitative and quantitative characteristics, underscoring the need for further exploration into how models can be fairly and comprehensively evaluated under such conditions.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Open-set noise Noisy labels
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/b038b562d317c63df7779be3bb00c8838b0d7dda.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Towards Better Understanding Open-set Noise in Learning with Noisy Labels
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission