Benign Overfitting of Long-Tailed Data Classification in Two-layer Neural Networks
two-layer neural networks;benign overfitting;long-tailed data distribution
I have read and agree with the venue's withdrawal policy on behalf of myself and my co-authors.
We re-examine benign overfitting in two-layer neural networks and prove that while short-tailed data is classified by explicit features, long-tailed data is classified through covariance-realted implicit features.
@misc{
xu2024benign,
title={Benign Overfitting of Long-Tailed Data Classification in Two-layer Neural Networks},
author={Ruichen Xu and Kexin Chen},
year={2024},
url={https://openreview.net/forum?id=lvxl6celCV}
}
Recent theoretical studies (Kou et al., 2023; Cao et al., 2022) have revealed a sharp phase transition from benign to harmful overfitting when the noise-to-feature ratio exceeds a threshold—a situation common in long-tailed data distributions where atypical data is prevalent. However, harmful overfitting rarely happens in overparameterized neural networks. Experimental results further suggested that memorization is necessary for achieving near-optimal generalization error in long-tailed data distributions (Feldman & Zhang, 2020). We argue that this discrepancy between theoretical predictions and empirical observations arises because previous feature-noise data models overlook the heterogeneous nature of noise across different data classes. In this paper, we refine the feature-noise data model by incorporating class-dependent heterogeneous noise and re-examine the overfitting phenomenon in neural networks. Through a comprehensive analysis of the training dynamics, we establish test loss bounds for the refined model. Our findings show that neural networks can benefit from "data noise", previously deemed harmful, and learn implicit features to enhance classification accuracy for long-tailed data. Experiments on both synthetic and real-world data validate our theory.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
~Ruichen_Xu1 ~Kexin_Chen4
Ruichen Xu Kexin Chen
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
two-layer neural networks benign overfitting long-tailed data distribution
I certify that there is no acknowledgement section in this submission for double blind review.
xu|benign_overfitting_of_longtailed_data_classification_in_twolayer_neural_networks
learning theory
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Benign Overfitting of Long-Tailed Data Classification in Two-layer Neural Networks
ICLR 2025 Conference Withdrawn Submission
ICLR.cc/2025/Conference/Withdrawn_Submission