Laplace Sample Information: Data Informativeness Through a Bayesian Lens
Sample informativeness;Sample Information;Sample  Difficulty;Long-tailed distribution;Leave-one-out retraining;KL Divergence
Yes
No ethics review needed.
None
1. Overall, this paper is well-written and easy to follow.
2. This paper considers the important task of estimating the sample information.
3. Experimental results illustrate the effectiveness of the proposed measure LSI.
This paper proposes a measure named Laplace Sample Information (LSI) to estimate the informativeness of individual samples in a dataset for deep learning methods. Specifically, LSI leverages a Bayesian approximation to the weight posterior and the KL divergence to measure the change in the parameter distribution induced by a leave-one-out sample from the dataset. Experimental results show that LSI is effective in ordering the data w.r.t. typicality, detecting mislabeled samples, measuring class-wise informativeness, and assessing dataset difficulty.
1. The novelty of the proposed method might be limited as the Laplacian approximation is widely used in the Bayesian machine learning community.
2. Theoretically, formal results are lacking to support the effectiveness of the proposed method.
3. Large experiments involving large datasets and models are lacking, which results in that this work might have little influence in practice.

Minor issues:
1. Typos exist in Lines 194-195: shownAppendix.
Yes
No ethics review needed.
Pleaser refer to the weakness section.
* **Originality and Quality**
The approach is novel because it uses a post-hoc Bayesian approximation to evaluate sample informativeness in standard neural networks. The authors include experiment results on multiple datasets, from CIFAR-10 to IMDb, showing that LSI can detect mislabeled samples and assess informativeness across classes. 

* **Clarity and Significance**
The paper is well-organized and easy to follow. As claimed in the paper, LSI is beneficial for identifying potentially mislabeled data, as well as evaluating dataset difficulty and informativeness. The proposed method could have a substantial impact, particularly for data-centric AI and applications where dataset quality is key.
This paper presents Laplace Sample Information (LSI), a metric to quantify how much individual data points contribute to a neural network’s learning process. By approximating a Bayesian model through the Laplace method, LSI uses KL divergence to capture each sample’s impact on the model’s parameters, which can be particularly beneficial for detecting mislabeled data, understanding dataset difficulty and informativeness. Empirically, the authors validate LSI across a range of datasets and demonstrate that it transfers well from smaller “probe” models to larger architectures.
**1. Computational Cost**

Calculating LSI requires Hessians, which may be expensive on large datasets. While the probe model helps, a more detailed breakdown of computation time, evaluation quality, and scalability would be helpful for readers to understand the effectiveness of LSI.

**2. Limitations of Probe Model** 

While using a smaller probe model makes computation faster, it could introduce discrepancies in sample rankings when applied to larger models. Could the authors provide more insight into how well LSI rankings from the probe model align with those from larger models?

**3. Understanding the Effectiveness of LSI**

During the investigation of low/high LSI samples, is it possible to have statistical measurements about the comparisons/differences between high and low LSI samples. It is not clear if the observation of the difference among a few samples can generalized to the larger scale. Besides, there are no baseline performances reported in this paper. (i.e., for the detection of mislabeled samples, I believe there are certain methods already working on it.)

**4. Experiments on Real-world Mislabeled Samples**

In the section ```Focusing on Mislabeled Samples```, it would be more beneficial if authors could test the effectiveness of LSI on real-world noise (CIFAR-10N [R1], or CIFAR-10H [R2]), instead of synthetic ones (random label flips). 

**References:**

R1: Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations. ICLR 2022.

R2: Human uncertainty makes classification more robust. ICCV 2019.
Yes
No ethics review needed.
Please see weaknessnes
- LSI offers a new perspective on sample informativeness, focusing directly on sample information flow rather than related metrics like sample difficulty. This approach addresses a gap in the current literature, where sample informativeness has often been approximated indirectly.

- LSI is adaptable across different model architectures, learning tasks (supervised and unsupervised), and data modalities (e.g., images, text). LSI has valuable applications in identifying mislabeled samples, estimating dataset difficulty, and detecting out-of-distribution data.

- By using small probe models, the authors make LSI computationally feasible for larger models and datasets. This approach yields a substantial efficiency gain without significantly compromising the integrity of the informativeness rankings.

- The authors evaluate LSI on various datasets, including CIFAR-10, CIFAR-100, ImageNet subsets, and text datasets, with extensive experiments validating LSI’s effectiveness in distinguishing sample informativeness. They also investigate LSI’s behavior under label noise.
The paper describes a novel approach to measuring the informativeness of individual samples in machine learning datasets, named the Laplace Sample Information (LSI). LSI is designed to quantify how much information each sample contributes to the model's parameters using a Bayesian approximation. The method applies the Laplace approximation to generate a distribution over model parameters and subsequently employs Kullback-Leibler divergence to measure each sample's unique contribution to this distribution. 

Experiments on CIFAR-10, CIFAR-100, ImageNet subsets (Imagewoof and Imagenette), a pediatric pneumonia dataset, and the IMDb text sentiment dataset. LSI revealed that only a small subset of samples contributes significant information to the model, while most samples have low informativeness. On CIFAR-10 with 10% label noise, mislabeled samples generally had higher LSI values than correctly labeled samples, showing that LSI can effectively highlight mislabeled data. Other experiments collectively validated LSI’s utility across various applications: ranking samples by informativeness, detecting mislabeled data, assessing dataset and class difficulty, and improving model generalization.
- The experiments mainly evaluate the proposed LSI measures. How does the proposed measure compare to other existing related measures, such as gradient similarity, influence functions, and TRAK (Park et al., ICML'23)? 

- Although LSI uses a probe model for efficiency, computing LSI still requires repeated training sessions, especially in large datasets. It would be better to describe its computation complexity and compare it with existing measures. 

- The reliance on the Laplace approximation assumes that the model parameters follow a multivariate Gaussian distribution. While the authors justify this through the central limit theorem, the approximation may still be less accurate for highly complex or non-convex loss landscapes, potentially affecting LSI’s accuracy.

- The method relies on approximations of the Hessian, particularly when using diagonal approximations for large models. It would be better to analyze how this approximation affects LSI in different neural architectures.
Yes
I would like author to discuss its potential negative implications of this methods such as data leakage issue.
No ethics review needed.
See the questions in weakness sections.
- I think this paper raised an important problem in explaining the important data sample. 
- The proposed method is theoretically principled with Bayes principle. 
- Extensive experimental results are presented with CV and NLP datasets (since this method is model agnostic). 
- The paper is clearly written and accessible.
This paper considered an information theoretical framework in understanding the data sample importance. Specifically, the intuition is further formulated as the difference of parameter distributions. Experiments on various and extensive experiments demonstrate the effectiveness of the proposed methods.
There are two major weak points within the paper. Therefore I currently provide a borderline score. 

1. About the novelty. How this method is different from the influence function based method. Both methods require some calculation on the higher-order derivative. Based on the given context, I find it a bit hard to differentiate the differences. 
2. About the calculation LSI in line 212, I could not understand how exactly \hat_{theta}^{-i} and \Sigma^{-i} are estimated. Here the technical detail is very limited, therefore I could not understand why estimating these can be done after training. Please give clear details how LSI is calculated. 
3. (Minor) The paper is a bit large by itself, 32Mb ! I could not properly read the results due to its large file.
This paper establishes a novel measure of sample informativeness based on the KL divergence applied on a quasi Bayesian approximaiton of the model parameters.
@inproceedings{
anonymous2024laplace,
title={Laplace Sample Information:  Data Informativeness Through a Bayesian Lens},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=qO6dk9KfIp},
note={under review}
}
Accurately estimating the informativeness of individual samples in a dataset is an important objective in deep learning, as it can guide sample selection, which can improve model efficiency and accuracy by removing redundant or potentially harmful samples. 
We propose $\text{\textit{Laplace Sample Information}}$ ($\mathsf{LSI}$) measure of sample informativeness grounded in information theory widely applicable across model architectures and learning settings.
$\mathsf{LSI}$ leverages a Bayesian approximation to the weight posterior and the KL divergence to measure the change in the parameter distribution induced by a sample of interest from the dataset.
We experimentally show that $\mathsf{LSI}$ is effective in ordering the data with respect to typicality, detecting mislabeled samples, measuring class-wise informativeness, and assessing dataset difficulty.
We demonstrate these capabilities of $\mathsf{LSI}$ on image and text data in supervised and unsupervised settings.
Moreover, we show that $\mathsf{LSI}$ can be computed efficiently through probes and transfers well to the training of large models.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Sample informativeness Sample Information Sample  Difficulty Long-tailed distribution Leave-one-out retraining KL Divergence
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/baaec8e596c3ab09fb6c4829d80dc16b326908e3.pdf
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Laplace Sample Information:  Data Informativeness Through a Bayesian Lens
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission