Calibrating LLMs with Information-Theoretic Evidential Deep Learning
evidential deep learning; information bottleneck; calibration; large language models
Yes
No ethics review needed.
See and clarify the weakness point above.
- Relevance: The paper addresses an important problem; the uncertainty calibration of LLM outputs (at the token level), which is an important component for providing safe deployment and error estimation for LLMs.

- Theoretical Soundness: The paper takes a very principled approach, identifying precise issues in the application of EDL to LLMs and addressing them with mathematically derived solutions based on reasonable starting assumptions.

- Experimental Soundness: The experiments are more than sufficiently extensive: they target the right settings and metrics to validate the claims of the paper and comprehend great breadth of models, tasks and mode of evaluation (acc+ECE, OOD detection). The claims of improved uncertainty calibration are strongly backed by the experiments.
The paper present a method to effectively apply Evidential Deep Learning (EDL), a parametric uncertainty estimation approach, to LLMs. The approach introduces two features to standard EDL that make it practical and effective for application to LLMs; i) the introduction of an information bottleneck cost function, which mitigates over-fitting and ii) the use of the LLM outputs as the intermediate "layer" Z where to impose the bottleneck, which simplifies direct computation of certain terms in the mutual information lower bounds, part of the objective function.
The only main weakness of the paper is the clarity of scope:

From the experiments, it is clear that this is a method for uncertainty calibration for a specific task at the fine-tuning stage of an LLM. e.g., an LLM is fine-tuned for summarisation and the proposed method offers a way to obtain superior uncertainty calibration *for the fine-tuned task*. This is indeed very useful, but needs to be stated more clearly in the abstract and intro, as from all sections up until the experiments it is unclear if this is a method for pre-training and/or fine-tuning. In principle, I can see the method may be applied to pre-training too, but since this is not tested in the experiments it should not be included in the claims.
Yes
No ethics review needed.
Please see weakness.
The paper is well-presented and easy to follow.

The paper proposes a unified perspective on EDL methods by framing several existing EDL methods as special cases within the IB-EDL framework.

The experimental results are strong and covers multiple LLMs, showing that IB-EDL addresses the overconfidence problem well.
This paper proposes IB-EDL, combining Evidential Deep Learning (EDL) with information bottleneck (IB) for uncertainty estimation in large language models (LLMs). This approach aims to enhance model calibration by reducing overconfidence, which is a common issue in fine-tuned LLMs. The experimental results show that IB-EDL achieves improved calibration with minimal computational overhead.
The pipeline of applying IB to EDL ($x → f(x; θ) → e˜ → e →
α → π → y$) seems computational heavy, it would be informative to include an inference time comparison of IB-EDL against other methods. 

Although the training overhead of IB-EDL is relatively small compared with the pretraining step, the paper lacks comparison of training time compared with LoRA, which is the fine-tuning backbone of IB-EDL.

For out-of-distribution (OOD) scenarios, the paper only compares the OOD detection capabilities with AUROC. The expected calibration error (ECE) is missing, which is also an indicator of OOD performance as the calibrated model should maintain relatively low ECE score on OOD datasets.

The novelty is not clear for combining IB with EDL, which are two well established methods.
Yes
No ethics review needed.
* While it appears the training methods/baselines tested are normalized for number of training steps, I am curious about comparison in terms of the number of training flops. Do the new mean and covariance prediction heads require separate training? 
* Is there a benefit to having two parameters estimating the distributional shift (mean and variance) of the token predictions vs. just one?
* What happens to calibration if we finetune on multiple datasets simultaneously?
* How does this fine-tuning procedure effect LLM performance on open-ended generation tasks?
* This work is well-motivated theoretically, pointing out deficiencies with previous EDL losses (e.g. paragraph *Challenges when applying IB to an internal layer of an LLM*). 
* The final regularization term $\mathcal{L}_{IB-Info}$ seems relatively simple, although I do have some questions about practical implementation (see questions)
* Strong experimental results; the OOD detection and noise injection experiments (tables 3,4) in particular seem support the argument for this additional regularization during calibration
In this work, authors propose a new loss regularization term to reduce overconfidence in LLMs when training via the evidential deep learning (EDL) framework. This regularization is based on the information bottleneck loss, which maximizes mutual information between token logits and ground truth prediction distribution while minimizing M.I. between predictions and inputs. This is achieved by minimizing a KL divergence between a learned conditional logit distribution and a Gaussian prior. This work demonstrates that fine-tuning via this loss produces better calibration (ECE, AUC) with equivalent accuracy to baselines. Authors also illustrate this method is robust to noise in the training data and distribution shifts.
* This work seems only applicable for a discrete set of mutually exclusive classes. I'm not sure how this extends to the case of open-ended generation where multiple responses may be semantically equivalent or have different logical relationships.
* While experiments show improvements in OOD detection when moving from OBQA--> ARC,CSQA datasets, I'd be interested to see the performance when the datasets are 'futher apart' semantically (e.g. moving from a reading comprehension to a math task).
Yes
No ethics review needed.
Please refer to the strengths and weaknesses. Additionally, how many bins did the authors use to compute the ECE? It is beneficial to conduct a sensitivity analysis on the number of bins.
The paper is well-written and easy to understand, trying to address the important task of calibrating fine-tuned large language models. The empirical experiments are comprehensive, testing across multiple LLMs including Llama2-7B, Llama3-8B, and Mistral-7B. The paper covers most of the baselines. The LoRA-based Mixture of Experts idea [1] could also be relevant and serve as a baseline for comparison. The empirical performance appears promising.

[1] Li, Dengchun, et al. "Mixlora: Enhancing large language models fine-tuning with lora based mixture of experts."
The paper presents the Information Bottleneck (IB) - Evidential Deep Learning (EDL) method, designed to regularize EDL for better calibration of large language models (LLMs). The empirical experimental results demonstrate that IB-EDL enhances calibration in fine-tuned LLMs.
The novelty of the paper is limited, as both EDL and the IB method are well studied in the literature. The primary contribution of the paper appears to be the implement IB on EDL. The authors explain that the IB objective allows the latent variables to retain the most predictive information about the target variable while discarding irrelevant information from X. However, a more detailed discussion on how this relates to improved calibration of LLMs is missing. Additionally, there is a lack of theoretical guarantees supporting this approach. It is beneficial to provide a theoretical bound on calibration error or a proof of convergence.

In terms of the empirical analysis, the paper highlights computational efficiency as a key contribution of the proposed method, yet it lacks experimental comparisons of computational time and memory costs to validating this claim. The paper also lacks the calibration performance analysis in out-of-distribution scenarios.
This paper introduces an information-theoretic regularization for evidential deep learning, substantially enhancing the calibration of fine-tuned large language models.
@inproceedings{
anonymous2024calibrating,
title={Calibrating {LLM}s with Information-Theoretic Evidential Deep Learning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=YcML3rJl0N},
note={under review}
}
Fine-tuned large language models (LLMs) often exhibit overconfidence, particularly when trained on small datasets, resulting in poor calibration and inaccurate uncertainty estimates. 
Evidential Deep Learning (EDL), an uncertainty-aware approach, enables uncertainty estimation in a single forward pass, making it a promising method for calibrating fine-tuned LLMs. However, despite its computational efficiency, EDL is prone to overfitting, as its training objective can result in overly concentrated probability distributions. 
To mitigate this, we propose regularizing EDL by incorporating an information bottleneck (IB). Our approach IB-EDL suppresses spurious information in the evidence generated by the model and encourages truly predictive information to influence both the predictions and uncertainty estimates. Extensive experiments across various fine-tuned LLMs and tasks demonstrate that IB-EDL outperforms both existing EDL and non-EDL approaches. 
By improving the trustworthiness of LLMs, IB-EDL facilitates their broader adoption in domains requiring high levels of confidence calibration.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
evidential deep learning; information bottleneck; calibration; large language models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/85dcf012b295d26acf68cd46b7bf9e242123e9ce.pdf
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/c3defaa446269180442be6a8331526d34fc5b49a.zip
Calibrating LLMs with Information-Theoretic Evidential Deep Learning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission