Calibrated Decision-Making through Large Language Model-Assisted Retrieval
calibration;RAG;LLMs
Yes
No ethics review needed.
Please refer to the weaknesses.
1. The model design in the paper appears well-constructed and reasonable.
2. Introducing a confidence score in LLM responses can effectively support human decision-making.
In this paper, the authors build upon Band et al.'s (2024) work on linguistic calibration for long-form generation by designing an LLM-based Calibrated Retrieval-Augmented Generation framework. This framework ensures that decisions informed by the retrieved documents are well-calibrated. The authors validated the effectiveness of their approach in four QA datasets.
1. Section 2 lacks coherent organization, which limits its ability to effectively link the relevant background to the authors’ contributions. For example, the three issues identified at the end of Section 2.1 are not clearly addressed in the proposed approach. It would be helpful to clarify how this approach resolves each of these problems. Additionally, when formalizing the problem in Section 2.1, the authors should explicitly outline its relationship to previous work. Without this context, researchers in automated QA may struggle to understand the distinction between the original question and the open-ended query, as well as the necessity of introducing the open-ended query. Additionally, Section 2.2 should provide a clearer explanation of the calibration error, as it is challenging to intuitively grasp how performance differences impact decision-making outcomes when interpreting Figure 1. Moreover, while Figure 1a illustrates the effectiveness of calibRAG, Figure 1b does not, and despite the presentation of calibRAG’s results in Figure 1a, there is no corresponding description in the text.
2. The authors should expand the discussion on open-ended queries. Specifically, is it necessary for users to create these queries, or could they be generated by the LLM itself? What impact would LLM-generated queries have on the outcomes? Additionally, examining how variations in open-ended queries influence model performance would add valuable insights.
3. The authors incorporate confidence scores in long-form generation to enhance human decision-making. However, there is insufficient empirical analysis to demonstrate this benefit—specifically, a comparison of human judgment accuracy with and without confidence scores is missing. Automated evaluation alone is inadequate to validate this claim. Moreover, while the authors prompt the LLM to express uncertainty using either an integer scale from 0 to 10 or linguistic terms of certainty, the impact of these differing methods on model performance remains unclear. Further analysis is needed to clarify how these approaches influence outcomes.
4. The paper lacks comprehensive experimental validation. Evaluating the model on the complete BEIR benchmark would provide a more thorough assessment. Additionally, how does the model perform in non-zero-shot scenarios? For instance, is the hyperparameter setting—particularly the number of retrieved passages (K)—consistent across datasets, or does it vary? Stage 3 in Section 3.4 requires a more detailed description, specifically regarding how the query is reformulated in the experiments. Furthermore, how is the predefined threshold ϵ selected, and what effect does its value have on performance? The dataset verified by human annotators is too limited to robustly support the conclusions drawn from the experimental analysis. It would also be beneficial for the authors to showcase the performance of different models on human-annotated data.
Yes
No ethics review needed.
Please refer to the details in the Weakness section:

1. Human decision-making is far more complex than described in this paper. How can we assume that if LLMs can handle document selection, they are also capable of decision-making?

2. How do other related baselines perform?
1. Proposes a new dataset by creating labels that indicate whether decisions made using retrieved documents correctly answer the questions, essential for training the forecasting function.

2. The proposed method outperforms existing uncertainty calibration baselines across various tasks involving RAG context using the Llama-3.1 model in decision-making scenarios.
This paper proposes the Calibrated Retrieval-Augmented Generation (CalibRAG) framework. CalibRAG allows an LLM using RAG to not only select relevant information to support user decision-making but also provide confidence levels associated with that information by utilizing a forecasting function, ensuring well-calibrated decisions based on the retrieved documents.
1. Many important baselines in RAG that have the same motivation with this paper are missing in experiments. This paper is actually aims to improve the robustness of RAG, which can select useful documents and prevent interference from noisy documents. So many important baselines such as RetRobust [1], Self-RAG [2], CRAG [3] should be discussed and compared but this paper overlooks them.

2. Technical contributions are limited. This paper uses a surrogate model that predicts the probability of whether the user’s decision based on the guidance provided by RAG will be correct. Many existing methods are similar to this such as Self-RAG [2] and CRAG [3]. I cannot learn any insightful points from this paper that is distinguishable enough from existing methods.

3. The writing should be improved. The Introduction section has so many contents about the existing methods that have been well-known in RAG area while there is too little contents about the methods and contributions for this paper.

4. The term "Decision Making" is too broad to describe only the evidence selection task in open-domain question answering. It is typically used in a much wider context.

[1] Making Retrieval-Augmented Language Models Robust to Irrelevant Context

[2] Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection

[3] Corrective Retrieval Augmented Generation
Yes
No ethics review needed.
* Are there any fundamental differences or novel issues in confidence calibration for Retrieval-Augmented Generation (RAG) compared to calibration in generation models without retrieval augmentation?
- It is a important research topic to calibrate LLM confidence in a retrieval-augmented generation (RAG) system. Confidence calibration of RAG is highly relevant for building trustworthy LLM systems, as in practical applications, LLMs are typically used with input knowledge.
- The proposed method demonstrates significant improvements in the uncertainty calibration task. The experimental setup includes diverse datasets, multiple baselines, and several standard evaluation metrics.
This work focus the problem that the confidently incorrect information provided by LLMs may mislead human decision-making process. The target is to calibrate confidence of LLM responses with the correctness of decision-making based on these responses. The proposed CalibRAG framework, which contains synthetic supervision data generation and training, show notable improvement in confidence calibration and reranking tasks.
- The paper appears to equate "the confidence of LLM response" with "the relevance between the query and the document."
	- During training (Section 3.2), the forecasting function relies solely on information from the query and the document, excluding the actual LLM-generated guidance ($z$). Since the LLM-generated guidance may omit or alter important information from the document, using only the query and document information to substitute for the LLM-generated guidance is unreasonable. Therefore, the proposed method seems more akin to training a specialized rerank model rather than calibrating the confidence of the LLM response.
- The reranking experiments are not entirely fair due to the use of additional fine-tuning data compared to baseline methods. Moreover, the synthetic supervision data uses the correctness of user decisions (surrogate model in this paper) as the supervision signal, which results in a smaller gap and aligns more closely with the testing procedure compared to baseline methods.
- The results in Section 4.3 do not convincingly that the LLM can effectively simulate human decision-making. In the 0-20 and 40-60 bins, the agreement rate is only approximately 60%, indicating that a substantial portion of the data cannot be simulated by the LLM. Considering that in Table 2, the proposed method improves accuracy by about 2% compared to LLM-rerank on the BioASQ dataset, the approximately 30% disagreement between the LLM and human decision-making is significant.
- Figure 3 should be a vector image (import as pdf/svg) to ensure high quality.
Yes
No ethics review needed.
See above.
1. The topic of confidence estimation in RAG systems (for decision making) is an interesting extension of existing work. 

2. In experiments, baselines include both calibration methods and reranking methods, which are two good perspectives to study.
The paper proposes CalibRAG, which extends Band et al. 2024 to the RAG use case, where LLM generates calibrated generations for better decision making. The main component is to train a forecasting function f that provides a confidence estimation of the query and a retrieved document.  Experiments are conducted on 4 existing QA datsets with baselines that were not specifically designed for RAG or reranking methods without considering calibration.
1. The reviewer feels the writing of the paper is not clear.  The paper does not clearly describe the use case (LLM-assisted decision making). More specifically, almost half of abstract and intro are about very basic concepts about LLM and RAG. The discussion of calibration is also very general and does not talk about anything specific in this work. By the end of abstract and intro, the reviewer is not clear what the paper wants to do. Section 2.1 then immediately starts with Band et al. 2024 and the notations become confusing since the problem setup is unclear. The authors seem to assume the readers know Band et al. 2024 very well. In fact, the reviewer has to read that paper to understand what the problem setup is. The authors are suggested to refer to the paper on how to set up the problem.

2. The novelty and technical contribution of this work looks borderline. More specifically, extending to RAG is meaningful, as indicated above. The work heavily builds upon Band et al. 2024. The arguments of the weaknesses of Band et al. 2024 look reluctant expect for the RAG part. To the reviewer, the real major contribution of the work is to the extension to RAG use case, which is not very technically deep (but again, still meaningful). 

3. Many technical details seem to be missing. For example, there's pretty much no details about the cross-encoder baseline. The LLM-rerank baseline is also a very sensitive method. Without more details. the experimental results are not very trustworthy to the reviewer.
In this paper, we propose uncertainty calibration method, called CalibRAG, for the RAG guided decision-making scenario.
@inproceedings{
anonymous2024calibrated,
title={Calibrated Decision-Making through Large Language Model-Assisted Retrieval},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=nNQmZGjEVe},
note={under review}
}
Recently, large language models (LLMs) have been increasingly used to support various decision-making tasks, assisting humans in making informed decisions. However, when LLMs confidently provide incorrect information, it can lead humans to make suboptimal decisions. To prevent LLMs from generating incorrect information on topics they are unsure of and to improve the accuracy of generated content, prior works have proposed Retrieval Augmented Generation (RAG), where external documents are referenced to generate responses. However, traditional RAG methods focus only on retrieving documents most relevant to the input query, without specifically aiming to ensure that the human user's decisions are well-calibrated. To address this limitation, we propose a novel retrieval method called Calibrated Retrieval-Augmented Generation (CalibRAG), which ensures that decisions informed by the retrieved documents are well-calibrated. Then we empirically validate that CalibRAG improves calibration performance as well as accuracy, compared to other baselines across various datasets.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
calibration RAG LLMs
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/d7efb2811ba67816fb7b76770ca8afee86d57f1a.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/aa73011046b2e8473e287bef264d6428db73f230.zip
Calibrated Decision-Making through Large Language Model-Assisted Retrieval
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission