Is Factuality Enhancement a Free Lunch For LLMs? Better Factuality Can Lead to Worse Context-Faithfulness
Large Language Models;In-Context Faithfulness;Factuality Enhancement
We greatly thank the reviewer Fg7q for his/her helpful and insightful comments. We provide our responses to the comments as follows.

*Weakness 1: In Algorithm 1, the None checks for P_new and P_para mean that once these variables are assigned a value, the algorithm stops capturing additional tokens. This setup results in identifying only the first differing token between S_new and S_para, while ignoring any subsequent distinctions. Is this interpretation correct? If so, this approach seems limited because of ignoring the subsequent tokens. Please correct me if I make it wrong.*

Thank you for your insightful feedback on our Knowledge Token Capturing algorithm, which has helped us realize that the algorithm's nuances might not have been fully conveyed.

You are correct; **we only capture the first knowledge-distinguishing token, which means that once it is captured, we no longer focus on subsequent tokens. Instead, we observe the logits distribution and hidden states of this first token only.**

As described in our paper, **we concentrate on the distinctive elements, ignoring repetitive or non-meaningful parts.** To clarify this approach, let’s consider an example. Given the question, “In which country is London located?” an expected LLM output with injected context (“London is a city in America”) might be “Answer: United States,” while the original parametric output without the injected context would be “Answer: United Kingdom.” Here, $S_{new}$ is “Answer: United States,” and $S_{para}$ is “Answer: United Kingdom” in Algorithm 1.

When capturing parametric knowledge from $S_{para} = \text{“Answer: United Kingdom.”}$, the first knowledge-distinguishing token we expect to capture is “King-.” We avoid capturing “Answer:” as it holds no factual meaning, nor do we focus on “United” since it is repetitive and does not reflect the core difference. **Additionally, after capturing “King-,” we do not further consider the following tokens, such as “-dom” or the punctuation.**

Capturing only the first distinctive token, without focusing on subsequent tokens, is a reasonable approach because, as widely acknowledged in existing literature, the first token plays a decisive role in sequence generation. For instance, in jailbreak studies, researchers induce LLMs to produce a non-refusal token initially, which effectively “unlocks Pandora’s box.” A clearer example emerges in the logits distribution of generated tokens. When the LLM is uncertain about which token to generate, there are often two high-probability choices—such as “States” at 0.44 and “King-” at 0.45—while the probabilities of other tokens are close to zero. This reflects an "illusion" of choice between parametric and contextual knowledge, as both tokens appear "equally competitive." However, once we select “King-” as the next token, forming a new context for subsequent iterations, we observe that the likelihood of generating “-dom” next approaches 100%. This pattern is widespread and not limited to word suffixes, reinforcing that once the initial distinguishing token is chosen, the model’s generation follows this preference with strong confidence.

**This demonstrates that once the first distinctive token is generated, the subsequent tokens are generated with conviction, as if the LLM has made a definitive knowledge preference based on that initial choice.** Thus, our algorithm aims to capture the first distinctive token for interpretive exploration, as it best reflects the knowledge preference—whether towards parametric knowledge or contextual knowledge.

*Weakness 2: It could benefit from a forward-looking research agenda or recommendations for addressing this challenge. Practical suggestions or potential strategies for mitigating these trade-offs would strengthen the contribution and provide guidance for future work.*

Thank you for your valuable suggestion on proposing future work, which aligns closely with our original intent to encourage the community to focus on the trade-off between factuality and context-faithfulness, ultimately promoting the development of more trustworthy large language models.

In the final section of our paper, we offer some forward-looking insights. We believe that a high-performing LLM should integrate both factual accuracy and context-faithfulness. Therefore, we recommend that future research efforts focus on devising strategies to effectively and controllably balance these two crucial aspects. By tackling this duality, we can move closer to developing LLMs that provide accurate information while adhering faithfully to contextual cues, thereby enhancing their reliability and practical utility.

We will continue to work in this direction, and a revised version with additional recommendations would indeed strengthen the contribution.

Your constructive input remains invaluable to us, and we appreciate your dedication to enhancing the quality of our manuscript. Once again, thank you for your insightful comments.
Response to reviewer Fg7q
We greatly thank the reviewer YFbN for his/her helpful and insightful comments. We provide our responses to the comments as follows.

*Weakness 1: If I understand it correctly, Algorithm 1 assumes there is one token difference between S_new and S_para. Is there any rationale behind this assumption? Additionally, if this is true, I wonder if the findings in section 5 generalize to settings where S_new and S_para different by more than 1 token?*

Thank you for your insightful feedback on our Knowledge Token Capturing algorithm, which has helped us realize that the algorithm's nuances might not have been fully conveyed.

First, we would like to clarify that **we do not restrict the difference between $S_{new}$ and $S_{para}$ to only one token.** Rather, $S_{new}$ and $S_{para}$ represent the LLM’s textual responses to knowledge queries. For instance, consider the question: “In which country is London located?” An expected LLM output with injected context ("London is a city in America") might be “Answer: United States,” whereas the original parametric output without injected context would be “Answer: United Kingdom.” **Here, $S_{new}$ is “Answer: United States,” and $S_{para}$ is “Answer: United Kingdom.”**

As described in our paper, we focus on the distinctive portions, ignoring repetitive or non-meaningful elements. For example, we avoid capturing “Answer:” as it holds no factual meaning, nor do we focus on the token “United” as it is repetitive and does not reflect the key difference. Instead, our goal is to capture tokens that hold clear factual significance—those that differentiate the newly introduced knowledge from the parametric knowledge inherent in the model. In this example, a token like “Kingdom” serves as a critical marker of distinction, effectively highlighting the divergence between contextual information and the model's existing knowledge.

Capturing the first distinctive token is essential because, as acknowledged in much existing literature, **the first token plays a decisive role in the generation of a sequence.** For instance, in the domain of jailbreak attempts, researchers try to induce LLMs to output a non-refusal token as the first token, which effectively “unlocks Pandora's box.” A clearer example can be observed when logging the logits distribution of generated tokens. For tokens that the LLM is uncertain about, there are often two high-probability tokens—for instance, “States” with a probability of 0.44 and “King-” with a probability of 0.45—while the probabilities of the remaining tokens are close to zero. This scenario highlights an "hallucination" of choice between parametric knowledge and contextual knowledge, as both knowledge tokens appear "equally competitive." However, if we select “King-” as the next token, forming a new context for the next generation iteration, we find that the probability of the subsequent token “-dom” approaches 100%. This pattern is quite common and is not limited to word suffixes.

**This demonstrates that once the first distinctive token is generated, the subsequent tokens are generated with conviction, as if the LLM has made a definitive knowledge preference based on that initial choice.** Thus, our algorithm aims to capture the first distinctive token for interpretive exploration, as it best reflects the knowledge preference—whether towards parametric knowledge or contextual knowledge.

*Weakness 2: I do not find other critical flaws in the paper other than some typos.  For example, a closing parenthesis is missing from equation 2.  In Figure 3, the legend of the green bar should be “Factuality”.  There are some other few types like repeating words across the paper but these do not hinder the understanding of the paper.*

Thank you for your careful review.  We will address the errors you identified and make the necessary corrections to improve our paper in the revised version.

*Question 1：In algorithm 1, how did you obtain S_new and S_para? Are they both contained in the MQUAKE-STUBBORN dataset?*

Yes, the MQUAKE-STUBBORN dataset naturally contains both newly injected contextual knowledge and original factual knowledge, which we use as $S_{new}$ and $S_{para}$ for capturing the corresponding tokens.

Your constructive input remains invaluable to us, and we appreciate your dedication to enhancing the quality of our manuscript. Thank you for your time and consideration.

We sincerely hope that your impression and recognition of our work will be enhanced! Thank you very much!
Response to reviewer YFbN
Yes
No ethics review needed.
- In algorithm 1, how did you obtain S_new and S_para? Are they both contained in the MQUAKE-STUBBORN dataset?
- The findings presented in the paper are interesting and insightful, bringing the trade-off of faithfulness and factuality to the community’s awareness.
- The analysis conducted in this paper is rich and sufficient.
- The paper is overall well written.
The paper discusses the trade-offs between enhancing factual accuracy and maintaining context-faithfulness in large language models (LLMs). The authors argue that current methods aimed at reducing hallucinations and improving factual accuracy in LLMs tend to undermine the models’ ability to remain faithful to the given context. Through a series of evaluations, they find that while factual accuracy might show inconsistent improvement, context-faithfulness suffers significantly, with notable declines observed—up to 69.7%. The analysis of hidden states and logit distributions highlights the limitations of the current approaches, emphasizing the complex balance required between factual knowledge and context adherence. The authors suggest that future research should focus more on minimizing the adverse effects on context-faithfulness when developing methods for enhancing factuality in LLMs.
- If I understand it correctly, Algorithm 1 assumes there is one token difference between S_new and S_para. Is there any rationale behind this assumption? Additionally, if this is true, I wonder if the findings in section 5 generalize to settings where S_new and S_para different by more than 1 token?

- I do not find other critical flaws in the paper other than some typos. For example, a closing parenthesis is missing from equation 2. In Figure 3, the legend of the green bar should be “Factuality”. There are some other few types like repeating words across the paper but these do not hinder the understanding of the paper.
Yes
No ethics review needed.
See the weakness above
1. The papers provides valuable insight on the dual trade-off between factuality enhancement and context-faithfulness, recommendation future research to reduce the sacrifice of context-faithfulness.

 2. The authors employ a comprehensive experimental setup, testing various methods such as contrastive decoding and representation editing.

3. The presentation of the paper is well-organized and clear, facilitating easy comprehension of experimental findings and details.
This paper aims to analyze the trade-off between context-faithfulness and factuality enhancement. They first revisist factuality enhancement method and assess their effectiveness. Then, they perform experiments to show that these methods might have a severe decline in context-faithfulness. They finally provide analysis on hidden states and logic distributions to explain the limitation of current methods.
1. In Algorithm 1, the None checks for P_new and P_para mean that once these variables are assigned a value, the algorithm stops capturing additional tokens. This setup results in identifying only the first differing token between S_new and S_para, while ignoring any subsequent distinctions. Is this interpretation correct? If so, this approach seems limited because of ignoring the subsequent tokens. Please correct me if I make it wrong.

2. While the paper effectively highlights the issue of reduced context-faithfulness with factuality enhancement, it could benefit from a forward-looking research agenda or recommendations for addressing this challenge. Practical suggestions or potential strategies for mitigating these trade-offs would strengthen the contribution and provide guidance for future work
Yes
No ethics review needed.
please see above
+ exploring the interplay between parametric knowledge and context faithfulness is promising
+ the experiments are comprehensive
This work proposes to examine the effect of "factuality enhancement" approaches that better exploit LLM parametric knowledge, specifically representation editing and contrastive decoding approaches. Experiments find that they could lead to better factuality and worse context faithfulness, while analysis of model internals provide some indicators.
- I'm not sure if I could agree with the framing of "factuality enhancement" methods in this work. In the abstract, the authors claim that "factuality enhancement can lead LLMs to become overly confident in their parametric knowledge" (lines 17-18). However, retrieval-augmented generation is also a way of "factuality enhancement", and that certainly steers models away from parametric knowledge to varying extents. Although the broad term of "factuality enhancement" is employed, the investigated approaches are just a narrow subset.

- Following up on the previous point, the included approaches "representation editing" and "contrastive decoding" are not representing broad walks of "factuality enhancement approaches", but rather approaches that specifically steer the models to better tap into parametric knowledge. As models better rely on internals, they are naturally less faithful to external contexts, hence the conclusion that "models become less context-faithful after you apply these approaches" become trivial. Essentially: if you use approaches that emphasize model internals, they will certainly be less reliant on external context, which is not a surprising/consequential finding.

- Aside from representation editing and contrastive decoding, some earlier prompting-based approaches also claim to better elicit and employ parametric knowledge to enhance factuality, such as [1-2]. I'm not sure if they should be included as baselines, but I wonder if they might change the conclusion as to context faithfulness.

- Section 2.1 is somewhat trivial and don't need to appear in the main paper.

- Again, the authors claim that "Current methods for enhancing factuality do not modify the underlying structure or parameters of LLMs." (lines 188-189) Why would factuality SFT [3] and factuality DPO [4] approaches not count as "current methods for enhancing factuality"? Readers might say that this is twisting the definition of "methods for enhancing factuality" to fit a pre-defined conclusion.

- typo on line 499: "Knowledge Conficts"

- Let's say if indeed "factuality enhancement" and "context-faithfulness" are two conflicting objectives. I wonder if this might be fixed by inference-time approaches that enhance context-faithfulness such as [5]. Essentially, take an LLM, do some "factuality enhancement" which might harm context faithfulness, and then apply things like [5] to patch it and get the best of both worlds.

- The model internal analysis part is great and potentially useful. I wonder if the authors could offer some crisp conclusions/takeaways in addition to presenting some numbers and figures. In the abstract it says "To explain these declines, we analyze the hidden states and logit distributions for the tokens representing new knowledge and parametric knowledge respectively, highlighting the limitations of current approaches." In the conclusion it says "By examining the logits distribution and hidden states, we gained insights into the underlying causes of this decline", which are both very vague. What are "the limitations of current approaches"? What are "the underlying causes of this decline"? Most importantly, I wonder if the authors have some actionable next steps or potential fixes as validated by this internal analysis part.

- Perhaps there is a way to tone down the definition of "factuality enhancement" and make the claims/findings more rigorous or better scoped. This would be essential in making a decision about this submission.

[1] Liu, Jiacheng, et al. "Generated Knowledge Prompting for Commonsense Reasoning." ACL 2022.

[2] Sun, Zhiqing, et al. "Recitation-Augmented Language Models." ICLR 2023.

[3] Hron, Jiri, et al. "Training Language Models on the Knowledge Graph: Insights on Hallucinations and Their Detectability." COLM 2024.

[4] Tian, Katherine, et al. "Fine-Tuning Language Models for Factuality." The Twelfth International Conference on Learning Representations. ICLR 2024.

[5] Shi, Weijia, et al. "Trusting Your Evidence: Hallucinate Less with Context-aware Decoding." NAACL 2024.
@inproceedings{
anonymous2024is,
title={Is Factuality Enhancement a Free Lunch For {LLM}s? Better Factuality Can Lead to Worse Context-Faithfulness},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=asGQQc7gNo},
note={under review}
}
As the modern tools of choice for text understanding and generation, large language models (LLMs) are expected to accurately output answers by leveraging the input context.
This requires LLMs to possess both context-faithfulness and factual accuracy.
Extensive efforts have been made to enable better outputs from LLMs by mitigating hallucinations through factuality enhancement methods.
However, they also pose risks of hindering context-faithfulness, as factuality enhancement can lead LLMs to become overly confident in their parametric knowledge, causing them to overlook the relevant input context.
In this work, we argue that current factuality enhancement methods can significantly undermine the context-faithfulness of LLMs.
We first revisit the current factuality enhancement methods and evaluate their effectiveness in enhancing factual accuracy.
Next, we evaluate their performance on knowledge editing tasks to assess the potential impact on context-faithfulness.
The experimental results reveal that while these methods may yield inconsistent improvements in factual accuracy, they also cause a more severe decline in context-faithfulness, with the largest decrease reaching a striking 69.7\%.
To explain these declines, we analyze the hidden states and logit distributions for the tokens representing new knowledge and parametric knowledge respectively, highlighting the limitations of current approaches.
Our finding highlights the complex trade-offs inherent in enhancing LLMs.
Therefore, we recommend that more research on LLMs' factuality enhancement make efforts to reduce the sacrifice of context-faithfulness.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Large Language Models In-Context Faithfulness Factuality Enhancement
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/462db32b04b7839092df26fc0cdf6df4e8402b2d.pdf
interpretability and explainable AI
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Is Factuality Enhancement a Free Lunch For LLMs? Better Factuality Can Lead to Worse Context-Faithfulness
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission