Delta - Contrastive Decoding Mitigates Text Hallucinations in Large Language Models
Contrastive Decoding;Text Hallucination Mitigation;Large Language Models
Yes
No ethics review needed.
N/A
This work tries to tackle an important and interesting problem of reducing regular LLMs' hallucination by modifying the decoding algorithm at inference time. The experiments show good results on SQuAD. The writing is overall clear and structured.
This work aims at proposing an inference-time improvement to the regular decoding algorithms by leveraging contrastive-based decoding. Specifically, the authors propose Delta that contrastively decodes from two input contexts, an original context and a masked context with part of the input removed. Experiments in question answering show that the method can help mitigate hallucinations.
(1) This work misses a reference and comparison to context-aware decoding (Shi et al., 2023). The two methods seem very similar (e.g., Equation (5) in this work is similar to Section 2.2 in Shi et al., where a partial context is removed contrastively). The authors should consider adding a comparison of the two methods' performance and discuss any potential differences between the two. 

(2) This work seems to be experimenting with only one LLM, specifically LLama 3.1 8B. Different model families and different model sizes should be reported. Also, the used Llama 3.1 8B is a non-instructed version. It would be interesting to see experiments with e.g., Llama-3.1-8B-Instruct. 

(3) This work only shows performance improvements in SQuAD. In TriviaQA, Natural Question, CommonsenseQA, and MMLU, the proposed Delta method is worse than the baseline decoding method. 

(4) The mask(z) operation could benefit from exploring more design choices. For example, how would using tokens other than EOS work as the mask token? How would transformations other than masking work (e.g., reordering, summarization/back-translation, randomly copying in irrelevant contexts, etc.)?

References
Shi et al., 2023. Trusting Your Evidence: Hallucinate Less with Context-aware Decoding. https://arxiv.org/abs/2305.14739
Yes
No ethics review needed.
No further questions.
The method is intuitive and easy to implement.
The paper proposes a constructive decoding method to reduce hallucinations, which compares the original logits with the logits computed from corrupt inputs.
1. The paper lacks plenty of related works and baselines. I suggest the author at least compare the following de-hallucination baselines:
    + CAD [1]
    + ITI [2]
    + DoLa [3]
    + AD [4]
    + SEA [5]
2. The method and equation are similar to CAD [1], where the masking part is the corresponding context for ODQA tasks. However, the paper does not mention and compare this method in SQuAD v1.1/v2, TriviaQA, and Natural Question.
3. The random masking leads to incompatible with FlashAttention, preventing its practical applications.
4. The performance improvement of the method is not significant, e.g., a negative improvement on TriviaQA and Natural Question, 0.25 and 0.3 improvements on CommensenseQA and MMLU
5. The paper is not well written.

[1] Trusting your evidence: Hallucinate less with context-aware decoding

[2] Inference-time intervention: Eliciting truthful answers from a language model

[3] Dola: Decoding by contrasting layers improves factuality in large language models

[4] In-context sharpness as alerts: An inner representation perspective for hallucination mitigation

[5] Spectral Editing of Activations for Large Language Model Alignment
Yes
No ethics review needed.
see weaknesses.
Reducing LLM hallucinations is an import research topic and challenging. This paper tries to introduce constrastive decoding (CD) to tackle this problem. The motivation is intuitive and reasonable as CD has been verifyed in other tasks. Different from CD based on Gaussian noise etc. in visual tasks, this paper adopts token-level masking in text LLM. Experiments on SQuAD show the proposed CD method significantly improves the accuracy.
This paper introduces using constrastive decoding to mitigate hallucinations in LLM. The so-called Delta method works by randomly masking the input text and adjusting predictions using the masked output distribution. Experiments are conducted on several well-known benchmarks, such as SQuAD, TriviaQA, NQ, CommonsenseQA, MMLU etc. The method brings +4~6 EM scores on SQuAD and no improvements on other benchmarks. The paper is generally well-written and easy to follow.
Experiments are conducted on several benchmarks. However, the proposed method only works on SQuAD and shows no improvements on otherss. Therefore, the generalization capability of the proposed method is limited. Why the proposed method does not work well on others tasks is unknown as well.

In addition to visual tasks, CD is also used in other NLP tasks, such as  "Contrastive Decoding Reduces Hallucinations in Large Multilingual Machine Translation Models", "Mitigating Hallucinations and Off-target Machine Translation with Source-Contrastive and Language-Contrastive Decoding" etc..   Whether these existing methods is effective in LLM is unknown. It's better to compare these methods  (or some) with the proposed one to verfiy its effectiveness. 

Ablation studies on technique choices are missing. For example, why chosing masking tokens instead of other variations? what's the best masking strategy?  how the multiply factors in  eq(3) are determined? Why the APC is necessary and how to choose the beta value? etc.
Yes
No ethics review needed.
1. Have the authors considered an adaptive masking mechanism where the degree of masking adjusts based on the input’s complexity or ambiguity level? In addition, masking important tokens (e.g. entity) would be a better alternative than random masking. There is a concurrent work with very similar idea [1].
2. Table 1 and Figure 2 are quite duplicated. Why is Figure 2 needed?
3. The study uses the Llama 3.1 8B model with 4-bit quantization, which may not be the most competitive model. Had the authors considered conducting further experiments using larger models?



**Reference**

[1] Gema et al. DECORE: DECODING BY CONTRASTING RETRIEVAL HEADS TO MITIGATE HALLUCINATIONS
- Delta provides an efficient, inference-only solution to reduce hallucinations, avoiding retraining and keeping implementation straightforward.

- The method shows some promising result in improvement question-answering tasks.
This paper introduces Delta, an inference-time approach to mitigate hallucinations in large language models (LLMs) without requiring model retraining. The method leverages contrastive decoding by randomly masking portions of input prompts and comparing the output distributions of masked and unmasked inputs. By dynamically adjusting logits during token generation, Delta aims to reduce hallucinated content, particularly in context-driven question-answering tasks. Experimental results on datasets like SQuAD demonstrate significant improvements, with the approach showing particular strength in handling ambiguous contexts and unanswerable questions, achieving up to 14.56% improvement in exact match accuracy for scenarios with no definitive answers.
- The text sequence masking is currently using random masking. This is a very basic approach. A better approach would be masking important tokens (e.g. entities). 
- Although Delta performs well in ambiguous question-answering datasets like SQuAD v1.1 and v2, its effectiveness is less pronounced in tasks like TriviaQA and Natural Questions, where factual recall or logical inference is needed. 
- The exact match and F1 score improvements in CommonsenseQA and MMLU are marginal, which may limit Delta's broader application.
- Delta is proposed to mitigate hallucination, however, it is evaluated on QA datasets. The method should be extended to other tasks for example, summarization, where hallucination is known to be a problem. 
- The baseline in the paper is just the vanilla Llama 3 model. Delta should be compared with other baselines with similar approach which is missing in the current work:
  1. Contrastive decoding using multiple input contexts (including relevant and irrelevant) [1]; 
  2. Context aware decoding (CAD) [2]



**Reference**

[1] Zhao et al. Enhancing Contextual Understanding in Large Language Models through Contrastive Decoding

[2] Shi et al. Trusting Your Evidence: Hallucinate Less with Context-aware Decoding
Yes
No ethics review needed.
1. Have you evaluated the performance of this method with different hyperparameter combinations on lower-performing benchmarks, such as CommonsenseQA and MMLU?
1. The paper is easy to follow.
2. The method is computationally efficient, making it suitable for integration into existing systems where minimal adjustments are preferable.
3. The experimental setup on SQuAD and other datasets is well-executed.
This paper introduces Delta, an inference-time approach designed to reduce hallucinations in LLMs without requiring model retraining or additional data. Delta utilizes contrastive decoding by masking parts of the input prompt, and then comparing the output distributions generated from masked and unmasked inputs. Through this mechanism, Delta aims to detect and mitigate hallucinations effectively. The method was evaluated on multiple benchmark datasets, including SQuAD v1.1 and v2, and showed performance improvements of up to 6% in hallucination-prone scenarios.
1. In this method, the tokens to be masked are selected randomly, which may lead to inconsistent performance.
2. Delta’s performance declines in tasks that rely more on pre-trained knowledge than contextual cues, such as CommonsenseQA and MMLU. This limits its applicability in domains that require factual correctness without strong context-based cues.
3. The method's effectiveness appears sensitive to hyperparameter settings, including mask and logit ratios. This limits the method's practicality because it requires setting different hyperparameters for different tasks.
@inproceedings{
anonymous2024delta,
title={Delta - Contrastive Decoding Mitigates Text Hallucinations in Large Language Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=HNbNcDsgxL},
note={under review}
}
Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language processing tasks. Still, they are prone to generating hallucinations—factually incorrect or fabricated content that can undermine their reliability, especially in high-stakes domains such as healthcare and legal advisory. In response to this challenge, we propose Delta, a novel inference-time approach that leverages contrastive decoding to mitigate hallucinations without requiring model retraining or additional training data. Delta works by randomly masking portions of the input prompt, then contrasting the original and masked output distribution generated by the model, effectively mitigating hallucinations through inference-only computations. Delta was evaluated across multiple benchmark datasets, including SQuAD v1.1 and v2, concerning 4 and 6 percent improvements. Delta demonstrated substantial advancement of 14.56 percent more extract match outcome with no definitive answers within the SQuAD version 2 benchmark. These findings suggest that Delta is particularly effective when hallucinations arise from contextual ambiguity. Delta presents a computationally efficient and scalable solution for reducing hallucinations in real-world LLM applications by focusing on inference-time enhancements.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Contrastive Decoding Text Hallucination Mitigation Large Language Models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/557e17d6728a2331695f9fbecb6aaf2737f6b79f.pdf
other topics in machine learning (i.e., none of the above)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Delta - Contrastive Decoding Mitigates Text Hallucinations in Large Language Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission