KnowHalu: Multi-Form Knowledge Enhanced Hallucination Detection
Hallucination Detection; Large Language Model; Factual Checking; Multi-Form Knowledge
Yes
None
No ethics review needed.
In Tables 4 and 5, does KnowHalu perform better in unstructured scenarios than in structured ones? However, doesn't Table 3 present the opposite conclusion?
The "Off-Target Hallucination Checking" concept is indeed compelling as it tackles a critical shortcoming in the output of language models. The detection pipeline for addressing these hallucinations is not only effective but also yields encouraging results across question answering and text summarization tasks. Furthermore, the methodology section is articulated in a clear and comprehensible manner, which enhances the understandability of the approach taken.
This paper presents KnowHalu, a novel hallucination detection framework leveraging multi-form knowledge to enhance the factual accuracy of Large Language Models (LLMs) and mitigate the generation of hallucinated content. KnowHalu employs a two-phase approach: initially identifying off-target hallucinations through semantic alignment, followed by a multi-step, knowledge-based fact-checking process that includes reasoning, query decomposition, knowledge retrieval, optimization, judgment generation, and aggregation. Extensive evaluations demonstrate KnowHalu's superiority over state-of-the-art baselines, with over 15% improvement in question answering tasks. The study introduces the concept of off-target hallucinations and explores the impact of query formulations and knowledge forms on detection accuracy, proposing an aggregation method to refine judgments based on different knowledge forms. While KnowHalu shows promising results on standard datasets, the experimental section could be enhanced, particularly in handling more complex dialogues and longer responses, and the analysis could benefit from deeper insights into the nuances of hallucination detection.
The shortcomings of this paper are quite apparent. First, it lacks a thorough analysis of the RAG data sources. For instance, it does not address the specific impact of the knowledge base (KB) data source on the results, nor does it consider how data missingness or redundancy might affect the outcomes. Additionally, it would be beneficial to explore whether data conflicts could impact the model's robustness.

Moreover, Section 3.2 presents a rather lengthy description of the methods. It would be advisable to streamline this section to enhance overall clarity.
Yes
No ethics review needed.
See weakness.

1.Regarding the experimental results presented in Table 2, it is noteworthy that GPT-3.5 does not exhibit superior performance compared to other models. Isn't that somewhat counterintuitive?

2.The overall framework appears to combine various manually designed prompting strategies with established fact-checking techniques. For example, the approach involves decomposing the question into smaller components and verifying each part through retrieval on a one-by-one basis. From your view, what's the difference in the current hallucination detection task and the original fact-checking task? 

I would like to improve my score if my aforementioned questions are answered properly.
1.The figure is presented in a clear and organized manner, effectively incorporating a wealth of details that contribute to KnowHallu’s comprehensibility.

2.The proposed framework demonstrates effectiveness based on a substantial number of experiments conducted.
This paper presents a novel approach, referred to as KnowHallu, for detecting hallucinations in texts generated by large language models (LLMs). Firstly, it isolates a new type of hallucination, referred to as the off-target hallucination, and then it conducts a novel multi-form knowledge-based fact-checking through a comprehensive four-stage pipeline. Experimental results demonstrate the effectiveness of the proposed framework for detecting hallucinations.
1.Lack of clarification of the concept of ‘off-target’ hallucinations. The author claims in the abstract that they introduce a new category of hallucinations. However, it is noteworthy that prior research, such as HaluEval, has already categorized model responses into "unverifiable", "non-factual" and "irrelevant" classifications. Furthermore, earlier studies focusing on fact-checking have highlighted that not all statements are suitable for this process, which has led to the emergence of the claim detection task. Therefore, why is the concept of off-target hallucination emphasized in this context? Additionally, I am interested to know whether there is statistical data in Table 1 regarding the percentage of different types of questions present in actual LLM responses.

2.Usage of testing tasks. The original HaluEval framework includes both Dialogue and General scenarios; however, it is unclear why these two scenarios were not included in the current evaluation. The proposed design of step-wise reasoning is a common approach in addressing multi-hop questions. However, it raises the question of its applicability across all hallucination detection tasks.
Yes
No ethics review needed.
1. Is there a clear and strict definition of Off-Target Hallucination?
2. What is the reason for the performance difference of two query formulations? How are the two queries generated?
1. The authors conducted sufficient experiments and the method description is generally clear. The motivation of the proposed method is intuitive
2. The paper includes in-depth analyses of various components of KnowHalu. These studies may provide valuable insights into the contribution of each component.
This paper proposes a KnowHalu method to detect hallucination, which is roughly divided into two stages, including off-target hallucination detection and multi-step facutal checking. The authors conducted experiments on multi-hop QA and text summariztion tasks on the HaluEval dataset. Three LLMs including Mistral-7B, Starling-7B and gpt-3.5-turbo-1106 were used. The experimental results show that the accuracy is improved compared with baseline models such as self-consistency that do not rely on retrieval evidence.
1. Although the proposed multi-step reasoning process decomposes the query into multiple sub-queries to improve the detection accuracy, it involves multiple steps of knowledge retrieval, reasoning and aggregation, but it introduces engineering complexity and difficulty in reproduction, such as shared key-value caching and multi-form queries. I suggest that the authors open source the code and provide detailed reproducible instructions. While some latency comparisons are provided, a more thorough analysis of the computational overhead compared to simpler approaches would be valuable, especially for real-time applications.
2. The evaluation relies heavily on the HaluEval dataset, where hallucinations are generated by chatgpt and may not fully represent hallucination behavior in real-world scenarios. I would like the authors to test on a wider dataset with more sources of hallucinations or real sources of hallucinations (1000/500 given the limited size of the test set).
3. In addition, I hope that the author conduct manual evaluation tests on the detected hallucinations and reasoning processes, and conduct corresponding error analysis to gain insight into their limitations and potential improvement directions, such as errors in early stages (query or knowledge retrieval, rewriting queries) may be amplified through the pipeline. Robustness analysis of cascading errors is needed.
4. The authors are encouraged to take a systematic exploration of how different scales of model size affect performance.
5. The writing and presentation still need improvement. Such as Line 222 - one-hope, the size of Figure 7, Table 10, Table 15
We introduce "KnowHalu", a novel approach with two main phases (off-target hallucination detection and multi-step factual checking) for detecting hallucinations in text generated by LLMs, leveraging multi-form knowledge for factual checking.
@inproceedings{
anonymous2024knowhalu,
title={KnowHalu: Multi-Form Knowledge Enhanced Hallucination Detection},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Kap9vaGKwF},
note={under review}
}
As large language models (LLMs) become increasingly integral to a wide array of applications, ensuring the factual accuracy of their outputs and mitigating hallucinations is paramount. Current approaches, which primarily rely on self-consistency checks or post-hoc fact-checking, often fall short by disregarding the nuanced structure of queries and the diverse forms of contextual knowledge required for accurate response generation.
To address these shortcomings, we introduce KnowHalu (pronounced “No Halu”), the first multi-form knowledge-based hallucination detection framework. We also introduce a new category of hallucinations, off-target hallucinations, which occur when responses are factually accurate but irrelevant or nonspecific to the query (e.g., answering "What’s the primary language in Barcelona?" with "European language").
In particular, KnowHalu employs a rigorous two-phase process to detect hallucinations. In the first phase, it isolates off-target hallucinations by analyzing the semantic alignment between the response and the query. In the second phase, it conducts a novel multi-form knowledge-based fact-checking through a comprehensive pipeline of reasoning and query decomposition, knowledge retrieval, knowledge form optimization, judgment generation, and judgment aggregation.
Extensive evaluations demonstrate that KnowHalu significantly surpasses state-of-the-art (SOTA) baselines across diverse tasks, achieving over 15% improvement in question answering (QA) and 6% in summarization tasks when applied to the same underlying LLM. These results underscore the effectiveness and versatility of KnowHalu, setting a new benchmark for hallucination detection and paving the way for safer and more reliable LLM applications.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Hallucination Detection; Large Language Model; Factual Checking; Multi-Form Knowledge
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/721674752b29a541fc6ef0c0838701e8d69821a5.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
KnowHalu: Multi-Form Knowledge Enhanced Hallucination Detection
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission