Hallucination Detox: Sensitive Neuron Dropout (SeND) for Large Language Model Training
LLMs;Hallucinations;Dropout;Reliability;Efficiency
Yes
No ethics review needed.
1. What guided the specific selection of neuron dropout parameters (e.g., dropout percentage, sensitivity threshold)? Could the authors provide insights into how the dropout parameters for SeND were chosen? Was there an empirical process for selecting these values, and did the team explore different configurations to determine the optimal settings? 
2. What impact do Sensitive Neurons have on downstream tasks, especially when a high percentage is dropped?
3. Can the authors share any qualitative examples of how SeND changes model outputs? Including specific examples of model outputs before and after training with SeND, particularly for hallucination-prone prompts, would help illustrate the model’s qualitative improvements.
1. Innovative Approach: Introduces SeND, a new training method, and EES, an efficient hallucination detection metric.
2. Robust Evaluation: Demonstrates SeND’s effectiveness across multiple models and datasets.
3. Computational Efficiency: EES is scalable, supporting application in large LLMs without adding significant computational costs.
4. Clear Methodology: The paper clearly explains the theoretical background and provides step-by-step details for SeND implementation.
The paper introduces Sensitive Neuron Dropout (SeND), a novel training protocol aimed at reducing hallucinations in large language models (LLMs) by minimizing variance during training. Unlike existing post-hoc hallucination mitigation methods, SeND operates during training, specifically targeting neurons—referred to as Sensitive Neurons—that exhibit high variability across training epochs. By selectively dropping these neurons, SeND helps stabilize model outputs, thereby enhancing factual confidence and reducing the likelihood of confabulations, which are hallucinations where models inconsistently produce factually correct and incorrect responses.
1. While the paper introduces Efficient EigenScore (EES) as an approximation of the EigenScore metric for hallucination detection, it largely focuses on a single metric. Expanding the scope of metrics could provide a more comprehensive understanding of SeND’s performance. For instance, incorporating metrics like Semantic Entropy or FactScore alongside EES would allow a nuanced evaluation of hallucinations across different aspects of factuality and consistency. 
2. The paper’s experimental setup lacks an ablation study on SeND’s dropout parameters, such as the percentage of neurons dropped and the interval for identifying sensitive neurons.
3. Although the paper tests SeND on the Pythia model series, this restricts its applicability to similar architectures. Testing SeND on diverse LLM architectures, such as LLaMA, would better establish its generalizability across model types with varying parameters and configurations.
Yes
I believe there are no ethical concerns in this paper.
No ethics review needed.
See the Weaknesses
● Innovation: This study proposes a new training protocol, SeND, which may have a significant impact on the reliability and security of LLMs, making it an important research area.
● Practical application: Empirical evaluations on Wikipedia and medical datasets have demonstrated the potential of SeND in improving factual accuracy, which is particularly important for applications in high-risk industries.
● Computational efficiency: The development of EES has significantly improved the computational efficiency of hallucination detection, which is particularly important for large models and datasets.
● Paper writing: This paper has a smooth writing structure and clear logical expression, allowing readers to quickly understand the relationship between this paper and previous related works.
In this paper, the authors are attempting to solve the hallucination problem called confabulations where the LLM generates different responses given the same or similar inputs. Specifically, the authors propose two main contributions including the triaining protocal named Sensitive Neuron Dropout (SeND) and the enhanced unsupervised hallucination detection metric namd Efficient EigenScore (EES).
In SeND, a novel training protocol, aimed at alleviating the phenomenon of hallucinations in large language models (LLMs) by reducing the variance during the training process. This method reduces the variance of illusions and enhances the factual certainty of the model by deterministically discarding neurons with significant variability on the dataset (known as sensitive neurons) as a regularization technique. The developed an unsupervised hallucinations detection metric EES that is twice as fast as traditional EigenScore while minimizing its impact on accuracy. This efficient metric is integrated into the SeND protocol, making SeND computationally scalable and effective in reducing illusions. In the experiments, the study demonstrated that its method improved the reliability of LLMs during testing, increasing reliability by up to 40% compared to normal training, and providing an effective approach to improve real-world accuracy when adapted to fields such as Wikipedia and medical datasets.
● Lack of discussion on other training stages. The authors assume that the existing research that focuses primarily on post hoc detection and mitigation strategies. However, the training stages in current works mainly contain three import paradigms including pre-training, continue pretraining and SFT. All of them may produce the hallucination phenomenon, and thus the discussion about other two training stages should be considered.
● To evaluate the OSCILLATORY BEHAVIOUR, the authors use the two tasks including self-consistency and summarization, the other important metrics (e.g., PPL) or tasks (e.g. QA) should be considered.
● Mismach parameters size between SENSITIVE NEURONS discussion and main experiments. In experiments settings, the range of paramerts' size of LLMs is from 70M to 12B. However, the theoretical analysis and experimental results of SENSIIVE NEURON are conducted using the Pythia 1B model in the main body (Sec. 3), and there is concern about the lack of generalization of the SeND to larger scales model.
● The effectiveness of SeND experiment is weak. Firstly, the authors only select two datasets (general domain and medical domain), and the effectiveness of this method needs to be proven on more authoritative hallucination benchmarks. In addition, as shown in Fig. 4, the FT method is not too weak compared to the SeND, and thus more datasets are needed to prove the effectiveness of the method.
Yes
No ethics review needed.
1. Could the authors provide results on a significantly larger test set beyond 100 datapoints?
2. What is the correlation between EES improvements and actual hallucination reduction as measured by standard benchmarks?
3. How does SeND compare to other hallucination reduction methods?
The paper proposes a novel approach tackling hallucinations during training rather than post-hoc, representing a interesting shift in addressing this critical challenge.
The foundation is solid, with clear mathematical derivations for both SeND and EES.
The development of EES shows practical value by providing a computationally efficient approximation for hallucination detection with demonstrated speedup.
This paper proposes a novel training protocol called Sensitive Neuron Dropout (SeND) to address hallucinations in Large Language Models (LLMs). The work presents three main contributions: (1) empirical validation of oscillatory hallucination behavior during training, (2) development of SeND for reducing hallucination variance during training, and (3) introduction of Efficient EigenScore (EES), a computationally efficient approximation of EigenScore for hallucination detection. While the theoretical framework is interesting, the empirical validation relies heavily on proxy metrics and limited evaluation data, making it difficult to assess the real-world impact on hallucination reduction.
The empirical evaluation is severely limited with only 100 test datapoints and lacks validation on more than one established hallucination benchmarks, like HaluEval, raising concerns about result reliability.
The work relies heavily on EES as a proxy metric without sufficient evidence that improvements in EES correlate with actual reduction in model hallucinations.
Yes
No ethics review needed.
1. For line 227, the meaning of H needs to be further explained.
2. For line 229, a citation is needed to support the claim.
3. Based on my understanding, Sensitive Neurons refers to specific indices, rather than neurons. This name could be misleading.
4. For line 284, the details of removing operation are unclear.
1. Different from existing post-hoc detection and mitigation strategies, this paper focuses on the relationship between the training process and the emergence of hallucinations, trying to provide interpretation from a relatively new perspective.
2.  Before introducing the specific method, this paper conducts motivational experiments, consolidating the rationale of the method.
This paper empirically validates the oscillatory nature of hallucinations during the training process of LLMs, despite being discovered by previous work. Subsequently, this paper introduces Sensitive Neuron Dropout (SeND), a training-time method for hallucination reduction, and  Efficient EigenScore (EES), a more efficient hallucination detection metric.
1. Despite focusing on hallucination, this paper does not test on any hallucination dataset and metrics directly, but utilizes some indicative alternatives. This reduces the credibility of the results.
2. Despite the claims of reduced hallucination, it's unclear whether this technique would hinder the performance of models.
3. For LLMs, it's rare to train models for several epochs to prevent overfiting and catastrophic forgetting, while it seems that this method can only be used for multi-epoch training settings.
4. Xsum is not suitable to evaluate hallucination of LLM, and Rouge1 score is a bit out-of-date/ineffective to evaluate the performance of LLMs.
5. The writing for "Sec. 1.2 Related Work" is quite strange. Here, the 2nd and 4th paragraphs focus on motivation and implementation details instead of the comparison with peer methods.
6. In Sec. 2.2, I observed that generally, the metrics change positively with the increase of LLM sizes, inconsistent with the observations of authors. Could you please provide further explanation?
7. The number of models and datasets is too small (i.e., only 1) to validate the robustness of the method.
8. There is a lack of baseline and performance comparison with post-hoc solutions.
We introduce SeND, a method that reduces hallucinations in language models by dropping sensitive neurons, and EES, a faster detection metric, improving efficiency and reliability by up to 2x and 40%.
@inproceedings{
anonymous2024hallucination,
title={Hallucination Detox: Sensitive Neuron Dropout (Se{ND}) for Large Language Model Training},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=6L8OdH5PBu},
note={under review}
}
As large language models (LLMs) become increasingly deployed across various industries, concerns regarding their reliability, particularly due to hallucinations—outputs that are factually inaccurate or irrelevant to user input—have grown. Our research investigates the relationship between the training process and the emergence of hallucinations  to address a key gap in existing research that focuses primarily on post hoc detection and mitigation strategies. Using models from the Pythia suite (70M–12B parameters) and several hallucination detection metrics, we analyze hallucination trends throughout training and explore LLM internal dynamics. We introduce SEnsitive Neuron Dropout (SeND), a novel training protocol designed to mitigate hallucinations by reducing variance during training. SeND achieves this by deterministically dropping neurons with significant variability on a dataset, referred to as Sensitive Neurons. In addition, we develop an unsupervised hallucination detection metric, Efficient EigenScore (EES), which approximates the traditional EigenScore in 2x speed. This efficient metric is integrated into our protocol, allowing SeND to be both computationally scalable and effective at reducing hallucinations. Our empirical evaluation demonstrates that our approach improves LLM reliability at test time by up to 40\% compared to normal training while also providing an efficient method to improve factual accuracy when adapting LLMs to domains such as Wikipedia and Medical datasets.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
LLMs Hallucinations Dropout Reliability Efficiency
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/e82f95f11ff5efadf81b89ec2453c2adb9bed2a2.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Hallucination Detox: Sensitive Neuron Dropout (SeND) for Large Language Model Training
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission