SCOPE: A Self-supervised framework for Improving Faithfulness in Conditional Text Generation
faithfulness;hallucination;conditional text generation;natural language processing;large language models
Yes
No ethics review needed.
We are moving towards general purpose models, instead of a task-specific model. How does the proposed method generalize to other tasks, while achieving higher factuality/faithfulness for data-to-text generation and summarization? Will it be much harder to tune the value of alpha when tried with multiple tasks together?

SFT on a specific task often  suffers from catastrophic forgetting. It would be interesting to see how post training baselines do on these tasks. The post training often aims to improve faithfulness along other criteria. 

Could you add a detailed analysis of synthetically generated data wrt faithfulness and fluency? Also how does the proposed method compare with other synthetic data generation methods? 

Reference responses often have hallucinations. For example, XSum gold summaries often have hallucinated content. How does this effect the data generation? How do you ensure that an example is a positive example with respect to faithfulness?
The paper is trying to address generative hallucination, a common problem in generative models. The paper is fairly well written and easy to follow. 

The paper brings new insights on the behavior of preference-tuning with the synthetically generated negative examples.

The proposed approach appears to improve response faithfulness, at a minor cost of fluency.
LLMs often struggle to generate fully grounded information to a provided context. There is a rich literature on the hallucination issues in generative models (https://arxiv.org/abs/2005.00661). This has become an even bigger priority as LLMs are being adapted widely. 

This paper addresses the hallucination issues in LLMs and proposes a self-supervised method for synthetically generating a training set of negative responses. Using these negative examples, models are then trained to encourage the generation of grounded outputs over unfaithful ones using preference-based training. 

The authors empirically demonstrate that the proposed method helps models generate more faithful texts, when evaluated automatically and by humans, on data-to-text generation and summarization.
The method for synthetically generating negative examples is not novel, as the authors claim. Several papers (see some pointers below) in the past have used similar methods for generating negative or diverse sets of samples to better calibrate their models. 

The paper should present a detailed analysis of the synthetic data being generated. For example, it would be nice to see faithfulness score distributions in negative examples. Also a qualitative analysis showing types of hallucinations being generated would have been very interesting. 

Faithfulness is a pointwise metric, but the human evals mostly focus on SxS ratings. It would have been nice to see pointwise evaluations of faithfulness. 

It would have been nice to see comparisons to other ways of generating negative examples, for example, prompting models to generate negative examples.

Other interesting work in this space that might be interesting to discuss here: 

Brio or Slic: https://arxiv.org/abs/2203.16804, https://arxiv.org/abs/2210.00045
Calibrating models for faithfuness: https://arxiv.org/pdf/2310.08764
https://renzhaochun.github.io/assets/pdf/26596-Article%20Text-30659-1-2-20230626.pdf
https://arxiv.org/pdf/1910.08684 from the point of sub-sequence sampling
Comparison with CoT or planning based methods for controlling hallucinations.
Yes
No ethics review needed.
1. Have you investigated whether the method's effectiveness varies with model size? The limitation to 7B models leaves open questions about scalability.
2. The choice of α in [0.4, 0.6] seems empirically driven. Could it vary for different tasks / models?
- They study an important problem in the subfield -- factuality and faithfulness issue in conditional text generation. The research is motivated.
- Interesting two-stage training pipeline, and the approach to generate synthetic unfaithful samples.
- Provides analysis of preference training dynamics as well as the ablation for the noise parameter $\alpha$
Scope is a self-supervised framework for improving the faithfulness of conditional text generation in LLMs. The approach uses a novel two-stage process: first fine-tuning on half the dataset, then using a mixture of that model and the base model to generate synthetic unfaithful samples for preference learning. The authors evaluate their method on data-to-text and summarization tasks.
- The preference learning stage uses β=0.1 for all models and datasets (A.2) without justification or ablation, despite this being a critical hyperparameter in preference learning.
- No evidence the method works on larger models. For their main claimed contribution ("improving faithfulness" in finetuning), testing on 7B models seems insufficient. Is it possible to include a few larger models, e.g. 14B, if not 70B scales.
Yes
No ethics review needed.
What are the impacts of training set size on the SCOPE methodology? It seems that for a small dataset, utilizing half of the dataset for finetuning and utilizing the other half for preference optimization may not be as effective as traditional finetuning.

Is there a way to rate the quality of the dispreferred samples in 3.2? For example, are there generated texts with more inconsistencies that are worse than other samples? Is there an impact on the potential difference in quality of the generated texts?
Relatively novel methodology that uses the process of collaborative decoding for the purpose of generating dispreferred text. The method of collaborative decoding has been used in the past to combine text from a general LLM and an expert LLM to improve performance. In this setting, however, collaborative decoding is utilized to generate inconsistencies in the text randomly based on a threshold alpha.

Experiments in the tasks of summarization and data-to-text in several datasets which highlight the strength of SCOPE compared to traditional finetuning and other baseline decoding methods. Additionally, the large variety of metrics, human evaluation, and GPT-4 evaluation indicates that SCOPE is effective at increasing the faithfulness of the model for these two specific tasks.
This paper introduces a new learning framework for conditional text generation for the purpose of increasing the faithfulness of generated text with respect to the context input. The learning framework consists of two main steps: finetuning and contrastive learning. For contrasting learning, dispreferred samples are created in an unsupervised manner by utilizing a decoding method for generating certain tokens from the untrained language model, thereby introducing tokens which are not contextually consistent with the task. In several experiments, SCOPE is shown to be more effective than other baselines, including various decoding methods. SCOPE shows to be more faithful than traditional finetuning and SCOPE output is more preferred by humans and by GPT-4 judges.
Lack of human analysis of the unsupervised generated data: it is unclear whether the the dispreferred samples are worse due to the changes in the language distribution, or if the dispreferred samples contain information or text that is inconsistent with the context. If the unwanted generation mainly comes from the changes in the language distribution, then SCOPE is not learning to be more faithful to the context. This is a key distinction which may limit the interpretability of SCOPE for being a more faithful method of conditional text generation. One way to mitigate this concern would be to conduct a human analysis on a sample of generated data and determine if the generated data contains text inconsistent with the context or if the inserted text from the base LM is merely different in language distribution.

Lack of comparisons to instruction-tuned models. The methodology of SCOPE, at its essence, is very similar to the standard method of reinforcement learning with human feedback, where the model is first finetuned and then preference optimized with binary data consisting of a preferred sample and an unwanted sample. Due to the similarities, it would be valuable to compare SCOPE to RLHF methods and compare the degree of faithfulness of the model. There is evidence that instruction-tuned models are more "helpful" to users' inputs in the prompt, which correlates highly with faithfulness. Using the instruction-tuned model as a baseline, or running SCOPE on an instruction-tuned model is important to validate SCOPE as a novel, effective method.
Yes
No ethics review needed.
1. In the noisy generation step, is fluency in the output ensured? Since each token is selected based on the weighted sum of the original and fine-tuned LM distributions, could it happen that the token with the highest probability from the combined distribution is not among the top tokens in either individual distribution? For instance, if the two models’ distributions differ significantly, could an unsuitable token receive a relatively high score in each distribution and then emerge as the highest-scoring token after the weighted sum?
2. Is the generated negative examples all unfaithful? it would be useful to provide statistics or evaluation results indicating the proportion of genuine negative samples. This could potentially be estimated using the evaluation metrics applied in the experiments.
1. The research problem addressed in this paper is significant, as faithfulness remains a major challenge for contextual generation in LLMs.
2. The authors propose an innovative method for generating negative examples for DPO training, utilizing two models—the fine-tuned model and the pre-trained model—selecting the next token based on a weighted combination of their distributions. This approach is independent of external tools like NER.
3. The experimental results demonstrate that their method outperforms baseline models across multiple tasks.
The paper investigates the common issue of hallucinations in large language models (LLMs) during conditional text generation. To enhance faithfulness, the authors introduce SCOPE, a novel self-supervised fine-tuning approach that generates unfaithful samples and then trains the model to prefer context-grounded outputs over these fabricated examples. By using this approach, SCOPE achieves substantial improvements in generating grounded and faithful responses, as validated by automatic metrics and evaluations from both LLMs and human judges. The study's findings demonstrate SCOPE’s effectiveness across diverse tasks, including data-to-text and summarization, where it consistently outperforms existing self-supervised techniques.
1. Some of the baselines appear to be training-free decoding methods, while others require training, similar to the method proposed in the paper. The authors should clarify which methods are training-free and which are not, as they require different resources to implement.
2. Additional ablation studies are needed, such as applying the same training framework with other negative sampling methods like NER replacement or negation. Otherwise, the effectiveness of the negative sampling itself remains uncertain.
3. Experiments should include the faithfulness score of the generated negative examples on the training set to directly demonstrate the impact of negative sampling.
Yes
No ethics review needed.
1. What is the rationale behind splitting the dataset into 50-50 for the two phases? Would other split also work?
2. Do you anticipate that SCOPE would perform similarly on domain-specific datasets, like those in healthcare or finance?
3. Can SCOPE’s approach to unfaithful sample generation and preference tuning generalize to larger or different model architectures?
4. The $\alpha$ for CAD is set to very small values for tasks other than XSum. Why is it the case?
5. The main evaluation metrics used are BLEU/NLI/ROUGE/AL. Did you consider any additional faithfulness metrics or error types that might capture different facets of faithfulness?
- An innovative self-supervised approach to improving LLM faithfulness through synthetic unfaithful sample generation
- The method shows strong performance across multiple benchmarks, with notable improvement in data-to-text tasks. 
- The paper explains the methodology and experimental setup well, using helpful figures and clear explanations of core ideas, such as the noisy sample generation and preference tuning.
The paper presents SCOPE, a self-supervised framework that enhances the faithfulness of LLMs in conditional text generation tasks like summarization and data-to-text generation. The framework employs a two-stage approach: initial fine-tuning followed by preference tuning using contrastive learning. The key innovation lies in its method of automatically generating synthetic unfaithful samples by blending grounded and context-free generation, which are then used to train the model to prefer context-aligned outputs over hallucinated ones. Through comprehensive evaluation across multiple datasets, SCOPE demonstrates significant improvements in output faithfulness, achieving up to 14% gains according to automatic metrics, with additional improvements confirmed through both GPT-4 and human evaluations.
- While SCOPE performs well on general summarization and data-to-text tasks, evaluating it on domain-specific datasets (e.g., biomedical or financial) could better demonstrate its robustness in high-stakes settings where hallucinations are especially problematic.
- SCOPE is tested on 7B models only; applying it to models of different sizes would clarify its scalability and help determine whether it generalizes effectively to larger different models. Perhaps the improvement using SCOPE on larger models will be marginal. 
- The evaluation on summarization task used ROUGE and AlignScore (an entailment metric). While SCOPE scores slightly lower ROUGE scores than other baselines, the authors did not use other faithfulness metrics in summarization research to further evaluate.
We propose a self-supervised method for faithfulness enhancement for conditional text generation.
@inproceedings{
anonymous2024scope,
title={{SCOPE}: A Self-supervised framework for Improving Faithfulness in Conditional Text Generation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=dTkqaCKLPp},
note={under review}
}
Large Language Models (LLMs), when used for conditional text generation, often produce hallucinations, i.e., information that is unfaithful or not grounded in the input context. This issue arises in typical conditional text generation tasks, such as text summarization and data-to-text generation, where the goal is to produce fluent text based on contextual input. When fine-tuned on specific domains, LLMs struggle to provide faithful answers to a given context, often adding information or generating errors. One underlying cause of this issue is that LLMs rely on statistical patterns learned from their training data. This reliance can interfere with the model’s ability to stay faithful to a provided context, leading to the generation of ungrounded information. We build upon this observation and introduce a novel self-supervised method for generating a training set of unfaithful samples. We then refine the model using a training process that encourages the generation of grounded outputs over unfaithful ones, drawing on preference-based training. Our approach leads to significantly more grounded text generation, outperforming existing self-supervised techniques in faithfulness, as evaluated through automatic metrics, LLM-based assessments, and human evaluations.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
faithfulness hallucination conditional text generation natural language processing large language models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/a3f8afa87747bb43c9a0bd3b2d081e8d8b16dccd.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/c944ef629113cddf1a46a6ea5c8b100713bca575.zip
SCOPE: A Self-supervised framework for Improving Faithfulness in Conditional Text Generation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission