On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models
Trustworthy Machine Learning;Explainability;Interpretability;Faithfulness;Large Language Models
Yes
No ethics review needed.
- Q1. Do you expect activation steering to be more or less effective for other models/architectures?
- Q2. Will you be releasing code/data for how faithfulness was calculated in this particular case?
- Q3. Do you expect your results to be consistent across how faithfulness metric was defined? So, for example, experimenting with faithfulness metric with paraphrasing vs. early answering strategy?
Overall, I thought the paper was strong and has potential for broad impact, because it connects so many concepts that are disparately considered. There has been a significant gap in evaluating *for* interventions, and this work systematically investigates the common and practical techniques for interventions.
- S1. Comprehensive evaluation of intervention methods for a widely used technique, CoT reasoning. Since this is how many researchers as well as practitioners interact with LLMs, this work is widely applicable and can have broad impact in considerations for AI safety.
- S2. I thought the introduction was particularly well motivated and the paper was generally well written.
- S3. Finetuning strategies were tested with multiple sampling strategy of their design. Adding faithfulness metric to the finetuning dataset creation was a particularly convincing experimental strategy.
- S4. Also introduces novel strategy for activation editing based on aligning on faithfulness vectors
- S5. The paper includes salient results, with most of these methods getting partial success. ICL or activation editing seem to get either accuracy or faithfulness performance enhancements, but rarely both. It seems that more finetuning on faithful datasets can improve both more so than ICL and activation editing
In recent years, there have been concerted effort in making language models more faithful and robust with methods such as finetuning, in-context learning and activation editing. This work investigates whether these 3 methods can make CoT reasoning more faithful. Their findings suggest that all of them achieve very limited performance improvements, with activation editing achieving only minimal improvements. Finetuning and in-context learning can be slightly more effective, though they seem to fail to generalize across tasks.
- W1. It seems that activation editing was only experimented with LLaMA-3B. I wonder if this could have been an issue with this particular model, particularly because activation editing could have vastly different results depending on the architecture. For that reason, I think this result could be made more robust by adding other models for comparison such as Gemma or OLMo.

- W2. "Fine-tuning using most faithful explanations achieve better accuracy-faithfulness trade-offs." This seems like an expected result, but I wonder if this holds true across domain. If there could have been a more comprehensive strategy such as sampling by length for comparison, I wonder if there were any observable differences across domain.

- W3. There's slew of methods proposed by lanham et al, but I think this paper only discusses faithfulness with respect to early answering strategy. Faithfulness metric could result in different behavior based on the metric definitions: early answering vs. adding mistakes vs. paraphrase.

- W4. The faithfulness based activation editing strategy was introduced, but the results on it were not included in the paper.
Yes
No ethics review needed.
- What is the formal definition of faithful (CoT) reasoning in LLMs? Unless, I am missing something this was stated to be formally defined in line 93, but I fail to find this definition later in the manuscript.
The paper targets an important and timely challenge in LLM. While massive effort is dedicated towards enhancing LLM's capability to reason or even demonstrating that it can reason, it still represents a major bottleneck and prevents using LLM to create AGI.  

Overall, the paper reads well and is well organised. The overall contribution is more technical and focuses on empirical studies of various combined techniques implemented, resulting in comprehensive experimental evaluations.
Recent advances of foundation models, in particular Large Language Models (LLMs) have demonstrated impressive performances in many natural language processing tasks. Nevertheless the capabilities of LLMs at reasoning tasks are still limited and raises significant debate (see [1]). A line of recent works proposed prompt-based techniques to improve LLM capability including, but not limited to, reasoning.  Notably, the most popular techniques are: *chain-of-thought* (CoT) by adding the phrase  'think/solve step by step' at the end of the prompt, and *in-context learning* by including illustrative examples in the prompt to inspire or assist the LLM about the specific context of the query to solve; another line focuses on fine-tuning the LLM on formal reasoning benchmarks data, mathematical problems (Algebra, Geometry, calculus and so on).  

This work combines the three aforementioned techniques to improve LLMs in producing what is referred to as *faithful* CoT reasoning and rational explanations to the delivered output. Moreover, it define a metric to assess the concept of faithful CoT reasoning.  



[1] Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, Mehrdad Farajtabar: GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models. EMNLP 2024
- The  paper constitutes incremental research work. The proposed solution is simply combining applied techniques in LLMs, which render the contribution incremental and straightforward. Technically, the contribution lacks in rigor, and many of the applied strategies are not formally justified.   

- The aforementioned techniques have shown several limitations, in past works, and more importantly in many cases techniques like activation patching are deteriorating the LLMs accuracy.   

- Several notions and techniques that this work builds upon, are not formally defined or described earlier in the paper, making it less accessible to a broader audience.
Yes
No ethics review needed.
1. What is the sample size of the benchmark? Correct me if I am wrong but lines 339 - 348 describe original datasets' statistics instead. 
2. When selecting N ICL demonstrations, are we considering questions' similarities or just using faithfulness as the single index? 

Minor:
1. Figures' notation requires browsing around.
2. Please avoid directly using acronyms, a full expression would be more reader-friendly. e.g. out of distribution for OoD in line 303 
3. Please check typos in the manuscript, such as:
a. line 312, Figure 4?
b. line 354 asking the question *without* invoking?
1. **Detailed Experiment** The paper conducted thorough experiments across in-context learning, fine-tuning, and activation editing.

2. **Insights from the Experiment** The empirical experiments provided meaningful insights, which might inform a better LLM alignment methodology for achieving faithful intermediate states in the future.
The paper systematically examines the Chain-of-Thought (CoT) behavior in large language models (LLMs) through experiments involving in-context learning, fine-tuning, and activation editing. The results indicate that activation editing had limited success, while in-context learning and fine-tuning led to only slight, non-generalizable improvements. The authors argue that the training process of these models does not prioritize faithfulness, which contributes to the generation of more self-aware content.
**Novelty in methodology** is the weakness of this position paper. As author(s) stated, changes are made for referenced methods/procedures, some theoretical/numerical supports can better validate the proposal. Please let me know if the following points are biased.

Here are some directions to consider:
1. To measure faithfulness, the Area Over the Curve (AOC) metric from [1] is adopted while the paper proposed to use probability scores for each instance instead of on the dataset level. However, section 2.3.1 of [1] also stated "AOC values are calculated as a weighted sum", thus [1] should also work on the instance level. I suggest editing line 166 to prevent confusion if this is the case.
2. For activation editing, this work selected top-K heads based on faithful probing results instead of top-K truth-relatedness heads in [2], they sound serving similar purposes to me. Can we compare these methods or see if they are transferable?

Reference
[1] Lanham, T., Chen, A., Radhakrishnan, A., Steiner, B., Denison, C., Hernandez, D., ... & Perez, E. (2023). Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702.
[2] Li, K., Patel, O., Viégas, F., Pfister, H., & Wattenberg, M. (2024). Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36.
Yes
No ethics review needed.
Insights on why faithfulness is difficult to learn, either in the form of mathematical theorems, or carefully designed experiments would be helpful.
The paper provided experimental results indicating that in-context learning, fine-tuning, and activation editing did not result in substantial improvement in faithfulness of chain of thought reasoning. This suggests that other techniques may be required if this type of faithfulness is required.
The paper did an empirical study on whether chain of thought reasoning can be made to accurately reflect the underlying reasoning done by the LLM (i.e. whether it can be made faithful) by in-context learning, fine-tuning, or activation editing. The faithfulness measurement tries to measure whether stopping the chain of thought early would results in different outcomes compared to using the full chain to answer the question; if it does not, it is an indication that the LLM already knows the answer before generating the chain and is doing post-hoc explanation of its reasoning in the chain rather than computing the answer within the chain. The study found that in-context learning, fine-tuning, and activation editing are all not successful in substantially improving faithfulness.
The paper provides negative results -- this is fine. However, to make a strong paper, insights that are supported by evidence on why the results are negative would be helpful.
Yes
No ethics review needed.
As the authors claim that "our work underscores the inherent difficulty in eliciting faithful CoT reasoning from LLMs, suggesting that the current array of approaches may not be sufficient to address this challenge", I wonder what could be revealed from the evaluation about the fundamental cause of the limitation for current LLM paradigms? Further, what could be the potential way to address them?
(1) The topic of faithful reasoning of LLMs is interesting and sounds reasonable to investigate.

(2) By demonstrating the limited success of conventional strategies, the paper highlights the intrinsic difficulty of faithful reasoning in LLMs, which provides a strong basis for future exploration. ​

(3) The presentation is generally clear and easy to follow.
This paper investigates the challenge of generating faithful Chain-of-Thought reasoning in large language models, specifically focusing on approaches like in-context learning, fine-tuning, and activation editing. While the authors highlight the importance of faithfulness in CoT reasoning for trustworthiness in high-stakes domains like healthcare, their empirical results suggest that none of these methods yield significant improvements in CoT faithfulness.
(1) The paper evaluates standard techniques like in-context learning, fine-tuning, and activation editing to improve CoT faithfulness, but these methods have already been extensively studied in other contexts such as improving accuracy, bias reduction, and factual consistency. The paper does not present any substantial technical adaptations or theoretical contributions to these methods specifically for faithful CoT reasoning. For example, while activation editing is discussed, it largely follows the framework of existing works like Li et al. (2024) without offering any new insights. The novelty of merely applying them to faithful CoT seems limited, and the contribution does not significantly advance the field beyond the status quo.

(2) The "early answering" metric used to evaluate faithfulness is based on whether truncating CoT reasoning affects the model's final output. However, the reason for taking it as the best way to measure faithfulness remains unclear, particularly given the complexity of CoT explanations. The measure seems too simplistic, as it fails to capture nuances in reasoning that may be faithful but not necessarily immediately reflected in the final answer. This could raise a misalignment between the metric and the goal of the research, which is to assess whether CoT explanations reflect the internal logic of the LLM.

(3) Although the paper acknowledges that none of the explored methods significantly improve CoT faithfulness, it does not provide a deep analysis of why these methods fail. For example, the results show only marginal gains in faithfulness, but the paper does not dive into what specifically causes this limitation—whether it is the inherent architecture of LLMs, the quality of training data, or other factors.

(4) While the paper claims to "lay the groundwork" for future research in trustworthy CoT reasoning, it does not propose concrete next steps or actionable insights based on the experimental findings. The conclusion merely restates that current methods are insufficient without suggesting innovative ideas or frameworks that could be explored in the future. This lack of direction limits the potential impact of the paper in advancing the field.
Yes
No ethics review needed.
Please refer to the Weaknesses
1. This paper tackles the issue of enhancing the faithfulness of reasoning in large language models, which is vital for applications requiring high reliability.

2. The study is methodologically sound, with rigorous experiments across different models and datasets, demonstrating the limited effectiveness of current strategies in improving reasoning faithfulness.

3. The findings are impactful, highlighting the need for new methodologies to make LLMs more transparent and trustworthy, which is crucial for their adoption in high-stakes domains.
The paper examines the difficulty of making large language models produce reasoning that accurately reflects their internal processes. It tests methods like in-context learning, fine-tuning, and activation editing and finds they only marginally improve a model's ability to produce faithful reasoning. The study concludes that current techniques are insufficient to ensure reasoning transparency in language models.
1. The study focuses on a limited number of benchmarks. It would benefit from expanding the range of datasets to better understand how these findings generalize across different types of reasoning tasks and domains.

2. The paper could benefit from a more robust theoretical framework that explains why certain strategies might improve faithfulness while others do not.
We explore approaches to improve faithfulness of CoT reasoning generated from large language models, and present their shortcomings.
@inproceedings{
anonymous2024on,
title={On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=1OyE9IK0kx},
note={under review}
}
As Large Language Models (LLMs) are being increasingly employed in critical domains such as healthcare, it is essential to make these models trustworthy. In this pursuit, Chain-of-Thought (CoT) prompting has emerged as a potential source of transparency in LLMs. While CoT reasoning is appealing to humans, prior studies have shown that these reasoning chains are not faithful i.e.; they do not accurately reflect the underlying LLM's behavior. Ensuring the faithfulness of LLM-generated CoT reasoning is crucial for decision-makers, who rely on them to determine if, when, and to what extent, trust the recommendations made by these models. While several works proposed strategies to enhance accuracy and truthfulness in LLMs, there has been a lack of exploration on the effectiveness of these common strategies to enhance the faithfulness of chain-of-thought (CoT) reasoning. Specifically, we explore the promise of in-context learning, fine-tuning, and activation editing to improve the faithfulness of the CoT reasoning. Our empirical analyses on benchmark tasks indicate that these strategies offer limited success in improving the faithfulness of the CoT reasoning, with only slight performance enhancements in controlled scenarios. Activation editing demonstrated minimal success, while fine-tuning and in-context learning achieved marginal improvements that failed to generalize across reasoning and truthful question-answering benchmarks. We subsequently analyse what makes faithful CoT reasoning challenging, and present findings to lay the groundwork for future research in trustworthy reasoning from LLMs.  In summary, our work underscores the inherent difficulty in eliciting faithful CoT reasoning from LLMs, suggesting that the current array of approaches may not be sufficient to address this challenge.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Trustworthy Machine Learning Explainability Interpretability Faithfulness Large Language Models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/2f95d56be8849edab629ad64c777a7227d73d1eb.pdf
interpretability and explainable AI
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/cf379a1ae57da762eafba539966b2e408c4114b7.zip
On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission