From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step
implicit reasoning;chain of thought
Few-shot NO CoT is extremely hard. As shown in the final version of chain of thought paper, even the 540B PaLM only gets about 20% accuracy on GSM8K under NO CoT few shot prompting, as shown in figure 4 - top row of https://arxiv.org/pdf/2201.11903 (standard prompting corresponds to our NO CoT setting).
Just to make sure we are on the same page---you are talking about NO CoT settings (in which models do NOT output intermediate steps) right? The 92.0% of GPT4 in its technical report is under the CoT setting. As for the GPT 3.5's number of 57.1%, did you mean it's under the NO CoT setting? While the paper didn't clearly specify the setting, a standard practice is to default to few-shot CoT setting (after CoT came out).
Thank you for the response. From the technical report of GPT-4 (https://arxiv.org/pdf/2303.08774v5), it seems like GPT-3.5 achieved 57.1 on GSM-8K with 5-shot No-CoT, while GPT-4 got 92.0 with 5-shot CoT (Table 2, last row). For Mistral, it looks like GSM-8K scores are consistent with 5-shot maj@1 greedy sampling. Look forward to the rest of your responses.
Thank you for your feedback. Could you clarify your point regarding "The comparison with No-CoT baselines is weak since those models are not fine-tuned like the ICoT models"? In our view, the comparison to No CoT baselines is fair, as we finetuned all models for 24 hours (we set a maximum of 200 epochs, but this limit wasn't reached in most experiments). Are you suggesting that we should first finetune the No CoT models on full CoT examples, then remove all CoT tokens and continue fine-tuning? Or is there another approach you have in mind for creating a more comparable No CoT baseline?
follow-up question
Thank you for your feedback. We'd like to quickly clarify a few points you raised.

- Validation Accuracy Interpretation: The validation accuracy is based on **exact match** accuracy, meaning any deviation, even by a single digit, results in a zero score. However, token-level accuracy (under teacher forcing) almost always remains above 90%. To make this clearer, we will include a token-level accuracy curve in our revised draft. Additionally, to our knowledge, this is the first instance of a GPT-2 model trained to perform 20-digit by 20-digit multiplication without CoT during inference, an achievement that No CoT training alone has not previously accomplished.

- "No CoT" Setting in Table 3: For Mistral 7B, the No CoT setting is achieved through finetuning on the same augmented GSM8K dataset, which yields better results than simple prompting. For GPT-3.5 and GPT-4, we use a 5-shot prompting setup with example demonstrations but without CoT (we used $\dagger$ to mark this distinction). This setting is inherently challenging, which is why the numbers are low. To illustrate the difficulty of the No CoT setting, please see this example: https://bit.ly/gpt4_system1. In fact, even with very simple prompts like "Which is closer to Ogden, Utah—the Empire State Building or the moon? Directly give the answer. Do not think step by step," a strong model like GPT-4o can still produce errors.

We appreciate your feedback and will provide a more detailed response in our forthcoming rebuttal.
Quick Clarification
Thank you for your comments. We would like to first clarify the following points regarding your question "I find it interesting that GPT-4 with 5-shot no-CoT only got 44%, while publicly reported GPT-3.5 got 57.1%, and Mixtral got 58.4%. I wonder if the authors have verified that the inference pipeline used can produce a performance that matches the public score for these models.":

- Prompting Method: The 58.4% accuracy reported for Mixtral 8x7B on GSM8K, as referenced from Mistral AI's announcement (https://mistral.ai/news/mixtral-8x22b/), was achieved using few-shot **CoT** prompting, which is a default practice on GSM8K. In contrast, our reported accuracy was obtained under a few-shot **No CoT** setting, which is way more challenging. As for the 57.1% accuracy of GPT-3.5, can you provide a source such that we can check if they are considering a CoT or No CoT setting?

- Difficulty of No CoT Setting: Achieving high accuracy on GSM8K without CoT is notably difficult, even for advanced models like GPT-4. To illustrate this, we have provided an example at https://bit.ly/gpt4_system1, demonstrating that even GPT-4 cannot solve very simple problems in this way. To the best of knowledge, our reported 52% accuracy under the No CoT setting represents the highest performance achieved in this specific configuration (No CoT).

We appreciate your feedback and will provide a more detailed response to address your other concerns in our forthcoming rebuttal.
Quick Clarification
Yes
No ethics review needed.
1. How does the "No CoT" setting in Table 3 prevent Mistral 7B, GPT-3.5, and GPT-4 from outputting intermediate steps? Is it achieved by using a prompt such as "Do not output intermediate steps, just provide the answer"? I couldn't find any related explanation. Additionally, considering that LLMs are quite sensitive to prompts, I suspect that the low metrics for these three models are due to poor prompts.

2. Given that ICoT-SI has been extensively trained on the GSM8K dataset, it may not be entirely fair to compare it with the No CoT version of Mistral 7B. A more appropriate comparison would be between ICoT-SI and the Mistral 7B model that has been trained with CoT data, using an effective prompt that encourages it to produce only the final answer.
1. Overall, the paper is well written and easy to follow.

2. The idea of gradually reducing the intermediate steps to internalize CoT is quite intuitive and seems reasonable.
The paper introduces a stepwise internalization approach that encourages a model, initially trained for explicit CoT reasoning, to internalize the reasoning process by gradually removing the intermediate steps during fine-tuning. As a result, the trained model achieves high performance while significantly reducing inference costs compared to explicit CoT.
1. Figure 3 illustrates that the validation accuracy steadily declines as tokens are progressively removed, until all tokens are eliminated, at which point the accuracy begins to gradually improve. This observation raises concerns that the model may have learned a shortcut instead of genuinely internalizing the CoT.

2. The analysis in Section 6.1 lacks details, such as how the probe model is trained and which layer's hidden states are analyzed. Furthermore, it may be necessary to conduct a probe analysis on the pretrained model to demonstrate that it does not internalize the CoT.
Yes
No ethics review needed.
Can implicit CoT reasoning match or surpass explicit CoT reasoning in all reasoning tasks? Modeling-wise, what will be the driving factors that decide whether explicit or implicit CoT may outperform each other?
The technical idea is original, well-motivated, and sound.

The experiments are solid, including compelling results and insightful analysis.

The paper is well-written.

This work is a solid contribution to the community.
The paper proposes a way to train an LLM to internalize its CoT steps. They start with a model trained for explicit CoT, and then gradually remove the intermediate steps during finetuning. This approach is simple yet enables small models to achieve stronger performance than explicit CoT in a range of problems, including GSM8K.
The models used in this paper are all small: the largest is mistral 7b; no other comparable-size models are used, like llama. 

Most of the tasks are somewhat synthetic (e.g., 20 x 20 multiplication).
Yes
No ethics review needed.
1. What are the specific motivations for internalizing CoT steps beyond improved latency? How does this process simplify reasoning without losing flexibility?
2. Have you considered the implications of losing token-based 'working memory' provided by explicit CoT methods, particularly for tasks requiring complex compositional reasoning and multiple steps?
3. Can you compare the effectiveness of this method against other synthetic data approaches, such as deterministic transformations of problem statements, e.g. changing the numbers of problem statement?
4. Why was the method not evaluated on larger models and more complex datasets to provide a broader understanding of its efficacy?
- The paper presents an innovative approach that shows potential by internalizing CoT reasoning, reducing LMs' inference time for multiplication and grade school math problems.
- This work demonstrates that smaller models can be trained to perform reasoning tasks effectively by leveraging internal states for reasoning.
- The Stepwise Internalization method shows significant speed enhancements over explicit CoT approaches and improved performance over models not using CoT reasoning.
The paper introduces a method called Stepwise Internalization to improve the reasoning capabilities of language models by internalizing chain-of-thought (CoT) steps. This approach involves initially training a model with explicit CoT reasoning, gradually removing these intermediate steps, and then fine-tuning the model to enhance its ability to perform implicit CoT reasoning. This method enables smaller models, like GPT-2 Small, to achieve high accuracy on tasks such as 20-by-20 multiplication while being significantly faster than models relying on explicit CoT reasoning.
- Internalized CoT models have worse performance on most of the evaluated tasks compared to the performance of explicit CoT prompting. The power of CoT prompting lies in its simplicity and ease of use without requirement for additional training, which allows for high performance gains across various tasks, which the purposed ICoT method does not offer.
- The comparison with No-CoT baselines is weak since those models are not fine-tuned like the ICoT models. A more appropriate baseline would involve fine-tuning models with CoT examples, without using the internalization technique, for a comparable number of epochs to the ICoT models.
- The choice of intermediate CoT steps as synthetic training data is not justified; other simpler methods of generating synthetic data, such as changing numerical values of the problem, might prove more effective in decreasing latency and increasing accuracy of the models .
- The experimental evaluation is limited in scope, focusing mainly on tasks with simple reasoning patterns like multiplication and basic math problems. This limits the applicability of the findings to compositional tasks that require a higher number of intermediate steps and working memory.
- The approach is not tested on larger models, such as Llama 3.1, or more complex datasets, such as the MATH dataset. This raises questions about its effectiveness compared to just scaling model size or training data, especially given that existing large models already perform well on simple math tasks.
Yes
No ethics review needed.
1. I find it interesting that GPT-4 with 5-shot no-CoT only got 44%, while publicly reported GPT-3.5 got 57.1%, and Mixtral got 58.4%. I wonder if the authors have verified that the inference pipeline used can produce a performance that matches the public score for these models.
1. Considering only the test time, the Stepwise Internalization (SI) approach greatly enhances the efficiency of reasoning tasks, allowing models to handle problems like multi-digit multiplication faster than explicit CoT reasoning.
2. SI models can achieve the same or close to the accuracy of explicit CoT on parity, coin flip, and multi-digit multiplication. They outperform the knowledge distillation approach and no-CoT inference.
3. The interpretability analysis provides some interesting insights into the internalized reasoning process. The paper finds that implicit CoT models can replicate partial product steps internally as explicit CoT reasoning. However, the internalized reasoning does not yet fully replicate the explicit CoT reasoning. These findings can potentially motivate future research in this direction.
The paper proposes a method to achieve implicit chain-of-thought (CoT) reasoning through stepwise internalization training. Through this training, language models learn to reason implicitly without explicit intermediate steps. The authors evaluate the proposed method on various reasoning tasks, such as arithmetic and grade-school math. This approach maintains accuracy while achieving faster inference speeds. Overall, it outperforms inference without CoT and implicit CoT through knowledge distillation while achieving faster inference than traditional explicit CoT methods.
1. The proposed implicit CoT's performance is pretty behind the explicit CoT method. Although the paper claims that there is a trade-off between performance and speed and that the proposed method is faster than explicit CoT, I do not view this as a major advantage of the proposed method. In particular, the proposed implicit CoT requires additional step-wise internalization in training time. At the same time, explicit CoT can be applied to models in test time, which allows the explicit CoT to be much more generalizable to a variety of reasoning problems. 
2. The paper seems to show only in-distribution experiments that perform step-wise internalization training and testing on the same data distributions. Thus, it is unclear whether the proposed method is generalizable to other reasoning benchmarks and applications. Meanwhile, the strong generalizability of explicit CoT has been well-established in previous studies [1-4].
3. Just to confirm, "remove 8 tokens per epoch." Here, one epoch is a full training run on the entire dataset, not a step of gradient update on a single batch, right? If so, it seems like the internalization process requires a very large amount of training epochs, and this number only increases when longer thoughts are needed. The efficiency seems limited in this case, especially when both the number of thought tokens and the number of model parameters rise.

**References**

[1] Wei, Jason, et al. "Chain-of-thought prompting elicits reasoning in large language models." Advances in neural information processing systems 35 (2022): 24824-24837.

[2] Nori, Harsha, et al. "Can generalist foundation models outcompete special-purpose tuning? Case study in medicine." arXiv preprint arXiv:2311.16452 (2023).

[3] Team, Gemini, et al. "Gemini: a family of highly capable multimodal models." arXiv preprint arXiv:2312.11805 (2023).

[4] Dubey, Abhimanyu, et al. "The llama 3 herd of models." arXiv preprint arXiv:2407.21783 (2024).