SuperCorrect: Supervising and Correcting Language Models with Error-Driven Insights
Large Language Models;LLM Reasoning
Yes
No ethics review needed.
See Weaknesses.
1. The paper proposes hierarchical thought template that more effectively captures the underlying reasoning mechanisms of powerful LLMs compared to simply prompting them to generate reasoning steps.
2. Cross-model collaboration applies step-level correction and DPO, which can provide small language models more detailed supervise signals and than solution-level training, thus leading to more powerful reasoning capabilities.
3. Extensive experiments are convicing, demonstrating the effectiveness of SuperCorrect.
4. The paper is well-structured and well-written.
The paper proposes SuperCorrect, mainly focuses on addressing the issue of unability of detecting erroneous steps in their reasoning steps of small language models. SuperCorrect firstly prompts powerful LLMs to generate correct CoT reasoning steps and their detailed thought templates, then employs cross-model collaborative DPO to enhance the inner reasoning and self-correct abilities of small language models. Extensive experiments have demonstrate the effectiveness of proposed SuperCorrect framework.
Teacher LLMs generated contents lack evaluation. For example, the logic flaws and errors found by teacher LLMs in student LLMs generated reasoning steps (Line 331). In Line 333, are the analysis $a_i$ and correction $c_i$ annorated by humans or generated by teacher LLMs? If it’s the latter case, what are the quality of these generated contents?

It’s seems like the cross-model collaborative DPO is nearly identical to the Step-DPO [1] paper, with the main difference being that in Step-DPO, $a_i$ and $c_i$ are sampled directly from the policy model, whereas in this paper, they are annotated/generated by humans/teacher LLMs. Can you explain the core differences? Or are these simply parallel works?

[1] Lai, X., Tian, Z., Chen, Y., Yang, S., Peng, X. and Jia, J., 2024. Step-dpo: Step-wise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406.18629.
Yes
No ethics review needed.
**(a)** It is stated multiple times that self-correction of reasoning steps is additionally difficult for mathematical tasks. While I mostly agree with this, the current proposed method has no part in its design that limits itself to Math benchmarks; the proposed Hierarchical Thought Template, to some extent, can be used for any task and the cross-model correction with a teacher model can also be used in other tasks. Therefore, it currently seems strange to me that the paper only evaluates on the math benchmarks and only for smaller LM models tuned for math questions. I would like to ask the authors to clarify this.
**(a)** I think the paper is very well-written and even someone who is not entirely familiar with self-correction and reflection concepts can follow. The formatting is also clean and makes the story again easier to follow. The preliminary concepts such as DPO and how it relates to other methods such as RLHF is again very well presented.

**(b)** The cross-model DPO, to me, seems quite close to a distillation framework of teaching the smaller model how to identify wrong steps from a larger model. I think it is very simple and quite applicable to many domains with different tasks (see also Question **(a)** below).
 
**(c)** The quantitative results seem quite promising, especially in Table 1 and 2. Both HTSF and cross-model DPO seem to give significant gains. (see also weakness **(c)** below)
The paper tries to improve the mathematical reasoning of small LLMs by (1) improving their thought template using their proposed Hierarchical Thought  template and (2) enhancing the self-correction of the model by making it to learn how to self-correct like a larger SOTA model using the proposed cross-model DPO. The experiments are done over three small LLMs on two math benchmarks.
**(a)**  While I find the quantitative results quite promising, I think there are a few tiny tables that can be added to further prove the point and support the proposed intuition. Specifically, the way I see it, is that the self-correction ability of the smaller (student) LLM is improved by comparing it with the correction provided from a larger, more powerful teacher, as if the paper is ‘distilling’ the self-correction. While it seems to work well for final accuracy, it is not quantitatively supported that this distillation was successful. For example, I would like to see if the two models agree on each reasoning step after the distillation and can locate correct and wrong steps the same way. This should be quantifiable, as you already have discrete reasoning steps in XML format. Specifically, I suppose you can measure the ratio of steps that are predicted the same way (both larger LM and smaller LM) say `correct’ or `wrong’ over the entire number of steps. Then, based on what the paper claims, I expect this ratio to increase after the cross-model collaborative DPO stage. I let the authors decide what sort of metrics demonstrates this the best, but I think right now such an inspection is missing.


**(b)** Regarding the Thought Template, the main contribution seems to be the hierarchical aspect of the thought. However, I think this needs to be supported by measuring variance over different prompt styles, while I would maintain the hierarchical aspect. For example, would we get the same benefits if the <Generalized> statement is moved to be before all of the <Step>s? (I understand if one works better than the other, but overall I would expect this should still give similar gains?) The other way to test this would be to only remove the Generalized prompt (or negatively prompt the model to not provide this). Or you could even use the same exact prompt as the baseline, and only improve it with a statement to include the <Generalized> template as well. In short, I’m currently finding the gains from HSFT alone, compared to SFT, hard to associate **only** to the hierarchical aspect and not to the different prompt formatting used.

**(c)** I would like to ask the authors to clarify why Table 2 only shows numbers for the Qwen model and not for the DeepSeek and Llama, as this would not require any additional experiments!? In the current version the individual gains for HSFT(without cross-model DPO) is only demonstrated for Qwen and not the other two models.
Yes
No ethics review needed.
1.	Could you elaborate on the potential strategies for scaling the SUPERCORRECT framework to even larger language models, and how you might address the computational efficiency challenges that come with such an increase in model size?

2.	How does SUPERCORRECT handle systematic biases or errors that may be present in the teacher model's corrections? Additionally, could you discuss how the framework evaluates and ensures the robustness of the student model against such potential inaccuracies in the supervision process?

3.	In the context of the two-stage training process, have you observed any stability issues or challenges in long-term training? If so, what techniques or modifications are employed to ensure the stability and convergence of the models?
1. By utilizing the teacher model to identify and correct errors in the student model's reasoning, SUPERCORRECT not only corrects mistakes but also teaches the student model to avoid and rectify specific errors. This approach breaks the bottleneck of the student model's thought process and equips it with new skills and knowledge to tackle challenging problems.

2. When leverage a large teacher model to supervise and correct a smaller student model, SUPERCORRECT combines hierarchical thought templates and cross-model collaborative direct preference optimization (DPO). I think this method is innovative.

3.  This advancement is particularly significant as it shows the potential for smaller models to compete with or even surpass the capabilities of much larger models in complex mathematical reasoning tasks.
The paper introduces SUPERCORRECT, a two-stage framework aimed at boosting smaller LLMs' reasoning and self-correction abilities. It leverages guidance from larger teacher models, such as GPT-4 or LLaMA, in two core phases: hierarchical thought templates and cross-model collaborative Direct Preference Optimization. Experiments on MATH and GSM8K benchmarks reveal that SUPERCORRECT consistently outperforms traditional reflection and fine-tuning approaches. Notably, the SUPERCORRECT-7B model exceeded DeepSeekMath-7B by 7.8% on MATH and 5.3% on GSM8K, showcasing improvements in both accuracy and stability.
1. The paper primarily focuses on 7B models. It may not be immediately clear how well the SUPERCORRECT framework would scale to larger models or generalize across different types of reasoning tasks beyond mathematical problems.

2. The success of SUPERCORRECT relies heavily on the quality of the fine-tuning datasets and the paired correction traces. The paper mentions constructing high-quality datasets, but it may face challenges in scenarios where such curated datasets are not available or the domain of interest is very niche.
Yes
No ethics review needed.
1. Are the training data for Qwen2.5-Math-7B-Instruct, DeepSeekMath-7B-Instruct, and Meta-Llama3.1-8B-Instruct the same as the data used for training SUPERCORRECT-7B as mentioned in the paper? Including the types of datasets and the data volumes used.
2. What LLM was used to generate the high-quality dataset ? Considering that LLM can still generate incorrect data (both of Hierarchical Thought process and error correction data), were any measures taken to address this issue?
1. The writing of the paper is smooth and clear, making it easy to understand.
2. Significant performance improvements were achieved in mathematical tasks.
3. The paper has open-sourced 10K high-quality SFT data and 1K preference alignment data.
The paper introduces SuperCorrect, a two-stage framework to improve the mathematical reasoning of smaller models like Llama-3-8B and DeepSeekMath-Base. While large language models such as GPT-4 and PaLM excel, smaller models struggle with error detection and correction. SuperCorrect addresses this by using a large teacher model to guide the student model through reasoning and reflection. The first stage extracts hierarchical thought templates from the teacher model to refine the student's reasoning. The second stage uses cross-model direct preference optimization (DPO) to enhance the student's self-correction abilities. SuperCorrect-7B achieves state-of-the-art results, outperforming DeepSeekMath-7B and Qwen2.5-Math-7B on MATH and GSM8K benchmarks.
1. The paper introduces two strategies for data augmentation using the teacher model (corresponding to SFT and DPO training), but there is a lack of a clear logical connection between these two methods, as they appear to be separately designed.
2. The motivation behind the design of the "Hierarchical Thought Template" is insufficiently explained. In the Introduction, author mention that the capability deficits in smaller models stem from "failing to effectively identify and correct reasoning errors". However, this doesn't naturally lead to the idea that hierarchical reasoning, incorporating both generalization and details, would be effective.
3. Using a teacher model to correct the student's reasoning results is not a novel idea. Although this paper emphasizes the use of DPO training with these data, there is a possibility that the performance gains primarily come from the expansion of data after correction, and using the SFT approach on this data may achieve similar results.
@inproceedings{
anonymous2024supercorrect,
title={SuperCorrect: Supervising and Correcting Language Models with Error-Driven Insights},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=PyjZO7oSw2},
note={under review}
}
Large language models (LLMs) like GPT-4, PaLM, and LLaMA have shown significant improvements in various reasoning tasks. However, smaller models such as Llama-3-8B and DeepSeekMath-Base still struggle with complex mathematical reasoning because they fail to effectively identify and correct reasoning errors. Recent reflection-based methods aim to address these issues by enabling self-reflection and self-correction, but they still face challenges in independently detecting errors in their reasoning steps.
To overcome these limitations, we propose SuperCorrect, a novel two-stage framework that uses a large teacher model to supervise and correct both the reasoning and reflection processes of a smaller student model. In the first stage, we extract hierarchical high-level and detailed thought templates from the teacher model to guide the student model in eliciting more fine-grained reasoning thoughts. In the second stage, we introduce cross-model collaborative direct preference optimization (DPO) to enhance the self-correction abilities of the student model by following the teacher's correction traces during training. This cross-model DPO approach teaches the student model to effectively locate and resolve erroneous thoughts with error-driven insights from the teacher model, breaking the bottleneck of its thoughts and acquiring new skills and knowledge to tackle challenging problems. Extensive experiments consistently demonstrate our superiority over previous methods. Notably, our SuperCorrect-7B model significantly surpasses powerful DeepSeekMath-7B by 7.8\%/5.3\% and Qwen2.5-Math-7B by 15.1\%/6.3\% on MATH/GSM8K benchmarks, achieving new SOTA performance among all 7B models.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Large Language Models LLM Reasoning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/294cfdd00c333860cc4300a53c634178da1ff9b2.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/5e6de82afb52835e5b5c73e26ad4a80b77ba8881.zip
SuperCorrect: Supervising and Correcting Language Models with Error-Driven Insights
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission