Step-Controlled DPO: Leveraging Stepwise Errors for Enhancing Mathematical Reasoning of Language Models
large language model;mathematical reasoning;alignment with relative feedback
Yes
No ethics review needed.
the authors generate positive & negative examples by sampling. For some of the more challenging reasoning problems it is probably impossible to get a single positive example out of the 100 samples. Can the authors provide the statistics on that?
The approach presented is simple, effective and immediately applicable to a wide range of logical/mathematical reasoning benchmarks.
This paper proposes Step-Controlled DPO, a new data-curating methodology that improves the effectiveness of DPO fine-tuning on reasoning tasks. When DPO is applied to fine-tune LLMs on reasoning datasets, one naive way of curating the dataset is to sample pairs of correct/wrong reasoning chains from LLMs. The authors of this work make the observation that when LLMs generate incorrect reasoning chains, they usually start making mistakes after a specific position; in particular, a reasoning chain is incorrect does not mean that the whole chain is incorrect, and its prefix could potentially be correct. Hence, the authors propose to generate negative examples by taking a correct reasoning chain and resample its suffix starting from some position k, which they refer to as SC-DPO. The authors combine the naively generated data with the data generated from SC-DPO, and conduct DPO fine-tuning. Compared to the baseline where only naively generated data is used, their approach show consistent improvement across different models and datasets.
The contribution overall seems relatively minor: SC-DPO is not really a new fine-tuning objective but instead the original DPO objective combined with a rather specific way of generating negative examples; the empirical gains are also relatively small.

Most of the information from Table 1. is probably not super important, e.g. why do the authors need to compare the performance of InternLM2-SFT-SCDPO against the listed open-source LLMs? how is this comparison related to SC-DPO?

Section 3 mentions MDP without defining what it refers to.

Eq (3) (4) and (6) seem somewhat redundant. E.g. to me Eq. (6) seems to be Eq. (4) expanded to its token-level and Eq. (6) itself is probably enough for describing the approach and explaining the intuition.
Yes
No ethics review needed.
See weaknesses above.
- The motivation of this paper that including step supervision in DPO for solving math problem is straightforward but reasonable.

- The step supervision is efficiently introduced by self-generation of LLMs, reducing the cost of annotation.

- The authors have demonstrated the effectiveness of the proposed method comparing to baselines on some regular benchmarks such as GSM8K and MATH
The paper presents Step-Controlled Direct Preference Optimization (SCDPO), a method designed to enhance the mathematical reasoning capabilities of large language models (LLMs). SCDPO extends Direct Preference Optimization (DPO) by incorporating step-by-step error supervision data. The negative (dispreferred) samples in SCDPO is generated by model itself by adjusting the model's sampling temperature until reasoning mistakes are introduced at specific steps.  SCDPO was tested on the InternLM2-20B and Mistral-7B, achieving competitive performance on benchmark datasets like GSM8K and MATH.
- Ths most concern is the lack of many baselines (that are kind of resembling the proposed method) in experiments. There are some previous works, e.g., Step-DPO [1], introduce the idea of step supervision in DPO for enhancing the math capabilities of LLMs. This work does make some differences by using temperature to controll the step errors, the comparison should be made with previous step-supervised SFT/DPO methods to demonstrate the effectiveness.

[1] Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs


- The idea of using softmax temperature to controll LLMs to make errors at certain steps is interesting but kind of limited. The naive problems can be accurately solved by LLM step by step, while the authors carefully craft the decoding temperature to "mislead" the LLMs to generate wrong answer at certain steps. Although the step supervision can be collected between the right and wrong answers, the LLM seems to hardly be improved as it can naively solve the problem and the step supervision just reduces some potential errors it may make from random decoding. In other words, the LLM is naively easier to give the right answer but harder to give the wrong answer (need to tune the temperature), making it doubtful if the LLM can achieve much improvment from such step supervision. Have you tried using a larger LLM to generate the step supervision data to improve smaller LLMs? It may be better than the self-improvement of smaller LLMs.

- The paper writing and structure can be improved. Some important informations are missed in this paper,  for example, it's unclear what the hyperparameters (top-p, top-k) used for generating training data, and how many samples the final training data contains. The cite of  Process Reward Synthesizing in related work is also missed.
Yes
No ethics review needed.
1. Since it's not guaranteed to have errors at the first generated step, as mentioned in the limitation, it would be good to add quantitative and qualitative analysis on which step the model generates error, to understand the granularity of the training signals. Additionally, it would be interesting to see if manipulating the distribution of the step index it generates error would cause any difference in performance.
2. Mathematical reasoning is one of reasoning types. It would interesting to see if it can expand to other reasoning tasks as well.
3. There are many "-" in table 1. Any reason?
1. The proposed methodology provides fine-grained, step-wise training signals, which is missing in most DPO and RFT work done in mathematical reasoning task.
2. The explanation of the 2-step pipeline is straightforward and clear.
3. The authors conduct experiments using various models including InternLM2 and Mistral, showing better and generalizable performance  compared to SFT/DPO baselines and various open-source and closed-source models with zero-shot prompting. They also did ablations to understand the impact of temperature on negative sample generation and the benefits of integrating naive DPO with SCDPO.
The authors propose Step-Controlled DPO (SCDPO), which, different from DPO, provides fine-grained, step-wise training signals by creating negative samples of mathematical reasoning rationales with errors after a specified step. The authors conduct experiments using InternLM2 and Mistral customized versions, showing better performance across seen and unseen datasets, compared to SFT and DPO baselines. They also did ablations to understand the impact of temperature on negative sample generation and the benefits of integrating naive DPO with SCDPO.
1. In section 4.2, the authors validate the generalizability of SCDOP on three baseline SFT models, but only show performance on the datasets that have been used for SCDOP training. It would be good to also show performance on the datasets that are not used for training, similar to table 1, to better understand its generalizability not only across models but also across datasets.
2. The paper mentions that they determine the correctness of reasoning steps by matching the final answer with the ground truth. However, it is not clear how they do the matching. Since the matching mechanism cannot be perfect as the same answer could be in different expressions, e.g. "increasing from 4 to 5" is equivalent to "increasing by 25%", it would be good to add quantitative or qualitative analysis of the matching strategy. In the other words, the paper currently only contains analysis on false negatives but not on false positives.
3. [Minor point] Typos: 
    a. line 121: "We" -> "we"
    b. figure 1.a, the order of i3 and i2 should be switched 
    c. line 366: 78 -> 78%
Yes
No ethics review needed.
1. (Following up on Weakness 2) Could the authors provide an analysis of the distribution of the dispreferred data? Specifically:

a. What is the distribution of the step $k$ (referenced in Line 138) that is used as the starting point for generating errors in the resulting dispreferred data?

b. What is the distribution of the "actual error" location within the dispreferred reasoning chains?

2. Line 467: There appears to be an unlinked reference.
1. The paper is well-written and easy to follow.
2. The method for generating erroneous steps in SCDPO is both interesting and novel. It provides an effective approach to achieve process-based supervision without human annotation on intermediate steps. This approach allows for error localization within certain steps, which—while not precise—represents a step towards finer-grained, automatic process-based supervision.
3. The experiments are solid and convincing, the proposed method demonstrates strength across different models and benchmarks.
This work proposes SCDPO, a method leveraging stepwise supervision without human annotation to enhance mathematical reasoning in models. The approach uses a model that is trained in math domains to generate both positive and negative samples. To produce dispreferred samples, the model is prompted with the initial portion of correct reasoning chains, then asked to complete the reasoning at a high temperature setting, leading to potential errors in the final answer. In the SCDPO loss, only the latter part of the reasoning chains is considered. Empirical results demonstrate that models trained with SCDPO outperform those trained with other methods.
1. The concept of a “step” in the reasoning chain is not clearly defined in the paper. It would be more clear if the authors could provide a more precise explanation of how they segment steps within a reasoning chain. 
2. Potential positional bias in error generation. The approach uses an ascending temperature strategy to create erroneous data, which may unintentionally introduce a bias where errors occur more frequently in the latter half of the reasoning chain. It would be helpful if the authors could investigate whether the distribution of errors indeed skews toward the end.
Since errors cannot be automatically located within the chain, manually sampling and inspecting some dispreferred solutions could provide insights into any potential biases.
Yes
No ethics review needed.
Line 466 misses a citation.
1. The SCDPO approach introduces a unique method for improving mathematical reasoning by targeting stepwise errors, which diverges from the traditional focus on outcome-based supervision in DPO.
2. The paper conducts extensive experiments, demonstrating great performance across several mathematical reasoning benchmarks, including GSM8K and MATH. 
3. This paper provides clear theoretical explanation of SCDPO, which is necessary and provides theoretical support for this method.
4. This paper is well-structured and clear in its presentation.
The paper introduces Step-Controlled DPO, a novel approach to improve the mathematical reasoning abilities of LLMs by leveraging stepwise errors. Unlike Direct Preference Optimization, which relies on final answers to generate preference feedback, SCDPO focuses on stepwise error generation. By creating solutions that introduce errors at specific reasoning steps, SCDPO provides more granular supervision to the model, allowing for better alignment and accuracy in mathematical reasoning. The method is evaluated on several mathematical reasoning benchmarks, showing competitive results with open-source models.
1. The paper mentions SCDPO works well on both CoT and Code-integrated solutions, but the results is not demonstrated this comparison. The author needs to add a table that highlights the comparison between these two different types of solutions.
2. This work lacks of the experiments on show generalisation ability of the method. The author could do an evaluation on out-of-distribution data, comparing the results with the in-distribution data.
3. This work is quite similar to another work "Step-dpo: Step-wise preference optimization for long-chain reasoning of llms", but the differences between these paper are not clearly demonstrated. The authors need to elaborate more about the comparison about this.
@inproceedings{
anonymous2024stepcontrolled,
title={Step-Controlled {DPO}: Leveraging Stepwise Errors for Enhancing Mathematical Reasoning of Language Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=ZRDa2IT1sQ},
note={under review}
}
Direct Preference Optimization (DPO) has proven effective at improving the performance of large language models (LLMs) on downstream tasks such as reasoning and alignment. In this work, we propose Step-Controlled DPO (SCDPO), a method for automatically providing stepwise error supervision by creating negative samples of mathematical reasoning rationales that start making errors at a specified step. By applying these samples in DPO training, SCDPO can better align the model to avoid reasoning errors and output accurate reasoning steps. Qualitative analysis of the credit assignment of SCDPO and DPO demonstrates the effectiveness of SCDPO at identifying errors in mathematical solutions. We then apply SCDPO to an InternLM2-20B model, resulting in a 20B model that achieves competitive scores of 88.5\% on GSM8K and 58.1\% on MATH, rivaling all other open-source LLMs, showing the great potential of our method. The code, models and data are released to inspire future work.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
large language model mathematical reasoning alignment with relative feedback
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/cfb533e4b754b950bd1b26c6d64b9f353a65bb65.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/e4f8a3d3605d1a0b232f5f7bde042597cfcfdab1.zip
Step-Controlled DPO: Leveraging Stepwise Errors for Enhancing Mathematical Reasoning of Language Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission