Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verification
Verifier;Math Reasoning;LLMs
Dear Reviewer FHq1,

Thank you for your constructive feedback on our work. We appreciate the opportunity to clarify and enhance our paper based on your suggestions.

---

### Comment:
> **The approach seems to be applicable generally for reasoning domains, but the authors only test on math reasoning. To what extent can the technique be applied to other domains?**

**Response:**
We agree that testing the technique across multiple reasoning domains could strengthen the generalizability claims of our approach. In addition to math reasoning, we have also evaluated our model on code reasoning tasks, which is another critical area within reasoning domains. As shown in Table 2, our approach demonstrates robust performance across both math and code reasoning, suggesting its applicability to diverse reasoning tasks.

---

### Comment:
> **While there are improvements above the baselines, they vary dramatically between models.**

**Response:**
Our method is designed to develop stronger verifiers on top of existing reasoners, so the performance can indeed vary depending on the specific reasoner employed. However, when we use a fixed backbone reasoner, our method consistently achieves significant improvements over various baselines. This consistency is demonstrated in Table 3, where using the same reasoner shows reliable performance enhancements, highlighting the robustness of our verifier design.

---

### Comment:
> **To what extent can the technique be applied to other domains?**

**Response:**
Thank you for this question. The verification approach outlined in our paper has significant potential for broader application in two key domains:

1. Output Selection and Verification:

LLMs face inherent limitations as auto-regressive generators, where they only have limited context during the early stages of generation. When an error or suboptimal strategy is introduced early in the output sequence, it becomes challenging for the model to produce a correct final response. Additionally, due to their stochastic nature, LLMs often generate multiple varied responses to the same input. Techniques like sample-then-rank or best-of-N strategies are therefore commonly used to improve the consistency and accuracy of outputs, as seen in models like GPT-o1. By using a verifier, which has access to the entire generated response upfront, it is possible to provide more precise feedback on sampled candidates, leading to more reliable output selection.

2. Synthetic Data Generation:

As sourcing high-quality training data from real-world resources becomes increasingly challenging, state-of-the-art models (e.g., LLaMA 3.2) are increasingly relying on synthetic data generated by LLMs. A critical aspect of this approach is maintaining the quality of generated data. The verifier method serves as an effective filtering tool to eliminate low-quality data, thus facilitating a more robust self-learning cycle and enhancing knowledge distillation and transfer for LLMs. By incorporating a verification mechanism, synthetic data quality can be better controlled, making it a valuable component in training pipelines for future LLMs.

---

### Comment:
> **The paper could benefit from more analysis on what is learned by the verifiers, for example if there are certain types of solutions where the verifiers perform better?**
> 
> **Are there certain types of programs/solutions where the verifier is better at judging?**

**Response:**
Thank you for this insightful suggestion. Figure 6 in our paper illustrates examples where our verifier effectively detects reasoning errors in solutions, showing that the verifier is sensitive to errors involving incorrect use of numbers, operators, or knowledge points.

We have also included additional descriptions to explain that while model-based verifiers can detect surface-level logical errors, they are less sensitive to subtle calculation mistakes or deeper logical inconsistencies. For example, the verifier may score incorrect answers highly if they involve only minor calculation errors. As shown in Figure 6, our verifier sometimes assigns a high score to equations like \(3.5 + 2.5 + 4.5 + 1.5 = 13\), even though the correct result should be 12. This limitation motivated the development of our CoTnPoT method, which aims to improve detection of these finer-grained mistakes.

**Revision Details:** The above points have been clarified in the updated manuscript on Lines 221–227.

---

We hope that these revisions and clarifications address your concerns. Thank you once again for your valuable feedback.


Best regards,

Authors of ICLR 5271
## Part 2 of Our Rebuttal

**Comment:**  
There is a notable performance imbalance between CoT reasoning and PoT reasoning.

**Response:**  
Thank you for pointing out this discrepancy. Our method actually benefits from cases where PoT outperforms CoT. In these instances, the PoT solution acts as a strong indicator of correctness, supporting the collaborative verification process.

Our approach primarily focuses on verifying CoT solutions, regardless of whether CoT or PoT performs better on a given task. However, we recognize that PoT can be preferable in certain tasks, and for such cases, we suggest directly utilizing PoT and applying our verification method to code reasoning tasks, such as MBPP, as shown in Table 2.

---

**Comment:**  
What is the performance improvement achieved by fine-tuning the base model on the generated dataset?

**Response:**  
Thank you for this question. We are currently implementing this experiment and will present the results in the rebuttal as soon as they are available.

---

**Comment:**  
How does the performance curve change with varying sizes of sampled solutions?

**Response:**  
We appreciate this suggestion. We are currently working on adding this performance curve and will include the results in the rebuttal as soon as possible.

---

**Comment:**  
It seems that CoTnPoT sacrifices some recall, which may degrade the upper bound of LLM reasoning performance. Are there any potential solutions to address this issue?

**Response:**  
We acknowledge that CoTnPoT, by filtering solutions, can lead to a recall reduction. As a potential solution, we propose using CoTnPoT as a scoring mechanism rather than a strict filter. Instead of outright removal, each solution would be assigned a score based on CoTnPoT results (1 for passing, 0 for failing). By integrating these scores with SimPO verifier scores, we can retain more solutions and alleviate the recall degradation.

---

Respectfully,

Authors of ICLR 5271
## Part 1 of Our Rebuttal

Dear Reviewer AjnZ,

Thank you for your thoughtful and detailed feedback. We have addressed each point carefully and believe our responses clarify the novel contributions and effectiveness of our proposed methods. We hope these clarifications and planned updates meet your concerns. 

**Comment:**  
The technical contributions of this paper are somewhat limited. For example, the introduction of a verifier has already been highlighted; Comparing the CoT reasoning with its formal version have also been explored.

**Response:**  
Thank you for your valuable feedback. We acknowledge the prior work in verifiers within math reasoning. We respectfully clarify that the paper offers two primary contributions with unique insights:

1. **Systematic Comparison of Verifier Training Methods**: We agree that scaling inference-time compute and employing verifiers are existing approaches, yet the field lacks a comprehensive overview of verifier training techniques. Through our dataset and verifier comparison, we conclude that the SimPO method is the most effective approach among current methods, offering valuable insights and benchmarks for future verifier-building efforts.
   - **Revision**: We have revised the abstract to accurately position this contribution.

2. **Novel CoTnPoT Verification Approach**: We identify a key weakness of LLM-based verifiers: they often overlook subtle calculation errors and inconsistencies in math reasoning and struggle to verify highly abstract and structured code. To address these limitations, we propose a novel method called CoTnPoT, designed to make verification more comprehensive and robust. Our approach fundamentally differs from previous work [1] and [2], as we focus on translating between math and code rather than explicitly verifying language solutions in code format. Translation, in this context, is a more feasible task than direct verification, especially for complex math problems where verification can be challenging. For example, while the major experiments in [1] and [2] used simpler datasets like GSM8k, MultiArith, and MMLU, achieving a performance improvement of 63.69 -> 73.54 on MATH, our method achieves a notable improvement from 59.7 -> 76.9. Additionally, unlike prior studies focused mainly on math reasoning, our approach is effective in both math and code reasoning, demonstrating a broader application potential.
   - **Revision**: Lines 99-100 and 509-515 clarify the motivation and distinction of our approach, with the contribution statement revised accordingly.

[1] Don’t trust: Verify–grounding llm quantitative reasoning with autoformalization. ICLR 2024.

[2] Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification. ICLR 2024.

---

**Comment:**  
The improvement of collaborative verification is relatively marginal compared with SimPO.

**Response:**  
Thank you for highlighting this point. We would like to emphasize that while SimPO is indeed effective, CoTnPoT contributes unique strengths that complement the overall verification process, especially in complex tasks.

For math reasoning, model-based verifiers are proficient at catching surface-level logical errors, such as operator misuse or numerical inaccuracies (see Figure 6). However, they often miss subtler mistakes, like calculation errors or deeper logical inconsistencies. For instance, the verifier frequently assigns a high score to expressions such as $3.5+2.5+4.5+1.5=13$, where the correct solution should yield 12.

Although the improvement from SimPO verifier is significant, CoTnPoT addresses a different category of error, providing a more thorough and nuanced approach to detecting LLM errors. By focusing on distinct error types, CoTnPoT enhances the robustness of the verification process and bolsters overall accuracy.

---

**Comment:**  
Simply merging these models' knowledge, even without verification, could also enhance the reasoning performance of the LLMs.

**Response:**  
We appreciate this insight and agree that knowledge distillation from multiple strong models could improve performance. However, our study finds two limitations with this approach:

Scaling training resources may lead to performance saturation, which we reference in [3], making it less effective than scaling inference resources.

Verifier training allows weak-to-strong generalization, enabling improved reasoning in weaker models. For instance, our experiments demonstrate that using data from models like Mistral, Phi3, and InternLM2-Math to train SimPO yields improvements in larger models, such as LLaMA-70B and Qwen2-72B, which traditional knowledge distillation cannot achieve.

Most importantly, we see verifier training and knowledge merging as orthogonal approaches that, when combined, could further enhance reasoning performance.

[3] Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters

---


(Connect to Part 2)
Dear Reviewer R5WH,

Thank you for your insightful feedback, which has been valuable in refining our work. We look forward to further clarifying and enhancing our study based on our discussion!

---

### Comment 1:
**"Have the authors tried to compare their approach with verification based on say 50% of the test cases?"**

**Our Response:**

We appreciate this thoughtful question and understand there may be a slight misunderstanding regarding our approach. Our verification process for coding tasks relies solely on the coding problem, the solution, and the CoTnPoT commentary, with no test cases utilized in the verification phase. Instead, test cases are employed only to compute the final performance, as reported in Table 3, in alignment with prior studies on MBPP and MBPP+.

In our study, we evaluate the pass@1 rate in Table 3 to measure our verifier's ability at Best-of-N, highlighting the verifier’s effectiveness without incorporating any test cases during verification.

---

### Comment 2:
**"What is the point of doing CoTnPoT for code? How to do?"**

**Our Response:**

We apologize for any lack of clarity in describing CoTnPoT within the context of code reasoning. Here, we aim to clarify both the rationale and implementation for CoTnPoT in coding tasks:

1. **Rationale for CoTnPoT in Code Reasoning**:  
   As noted in lines 226-229 of our paper, the abstract and structured nature of code often renders it challenging for verifiers to fully understand, leading to similar scores for different solutions. This similarity indicates difficulty in accurately identifying nuanced errors within the code. CoTnPoT aids in mitigating this issue by providing descriptive commentary to enhance interpretability and reduce scoring ambiguities for the verifier.

2. **Implementation of CoTnPoT in Code Reasoning**:  
   We have revised Figure 2 to better illustrate how CoTnPoT functions across math and coding tasks. Additionally, Section 2.3 will be expanded to distinguish the application of CoTnPoT in these different domains.

   - **CoTnPoT for Math**: We sample multiple CoT solutions ($S_{CoT}$), translate each into PoT format ($S_{PoT}$), filter out any $S_{CoT}$ solutions if their answers don’t match $S_{PoT}$, and subsequently apply an LLM-based verifier on the remaining $S_{CoT}$.

   - **CoTnPoT for Code**: For coding tasks, we first sample multiple PoT solutions ($S_{PoT}$), then write a descriptive commentary ($S_{Des}$) based on $S_{PoT}$. This commentary is concatenated with $S_{PoT}$, providing a richer input for the LLM-based verifier to analyze.

---

### Comment 3:
**"Even so, it seems that SimPO is already very strong and CoTnPoT may not be necessary."**

**Our Response:**

Thank you for highlighting this point. We would like to emphasize that while SimPO is indeed effective, CoTnPoT contributes unique strengths that complement the overall verification process, especially in complex tasks.

For math reasoning, model-based verifiers are proficient at catching surface-level logical errors, such as operator misuse or numerical inaccuracies (see Figure 6). However, they often miss subtler mistakes, like calculation errors or deeper logical inconsistencies. For instance, the verifier frequently assigns a high score to expressions such as $3.5+2.5+4.5+1.5=13$, where the correct solution should yield 12.

Although the improvement from SimPO verifier is significant, CoTnPoT addresses a different category of error, providing a more thorough and nuanced approach to detecting LLM errors. By focusing on distinct error types, CoTnPoT enhances the robustness of the verification process and bolsters overall accuracy.

---

### Comment 4:
**"Same as B3, CoTnPoT should be compared with Best-of-2N, from the perspective of computational resource."**

**Our Response:**

We appreciate the reviewer’s suggestion regarding computational resources, and we acknowledge the importance of fair comparison. In response, we have initiated additional experiments to compare CoTnPoT with Best-of-2N and will share these results in our final rebuttal submission.

Best,

Authors of ICLR 5271
# Part 2 of our Rebuttal

### Comment:
> _"Grammar issues on line 219 (preference-tuning) and line 128 ('surpassing the performance of existing verifiers')."_

**Response**: We have corrected these grammatical issues:
- Line 219: Updated “reference-tuning” to “preference-tuning.”
- Line 128: Revised wording to improve clarity.

---

### Comment:
> _"Inaccurate terminology: 'recall rate' in Figure 1 should be 'Pass@64'; 'sample-then-rank' should be 'best-of-N.'"_

**Response**: We appreciate this feedback and have revised terminology for clarity:
   - **Revisions**: Lines 60-61 and 75-76 now use the terms “Pass@64” and “best-of-N,” respectively.

---

### Comment:
> _"Why was MAmmoTH-7B-plus used for math reasoning verification but not in previous sections?"_

**Response**: We chose MAmmoTH-7B-plus for two reasons:
1. Enhanced performance with an upgraded model.
2. Demonstrates the generalizability of the training method across different models. We acknowledge that this could be confusing but believe it does not affect the paper’s conclusions.
   - **Revision**: We expanded this explanation in Lines 370-374.

---

### Comment:
> _"Figure 2 is misleading in its comparison."_

**Response**: We acknowledge that Figure 2 could cause misunderstanding. To improve clarity, we have removed Figure 2, and we emphasize the clearer comparisons in Table 3.
   - **Revision**: Removed Figure 2, referencing clearer comparisons in Table 3.

---

### Comment:
> _"Contradiction in reducing reliance on external LLMs but using them for math reasoning."_

**Response**: Apologies for the confusion. Our aim is to minimize reliance on external models in coding verification by using the same LLMs to write comments. However, many math models lack the capability to translate CoT to PoT, necessitating the use of external LLMs.
   - **Revision**: Further clarified in Lines 246-247.

---

### Comment:
> _"Inconsistent number of candidate solutions in Figure 4."_

**Response**: We accept this suggestion and have standardized the number of candidate solutions in Figure 4.
   - **Revision**: Updated Figure 4 to show consistent testing with 100 candidate solutions.

---

### Comment:
> _"Generalizability of the verifier, line 82."_

**Response**: Previous verifiers were often trained on outputs from a single model, making them specific to particular solution formats. In contrast, our verifiers are trained on diverse backbones, enhancing generalizability. We realize that our original statement may have caused confusion, so we have removed it for clarity.
   - **Revision**: Adjustments made on Lines 81-83.

---

Best regards,

Authors of ICLR 5271
Dear Reviewers,

We sincerely thank each reviewer for the thoughtful and constructive feedback, as well as the time and effort invested in evaluating our paper. Your insights have been invaluable in helping us refine and improve our work. Through your questions and comments, we gained a deeper understanding of areas for enhancement, which has allowed us to make meaningful revisions that we believe have strengthened our paper considerably.

We respectfully invite you to review our itemized responses and would be most grateful if you could share any further questions or concerns in the comments. We are fully committed to addressing each point and provide additional clarifications if needed. If our responses have satisfactorily resolved your initial concerns, we would greatly appreciate it if you might consider an update to the evaluation score.

Thank you once again for your valuable feedback and guidance, which have been instrumental in enhancing the quality of our work.

Best Regards,
Authors of ICLR 5271
# Part 1 of our Rebuttal

Dear Reviewer dNMD,

Thank you very much for the detailed and thoughtful review, which has been invaluable in improving our work. We are fully committed to addressing each of your concerns and clarifying any areas that may have lacked precision in our initial submission. Should you have any further questions or additional concerns, please feel free to reach out to us. We would be more than happy to engage in further discussion to ensure our contributions are presented as clearly and accurately as possible. Our responses:

---

### Comment: 
> _"The paper lacks significant novelty; the primary differences lie in the volume of data, variety of sources, and minor variations in training."_

**Response**: We acknowledge that certain statements may have overclaimed novelty in our draft. However, we respectfully clarify that the paper offers two primary contributions with unique insights:

1. **Systematic Comparison of Verifier Training Methods**: We agree that scaling inference-time compute and employing verifiers are existing approaches, yet the field lacks a comprehensive overview of verifier training techniques. Through our dataset and verifier comparison, we conclude that the SimPO method is the most effective approach among current methods, offering valuable insights and benchmarks for future verifier-building efforts.
   - **Revision**: We have revised the abstract to accurately position this contribution.

2. **Novel CoTnPoT Verification Approach**: We identify a key weakness of LLM-based verifiers: they often overlook subtle calculation errors and inconsistencies in math reasoning and struggle to verify highly abstract and structured code. To address these limitations, we propose a novel method called CoTnPoT, designed to make verification more comprehensive and robust. Our approach fundamentally differs from previous work [1] and [2], as we focus on translating between math and code rather than explicitly verifying language solutions in code format. Translation, in this context, is a more feasible task than direct verification, especially for complex math problems where verification can be challenging. For example, while the major experiments in [1] and [2] used simpler datasets like GSM8k, MultiArith, and MMLU, achieving a performance improvement of 63.69 -> 73.54 on MATH, our method achieves a notable improvement from 59.7 -> 76.9. Additionally, unlike prior studies focused mainly on math reasoning, our approach is effective in both math and code reasoning, demonstrating a broader application potential.
   - **Revision**: Lines 99-100 and 509-515 clarify the motivation and distinction of our approach, with the contribution statement revised accordingly.

[1] Don’t trust: Verify–grounding llm quantitative reasoning with autoformalization. ICLR 2024.

[2] Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification. ICLR 2024.

---

### Comment:
> _"Overclaimed contributions, such as 'To address this, we scale up inference-time computation…' and 'We develop inference-time verifiers to enhance the reasoning capabilities of LLMs.'"_

**Response**: We acknowledge these inaccuracies and have revised the statements to reflect our approach more accurately:
1. Revised to: “To address this, we adopt a widely used method to scale up inference-time computation by generating multiple reasoning paths and employing verifiers to assess and rank the generated outputs by correctness.” (Lines 17-18)
2. This item has been removed from the contributions list.

---

### Comment:
> _"CoTnPoT should be compared with Best-of-2N to ensure fair use of computational resources."_

**Response**: We appreciate this suggestion. We are currently conducting additional experiments to compare CoTnPoT with the Best-of-2N approach. Results will be included in the rebuttal once the experiments are complete.

---

### Comment:
> _"Clarify the distinction between CoTnPoT in math vs. coding tasks."_

**Response**: We have clarified this distinction by updating Figure 2 and providing a detailed description in Section 2.3.

- **CoTnPoT for Math**: Generate multiple CoT solutions \( S_{CoT} \), translate each to PoT solutions \( S_{PoT} \), and filter out \( S_{CoT} \) if its answer does not match \( S_{PoT} \), followed by LLM-based verification.
- **CoTnPoT for Code**: Generate multiple PoT solutions \( S_{PoT} \), generate comments \( S_{Des} \) based on \( S_{PoT} \), concatenate \( S_{PoT} \) and \( S_{Des} \), and apply LLM-based verification.

   - **Revision**: Figure 2 and Lines 256-274 have been updated to clarify this distinction.

---


(connect to Part 2)
Yes
No ethics review needed.
Here are some of my questions and comments other than the ones mentioned above: 

If i understand correctly "recall rate" in Figure 1 is basically Pass@64. If that’s the case, it would be clearer to simply use the more widely recognized Pass@64 term.

Similarly, the "sample-then-rank" approach could be replaced with the more widely used "best-of-N" sampling strategy.

In section 3.2 it is mentioned that "We upgraded the backbone model of our verifier in math reasoning
from Mistral-7B to MAmmoTH-7B-plus to enhance performance.". Why not use MAmmoTH-7B-plus for the previous sections of the paper?

Figure 2 is misleading, one should compare Maj@k of models (without verifiers) with verification-based approaches (models with verifiers). I don't believe this is mentioned in the paper.

Line 256 states "Because using the same LLMs for both code and description generation reduces over-reliance on external LLMs". However, for math reasoning problems, the proposed method is in fact using and relying on external LLMs for generation of $S_{PoT}$. Doesn't this seem contradictory?

It would have been better if the number of candidate solutions $k$ in experiments for performance of different verifiers was consistent. Figure 4 shows some models have been tested with 100 and some with 64 candidate solutions.

Line 82 states "In other words, these verifiers are not generally applicable across different tasks and backbone reasoners". However, the proposed method and trained verifiers are also based on a few backbones. How is the proposed verifier more general?
The compilation of a large dataset containing both correct and incorrect candidate solutions from multiple LLMs, covering math and coding tasks, offers a valuable resource for training solution verifiers.

The CoTnPoT verification strategy improves accuracy by leveraging the complementary strengths and synergies of both CoT and PoT formats, incorporating insights that each provides.

The paper effectively explores a range of verifier training methods, including outcome-reward models (ORM) classifiers and various preference optimization techniques. The experiments, conducted on several models, provide valuable insights into the effectiveness of these approaches and their impact on verification performance.

The authors perform rigorous verification experiments with several generators, including weighted majority voting and thorough hyper-parameter search.
The paper provides a dataset of correct and incorrect responses for mathematical and coding problems, which are generated by multiple large language models (LLMs). This dataset is used to train verifiers to judge the correctness of candidate solutions from models at test-time. The verification is done on both chain-of-thought (CoT) and program-of-though (PoT) candidate solutions for a problem. The trained verifiers for math and coding tasks show improvements on GSM8K, MATH500 and MBPP.
The paper lacks significant novelty; the primary differences lie in the volume of data, the variety of sources, and minor variations in training (and verification with both CoT and PoT candidates). The tone in the abstract and introduction suggests that the approach is a bit more groundbreaking than it actually is, particularly regarding verification and inference-time compute scaling. For instance, the paper states "To address this, we scale up the inference-time computation by generating multiple reasoning paths and employing verifiers to assess and rank the generated outputs by correctness." However, this approach is already an established direction in the literature. It would be more appropriate to position this as a current method for scaling inference-time compute, rather than framing it as a novel solution.

Another instance of this is “We develop inference-time verifiers to enhance the reasoning capabilities of LLMs”. This item may not be suitable for inclusion in the contribution list, as similar work has already been completed in the past.

The CoTnPoT method relies on PoT generation from an external LLM, but such code-based solutions may not always be feasible or reliable. Additionally, if the proposed method generates K more (code) solutions per query, it would be appropriate/fair to compare other methods with a best-of-2K budget to ensure a fair comparison. Similarly, for code problems, there are extra description tokens generated by the language model. For a fair comparison, other methods should have access to a similar budget for generation of candidate solutions.

The distinction between the CoTnPoT approach for math tasks and coding tasks could be clarified further in the paper. For example, Figure 3 explains the method only in the context of math reasoning problems, but it is unclear how this approach adapts to coding tasks, especially given that there is no final answer in coding problems. 

There are some grammatical or dictation errors in the paper. For instance, on line 219, I believe the authors meant to write preference-tuning instead of reference-tuning. 

The phrase "and surpassing the performance of existing verifiers" in line 128 seems grammatically awkward and could be omitted without losing meaning.
Yes
No ethics review needed.
- To what extent can the technique be applied to other domains? 
- Are there certain types of programs/solutions where the verifier is better at judging?
- The method is simple and effective, while providing improvements above the baseline model.
- There is good analysis on the success of the verifier (Table 4), which provides evidence for why the verifier 
- The paper provides relevant ablations (Section 3.4) that highlight the effectiveness of various components of their approach.
The paper presents a number of contributions to improve LLM reasoning:
- First, they create a dataset of math and code reasoning tasks used to train verifiers
- Next, they train various verifiers and compare their performance
- Finally, they use these verifiers to improve performance on math tasks.
- The approach seems to be applicable generally for reasoning domains, but the authors only test on math reasoning domains. The paper would benefit from an assessment on another domain as well. 
- While there are improvements above the baselines, they vary dramatically between models
- The paper could benefit from more analysis on what is learned by the verifiers, for example if there are certain types of solutions where the verifiers perform better?
Yes
N/A
No ethics review needed.
Please see Weaknesses section.
The paper tackles an important topic, improving LLM reasoning, through verification based approach. The motivation makes sense and the authors study different options for training the verifiers. The filtering of comparing if CoT and PoT agree is also reasonable to do.
This paper proposes two ways to improve LLM reasoning performance via verification. The first way is to train reward models / verifiers on correct and incorrect solutions. The reward models can either be trained with simple outcome-based or preference based. The second way is to filter solutions when the COT solution and PoT solution answer do not disagree with each other. The paper shows that by using these verification methods, the performance can be improved considerably on math and coding datasets.
There are a few concerns I have for the proposed approach and settings.

1. For code reasoning, many existing work assumes we have access to some if not all test cases for the problem. Because of this, pass@k is typically used to evaluate model performance. I wonder if the verification setting for code reasoning makes sense. Have the authors tried to compare their approach with verification based on say 50% of the test cases?

2. For code reasoning, what is the point of doing CoTnPoT if the problem we are solving is already a coding problem? Also I generally have a hard time understanding equation (2) and the text around it. The LHS is S_Des while the text mentions "we concatenate the description and the code as an integrated input for the verifier..."

3. While CoTnPoT is nice to help and always seems to help, it essentially doubles the amount of inference time compute which makes verification much heavier now (see Equation 1). For a fair comparison, CoTnPoT should be compared against the baseline of sampling twice the number of solutions as before. Even so, it seems that SimPO is already very strong (see Table 1) and CoTnPoT may not be necessary.