Generative Verifiers: Reward Modeling as Next-Token Prediction
LLM reasoning;reward models;verifiers
We believe that there might be some misunderstandings in regards to the contributions of our paper and its novelty and significance in relation to prior work, hence we wanted to kickstart discussion soon.

At the outset, we are unaware of prior work that utilizes inference-time computation using chain-of-thought (CoT) abilities of LLMs to improve verification (though of course, CoT have been used to improve generation). Our work makes it possible to improve verification by posing it as a generative modeling task. This contribution has been acknowledged by other reviewers as **“well-motivated” and “innovative”**. We would like to clarify the following:

> **First, I don't think such a process can be called a "verifier", as there is no rigor in the entire process.** 

To our knowledge, the term “verifier” is widely-used in the LLM community, established by the seminal GSM8K paper [1] and notable follow-ups [2, 3]. It’s unfair to be penalized for using the terminology widely-adopted by the community.

> **Second, the novelty, i.e., utilising the autoregressive ability rather than ranking, is not significant. The contribution is very incremental.**

We believe that there is perhaps a misunderstanding in this statement: we do not simply utilize autoregressive abilities rather than ranking, but unlock the ability to utilize inference-time computation for improving verification accuracy, by running multiple parallel chains of thought and majority voting. We are unaware of any prior work that uses chain of thought or majority voting for improving verification accuracy for learned verifiers. We are happy to revise novelty claims if there are suggestions regarding prior work demonstrating similar capabilities.

> **Third, I suppose the utilisation of another LLM will not always lead to positive impact. The study of its potential negative impact (e.g., the propagation of the vulnerabilities of multiple LLMs) may be more interesting than what presents in the paper.**

The above concern is applicable to most papers involving LLMs, and for most work involving LLM reasoning. We are happy to discuss this as a broad societal implication of this entire line of work but we do not think this should be a reason to reject this paper. 

> **No comparison and no discussion) with the methods on uncertainty estimation .. I don't see this "verifier" methodology offers significantly more than an uncertainty estimator.**

This concern also seems applicable to any work on LLM verifiers, and we believe that this should not be a reason to dismiss the contributions of this paper.  While we are not aware of work using uncertainty estimation approaches for verifiers, we do compare to prevalent verification approaches, including discriminative verifiers (classifiers), LLM-as-a-Judge, DPO, and self-consistency. 

Moreover, while “verifiers” can be viewed as uncertainty estimators, current LLMs are poor at judging “correctness” of their own responses on reasoning tasks, dubbed as “Generative AI paradox” [4, 5]. As such, using uncertainty estimations for verification seems like an interesting direction for future work.


**References:**

[1] “Training Verifiers to Solve Math Word Problems”, Cobbe et al, 2021.

[2] “Solving math word problems with process- and outcome-based feedback”, Uesato et al, 2022.

[3] “Let's Verify Step by Step”, Lightman et al, 2023.

[4] “The Generative AI Paradox: What It Can Create, It May Not Understand". ICLR 2023.

[5] “The Generative AI Paradox on Evaluation: What It Can Solve, It May Not Evaluate.” EACL 2024.
Author Rebuttal
Yes
No ethics review needed.
Suggested Improvements  
1. Broader Mathematical Validation: To strengthen the scientific claims, I suggest including results from more advanced math reasoning benchmarks, such as IMO tasks from OlympiadBench or relevant subsets of MMLU. Results from these additional benchmarks could significantly boost the paper’s credibility. BigBench has an induction and Identify Math Theorem https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/identify_math_theorems 
2. Justification for Larger Model Use: The reliance on a larger model (Gemini 1.0 Pro) to generate training data for smaller models needs a sound explanation, as it impacts reproducibility. Without this, the setup may seem biased.
3. CoT-Only Baseline: To isolate the effect of CoT reasoning in verification, a baseline experiment using CoT reasoning alone without verification reasoning steps could help confirm the added value of GenRM-CoT.
4. Consider Extending to Other Models: Testing GenRM-CoT on open-source models would help show that the approach generalizes beyond the proprietary Gemini/Gemma series.
5. Length Generalization: Generalizing to shorter problem lengths is not particularly noteworthy, as longer problem lengths often include shorter steps. Showing robustness across various task lengths would be more convincing.

I'm willing to revise if points addressed well.

citations
[1] Scratch pad: https://arxiv.org/abs/2112.00114 (cite too, they created CoT concurrently)
[2] OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems https://arxiv.org/abs/2402.14008
1. Innovative Approach: Reframing verification as a generative task, specifically with GenRM-CoT, is novel and shows promise for complex reasoning tasks.
2. Synthetic Rationale Generation: The use of the same model to generate both solutions and synthetic rationales offers a more streamlined and potentially scalable verification process.
3. Improved Performance: Results indicate that GenRM-CoT improves upon discriminative reward models, especially when using chain-of-thought (CoT)/ScratchPad [1] reasoning and majority voting.
This paper proposes Generative Verifiers (GenRM), a novel framework for verification in large language models (LLMs), which reframes reward modeling as a generative task. Specifically, the authors introduce GenRM and GenRM-CoT (Chain-of-Thought), where GenRM-CoT incorporates additional reasoning steps. They aim to improve verification by using the model (Gemini 1.0 Pro) for both solution generation and generating synthetic verification rationales and then train Gemma open source 2B, 7B, 9B models. Experiments are conducted on GSM8K, MATH and algorithmic tasks to demonstrate the effectiveness of this approach over discriminative reward models.
Scientific Reservations  
1. Limited Mathematical Task Scope: The reliance on GSM8K and limited algorithmic tasks raises concerns about generalizability. These datasets represent only basic levels of math reasoning (grade school and high school). Including results from more rigorous benchmarks, such as the IMO portion of OlympiadBench [2] or math subsets of MMLU, would strengthen the claims.
2. Over-Reliance on Proprietary Model (Gemini 1.0 Pro): By using Gemini 1.0 Pro to generate solutions and rationales for training smaller Gemma models, the paper introduces a dependency on proprietary resources, which might limit reproducibility. Showing results on more accessible, open-source models would be essential to add credibility.
3. Toy Nature of Algorithmic Tasks: The algorithmic tasks feel limited and not representative of real-world complexity. Including a more robust task, or additional toy tasks for variety, would better support the general claims.

Writing Reservations  
1. Inconsistency in Figures and Text: Figure 4 uses inconsistent colors (e.g., GenRM in blue but lines are cyan/green), which makes interpretation challenging. Additionally, the reported improvements (e.g., 73%-93.4% in the introduction vs. 16%-40% in the abstract) should be unified to avoid reader confusion.
2. Notation and Explanation Gaps: Section 3.1’s notation (e.g., inconsistent usage of x, y) creates confusion and requires more clarity. Specific variables, like I, need explicit definitions or cross-references to earlier sections to ensure readability.
3. Incomplete Background: Key concepts, such as "reference-guided grading" and "LLM-as-a-Judge," are insufficiently explained, causing unnecessary interpretive burden. Adding a background section for these terms, or moving some non-essential related work to the appendix, could improve clarity.
4. Confusing Terminology: The paper should clarify that “CoT Verifiers” refers to CoT reasoning in the verification process, not the solutions themselves, which also contain CoT. Renaming these methods would reduce ambiguity.
5. Inference and Training Separation: The distinction between training and inference (lines 211-241) is blurred. Separating these sections would make the methodology clearer.
6. Inconsistent Use of Majority Voting: The term "majority voting" implies selecting the most frequent result, yet the paper uses an averaging approach. Clarifying this terminology would prevent misunderstanding.
Yes
No ethics review needed.
In all evaluations are the actual generations are the same, the only difference is the verifier in each method? I want to confirm that you aren’t using the fine tuned generator for GenRM (it would be great to make this clear in the text).

Did you try any PRMs? How do they compare?

Did you try verifying sample collections larger than 32?

Couldn’t reference guided rational training introduce a train/test mismatch? Ie at training the verifier objective is conditioned on a correct answer, but it isn’t at test time?

Are the CoTs faithful? Ie. is the reasoning for yes/no accurate to the actual problem?
The authors examine a very relevant and important problem of learning good verifiers for LLM generations. 

They present a new method that is well-motivated (increasing verification compute and framing verification to match the original LLM objective).

The authors conduct a lot of key experiments exploring these dimensions, the paper is explained clearly and is very easy to understand.
- In particular, it’s great to see experiments measuring the generation performance change as well as experiments investigating the scaling properties of more verification inference compute.
The authors investigate training LLMs to act as verifiers using a generative objective (training the LLM to verify if a solution is correct by directly predicting the yes/no token). Notably, they investigate the implications of this modification for scaling inference compute and for jointly optimizing solution generation and verification.
In many of the plots (figure 1, 4, 5, 6), the y-axis scaling changes from plot-to-plot and is often very restricted (ie. sometimes spanning only 4%). This is misleading when comparing results, and it would be great to standardize it more.

GenRM does improve over the baselines (it seems like more on harder tasks which is worth highlighting more!) but a lot of times the improvement is relatively small (ex. 1% for gsm8k over discriminative).

"In Figure 8, we show that generative verifiers, especially GenRM-CoT, exhibit
better scaling behavior than discriminative RMs,"
- Unclear from figure 8 that generative verifiers scale better -> the boosts are very similar for the discriminative RM.

Although the authors explore a lot of baselines which is great, there are some key verification methods that are missing. Specifically process reward models are generally better than the ORMs studied, which is important given that the performance of ORMs in some settings are close to GenRM.

Verification is only done on a max of 32 generated solutions. Although this is done in many other past works, given that the best-of-n performance scales to thousands of samples for some datasets it would be great to see the scaling properties along this dimension.

I am willing to raise my score if some of these concerns are addressed! In particular, the presentation of results and/or adding stronger baselines.

Nits:
- Figure 4, the color is wrong for GenRM
Yes
No ethics review needed.
see the weaknesses.
An easy to understand method that works in the few experiments of the paper.
The paper proposes a way to automatically verify whether a solution generated by a LLM is correct or not. Instead of replying on a third-party verifier which ranks multiple solutions (as the best-of-N method), it suggests the verifier should use a CoT step, followed by a majority voting, to determine the correctness probability of the solution.
I have several concerns about this paper. First, I don't think such a process can be called a "verifier", as there is no rigor in the entire process. Especially, we do not have any guarantee on the final probability value. It completely relies on the quality of the other LLM to evaluate the solution, and as mentioned at the beginning of the paper, " ... often confidently make logical and factual mistakes ". I understand this is what the community is doing, but on the other hand, this paper does not make any progress on this aspect. 

Second, the novelty, i.e., utilising the autoregressive ability rather than ranking, is not significant. The contribution is very incremental. 

Third, I suppose the utilisation of another LLM will not always lead to positive impact. The study of its potential negative impact (e.g., the propagation of the vulnerabilities of multiple LLMs) may be more interesting than what presents in the paper. 

Finally, there is no comparison (and no discussion) with the methods on uncertainty estimation, which generates a confidence when producing a solution. I don't see this "verifier" methodology offers significantly more than an uncertainty estimator. 

In summary, I found the method proposed in the paper is incremental, without tackling the major issues (guarantees, negative impacts, etc) of this problem.
We frame reward modeling as a next-token prediction task, incorporating Chain-of-Thought reasoning and Majority Voting within verifiers.
@inproceedings{
anonymous2024generative,
title={Generative Verifiers: Reward Modeling as Next-Token Prediction},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Ccwp4tFEtE},
note={under review}
}
Verifiers or reward models are often used to enhance the reasoning performance of large language models (LLMs). A common approach is the Best-of-N method, where N candidate solutions generated by the LLM are ranked by a verifier, and the best one is selected. While LLM-based verifiers are typically trained as discriminative classifiers to score solutions, they do not utilize the text generation capabilities of pretrained LLMs. To overcome this limitation, we instead propose training verifiers using the ubiquitous next-token prediction objective, jointly on verification and solution generation. Compared to standard verifiers, such generative verifiers (GenRM) can benefit from several advantages of LLMs: they integrate seamlessly with instruction tuning, enable chain-of-thought reasoning, and can utilize additional test-time compute via majority voting for better verification. We demonstrate that GenRM outperforms discriminative, DPO verifiers, and LLM-as-a-Judge, resulting in a 16-40% improvement in the number of problems solved with Best-of-N on algorithmic and math reasoning tasks. Furthermore, we find that training GenRM with synthetic verification rationales is sufficient to pick out subtle errors on math problems. Finally, we demonstrate that generative verifiers scale favorably with model size and inference-time compute.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
LLM reasoning reward models verifiers
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/dea353f9b3553e107fde3684e0a441ef2ab3897a.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Generative Verifiers: Reward Modeling as Next-Token Prediction
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission