Training Language Models to Critique with Multi-Agent Feedback
Critique;LLM;preference-based RL
Yes
N/A
No ethics review needed.
- Table 4 is somewhat unclear to me. Could you elaborate on what was specifically done in this ablation study? Are you fine-tuning from the same baseline models and then using different datasets, such as MultiCritique-SFT versus critiques generated solely by GPT-4-turbo and other individual models? Additionally, are these individual critiques extracted from the MultiCritique dataset, or are they generated separately?
- The paper is well-written and effectively presented, with technical details that are clearly explained and easy to follow. Figure 1 is particularly effective in illustrating the entire MultiCritique pipeline.
- The evaluations and ablation studies are comprehensive, providing a thorough analysis of the model's performance and the impact of various components.
- The MultiCritique SFT dataset, if made open-source, could be a good contribution to the field due to its careful construction aimed at reducing variances and biases.
- The limitations section is well-prepared, addressing most concerns that might arise from the study.
This paper proposes a critique data generation pipeline called MultiCritique to enhance the critique capabilities of language models. The process is divided into three steps. In Step 1, the authors selected queries from six datasets and used 11 language models to generate responses. These responses were rated by InternLM2-20B-reward into three quality levels, and GPT-4 was used to extract and summarize crucial information (CI) for each query-response pair. In Step 2, the query-response-CI tuples were input into multiple language models to generate aspect-specific critiques, referred to as Analytical Critique Units (ACUs). GPT-4 then classified these ACUs into seven quality categories and aggregated them into a final critique summary, which could be used for SFT of language models. In Step 3, the ACUs generated in Step 2 were used to construct preference pairs, which were then employed for reinforcement learning fine-tuning using PPO.
The evaluation of this pipeline was conducted on two benchmarks: CriticEval (unpublished) and CriticBench (ACL Findings '24).
- Dependence on GPT-4: The project heavily relies on GPT-4 for classifying and summarizing critiques, which may introduce biases into the final critique summaries. It is not hard to say that the entire project is feasible because of the existence of GPT-4. This reliance potentially undermines the benefits of using multiple agents in earlier steps. I know it might be a bit too much to ask, but it would be interesting to explore the impact of using a different model, such as Claude 3.5 Sonnet, to see how the results might differ.
- Specificity Towards Critique Ability: The model's focus on improving critique ability raises concerns about overfitting to these specific tasks. It would be beneficial to assess whether this specialization affects other capabilities, such as chat performance. Evaluating the model on benchmarks like AlpacaFarm Arena-Hard could provide insights into any potential degradation in general capabilities.
Yes
No ethics review needed.
- What is the generalizabilty of this kind of fine-tuning of the critiquing ability. Can the authors show whether finetuning on critiquing dataset generated from general alignment datasets (like OpenHermes, OpenAssistant) alone can still help in generating critiques in unseen domains like math and code.
- The performance improvement of the MultiCritiqueDataset-RL over MultiCritiqueDataset-SFT is not very clear - for some datasets there seems to be improvement while for others it is hurting or improvement is negligible. Can the authors comment on why this happening? Also what about including both MultiCritiqueDataset -SFT and -RL?
- Minor concern about terminology - I am not quite sure why the authors stress on “agent” feedback — this work on critique generation seems like yet another instruction tuning task, this in general can be helpful in agentic tasks but I don’t see why this is termed “multi-agent”
- It is a Practically useful work
- Good improvement in results in CriticEval and CriticBench benchmarks
This paper proposes a data generation pipeline utilising some of the SoTA LLMs (OpenAI, Claude, Qwen etc) both for SFT and RL tuning in order to improve the critiquing ability of LLMs. The authors experiment on CriticEval and CriticBench, where their SFT and RL datasets provides a good boost in performance.
- Mostly this paper feels like just a bit of a formalisation for distilling “critiquing” ability from GPT-4 by generating critiquing data both for SFT and RL tuning. The data generation process is also quite heavily dependent on GPT-4 abilities.
- Novelty of this work is not very clear to me
Yes
No ethics review needed.
- No major questions, I think the paper ideas a straightforward and easy to follow, the presentation is good, and the ideas are well executed
- From a research perspective, I think novelty is lacking a bit, neverthless it is an important contribution
- overall addressing an important problem of developing small high qulaity critique model
- new multi-agent feedbakc approach to curate dataset
- creation of a new high qulaity multi-critique dataset, incorporating diverse queries,and crucial information collection
- naturla integration and use of RL
- Strong experimental valiadtion, showing 7b FT models perform at par or better with closed source and 70b model on criticeval and critic bench
The paper, tackles the inherent difficulty in enhancing critique capabilities within large language models (LLMs). Critique ability, a key aspect of human meta-cognition, is essential for LLMs as it allows them to identify and address flaws in responses. This skill is pivotal for reliable automatic evaluation and model self-improvement. However, training models to critique effectively poses challenges. Most existing methods rely on supervised fine-tuning (SFT) using critiques generated by a single LLM, typically a model like GPT-4. Although effective to a degree, these single-model critiques can propagate the original model’s biases and limitations, leading to inaccuracies and diminished critique quality.

To address these issues, the authors introduce a novel data generation pipeline, MultiCritique, that uses multi-agent feedback to enhance the quality and diversity of critiques. Rather than depending on a single model’s perspective, MultiCritique aggregates feedback from multiple advanced LLMs, allowing for a richer, more balanced dataset. This multi-agent approach is implemented in both the SFT and reinforcement learning (RL) stages, producing a dataset called MultiCritiqueDataset. With this dataset, the model undergoes fine-tuning and then reinforcement learning, where preference scoring further refines the critique quality based on feedback from several agents.

The study’s experiments show that a 7B model fine-tuned with the MultiCritiqueDataset significantly outperforms other models in critique tasks, even rivaling the performance of much larger, closed-source models like GPT-4. The authors argue that the multi-agent approach not only yields a higher-quality critique dataset but also preserves feedback diversity, making it a robust solution for training LLMs in critique ability. This advancement offers an effective pathway toward more self-improving, reliable, and critique-capable language models
- not very technically novel, looks like a fairly stratightforward principle of ensembling for dataset collection
- Performance is still limited by an upperbound of the the base LLMs, and it is not clear if we could sclae to 70b, would the performance be better
Yes
No ethics review needed.
1. While the pipeline effectively aggregates critiques, the reliance on multiple LLMs (including GPT-4) for meta-critique classification may introduce considerable computational and financial overhead. Could you provide quantitative result and  a discussion on efficiency, potential trade-offs, or future work focused on optimizing this aspect?

2. Although the multi-agent framework is well-justified, could you provide a more explicit differentiation from other multi-agent critique frameworks (e.g., Arena learning and Stable Alignment). Clearer explanations on how MultiCritique uniquely contributes to critique generation and feedback quality.
1. The paper introduces a multi-agent framework that significantly improves critique dataset quality by aggregating and refining feedback from diverse LLMs. The structured use of Analytical Critique Units (ACUs) and meta-critique classification addresses the limitations of single-agent feedback and provides a robust critique foundation. The structured use of Analytical Critique Units (ACUs) and meta-critique classification addresses the limitations of single-agent feedback and provides a robust critique foundation.

2. The MultiCritiqueDataset improves LLM critique ability considerably over traditional supervised datasets, evidenced by impressive gains on CRITICEVAL and CRITICBENCH

3. The paper has quite extensive evaluation and analysis that support the method.
The paper proposes MultiCritique, a novel data generation pipeline aimed at enhancing language models' (LLMs) critique abilities by leveraging multi-agent feedback during supervised fine-tuning (SFT) and reinforcement learning (RL) stages. The proposed MultiCritique pipeline aggregates critiques from multiple advanced LLMs, utilizing key meta-critique processes to classify and consolidate high-quality feedback. This results in the construction of a new MultiCritiqueDataset, which shows empirical improvements on CRITICEVAL and CRITICBENCH benchmarks, demonstrating superior performance in critiquing abilities. Notably, the approach effectively competes with models up to ten times its size, approaching the critique capabilities of models like GPT-4.
See question.
Yes
See above.
No ethics review needed.
See above.
The authors leverage multi-agent feedback to train models capable of critique, claiming that multi-agent feedback provides more accurate and critical information, which could address the inherent complexity of critique. However, the reviewer has expressed doubts about this claim. While the experimental setup is fairly robust, the descriptions of the experimental settings lack detail in most parts.
Overall, this paper is poorly written, describing certain motivations and methods that are difficult to understand.

> For example, in the abstract, the authors emphasize that the inherent complexity of model-generated critiques limits model performance, which they cite as a primary motivation for this work. However, MultiCritique originates from Multi-agent Feedback, which clearly adds to this complexity by involving multiple models.
1. In terms of research on critique, other researchers have already explored similar ideas with LLMs. For instance, Jan Leike at OpenAI proposed "Self-critiquing models for assisting human evaluators," which this paper does not discuss. Additionally, while the authors discuss RLHF, they surprisingly omit foundational literature on RLHF, such as *Training language models to follow instructions with human feedback*. It appears that the authors did not thoroughly investigate relevant literature, which results in some contributions being overstated.
   
2. The terms “meta-cognitive” and “critique as a greater challenge than criticism” are ambiguous. Could the authors clarify these?

3. In Line 173, the authors state, "In total, we sample 10.7K queries covering 123 diverse task scenarios." However, the reviewer does not know what these 123 tasks entail or where to find this information in the appendix in this paragraph.

4. The authors use InternLM2-20B for scoring. How accurate is this reward model in the authors’ study, and how does it align with human evaluations at the three labeled response levels? Furthermore, Section 3.1 lacks a clear definition of these three levels.

5. In Section 4.1, the authors present Equation 1, where they condition \( Q_i \) and \( R_i \) to generate \( C_I \) and \( C \). How does this differ from the fine-tuning approach in the Aligner [3] paper, or are the two approaches identical?

6. In the experiments, I noticed that the authors applied SFT on InternLM2-7B-Chat-SFT, essentially layering SFT on a model that was already SFT-trained. How did they address potential issues of hallucination in this setup? Moreover, why did they not consider using Llama3-8B-Instruct as a base model?

7. This paper is based on multi-agent feedback, but it is unclear how the multi-agent approach is implemented. My understanding is that the feedback primarily comes from multiple language models. How does this approach differ from the multi-agent debate setups described in [4][5]? If the multi-agent approach in this paper merely relies on feedback from multiple LLMs without any designed communication between the agents, it appears to lack sufficient innovation compared to existing works on critique and debate. I recommend that the authors clarify this aspect.

---

### References:
[1] Self-critiquing models for assisting human evaluators
[2] Training language models to follow instructions with human feedback
[3] Aligner: Efficient Alignment by Learning to Correct  
[4] Debating with More Persuasive LLMs Leads to More Truthful Answers  
[5] Improving Factuality and Reasoning in Language Models through Multiagent Debate
Yes
No ethics review needed.
The multi-agent critique and meta-critique process likely incurs substantial computational expenses, especially given the high reliance on GPT-4 for critique refinement. A practical analysis of the computational cost relative to the quality gains would provide more context on the feasibility of scaling this approach for widespread or real-time applications.
The paper presents using multi-agent feedback through the MultiCritique pipeline, a deviation from previous single-model critique training methods. This generates critique data by aggregating feedback from multiple advanced models, thus raising the quality and robustness of the generated critiques. 

This paper validates the model improvements on benchmarks CRITICEVAL and CRITICBENCH, with positive results. The inclusion of ablation studies demonsrates the contributions of each aspect of the proposed pipeline.

The authors provide a clear breakdown of the MultiCritique pipeline, detailing each stage from data preparation to the meta-critique and RL stages. The structured format of the ACUs is explained clearly, allowing readers to follow the critique generation and refinement processes.
The paper introduces MultiCritique, a data generation pipeline designed to enhance the critique ability of LLMs by using multi-agent feedback during both SFT and RL. MultiCritique aggregates feedback from multiple models, filtering out errors to produce a critique dataset. This method uses a structured approach where each model identifies specific flaws, followed by a meta-critique stage that refines the critiques for accuracy. The reinforcement learning stage further boosts critique quality using a preference-based dataset refined through multi-agent scoring of revision effectiveness. Experiments demonstrate that MultiCritique-trained models outperform existing models on benchmarks such as CRITICEVAL and CRITICBENCH.
The improvements demonstrated by the MultiCritique pipeline, while statistically significant, are relatively modest in scale, particularly in relation to the substantial computational complexity introduced by multi-agent / GPT4 feedback. 

Additionally, the paper would benefit from exploring where the MultiCritique pipeline's improvements become saturated. By examining performance gains across a broader range of model sizes or critique tasks, the authors could better identify the conditions under which MultiCritique delivers more substantial gains.
@inproceedings{
anonymous2024training,
title={Training Language Models to Critique with Multi-Agent Feedback},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=tciQfO8S8j},
note={under review}
}
Critique ability, a meta-cognitive capability of humans, presents significant challenges for LLMs to improve.
Recent works primarily rely on supervised fine-tuning (SFT) using critiques generated by a single LLM like GPT-4.
However, these model-generated critiques often exhibit flaws due to the inherent complexity of the critique. 
Consequently, fine-tuning LLMs on such flawed critiques typically limits the model's performance and propagates these flaws into the learned model.
To overcome these challenges, this paper proposes a novel data generation pipeline, named MultiCritique, that improves the critique ability of LLMs by utilizing multi-agent feedback in both the SFT and reinforcement learning (RL) stages.
First, our data generation pipeline aggregates high-quality critiques from multiple agents instead of a single model, with crucial information as input for simplifying the critique.
Furthermore, our pipeline improves the preference accuracy of critique quality through multi-agent feedback, facilitating the effectiveness of RL in improving the critique ability of LLMs.
Based on our proposed MultiCritique data generation pipeline, we construct the MultiCritiqueDataset for the SFT and RL fine-tuning stages. Extensive experimental results on two benchmarks demonstrate:
1) the superior quality of our constructed SFT dataset compared to existing critique datasets;
2) additional improvements to the critique ability of LLMs brought by the RL stage.
Notably, our fine-tuned 7B model significantly surpasses other advanced 7B-13B open-source models, approaching the performance of advanced 70B LLMs and GPT-4.
Codes, datasets and model weights will be publicly available.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Critique LLM preference-based RL
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/4a43c11e4e9fab9bbc29692450cfe35a8d1497d9.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/313e26c1825a3232c8e6a05359ac41e049690b68.zip
Training Language Models to Critique with Multi-Agent Feedback
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission