Multiagent Finetuning of Language Models
Language Models;Multi-agent Interaction;Self Improvement
Yes
No ethics review needed.
In addition to my primary concerns, there are also the following questions: 
1. In this paper, is each individual agent fine-tuned using the SFT (a combination of query and label response data) approach? If yes, can this method be integrated with DPO (a combination of query and human feedback preferences data) and PPO (a combination of query and reward signal data) approaches? Please provide a detailed discussion. 
2. How is the Combination of datasets implemented in line 25 of Algorithm 1 pseudocode? What does the hyperparameter denote? The authors need to clearly explain how the two types of data are used respectively in the training of the critic agent and what the role of the weight hyperparameter is.
3. The abstract mentions “multiagent society of language models”, yet the experiments only utilized 3-5 agents (and the improvement with 5 agents does not seem significant). How do the authors view the relationship between the number of agents and performance improvement, as well as the issue of number versus resource consumption?
4. The selection of baselines. From the setup of the baselines, this work seems more like an “A+B+C” approach. However, would it be fair to compare “B+C” with B alone, even if the A module is ablated?
1. The paper is easy to follow and the content is well-organized.
2. The paper proposes a method for agent self-improvement fine-tuning based on multi-agent collaboration, allowing for multiple rounds of self-improvement fine-tuning, which could be a promising approach.
This paper contributes to the field of self-improvement fine-tuning for LLMs by proposing a multi-agent cooperation approach. It replicates an LLM into multiple generation agents and corresponding critic agents. A multi-agent debate architecture is utilized to generate label responses for generation agents and critic agents. Each agent is then fine-tuned using its unique generated dataset. During the inference phase, the final result is generated following the multi-agent debate. The method demonstrates superiority over the baselines in a series of math-related language reasoning tasks.
My primary concerns with this paper are centered around the experimental section.

(Major)The first concern is regarding the selection of experimental datasets. The paper exclusively uses mathematical language reasoning tasks, and each task is not particularly challenging. Arithmetic is limited to arithmetic operations, GSM corresponds only to Grade School level difficulty, and MATH selects only the first three levels. If the tasks are not challenging enough, it may lead to questioning the need for incorporating multiple LLMs when a single LLM might be adequate. What’s more, conducting generalization experiments solely within mathematical datasets may not demonstrate the unique advantages in terms of generalizability, given the highly similar nature of the tasks. Including a graduate-level mathematical reasoning set or challenging datasets from other fields would make the experiments more convincing.

(Major)The second concern is that the performance improvement on datasets of modest difficulty is not significant, especially considering the introduction of multiple large models collaborating. It may be worthwhile to quantify other metrics beyond answer accuracy, for example, the KL divergence of each LLM from the original LLM distribution? Introducing multiple large models might allow agents to achieve better results without significant parameter changes.

*Overall, I believe the idea of the paper is commendable. However, as a practice-oriented paper lacking in theoretical explanation, its experimental design is insufficient. Without addressing these major concerns in the experiments, I'm afraid this is the highest score I can give.*

（Minor）The diversity of models: When reading the sentence " each model to capture parts of a task of interest.", I am very excited. However, the authors do not take any measures during the implementation to activate the heterogeneous characteristics among the generative agents. In other words, diversity is only brought about by the quantity of agents. The diversity cannot be theoretically guaranteed, nor can it be conceptualized and defined for specific agents after it appears. If the authors could enable different agents to collaboratively complete a complex task and truly capture the sub-tasks they are interested in or excel at, it might greatly enhance the contribution of this paper.
Yes
No ethics review needed.
See my questions in the "Major comments"
- Significance: This paper presents a promising approach to LLM self-improvement and could offer a valuable contribution.
- Clarity: Most of the Figures in the paper are clear and the paper is generally well-written.
This paper proposes a method to address a key limitation of LLMs: their dependency on static datasets, which limits their ability to self-improve. The authors propose a multiagent approach for LLM finetuning that enables iterative self-improvement through interactions with other models. By using distinct Actor (generation) and Critic (evaluation) agents, their approach improves feedback quality and response diversity, which results in better responses over iterations of finetuning. The authors empirically demonstrate their approach's effectiveness across 3 reasoning benchmarks (Arithmetic, MATH, GSM), showing performance gains of approximately 1-15% across 4 baselines. Additionally, the finetuned agents may demonstrate zero-shot generalization from MATH to the other two benchmarks.
There are several comments I would like the authors to address to make some details clearer and the paper more complete.

**Major comments**
1. Role Specialization: The paper introduces distinct roles for models (generation agents and critic agents). However, it would be helpful to clarify the specific objectives each role optimizes. Additionally, I suggest emphasizing that only two roles are used in this paper (generation and critic) to avoid confusion. 
2. Zero-shot Generalization: In Section 4.3 The authors claim zero-shot generalization on held-out benchmarks. I find the claims made in Section 4.3 not very convincing for three reasons:
- (a) The authors only evaluate on 100 random samples. Could the authors elaborate on why this specific sample size was chosen, and why not more?  If it is computationally possible to evaluate more samples (e.g. 1000), this would make the evidence for zero-shot transfer more convincing.
- (b) In Figure 5, the standard errors for performance (accuracy) are absent; could you please report these like was done in the Table 2
- (c) The choice of finetuning on MATH and evaluating on the GSM dataset is unclear; could you explain the rationale? Why not report all cross-dataset generalizations, such as MATH → GSM or Arithmetic → GSM? If the results are consistent across all cross-evaluations, this would strengthen the findings.
3. Computational Trade-offs: The limitations section mentions increased computational demands but does not quantify these trade-offs. Details on the additional costs or time requirements would help to evaluate the compute/performance balance.
4. Limitations Discussion: The current discussion of limitations is very short. Future researchers building on this work would benefit from more detailed insights into any bottlenecks or constraints of the approach. Based on these more detailed suggestions for future research directions would be valuable.
5. Response diversity: Looking at the slope, Figure 3 seems to indicate that all the benefits could be coming from the initial improvement in diversity. It seems important to mention the initial diversity for each ablation. 
6. Standard error: How exactly is the standard error in Table 2 computed?

**Minor comments** (that did not affect my score)
- Abstract: The sentence“ A set of language models are initialized from the same base model and then are specialized by independently updating each model using data generated by the model under multiagent interaction with other models” is somewhat lengthy and could be simplified for readability.
- Citation Formatting: Throughout the paper, `\cite{}` is used in places where `\citep{}` might be more appropriate (e.g., lines 137, 244, 448).
- Figure: In Figure 4, what are the multiple dots in each box for each method? I’m assuming that their order is also the order of performance across iteration but it’s not obvious.
Yes
No ethics review needed.
1.	Line 77: How does the proposed approach “promotes diversification within the society of models”? 
2.	What precisely does "Single-Agent" in "Fine-tuning Single Agent" refer to? Is it intended to indicate one generation agent or a single LLM? If the latter, a more fitting term might be “Fine-tuning Single LLM.”
3.	How is a fair comparison with baseline methods established?
4.	How does varying the number of agents affect the performance of the proposed method?
5.	It is pretty costly to train multiple LLMs, especially considering the inference-time compute and resources required to serve N LLMs. A straightforward possible strategy to avoid training multiple LLMs while also maintaining diversity is to include an unique identifier (e.g. an ID) or a special token in the input for each agent. How does this strategy compare to finetuning multiple LLMs?
1.	Jointly optimizing the LLM in the roles of generators and critics appears to be a robust method for enhancing the reasoning ability of LLMs.
2.	The work shows that finetuning multiple LLMs on independent datasets derived from multi-agent debate can preserve diversity, which is a critical challenge for LLM finetuning.
3.	The evaluation results show the strength of the proposed method.
This paper explores the fine-tuning of multiple LLM agents within the framework of multi-agent debate. The authors propose a method where multiple LLMs are fine-tuned and organized into generation and critic agents to facilitate multi-agent debate. Experiment results indicate that the proposed method outperforms baseline methods, and the authors further demonstrate that fine-tuning multiple LLMs helps preserve diversity compared to fine-tuning a single LLM.
1.	The title “Multiagent Finetuning of Language Models” may imply a broader scope than the paper addresses. Multi-agent applications of language models can indicate a much broader range of settings besides reasoning tasks and multi-agent debate, such as gaming and social simulation; however, this work focuses solely on multi-agent debate.
2.	The terms  “Single Agent” and “Multi Agent” is vague and unclear in this paper. For example, Sec 2.2 “Fine-tuning Single Agent”discusses scenarios involving multiple agents rather than a true single-agent setting.
3.	This work might not obey the standard training and evaluation procedure on GSM and MATH dataset, as only 500 examples are selected for training.
We illustrate how multiagent finetuning can improve language models
@inproceedings{
anonymous2024multiagent,
title={Multiagent Finetuning of Language Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=JtGPIZpOrz},
note={under review}
}
Large language models (LLMs) have achieved remarkable performance in recent years but are fundamentally limited by the underlying training data. To improve models beyond the training data, recent works have explored how LLMs can be used to generate synthetic data for autonomous self-improvement. However, successive steps of self-improvement can reach a point of diminishing returns, as the diversity of generations decreases, limiting further performance gains. In this work, we propose a complementary approach towards self-improvement where finetuning is applied to a multiagent society of language models. A set of language models are initialized from the same base model and then are specialized by independently updating each model using data generated by the model under multiagent interaction with other models. By training each model on independent sets of data, we illustrate how this approach enables specialization across models and diversification over the set of models. As a result, our overall system is able to autonomously improve over many more rounds of fine-tuning than single-agent self-improvement methods. We quantitatively illustrate the efficacy of the approach across a wide suite of reasoning tasks.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Language Models Multi-agent Interaction Self Improvement
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/b847321bd32ceb2accb33ae16f47746ec481c2e8.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Multiagent Finetuning of Language Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission