Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent System
llm agent;multi-agent;inference scaling law
Yes
No ethics review needed.
* what about systemic biases/safety misalignment arising from training?
* can we extend this to systems of more than 2 agents, and how does this scale?
* could you provide assurance that results are not from training on test data (lacks details)
The authors tackle an extremely timely problem with significant community interest - how to optimise LLM agents for collaborative tasks.
There are several algorithmic innovations, and the empirical evaluation is extensive.
The observed improvements are significant.
Last but not least, there is a very extensive treatment of related work.
The paper presents a collaborative training method for systems of LLMs, Optima. Optima features a carefully crafted reward function that incentivizes the communications readibility, task performance, and token efficiency. Additionally, a variant of Optima employs MCTS to sample diverse trajectories. Trajectories are then trained on using SFT or DPO. The authors evaluate their algorithm on diverse complex reasoning and information-asymmetric question answering tasks and observe performance improvements in both tasks performance and token efficiency.
* lack of theoretical examination of why the reward function used is working 
* lack of evaluation of models other than llama 8bn (other/smaller models would have been interesting to see).
Yes
No ethics review needed.
1.	Figure 3(a) need further elaboration. How does Optima influence MAS’s inference scaling law comparing to baseline?
2.	I would appreciate further discussion on the advantage of optimizing the effectiveness and efficiency of a MAS versus optimizing these aspects of a single LLM using similar reward functions.
3.	Would improvements in effectiveness and efficiency facilitate the design of communication topologies within MAS?
The paper is well-structured and easy to understand, addressing an important problem with a clearly articulated methodology. The data generation mechanism for MAS is interesting and may facilitate further improvements in MAS, including frameworks like AutoGen.
This work proposes a framework that improve the effectiveness and efficiency of LLMs in multi-agent dialogue systems by iteratively optimizing data and training LLMs using SFT/DPO. For iSFT, it use sampling to generate better data; for iDPO, it use MCTS to generate paired data. Additionally, a carefully designed reward function ensures the system's overall effectiveness and efficiency in task performance.
1.	While this paper offers a well-structured approach to enhancing the effectiveness and efficiency of MAS, iSFT and iDPO have been extensively explored in prior works. Thus, the main contributions here—reward function design and data improvement mechanism in MAS—offer limited novelty within the existing research landscape.

2.	A thorough and fair experimental comparison is very important if the novelty is limited. Another key weakness of this paper is unfair comparison. As this framework has trained LLM to maximize a delicate reward, it’s slightly unfair to compare with prompt-tuning method like CoT, SC, etc.

3.	This paper focuses on LLM-based MAS, but the definition of the problem and the specific methods do not adequately reflect the multi-agent aspect. It appears that the multi-agent scenario is only evaluated in the experiments. The methods are relatively general and do not address the key issues in MAS.

Other trivial weakness in this paper:

1.	Fact error in abstract: iSFT and iDPO with reward filters can be considered as RL, but SFT and DPO are not RL.

2.	The language modeling loss in equation 1 is not defined.
Yes
No ethics review needed.
As the weakness mentioned.
The method is written with significant detail, which is easy to follow.

Introducing an effective and efficient LLM-based frameework, which can be treated as a foundational model, seems important and interesting.

Experiments are provided for various environments and demonstrate promising results.
This paper investigates inter-agent communication and task inference effectiveness within LLM-based MAS. It introduces a framework based on an iterative generate, rank, select, and train paradigm to address these challenges. The core of this framework is iteration, and build iSFT and iDPO based on the iteration paradigm. The iSFT leverages prompt formats to create a diverse dataset. Subsequently, it removes these formats and fine-tunes the model based on the generated trajectories. The iDPO employs Monte Carlo Tree Search (MCTS) to generate diverse data through multi-agent conversations (MAD), alternating between MCTS-based data generation and model updates using DPO.
The authors evaluate this framework across several benchmarks using Llama 3 8B as the baseline model. They achieve promising results compared to Chain-of-Thought (CoT), SC (n=8), MAD, and AutoForm.
1. Unfair Comparison: The methods in OPTIMA do show less token consumption and higher accuracy compared to other approaches. However, the comparisons between iSFT, iDPO, or iSFT-DPO and other methods may be unfair. The authors did not clarify whether OPTIMA’s training is conducted online or offline (I assume it is offline). If it is online, the token consumption should not be lower than CoT (since MCTS requires searching across 24 trajectories). If it is offline, then the token consumption comparison is unfair, as the OPTIMA methods fine-tune the model on diverse prompts or multiple sampled data, which would have already consumed a significant amount of tokens during the post-training phase. The paper, however, only compares token consumption during the inference process.

2. Lack of Novelty: The methods iSFT and iDPO seem to lack innovation. iDPO merely combines MCTS from ToT with DPO, while iSFT simply adds a step of supervised fine-tuning (SFT) after removing the prompt. These methods seem incremental rather than novel, similar approaches can already be found, such as [1], [2], [3], [4], [5].

3. Reward Function: The authors mention the reward function multiple times, but it is only briefly defined in line 146. Meanwhile, in line 79, the reward is described as the core element of OPTIMA’s success. This undermines the credibility of the proposed method since the reward function is not thoroughly explained or explored.

4. Scalability: The authors frequently mention the scalability of OPTIMA, which seems exaggerated. There is no detailed discussion of the framework's ability to scale up in the experiments. Additionally, the token consumption during fine-tuning with iSFT and iDPO would increase significantly as the number of agents grows, which could limit the scalability of the framework.

[1]Wu T, Li X, Liu P. Progress or regress? self-improvement reversal in post-training[J]. arXiv preprint arXiv:2407.05013, 2024.

[2]Pang R Y, Yuan W, Cho K, et al. Iterative reasoning preference optimization[J]. arXiv preprint arXiv:2404.19733, 2024.

[3] Mukobi G, Chatain P, Fong S, et al. SuperHF: Supervised Iterative Learning from Human Feedback[J]. arXiv preprint arXiv:2310.16763, 2023.

[4]Xiong W, Dong H, Ye C, et al. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint[C]//Forty-first International Conference on Machine Learning. 2024.

[5] Pang R Y, Yuan W, Cho K, et al. Iterative reasoning preference optimization[J]. arXiv preprint arXiv:2404.19733, 2024.
Yes
No ethics review needed.
The method proposed in this paper is great, but still I have following concerns which may need further experiments:
1, More tasks study will make this proposed method more convincible, including using more complex tasks and more agents in a task. (StarCraft II, Hanabi and any other tasks which are broadly used in using LLM in MAS problems)

2, Try at least one more base model on all the tasks such as Llama 3.2 or Llama 3 70B
1, Clarity: The paper is well written and have full proof for their argument, the concepts in this paper are easy to follow and understand.
2, Quality: The performance of their designed pipeline is impressive and outperform the baseline a lot.
3, Originality: The paper introduced a new way to enable the development of MAS that are not only effective and efficient but also maintain interpretable communication patterns, which solves the core issues of communication efficiency and collective optimization.
The paper introduces OPTIMA, a novel framework designed to optimize Large Language Model (LLM)-based multi-agent systems (MAS), which shows the potential towards scalable, efficient, and effective LLM-based MAS.. The key focus is on enhancing communication efficiency and task effectiveness through an iterative generate, rank, select, and train paradigm. OPTIMA utilizes a hybrid of reinforcement learning (RL) techniques, including Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO), along with Monte Carlo Tree Search (MCTS)-inspired data generation. The framework is evaluated on two multi-agent settings: (a) information exchange, including information-asymmetric question answering and  (b) debate, encompassing mathematical and reasoning tasks. OPTIMA’s efficiency gains open new possibilities for leveraging inference-compute more effectively, potentially leading to improved inference-time scaling laws.
1, The method is only evaluated on two tasks: (a) information exchange, including information-asymmetric question answering and  (b) debate, encompassing mathematical and reasoning tasks.  Although the performance of proposed method is great on those two tasks, but not sure how this method will perform on the other more complex tasks such as: StarCraft II, Hanabi and so on. (those two environment are broadly used in reinforcement learning tasks and there are also works of using LLM to play those two games)

2, The scalability of this method. The number of agents in showed tasks are limited,. And because the proposed method are aimed to solve the problem of multi-agent tasks, using more agents in environment(maybe at least 4 or 5 agents?) will be more convincible.

3, They use Llama 3 8B as their base model across all benchmarks. However, can this method still have significant improvement compared to baseline if using other base models still remains as a question and need further proof.
We present Optima, a framework for training LLM-based multi-agent systems that boosts communication efficiency and task effectiveness. Using iterative training, we achieve significant token reduction and performance gains across diverse tasks.
@inproceedings{
anonymous2024optima,
title={Optima: Optimizing Effectiveness and Efficiency for {LLM}-Based Multi-Agent System},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=c4w1TqcSi0},
note={under review}
}
Large Language Model (LLM) based multi-agent systems (MAS) show remarkable potential in collaborative problem-solving, yet they still face critical challenges: low communication efficiency, poor scalability, and a lack of effective parameter-updating optimization methods for multi-agent collaboration. We present Optima, a novel framework that addresses these issues by significantly enhancing both communication efficiency and task effectiveness in LLM-based MAS through LLM training. At its core, Optima employs an iterative generate, rank, select, and train paradigm, incorporating a reward function that balances task performance, token efficiency, and communication readability. We explore various RL algorithms, including Supervised Fine-Tuning, Direct Preference Optimization, and their hybrid approaches, providing insights into their effectiveness-efficiency trade-offs for iterative LLM-based MAS training. Additionally, we integrate Monte Carlo Tree Search-inspired techniques for DPO data generation, conceptualizing conversation turns as tree nodes to explore diverse interaction trajectories. We evaluate Optima on common multi-agent tasks, including information-asymmetric question answering and complex reasoning. Our method demonstrates consistent and substantial improvements over single-agent baselines and vanilla MAS based on Llama 3 8B, achieving up to 2.8x performance gain with less than 10\% tokens on tasks requiring heavy multi-agent information exchange. Moreover, Optima's efficiency gains open new possibilities for leveraging inference-compute more effectively, potentially leading to improved inference-time scaling laws. By addressing fundamental challenges in multi-agent collaboration and providing a novel optimization framework, Optima shows the potential towards scalable, efficient, and effective LLM-based MAS.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
llm agent multi-agent inference scaling law
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/599758469d64b2f9e8633e5969be54c9cdbebeb9.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent System
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission