AgentRefine: Enhancing Agent Generalization through Refinement Tuning
agent;self-refine;diversity;generalization;data synthesis
Yes
No ethics review needed.
Refer to weakness section.
The proposed method's idea seems like meta learning, which trains the policy on diverse tasks for quickly adapting to novel tasks. This idea makes sense to me and seems new in agent domain. 

I appreciate authors' rethinking on the generalization of agent-tuning. The issue of memorizing trajectory leading to overfitting seems valid to me.

The experiment evaluates the performance of AgentRefine from wide range of perspectives.
The findings establish a correlation between agent generalization and multi-task agent training mechanism / self-refinement, providing a new paradigm for future research in agent-tuning.
The paper proposes a novel framework to improve the generalization capabilities of LLMs based agents. The authors identify that existing agent-tuning methods often overfit to specific environments and fail to generalize to new tasks. To address this, the paper introduces AgentRefine, which leverages a agent synthesis framework to encompass a diverse array of environments and tasks drawing upon extensive human persona data, enabling the model to learn from its mistakes through a process of refinement tuning. The experiments demonstrate that AgentRefine method outperforms state-of-the-art methods in terms of generalization, robustness to perturbations, and the ability to generate diverse thoughts during inference.
Overall AgentRefine is a simple and effective method. However, the main idea is not new, as discussed in related work, Agent-FLAN and AgentGen have proposed to train generalist agents using general data. The idea of refinement is also widely studied as discussed in introduction. I encourage authors to clearly differentiate AgentRefine from these prior works. Highlight unique aspects or improvements over existing methods. Consider incorporating a comparative analysis to demonstrate the advantages of AgentRefine.

I feel the procedure suffers from a high risk of generating low-diversity tasks, as the script generation is based on human persona data, which is limited in a certain domain. In contrast, a generalist agent is expected to complete any tasks. 

The goal of the proposed method is to build a LLM-based agent to generalize to novel tasks. However, this way to generate agent tasks does not bring new knowledge to LLMs, but enabling the LLMs to follow the output format more strictly, as it trains LLMs on the data generated by LLMs themselves. 

Besides, the source of performance improvement is not clear. For instance, why the LLM-generated trajectories can improve performance on novel tasks? Authors can provide some examples of the evaluation tasks, and examples of the generated tasks.
Yes
No ethics review needed.
see weaknesses
1. The paper is well-organized and easy to follow, with a clear progression from motivation to methodology.
2. The identification of the generalization gap in existing LLM-based agents and the proposal of a self-refinement approach to address it is a rational step forward in the field.
The paper presents a framework aimed at improving the generalization capabilities of Large Language Model (LLM) based agents through instruction tuning. The authors observe that existing agent training methods overfit to specific environments and struggle with new situations, leading to poor generalization. To address this, they propose AgentRefine, which incorporates self-refinement processes to enable the model to learn from its mistakes and adapt to diverse environments and tasks.
1. The problem of generalization in LLM-based agents has been extensively discussed in previous literature, making the contribution of this work less novel. For example,  [1]  investigates the robustness of accuracy measurements in large language models (LLMs) when the order of answer labels is shuffled, using the MMLU dataset as a testbed.
2. The methodology, while intuitive, lacks significant innovation, as the approach of enhancing generalization through data synthesis is not new [2].
3. The experimental results do not demonstrate a strong improvement over existing methods, which questions the practical impact of the proposed approach. An apple-to-apples comparison of your main results to show the advantage of the algorithm would make your results more straightforward and strong, instead of using a lot of underlined text to filter out the results where training data is sampled in the same environment as the task.

[1] Changing Answer Order Can Decrease MMLU Accuracy.

[2] Knowledgeable Agents by Offline Reinforcement Learning from Large Language Model Rollouts.
Yes
No ethics review needed.
1. Can you also provide more detailed statistics of your experiments, for example, the std of each task?

2. How does the agent get an error signal during the evaluation?

3. For the LLaMA-3-70B Series, the performance of AgentRefine is worse than the base model? Am I misunderstanding something?
1. This paper discusses the generalization ability of agents, which is a very important topic for the community.

2. The authors provide quantitative analysis to explain their insight, which is very convincing.

3. Synthesizing data with almost no task-specific information is a very practical setting, and the improvement of generalization ability in this paper is impressive.
This paper discusses using synthetic data to improve the generalization ability of agents on held-out sets. Previous agent-tuning work often chose to construct agent-tuning data on held-in sets. The authors demonstrate that although these methods can greatly improve the performance of agents on held-in sets, they usually lead to overfitting, which in turn affects the performance of agents on held-out sets. Based on this observation, the authors propose AgentRefine. This method does not use task-related information at all. Instead, it uses LLM to complete the entire data generation process, including task generation, trajectory generation, and verification to construct the agent-tuning dataset, thus avoiding the possibility of overfitting to held-in sets from the very start. In the constructed dataset, the authors emphasize the ability of the agent to correct errors based on the feedback, which further improves the agent's generalization ability. They validate AgentRefine in multiple scenarios, and the experimental results show that finetuned agents outperform other baselines on held-out sets.
1. The presentation of this paper should be improved and some grammar mistakes should be fixed.

2. Some important baselines, for example, Reflexion[1], are missing and should be included.

3. They only consider decision-making tasks in their experiments. However, as they claimed on the generalization ability, tasks of different types should also be included, for example, reasoning tasks. 

[1] Shinn, Noah, et al. "Reflexion: Language agents with verbal reinforcement learning." NeurIPS, 2023.
Yes
No ethics review needed.
See above weakness section
1. The introduction of a self-refinement process for agent tuning is a novel contribution to the field. By allowing agents to correct their mistakes based on environmental feedback, the authors propose an interesting alternative to traditional fine-tuning methods.
2. The use of diverse environments and tasks in data generation helps mitigate overfitting to specific scenarios, which is a common issue in LLM-based agents.
3. The experiments show that AgentRefine outperforms baselines in held-out tasks, suggesting that the approach has potential for improving generalization.
The paper proposes AgentRefine, a framework designed to enhance the generalization capabilities of large language model (LLM)-based agents through a self-refinement process. The core idea is to enable agents to learn from their mistakes by refining their actions based on feedback from the environment. The authors introduce a data generation pipeline that simulates diverse environments and tasks, followed by a refinement tuning process to improve agent robustness and generalization. Experimental results show that AgentRefine outperforms state-of-the-art methods in held-out tasks, demonstrating improved generalization and robustness.
1.  The paper relies heavily on GPT-4 for generating both scripts and trajectories. This raises several concerns:
   - The quality of the generated data depends entirely on GPT-4's ability to detect and correct errors
   - The method is not truly "self-refinement" since it requires external stronger models for error detection and correction
   - The authors should analyze what happens when using weaker LLMs for data generation and verification

2. The verification process has potential flaws:
  - It uses LLMs to verify the correctness of scripts and trajectories without human validation
  - The paper lacks analysis of verification failure cases or error rates
  - The authors should include human evaluation of the verification process accuracy

3. While the paper shows improved performance, it lacks analysis of whether this is simply distillation from GPT-4 rather than true generalization and how much of the improvement comes from the refinement process versus having access to GPT-4's knowledge

4. The experiments only scale up to 64k examples. Would the computational cost of generating refinement data with GPT-4 makes large-scale training difficult? Also, the authors should analyze the cost-benefit tradeoff of generating more refinement data

5. While the paper shows some robustness analysis, the perturbation experiments are limited to only action descriptions. More diverse types of perturbations should be tested. The analysis should include how different components (script generation, verification, refinement) contribute to robustness
The self-refine data can expand the search space of LLM agent and improve the reason quality, leading a generalized performance in agent tasks.
@inproceedings{
anonymous2024agentrefine,
title={AgentRefine: Enhancing Agent Generalization through Refinement Tuning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=FDimWzmcWn},
note={under review}
}
Large Language Model (LLM) based agents have proved their ability to perform complex tasks like humans. However, there is still a large gap between open-sourced LLMs and commercial models like the GPT series. In this paper, we focus on improving the agent generalization capabilities of LLMs via instruction tuning. We first observe that the existing agent training corpus exhibits satisfactory results on held-in evaluation sets but fails to generalize to held-out sets. These agent-tuning works face severe formatting errors and are frequently stuck in the same mistake for a long while. We analyze that the poor generalization ability comes from overfitting to several manual agent environments and a lack of adaptation to new situations. They struggle with the wrong action steps and can not learn from the experience but just memorize existing observation-action relations. Inspired by the insight, we propose a novel AgentRefine framework for agent-tuning. The core idea is to enable the model to learn the correct its mistakes via observation in the trajectory. Specifically, we propose an agent synthesis framework to encompass a diverse array of environments and tasks and prompt a strong LLM to refine its error action according to the environment feedback. AgentRefine significantly outperforms state-of-the-art agent-tuning work in terms of generalization ability on diverse agent tasks. It also has better robustness facing perturbation and can generate diversified thought in inference. Our findings establish the correlation between agent generalization and self-refinement and provide a new paradigm for future research.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
agent self-refine diversity generalization data synthesis
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/bac3eb83aeadb4ebbf4d37dddb99050f52328892.pdf
other topics in machine learning (i.e., none of the above)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/5c6e80f2a78ca04ce7eb2d7301cb7dccbc1d9086.zip
AgentRefine: Enhancing Agent Generalization through Refinement Tuning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission