Agent Skill Acquisition for Large Language Models via CycleQD
Large Language Models;Skill Acquisition;Quality Diversity
Yes
No ethics review needed.
I am also curious about any real-world application tests using the proposed method across different scenarios, given that the scenarios in the experiments are quite similar.
The paper presents an innovative approach to LLM skill acquisition, uniquely applying the Quality Diversity paradigm to cycle through task-specific optimizations, which is quite novel. 
The experimental results on the proposed CycleQD framework show a substantial performance gain, validating the model’s effectiveness and overall performance.
Overall, CycleQD introduces a scalable method to merge agent skills into LLMs, addressing critical challenges in agent-based LLM design.
This paper presents CycleQD, a novel framework for making llm acquire specific skills through Quality Diversity (QD) optimization. CycleQD addresses challenges in skill training, such as data imbalance and inadequate objective functions, by cyclically alternating quality measures for different tasks.  CycleQD shows competitive results, achieving performance on par with GPT-3.5-TURBO.
Although the CycleQD  is designed for the agent skill acquirment of LLMs, the experiments predominantly focus on computer science tasks, such as OS and DB. Its applicability to other fields as agents will futher strength this paper.

 The methodology, though effective, involves a multi-step process (crossover, mutation, cyclic quality alternation) that may be complex. The author may need to provide a clearer illustration of the methodology, such as an overview figure of the method pipeline.

The ablation study does not provide substantial evidence of the proposed method's effectiveness, as the performance gains of each component are minimal.
Yes
No ethics review needed.
Formatting: I see that a couple half-column figures are used (concatenated horizontally with text). Please make sure that this abides by the formatting rules and correct if not.

Is it normal for model merging approaches to test using just 2-3 tasks?
This work appears to be very original: (1) it is different from conventional fine-tuning and even model merging methods, and (2) it alleviates certain design decisions, like data mixing ratios and different objectives, when fine-tuning on multiple tasks.

I understand that previous evolutionary approaches don't update the weights directly and this method proposes to do so (using the framing of model merging). I think this is a significant contribution. I think the specific adaptations they propose are also positive contributions.

From what I can see, this method establishes sufficient gains over the alternative methods in the computer science tasks. Since this paper introduces several components, I was looking for (and was satisfied to find) the ablations in Table 3.
This paper introduces evolutionary approaches for merging large language models (trained on individual skills), as an alternative to fine-tuning (jointly on multiple skills). This is the first work to merge models using Quality Diversity (QD) algorithms. This work introduces a cyclic adaptation for QD: the optimization is conducted for one task at a time (in a cycle). Models are merged in the crossover operation by linearly combining task vectors. Finally, the mutation function modifies the "child" (from crossover) by sampling perturbations from the SVD components of its task vector.

This approach is tested using coding, OS, and DB skills. This evolutionary approach for model merging outperforms both fine-tuning based methods (by +5.4 pts) and other model merging methods (by +5.5 pts). The ablations show that each component of this method (cycle adaptation, SVD-based mutation, and elite sampling) lead to gains (on average). This method also generalizes to out-of-distribution coding tasks (more than fine-tuned, single-task experts) while also retaining performance on language skills. On the other hand, the merged model produced by this approach underperforms on individual tasks compared to fine-tuned experts for the domain of image segmentation.
I think the writing is mostly clear, but this paper assumes some extra background knowledge about evolutionary methods. I would really appreciate it if the authors (briefly) introduce such approaches and terminology from first principles, before the details of their augmentations.

It would also be great if the authors could again motivate the need for evolutionary approaches (rather than fine-tuning) in training or merging models. Is it just the avoidance of design decisions that I mentioned above? Are there more fundamental motivations (like the diversity of skills captured by this method, which was briefly mentioned)? Or is this direction of work primarily exploratory?

I am not already familiar with evolutionary approaches, so no significant methodological weaknesses are obvious to me. I will wait to see what other reviewers have to say, but generally hold a positive opinion for now.
Yes
No ethics review needed.
1. Questions on presentation issues: see above. 
2. Is it possible to combine finetuning and CycleQD, e.g., SFT after CycleQD? What would be the result.
3. Computation and time cost comparisons between CycleQD and gradient-based fine-tuning.
4. The number of generations is critical for selecting elites. How sensitive is CycleQD regarding this hyperparameter?
- The paper discussed a critical problem: achieving superior performance on multiple skills while retaining language capabilities. 
- The proposed method is novel.
The paper proposed CycleQD to achieve superior performance on multiple agent skills while retaining robust language capabilities.
Overall, the major weakness of this paper is its presentation. For the audience not familiar with evolutionary computation, the paper is hard to follow.
1. [Presentation] Problems in understanding the proposed method.
	- Explain the evolution of figure 4 by referring to algorithm 1. The figures are meant to give direct intuitions about how the algorithm works, but the current delivery fails to achieve this purpose. 
	- In the elite sampling (line 217), what does the formula intuitively means? Specifically, what is the meaning of $ \gamma$ . 
	- In the crossover (Sec. 3.2), there are no explanations for why $w$ are sampled from gaussian? From my understanding, $w$ are used to determine component weights for combining task vectors. The component weights should be greater than $0$ . But Gaussian samples has infinite range. 
	- In the mutation (Sec. 3.3), there are no explanations for why SVD can address the challenge of excessive exploration in parameter space or overfitting. In addition, clarify how the proposed SVD-based mutation allows finer control of $\theta_{child}$ (line 283).
	- In the model aggregation (Sec. 3.4), there are no explanations for why the proposed convex combination is adopted for obtain the final model. In particular, what is the connection between this design choice and the major goal of this paper, i.e., achieving superior performance on multiple tasks without degradation on other tasks.
2. [Contribution] CycleQD on SAM is irrelevant with the main topic of this paper, which focuses on language models.
3. [Soundness] The current experiments include only three tasks. Including more tasks to support the generality of CycleQD is better.
Yes
No ethics review needed.
1. Why is cyclic optimization more effective than simultaneous optimization of multiple objectives?
2. What is the theoretical basis for using SVD decomposition in mutation, how does it help explore the parameter space more effectively?
3. Can this method scale to combining skills from vastly different domains (e.g., math, creative writing, and visual reasoning)? Would it still outperform traditional SFT?
1. Demonstrates strong empirical results by achieving comparable performance to GPT-3.5-TURBO on coding, OS and DB tasks using only an 8B parameter model
2. Avoids the complex tuning of data ratios and objective functions by decomposing multi-skill training into single-skill training followed by model merging
3. Provides full ablation studies validating the necessity of key components including cyclic adaptation, SVD-based mutation, and elite sampling
This paper introduces CycleQD, a framework for enhancing large language models' skill acquisition through cyclical quality diversity optimization. CycleQD first trains expert models for individual skills, then combines them using model merging as crossover operations and SVD-based mutations, while cycling through different task metrics as quality measures. The method alternates between optimizing different skills by maintaining separate archives, each tracking models with diverse behavioral characteristics.
1. Elite sampling strategy lacks theoretical foundation and uses arbitrary hyperparameters (0.5-0.8 range) without clear justification.
2. Computational costs and efficiency not analyzed or compared with baseline methods.
3. Missing comparison with established multi-objective optimization approaches like NSGA-II or Pareto-based methods.
@inproceedings{
anonymous2024agent,
title={Agent Skill Acquisition for Large Language Models via Cycle{QD}},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Kvdh12wGC0},
note={under review}
}
Training large language models to acquire specific skills remains a challenging endeavor. Conventional training approaches often struggle with data distribution imbalances and inadequacies in objective functions that do not align well with task-specific performance. To address these challenges, we introduce CycleQD, a novel approach that leverages the Quality Diversity framework through a cyclic adaptation of the algorithm, along with a model merging based crossover and an SVD-based mutation. In CycleQD, each task’s performance metric is alternated as the quality measure while the others serve as the behavioral characteristics. This cyclic focus on individual tasks allows for concentrated effort on one task at a time, eliminating the need for data ratio tuning and simplifying the design of the objective function. Empirical results from AgentBench indicate that applying CycleQD to LLAMA3-8B-INSTRUCT based models not only enables them to surpass traditional fine-tuning methods in coding, operating systems, and database tasks, but also achieves performance on par with GPT-3.5-TURBO, which potentially contains much more parameters, across these domains. Crucially, this enhanced performance is achieved while retaining robust language capabilities, as evidenced by its performance on widely adopted language benchmark tasks. We highlight the key design choices in CycleQD, detailing how these contribute to its effectiveness. Furthermore, our method is general and can be applied to image segmentation models, highlighting its applicability across different domains.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Large Language Models Skill Acquisition Quality Diversity
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/dae6d3f49c63986d67911bb361da9fe899daa4cf.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/ad29c3ee781f63e5b58d9cffc93f06e6a63596fa.zip
Agent Skill Acquisition for Large Language Models via CycleQD
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission