Skill Expansion and Composition in Parameter Space
Skill expansion;skill composition;parameter-efficient finetuning;decision making
Yes
No ethics review needed.
Please refer to the questions above
The paper is easy to follow and clearly written. The experiment suite is diverse and proves the ability of SPEC to learn and compose skills.

Regarding originality and significance I particularly found interesting the usage if diffusion models to adjust the weight levels of the compositions of skills and the study for skill composiiton on the differenc spaces (parameter, noise and action spaces). It made very clear the thought process of the authors to design the framework.
The paper introduces the Parametric Skill Expansion and Composition (PSEC) framework, which allows RL agents to learn and combine new skills by efficiently using a "skill library". Instead of relearning tasks from scratch, PSEC uses LoRA modules—compact, adaptable components—that can be added to this library as plug-and-play skills. This design enables agents to adapt to new tasks by combining skills directly within the model's parameters, allowing them to leverage shared knowledge from previous tasks while avoiding "catastrophic forgetting", since each "skill" is stored as an independent "frozen" module. 

Authors test across different environments, including multi-objective tasks (where skills must be blended to meet multiple goals), settings with continual learning demands, and dynamic scenarios where the environment changes. Results show that PSEC enables efficient, flexible learning compared to vanilla RL.
My biggest concern with this paper is that this is not the first paper proposeing using LoRa for multiple task leanring, e.g. [1,2] and while SPEC is clearly different from previous existing approaches, some level of  comparison, theorethical or empirical would be greatly benefitial. Specifially, at present is difficult to discern what are the novel components within SPEC wrt previous skill learning frameworks.

It would be also good if authors could provide their though contrasting SPEC with existing works on RL agents that learn skills compositionally such as [3-6]. Particularly, [5] even points as an advantage over previous frameworks not having to learn and storing a different set of weight for every skill/sub-task. Since SPEC goes back to this form of learning it would be good that authors include a discussion on this topic.

[1] Ponti, Edoardo Maria, et al. "Combining parameter-efficient modules for task-level generalisation." Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 2023.

[2] Huang, Kaixin, et al. "Solving Continual Offline Reinforcement Learning with Decision Transformer." arXiv preprint arXiv:2401.08478 (2024).

[3] Araki, Brandon, et al. "The logical options framework." International Conference on Machine Learning. PMLR, 2021.

[4] Hill, F., Tieleman, O., von Glehn, T., Wong, N., Merzic, H., & Clark, S. Grounded Language Learning Fast and Slow. In ICLR 2021.

[5] León, Borja G., Murray Shanahan, and Francesco Belardinelli. "In a Nutshell, the Human Asked for This: Latent Goals for Following Temporal Specifications." ICLR 2022.

[6] Vaezipoor, Pashootan, et al. "Ltl2action: Generalizing ltl instructions for multi-task rl." International Conference on Machine Learning. PMLR, 2021.
Yes
No ethics review needed.
o See the concerns and questions from Weaknesses 
 
o If I understand correctly from the paragraph "Context-aware 
Composition" (Line 244-254), only $W_{k}$ is updated at skill k. If 
that's the case, What would be the impact of existing pre-trained LoRA 
weights? Is it possible that because there are two skills that are 
contradictory to each other, the LoRA fine-tuning on a new skill might 
be harder to train? It will be good to see such experiments, and see 
if the method still applies; or an additional experiment can conducted 
where the learning curriculum is shuffled. Will the result be 
consistent with the existing ones where learning happens from standing 
to walking, running? 
 
o Can this method scale to dozens of skills instead of three skills 
presented in this paper? An experiment on stress testing the number of 
skills it can handle will be able to support the effectiveness of the 
proposed method.
o This paper presents an interesting idea of how to compose skills in 
the parameter space, and provide a clear categorization of skill 
composition: parameter-level, noise-level, and action level. This 
paper shows a way where neural network-based skills can be composed at 
the parameter space level, instead of composing at the action output 
level. 
 
o In the long run, the proposed method can open up the possibility for 
efficient skill learning based on large models in decision-making 
domains
This is a paper presents a new skill composition method, which 
composes skills in the parameter space. Concretely, the paper 
introduces PSEC, which fine-tunes new skills using low-rank adaptation 
(LoRA) and argues the actions synthesized through this parameter-space 
composition greatly improves the performance, compared to two other 
composition methods termed noise-level composition and action-level 
composition.
o While the general motivation in the first paragraph makes sense, the 
example needs to be justified a bit better From online statistics, the 
time difference between child learning to walk and learning to stand 
without support is on average 2-2.5 months. Not everyone would call 
this "rapid" (Ln 32) 
 
o While the idea of parameter-level composition is interesting and 
also introduces fewer bottlenecks compared to methods like action 
level, the argument of parameter-level composition being superior is 
not well-grounded. The explanation from line 256 to line 304 lacks 
convincing evidence. 
 
o The t-SNE plot (Figure 4) does not directly reveal that the 
parameter space "shares knowledge" - if 4(a) can be explained to have 
shared knowledge, why is 4(b) not a plot that shows "shared 
knowledge"? The authors might try to design a more specific 
visualization that clearly illustrates the idea of "shared 
knowledge." 
 
o In the experiment sections, there are a lot of terms that authors 
either didn't specify clearly or misuse the words. 
 
    o It is unclear what "versatility" means. Authors should provide a 
    clear definition of this term(Line 354&473) 
 
    o Line 397, the authors shouldn't use the word "generated" to 
    describe other comparison methods, especially when the results 
    haven't been revealed. 
 
    o Line 374 "popular safe offline RL benchmark" is not 
    informative. The authors should replace this with  an explanation 
    of what problems does it contain (robot manipulation? autonomous 
    driving? a collecition of them?) and why is it a good benchmark to 
    test on (large-scale? diverse? or any other key features that make 
    them special) 
 
    o What does "safety" refer to? It's not explained well. The only 
    thing I can take a guess is the cost that shows in the table, 
    which does not really provide much information on how to interpret 
    those numbers. 
 
o The paper touches on multi-task learning and continual learning, and 
uses a skill library. 
All of these topics have been studied to some degree for decades.  Yet 
the majority of citations are from 2023 and 2024, with only two 
citations to work prior to 2016 - and those are just to energy-based 
models and t-SNE.   
Without putting the work in the proper context of prior work, its 
novelty and significance can't be properly assessed.  
 
o Minor: There are many small grammatical errors throughout the paper 
- including in the first sentence of the abstract!
Yes
No ethics review needed.
* One of the main strengths claimed for PSEC is that it "reduced computational costs and memory usage". How does this compare to the baselines?
The key features of PSEC include efficient skill learning and storage utilizing LoRA, direct skill synthesis in parameter space, adaptability through a context-aware module, and the ability for continuous skill expansion. Its effectiveness has been validated through various experiments, demonstrating versatility across different scenarios.
This paper introduces a novel framework called Parametric Skill Expansion and Composition (PSEC). PSEC employs Low-Rank Adaptation (LoRA) to learn and store new skills, and directly synthesizes skills in the parameter space. It incorporates a context-aware module to adaptively compose skills in dynamic environments and utilizes diffusion models for policy modeling. The framework has been evaluated across various scenarios, including multi-objective composition, policy shifts, and dynamics changes.
* While the strong assumption about the expressiveness of the pre-trained policy and the scalability issue of the skill library appear to be weaknesses, these seem to have been addressed in the paper's appendix.
* A potential limitation of the authors' proposed framework is its applicability only to environments where data is available. This could be considered a weakness of the paper. It is conceivable that if the framework could be extended to unseen tasks through transfer learning or fast adaptation techniques, it would demonstrate greater differentiation from previous research.
* Minor:
    * Generally, skill learning, as in  [1] and [2], refers to 'temporal abstraction', but the skill defined by the authors seems to differ from this. This may lead to confusion with existing skill learning concepts.

    * [1] Pertsch, Karl, Youngwoon Lee, and Joseph Lim. "Accelerating reinforcement learning with learned skill priors." Conference on robot learning. PMLR, 2021.
    * [2] Eysenbach, Benjamin, et al. "Diversity is all you need: Learning skills without a reward function." arXiv preprint arXiv:1802.06070 (2018).
Yes
No ethics review needed.
- I think the paper lacks general motivations and overview on the benchmarks used. Why does this paper uses an offline safe RL benchmark instead of just an offline RL benchmarks in general? What are the tasks and what do the recorded data look in the offline benchmarks? Why would they provide a characteristic examples to highlight the pros and cons of PSEC?
- PSEC has a very intuitive and stright-forward design that utilizes LoRA structure. While PSEC wouldn't be the only structure to use the LoRA as building blocks for skills composition, the authors provides tSNE analysis to show why their method encodes skills better than the alternatives
- The paper is very pleasant to read and the paper structure too, is very straightforward. The authors also covers the related works very well which makes PSEC's design choices to be convincing.
- The experimentations are very extensive.
- I found the graphics very easy to understand and they summarize the key points very well. From intro to methods, I could very easily follow the keypoints by reading the details and captions in the graphics beforehand.
This paper proposes Parametric Skill Expansion and Composition framework, which encodes skill primitives offline using LoRA modules and learning skill compositions with the generated LoRA library. The PSPEC framework is designed to cope with multi-objective composition as well as adapting to shift in policy and dynamics. The authors provides extensive experimentation on their proposed algorithm on the D4RL, DSRL, and DeepMind Control Suite.
- LoRA is one of the key elements of the PSEC's structure, and while LoRA is mentioned in the abstract, the authors do not explain what LoRA is until page 2. I think it would be better to say in the abstract what LoRA stands for or simply refer to the module as a skills module if not explaining what LoRA is.
- While the experiments are extensive, I think some of the assumptions and motivations were not well explained. I have included the details in the questions section.
- I think one of the possible limitations of composing skills as a weighted some of LoRA actions is that some skills are not simply some of sub-skills. For example, the skill of throwing a ball is more of a fluid movements where the a person builds up the momentum and transfer the kinetic energy to the ball, instead of some combinations of walking forward and rotating torso. I am curious how the PSEC would perform with more complex tasks.
A flexible framework that can efficiently reuse piror knoweledge to learn new tasks and progressively evolve during training.
@inproceedings{
anonymous2024skill,
title={Skill Expansion and Composition in Parameter Space},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=GLWf2fq0bX},
note={under review}
}
Human excels at reusing prior knowledge to address new challenges and develop skills while solving problems. This paradigm becomes increasingly popular in the development of autonomous agents, as it develops systems that can self-evolve in response to new challenges like human beings. However, previous methods suffer from limited training efficiency when expanding new skills and fail to fully leverage prior knowledge to facilitate new task learning. We propose Parametric Skill Expansion and Composition (PSEC), a new framework designed to iteratively evolve the agents' capabilities and efficiently address new challenges by maintaining a manageable skill library. This library can progressively integrate skill primitives as plug-and-play LoRA modules, facilitating efficient and flexible skill expansion. This structure also enables the direct skill compositions in parameter space by merging LoRA modules that encode different primitives, leveraging shared information across skills to effectively program new skills. Based on this, we propose a context-aware modular to dynamically activate different skills to collaboratively handle new tasks. Empowering diverse applications including multi-objective composition, dynamics shift, and continual policy shift, the results on D4RL, DSRL benchmarks, and the DeepMind Control Suite show that PSEC exhibits superior capacity to leverage prior knowledge to efficiently tackle new challenges, as well as expand its skill libraries to evolve the capabilities.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Skill expansion skill composition parameter-efficient finetuning decision making
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/4d27e7cd1948bfc605d6dfa614e511c2bbfbde0b.pdf
transfer learning, meta learning, and lifelong learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Skill Expansion and Composition in Parameter Space
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission