PaSEFT: Gradient-based Partial Parameter Selection in Efficient Fine-Tuning for Large Language Models
partial parameter;parameter efficient fine-tuning;large language models;multilingual capability
I have read and agree with the venue's withdrawal policy on behalf of myself and my co-authors.
@misc{
jiang2024paseft,
title={Pa{SEFT}: Gradient-based Partial Parameter Selection in Efficient Fine-Tuning for Large Language Models},
author={Shuoran Jiang and Wentao Zhu and Qingcai Chen and Xiangping Wu and Junjie Chen and Yang Xiang and Youcheng Pan and Kai Chen},
year={2024},
url={https://openreview.net/forum?id=jkZ4qOlnLZ}
}
As the severe unfairness of language distribution in the pre-training data, large language models (LLMs) require further fine-tuning for multilingual capability. However, existing parameter efficient fine-tuning (PEFT) methods are unable to enhance multilingual translation capabilities in the incremental learning for LLMs. This paper proposes a language specified gradient attribution (LanGA) method to select the trainable subset from the full-parameter LLMs, which is used for the multilingual translation PEFT. Specifically, LanGA first select a language feature specific subset from full parameters for training, and the selected subset is much smaller than full parameters. Next, LanGA reconstruct the AdamW as a sparse optimizer for parameter subset training. Our experiments demonstrate that LanGA outperforms different low-memory fine-tuning methods for multilingual fine-tuning. At the same time, LanGA costs less memory usage that existing PEFT methods. Importantly, LanGA keeps the translation performance on resource-rich languages in highest measure.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
~Shuoran_Jiang1 ~Wentao_Zhu5 ~Qingcai_Chen2 ~Xiangping_Wu1 ~Junjie_Chen5 ~Yang_Xiang4 ~Youcheng_Pan1 ~Kai_Chen34
Shuoran Jiang Wentao Zhu Qingcai Chen Xiangping Wu Junjie Chen Yang Xiang Youcheng Pan Kai Chen
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
partial parameter parameter efficient fine-tuning large language models multilingual capability
I certify that there is no acknowledgement section in this submission for double blind review.
jiang|paseft_gradientbased_partial_parameter_selection_in_efficient_finetuning_for_large_language_models
optimization
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
PaSEFT: Gradient-based Partial Parameter Selection in Efficient Fine-Tuning for Large Language Models
ICLR 2025 Conference Withdrawn Submission
ICLR.cc/2025/Conference/Withdrawn_Submission