DynMoLE: Boosting Mixture of LoRA Experts Fine-Tuning with a Hybrid Routing Mechanism
Large Language Model;Self-supervised Fine-Tune;Parameter-Efficient Fine-Tuning;Mixture of LoRA Experts
For using smaller margin to fit in the page limit.
Submission Desk Rejected by Program Chairs
@misc{
li2024dynmole,
title={DynMo{LE}: Boosting Mixture of Lo{RA} Experts Fine-Tuning with a Hybrid Routing Mechanism},
author={Dengchun Li and Naizheng Wang and Zihao Zhang and Haoyang Yin and Lei Duan and Meng Xiao and Yuanchun Zhou and Mingjie Tang},
year={2024},
url={https://openreview.net/forum?id=IpLXuqmP0C}
}
Instruction-based fine-tuning of large language models (LLMs) has achieved remarkable success in various natural language processing (NLP) tasks. Parameter-efficient fine-tuning (PEFT) methods, such as Mixture of LoRA Experts (MoLE), combine the efficiency of Low-Rank Adaptation (LoRA) with the versatility of Mixture of Experts (MoE) models, demonstrating significant potential for handling multiple downstream tasks. However, the existing routing mechanisms for MoLE often involve a trade-off between computational efficiency and predictive accuracy, and they fail to fully address the diverse expert selection demands across different transformer layers.
In this work, we propose DynMoLE, a hybrid routing strategy that dynamically adjusts expert selection based on the Tsallis entropy of the router's probability distribution. This approach mitigates router uncertainty, enhances stability, and promotes more equitable expert participation, leading to faster convergence and improved model performance. Additionally, we introduce an auxiliary loss based on Tsallis entropy to further guide the model toward convergence with reduced uncertainty, thereby improving training stability and performance.
Our extensive experiments on commonsense reasoning benchmarks demonstrate that DynMoLE achieves substantial performance improvements, outperforming LoRA by 9.6\% and surpassing the state-of-the-art MoLE method, MoLA, by 2.3\%. We also conduct a comprehensive ablation study to evaluate the contributions of DynMoLE's key components.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
~Dengchun_Li1 ~Naizheng_Wang1 ~Zihao_Zhang20 ~Haoyang_Yin1 ~Lei_Duan2 ~Meng_Xiao1 ~Yuanchun_Zhou2 ~Mingjie_Tang1
Dengchun Li Naizheng Wang Zihao Zhang Haoyang Yin Lei Duan Meng Xiao Yuanchun Zhou Mingjie Tang
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Large Language Model Self-supervised Fine-Tune Parameter-Efficient Fine-Tuning Mixture of LoRA Experts
I certify that there is no acknowledgement section in this submission for double blind review.
li|dynmole_boosting_mixture_of_lora_experts_finetuning_with_a_hybrid_routing_mechanism
/pdf/9b5b1b3d8da3a6bb105fa9919b8ce6d6be643892.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/de12ac55da3c456883c20e8cbcd6a43869ae94a1.zip
DynMoLE: Boosting Mixture of LoRA Experts Fine-Tuning with a Hybrid Routing Mechanism
ICLR 2025 Conference Desk Rejected Submission
ICLR.cc/2025/Conference/Desk_Rejected_Submission