Tuning Language Models by Mixture-of-Depths Ensemble
Large language model;Parameter-efficient fine-tuning;Interpretability
material on 11th page, not reproducibility or ethics statement, changed margins
Submission Desk Rejected by Program Chairs
@misc{
luo2024tuning,
title={Tuning Language Models by Mixture-of-Depths Ensemble},
author={Haoyan Luo and Lucia Specia},
year={2024},
url={https://openreview.net/forum?id=FLSWAJqTjE}
}
Transformer-based Large Language Models (LLMs) traditionally rely on final-layer loss for training and final-layer representations for predictions, potentially overlooking the predictive power embedded in intermediate layers. Surprisingly, we find that focusing training efforts on these intermediate layers can yield training losses comparable to those of final layers, with complementary test-time performance. We introduce a novel tuning framework, $\textit{Mixture-of-Depths}$ ($MoD$), which trains late layers as ensembles contributing to the final logits through learned routing weights. With the auxiliary distillation loss and additional normalization modules, we ensure that the outputs of the late layers adapt to language modeling. Our MoD framework, which can be integrated with any existing tuning method, shows consistent improvement on various lanaguage modelling tasks. Furthermore, by replacing traditional trainable modules with MoD, our approach achieves similar performance with significantly fewer trainable parameters, demonstrating the potential of leveraging predictive power from intermediate representations during training.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
~Haoyan_Luo1 ~Lucia_Specia2
Haoyan Luo Lucia Specia
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Large language model Parameter-efficient fine-tuning Interpretability
I certify that there is no acknowledgement section in this submission for double blind review.
luo|tuning_language_models_by_mixtureofdepths_ensemble
/pdf/46f5c9631eadbf31b68af5134f3c22e54d93a646.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Tuning Language Models by Mixture-of-Depths Ensemble
ICLR 2025 Conference Desk Rejected Submission
ICLR.cc/2025/Conference/Desk_Rejected_Submission