GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMs
large language models;parameter efficient fine-tuning;Gaussian noise;quantization
material on 11th page, not reproducibility or ethics statement
Submission Desk Rejected by Program Chairs
Fine-tuning weights in salient columns produces results comparable to those obtained using low-rank adapters
@misc{
zhelnin2024giftsw,
title={{GIFT}-{SW}: Gaussian noise Injected Fine-Tuning of Salient Weights for {LLM}s},
author={Maxim Zhelnin and Viktor Moskvoretskii and Egor Shvetsov and Venediktov Egor and Maria Krylova and Zuev Aleksandr and Evgeny Burnaev},
year={2024},
url={https://openreview.net/forum?id=InxxTu0Bfs}
}
Parameter Efficient Fine-Tuning (PEFT) methods have gained popularity and democratized the usage of Large Language Models (LLMs). Recent studies have shown that a small subset of weights significantly impacts performance. Based on this observation, we introduce a novel PEFT method, called Gaussian noise Injected Fine Tuning of Salient Weights (GIFT-SW). Our method updates only salient columns, while injecting Gaussian noise into non-salient ones. To identify these columns, we developed a generalized sensitivity metric that extends and unifies metrics from previous studies. Experiments with LLaMA models demonstrate that GIFT-SW outperforms full fine-tuning and modern PEFT methods under the same computational budget. Moreover, GIFT-SW offers practical advantages to recover performance of models subjected to mixed-precision quantization with keeping salient weights in full precision.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
~Maxim_Zhelnin1 ~Viktor_Moskvoretskii1 ~Egor_Shvetsov1 ~Venediktov_Egor1 ~Maria_Krylova1 ~Zuev_Aleksandr1 ~Evgeny_Burnaev1
Maxim Zhelnin Viktor Moskvoretskii Egor Shvetsov Venediktov Egor Maria Krylova Zuev Aleksandr Evgeny Burnaev
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
large language models parameter efficient fine-tuning Gaussian noise quantization
I certify that there is no acknowledgement section in this submission for double blind review.
zhelnin|giftsw_gaussian_noise_injected_finetuning_of_salient_weights_for_llms
/pdf/54960b9298b49a0e117388eb9986eec85b12f1e7.pdf
optimization
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMs
ICLR 2025 Conference Desk Rejected Submission
ICLR.cc/2025/Conference/Desk_Rejected_Submission