Layer-Wise Quantization: A Pragmatic and Effective Method for Quantizing LLMs
Layerwise Quantization of LLMs based on layer importance;memory-constraint quantization;variable decimal-point bit quantization based on memory availability;reduced model size for resource-efficient NLP systems
Violate anonymity policy.
Submission Desk Rejected by Program Chairs
Layerwise LLM quantization based on importance, with less critical layers in lower bits and key layers in higher bits, enables memory efficiency. Supports any quantization technique, enables decimal-point bit quantization for low-memory settings.
@misc{
dumitru2024layerwise,
title={Layer-Wise Quantization: A Pragmatic and Effective Method for Quantizing {LLM}s},
author={Razvan-Gabriel Dumitru and Vikas Yadav and Rishabh Maheshwary and Paul Ioan Clotan and Sathwik Tejaswi Madhusudhan and Mihai Surdeanu},
year={2024},
url={https://openreview.net/forum?id=eJVrwDE086}
}
We present a simple meta quantization approach that quantizes different layers of a large language model (LLM) at different bit levels, and is independent of the underlying quantization technique. Specifically, we quantize the most important layers to higher bit precision and less important layers to lower bits. We propose two effective strategies to measure the importance of layers within LLMs: the first measures the importance of a layer based on how different its output embeddings are from the input embeddings (higher is better); the second estimates the importance of a layer using the number of layer weights that are much larger than average (smaller is better). We show that quantizing different layers at varying bits according to our importance scores results in minimal performance drop with a far more compressed model size. Finally, we present several practical key takeaways from our variable layer-wise quantization experiments: (a) LLM performance under variable quantization remains close to the original model until 25–50% of layers are moved in lower quantization using our proposed ordering but only until 5–10% if moved using no specific ordering; (b) Adding layer importance to inherently dynamic quantization techniques can further improve their performance, showing that our approach is complementary to other dynamic quantization methods; (c) Quantizing LLMs to lower bits performs substantially better than pruning unless extreme quantization (2-bit) is used; and (d) Layer-wise quantization to lower bits works better in the case of larger LLMs with more layers compared to smaller LLMs with fewer layers.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
~Razvan-Gabriel_Dumitru1 ~Vikas_Yadav2 ~Rishabh_Maheshwary2 ~Paul_Ioan_Clotan1 ~Sathwik_Tejaswi_Madhusudhan2 ~Mihai_Surdeanu1
Razvan-Gabriel Dumitru Vikas Yadav Rishabh Maheshwary Paul Ioan Clotan Sathwik Tejaswi Madhusudhan Mihai Surdeanu
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Layerwise Quantization of LLMs based on layer importance memory-constraint quantization variable decimal-point bit quantization based on memory availability reduced model size for resource-efficient NLP systems
I certify that there is no acknowledgement section in this submission for double blind review.
dumitru|layerwise_quantization_a_pragmatic_and_effective_method_for_quantizing_llms
/pdf/b1862d650ad71eab1fb88cf708a02617952cbc33.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Layer-Wise Quantization: A Pragmatic and Effective Method for Quantizing LLMs
ICLR 2025 Conference Desk Rejected Submission
ICLR.cc/2025/Conference/Desk_Rejected_Submission