Attention-aware Post-training Quantization without Backpropagation
Quantization;Hyper-scale LLMs;Attention;Hessian
Yes
No ethics review needed.
Refer to 2 in weakness. What is the effectiveness of proposed approaches in terms of efficiency?
1. The proposed BOA consider inter-layer dependencies within the attention module when optimize a weight-rounding mechanism. It is beneficial to maintain higher quantization accuracy, especially at low-bit precision.

2. The proposed BOA method demonstrates impressive results, especially in the low-bit regime (e.g., INT2 quantization).

3. The paper includes extensive experiments across multiple model types and sizes, demonstrating  scalability across LLMs of different parameter counts.
This paper presents a novel post-training quantization (PTQ) method, termed BOA (Backpropagation-free Optimization for Attention-aware PTQ), targeting large language models (LLMs) without relying on backpropagation. The approach introduces attention-aware Hessian matrices that capture inter-layer dependencies within the attention module, aiming to improve quantization accuracy, especially at low bit-widths (e.g., INT2). BOA incorporates techniques like Hessian relaxation and efficient computation of inverse Hessians to mitigate the high computational costs. The method is benchmarked against existing PTQ approaches on LLMs, demonstrating improved performance in terms of perplexity and zero-shot task accuracy.
1. Novelty Limitations: The primary contribution, the attention-aware Hessian matrix, is an incremental improvement over existing Hessian-based PTQ methods. While capturing inter-layer dependencies within the attention module is beneficial, the idea is not a novel quantization paradigm. 

2. The authors introduce optimizations approaches like Hessian relaxation and efficient computation of inverse Hessians, but the results did not show the effect of these optimization methods.
Yes
No ethics review needed.
The advantage over existing methods is not substantial, suggesting the need for further validation, such as, SmoothQuant, LLMC,QuIP etc.
The topic of this paper is of significant importance and represents one of the most active and rapidly evolving research areas in the field. As LLMs grow increasingly complex, their deployment on resource-constrained devices requires innovative solutions to reduce computational and memory demands. Quantization, as a compression technique, has gained considerable traction for enabling efficient deployment of LLMs without sacrificing model accuracy.
The paper introduced the BOA post-training quantization algorithm designed for LLMs that overcomes the limitations of traditional quantization methods, which struggle with inter-layer dependencies and backpropagation requirements in LLMs. BOA leveraged attention-aware Hessian matrices to better capture inter-layer interactions within the attention module, enhancing performance, especially at low bit-widths. The algorithm employed Hessian relaxation and head-wise simultaneous quantization, to attempt to reduce computational and memory costs, making it feasible for quantizing LLMs without backpropagation.
The technical approach of this paper is relatively straightforward, lacking intricate or highly novel methodologies. Additionally, certain English terminology within the paper is used imprecisely, which may affect clarity and readability. The comparison methods are somewhat limited, providing a narrow benchmark for evaluating the proposed technique. Moreover, while the experimental results demonstrate some improvements, the advantage over existing methods is not substantial, suggesting the need for further validation, such as, SmoothQuant, LLMC,QuIP etc.
Yes
No ethics review needed.
1.How does the performance of BOA compare when tested on more advanced models, such as the LLaMA3 series, instead of the relatively outdated models used in the paper?
2.How does BOA's accuracy compare to more recent quantization methods, such as QuaRot and SpinQuant?
1.	The paper introduces an innovative PTQ method that cleverly captures inter-layer dependencies within attention modules through attention-aware Hessian matrices while avoiding backpropagation overhead. 
2.	BOA is compatible with other techniques, such as SmoothQuant and Z-FOLD, enabling further improvements in quantization accuracy by integrating different quantization strategies.
This paper presents a post-training quantization method called BOA that incorporates inter-layer dependencies without relying on backpropagation. BOA leverages attention-aware Hessian matrices to capture dependencies within the attention module, a relatively rare approach in existing PTQ methods. Additionally, BOA demonstrates compatibility with techniques like SmoothQuant and Z-FOLD, allowing for further enhancements in quantization performance. However, despite these strengths, BOA does not show sufficient memory and processing time benefits compared to existing PTQ methods. The experiments are conducted on outdated models, and the comparison methods lack recent advancements. Adding more experiments with up-to-date models and techniques would strengthen the paper.
1.	The experiments are primarily conducted on BLOOM, LLaMA1, and OPT models, which are somewhat outdated compared to current state-of-the-art models. The paper lacks validation on more recent models, such as the LLaMA3 series.
2.	Although the paper introduces various techniques to reduce computational overhead and claims to use a Hessian-based strategy to avoid time-consuming gradient-based optimization, as shown in Table 13, BOA’s actual overhead in terms of memory and processing time is greater than GPTQ. Additionally, in Tables 3, 4, and 5, even under 2-bit quantization, BOA's improvement over GPTQ is marginal. For Table 6, it’s worth noting that GPTQ can also integrate certain quantization algorithms, like QuaRot [1] and SpinQuant [2], to achieve better results. Including comparisons with these methods is recommended.	

[1] Ashkboos S, Mohtashami A, Croci M L, et al. Quarot: Outlier-free 4-bit inference in rotated LLMs. arXiv preprint arXiv:2404.00456, 2024.
[2] Liu Z, Zhao C, Fedorov I, et al. SpinQuant—LLM quantization with learned rotations. arXiv preprint arXiv:2405.16406, 2024.
Yes
No ethics review needed.
Please refer weaknesses for details.
1. Introducing inter-layer interaction in a training-free manner is innovative.
2. The paper is well-written.
This paper presents a training-free post-training quantization method based on GPTQ. It introduces inter-layer interaction by calculating Hessian matrices using an attention module instead of a simple linear module in LLMs. Additionally, the paper proposes techniques to improve the efficiency of Hessian matrix calculations.
1. The experimental setup is somewhat outdated. Additional experiments on newer models, such as LLama-2 and LLama-3, are needed.
2. Although the paper introduces a training-free PTQ method, it may be slower than training-based methods. For example, Table 2 shows that BOA takes 1 hour to quantize 2.7B models, while GPTQ quantizes larger 13B models in only 21 minutes. OmniQuant, a training-based method, requires only ~1.1 hours for 7B models. The paper should provide comprehensive comparisons of quantization times to demonstrate the proposed method's effectiveness.
3. The paper focuses on 2-bit per-channel quantization and mentions that "group-wise parameters result in additional memory costs and processing time during inference." However, weight-only quantization aims to alleviate memory constraints during the decoding stage. Group-wise quantization introduces negligible overhead but significantly improves performance and is a common practice in existing inference engines. Therefore, the paper should include results for group-wise quantization.
We propose a novel post-training quantization algorithm that considers inter-layer dependencies inside the attention module without relying on backpropagation.
@inproceedings{
anonymous2024attentionaware,
title={Attention-aware Post-training Quantization without Backpropagation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=0L8wZ9WRah},
note={under review}
}
Quantization offers a promising solution for deploying large-scale language models (LLMs) on resource-constrained devices. However, early quantization methods, developed for smaller networks like ResNet, rely on gradient-based optimization, which becomes impractical for hyper-scale LLMs with billions of parameters. While recently proposed backpropagation-free post-training quantization (PTQ) methods alleviate this issue, their performance is limited by a lack of inter-layer dependency consideration. In this paper, we introduce a novel PTQ algorithm that incorporates inter-layer dependencies without relying on backpropagation. The key innovation is the development of attention-aware Hessian matrices that capture inter-layer interactions within the attention module. Extensive experiments demonstrate that our approach significantly outperforms conventional PTQ methods, particularly at low bit-widths.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Quantization Hyper-scale LLMs Attention Hessian
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/1b6210fb7bc41a5685bb0f16f329e2cea4a831f1.pdf
other topics in machine learning (i.e., none of the above)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Attention-aware Post-training Quantization without Backpropagation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission