Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolbox
LLM;Quantization;Evaluation;OOD
> W1: This paper does not propose new algorithms but rather tests quantization algorithms proposed by other researchers before. Can the authors provide some more insights, such as: does the generalization performance of different quantization algorithms differ?

Thank you for your suggestion!

This paper primarily focuses on evaluating the effects of distribution shifts from calibration to test sets on quantized LLMs, rather than on algorithm design. In future work, we will build on this research to develop algorithms that improve calibration set selection, optimizing quantized model performance from a data perspective—an aspect that has not been explored in the quantization field thus far.

Due to space constraints, we did not elaborate on additional findings in the paper, so I’ll provide some supplementary insights here.

For experiments in S1 (Figure 2), we observe that GPTQ experiences significant performance drops at lower bits, such as 2-bit, whereas SPQR does not exhibit this issue and may even show performance gains. SPQR is specifically designed for low-bit scenarios, which may be related to its capability to identify and isolate outlier weights.

In the cross-dataset distribution shift experiments in S2 on BOSS (Table 2), we find that at 4-bit, GPTQ, SPQR, and AWQ achieve performance close to full precision, but at 3-bit, both GPTQ and AWQ suffer notable performance losses. Additionally, SmoothQuant consistently shows a greater performance drop relative to full precision, likely due to activation quantization, indicating that activation quantization remains a challenging issue. It is also noteworthy that while 3-bit shows minimal performance drop in S1, it encounters significant losses on BOSS, suggesting that quantized LLMs generally exhibit lower generalization on the BOSS dataset compared to datasets in S1.
From the prompt paradigm perspective, few-shot settings significantly recover model performance, restoring accuracy by up to 40%, demonstrating the performance-boosting effect of in-context learning for LLMs.
Regarding distribution shift characteristics, we observe that calibration dataset suitability varies across algorithms. For example, with GPTQ, the SQA dataset performs better on SQ tasks, while no single dataset stands out for AWQ. This variation may stem from differences in how each algorithm leverages calibration data internally.

Overall, from an algorithmic perspective, **generalization capability varies significantly across quantization algorithms**.
> W2: Visualization Issues: 
Radar charts (Figures 2, 5, and 6) lack marked magnitudes for the scores on the radius, and text overlays reduce clarity.
The task types, while indicated by background colors, are not explicitly labeled. An additional legend would make the visualizations more intuitive.

Thank you very much for your valuable suggestion!

In future versions, we will optimize these images to enhance clarity and readability, allowing readers to more easily see specific values and task types.
> Q: Did the authors experiment with different samples from the C4 dataset? Did authors measure variance even when using the same dataset, like C4, but with different examples? Understanding these aspects would provide deeper insights into the robustness and reliability of the quantization process.
In line 1249, the authors mentioned: We present the average results with random seeds 42 and 567. Why particular choose 42 and 567 as random seed? What if we use other random seeds, like 0 or 1?

Thank you for your question and suggestion!

In our experiments, we used C4 as the calibration set for the S1 scenario. For this experiment, we selected two random seeds, 42 and 567, to perform two different samplings on C4, with the results shown in **Figure 6**. By comparing the outcomes from the two different random seeds, we found that the experimental results showed almost no variation, aside from a few isolated cases.

To avoid further increasing our already extensive workload, we initially conducted the experiment with only two random seeds. However, to ensure rigor, we are currently expanding the experiment with additional random seeds, such as 0 or 1.
> W3: This article appears to be a superficial description and summary of experimental phenomena, lacking in-depth discussion.

Thank you for your suggestion!

In **Section 3.3**, we conducted an in-depth analysis that includes summarizing conclusions, formulating hypotheses, and conducting validations. First, we concluded from the experiments that: Consistency between calibration data and test distribution does not always yield optimal performance. Then, we proposed the hypothesis that LLMs may not require highly relevant data related to downstream tasks to recover performance loss due to quantization. Subsequently, we compared results using C4 as the calibration set against those using I.I.D downstream task datasets as the calibration set. We found that I.I.D downstream tasks as calibration sets do not outperform high-quality pretraining corpora as calibration sets, with both yielding comparable performance. This further supports our hypothesis that, unlike fields such as CV, highly relevant data as calibration sets do not significantly enhance performance.
> W2: The number of LLMs using for quantizing in the experiment is too small, and their size is relatively small (7B). This limits the generality of the experimental results to a certain extent.

Thank you for your comment!

First, we would like to note that we have also tested **LLaMA2-13B** and presented the results in **Table 15**, thereby expanding the range of models. Additionally, we are currently supplementing experiments with larger models to ensure a more comprehensive and robust evaluation.
> W1: A serious issue is that the authors claim that this article is the first to study the impact of the calibration set on the generative capacity of quantized large models. However, to my knowledge, similar work has already been done previously [1]. Therefore, the authors' statement is quite inappropriate.

Thank you for your careful review!

We would like to emphasize that we do not claim to be the first to study the impact of calibration datasets on quantized LLMs. Rather, we are the first to investigate distribution shifts between calibration and test datasets on quantized LLMs, as well as to conduct cross-subject distribution shift experiments, filling a gap in evaluating the generalization ability of quantized LLMs. If there is content suggesting otherwise, please indicate the specific location. We also discuss this paper [1] on lines 490 and 505 in Section 5, and Table 7 highlights the distinctions between our work and all prior studies, as noted in our response to Q1.

[1] Miles Williams and Nikolaos Aletras. 2024. On the Impact of Calibration Data in Post-training Quantization and Pruning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10100–10118, Bangkok, Thailand. Association for Computational Linguistics.
> Q2: Given that the evaluation could potentially be performed by extending existing toolboxes, what is the necessity of developing a new quantization and evaluation framework?

Thank you for your question!

To the best of our knowledge, no existing toolbox provides a framework for evaluating distributional shifts from calibration to test sets to assess the generalization ability of quantized models. Additionally, our toolbox implements a wider range of algorithms and datasets, enabling more extensive testing of quantized models. Our toolbox also supports the combination of multiple quantization algorithms, achieving better performance than using any single algorithm alone.
> Q4: The athors should further describe the definition of IID and OOD which appears abruptly. Does IID means the different or the same dataset under the same subject?

Thanks for this suggestion.

I.I.D (Independent and Identically Distributed) refers to a set of random variables that are both independent and follow the same probability distribution, meaning that each data point's value is unaffected by others and all data points come from the same distribution. OOD (Out-of-Distribution) refers to data whose distribution lies outside the range of the distribution seen by the model during training, meaning that the features or distribution of the test data differ from the training data, often leading to poor performance on such data. The concepts of I.I.D and OOD are well-established in the CV field but are relatively underdeveloped in NLP.

In this paper, our definitions of I.I.D and OOD follow and extend the setting in [1], considering distribution shifts from the calibration set to the test set. We have provided an explanation of I.I.D and OOD in this experiment in **Section 1** and **Figure 1**.  A test set that uses data with the same distribution as the calibration set is considered the I.I.D setting, while a test set using data from a different distribution than the calibration set represents the OOD setting, encompassing any form of distribution shift. Specifically, for cross-dataset distribution shift experiments, under each task type, using different splits of the same dataset as the test set constitutes an I.I.D setting, while using different datasets as the test set constitutes an OOD setting. For cross-subject distribution shift experiments, using data from the same subject of the same dataset as the test set constitutes an I.I.D setting, while using data from different subjects within the same dataset as the test set constitutes an OOD setting.
**In general, whether the setting is I.I.D or OOD depends on whether the distribution of the calibration set is the same as that of the test set.**

[1]LifanYuan,YangyiChen,GanquCui,HongchengGao,FangyuanZou,XingyiCheng,HengJi,ZhiyuanLiu,andMaosongSun.Revisiting out-of-distribution robustness in nlp: Benchmarks,analysis,and llms evaluations. AdvancesinNeuralInformationProcessingSystems,36,2024.
> Q3: For W3, one of the main contributions of a benchmark is to provide guidance for future work. Therefore, the authors should offer some appropriate suggestions based on the experimental results. For example, they should recommend which dataset is best suited as a calibration set for future quantization methods to achieve optimal results. In more detail, although different calibration sets may yield varying results, a comprehensively optimal dataset should be selected for calibration.

Thanks for this comment.

We suggest that using a high-quality corpus as the calibration set can effectively recover the performance lost due to model quantization. We discuss this issue in detail in **Section 3.3** and provide references for future work. **Table 4** demonstrates the performance differences between using a high-quality corpus and a downstream task dataset as calibration sets, showing minimal discrepancies. This indicates that using data with the same distribution as the test set does not significantly boost performance; LLMs are relatively robust to the distribution of the calibration dataset. Identifying the optimal calibration dataset is an area that warrants further exploration in future work. However, the focus of this paper is not on finding the optimal calibration dataset but on studying the impact of distribution shifts between the calibration and test sets on performance.
> Q1: For W1, apart from revising their statement, the authors also need to provide a detailed description of the differences between their research and the mentioned paper. Since the objectives and main content of this work and the mentioned one are extremely similar, failing to provide clear distinctions is a significant issue.

Thank you for your comment!

First, we want to clarify that we **did not claim to be the first to study the impact of calibration datasets**. Could you please indicate the specific location where this may have been misunderstood? Our work is the first to investigate distribution shifts between calibration and test datasets on quantized large LLMs, as well as the first to explore cross-subject distrbution shift experiments. This fills a gap in assessing the generalization capability of quantized LLMs.

Additionally, we discuss the related work in **lines 490 and 505**, specifically referencing [1], and **Table 7** highlights the distinctions between our work and prior studies. In particular, [1] primarily uses datasets from the pre-training corpus, employing different samples as calibration datasets for downstream tasks. However, their work does not include evaluations under I.I.D and OOD setting, nor does it extend calibration datasets from pre-training corpus data to downstream task datasets. It also does not address the distribution shift between calibration and test datasets, remaining limited to the S1 setting in our study. In contrast, our work not only varies the use of calibration datasets but also considers how distribution shifts between calibration and test datasets impact quantized LLMs, covering both the S1 and S2 settings outlined in our text.

[1] Miles Williams and Nikolaos Aletras. 2024. On the Impact of Calibration Data in Post-training Quantization and Pruning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10100–10118, Bangkok, Thailand. Association for Computational Linguistics.