DFRot: Achieving Outlier-Free and Massive Activation-Free for Rotated LLMs with Refined Rotation
large language models;rotational;quantization;orthogonal Procrustes transforms;Outlier-Free and Massive Activation-Free
Thank you for the authors' response. I would like to kindly address several claims that I find confusing.

**Q1**: The introduction of doesn't introduce any additional computational overhead.

**A1**: This is strictly accurate for $R_1$. Thanks to the property $RMSNorm(XR_1) = RMSNorm(X)R_1$ (which can be obtained from QuaRot, SliceGPT), $R_1$ can be merged by reparameterization into weight, changing ($W_{embedding}$, $W_{lm head}$, $W_q$, $W_k$, $W_v$, $W_o$,
$W_{up}$,$W_{gate}$, $W_{down}$) into ($W_{embedding}R_1$, $R_1^TW_{lm head}$, $R_1^TW_q$, $R_1^TW_k$, $R_1^TW_v$, $W_oR_1$,
$R_1^TW_{up}$,$R_1^TW_{gate}$, $W_{down}R_1$).


**Q2**: Is this operation fully W4A4? Additionally, if I'm correct, the QuaRot setting is K4V4Q16. Does this qualify as fully W4A4?

**A2**: First of all, you should understand that W4A4 as it is considered in LLM does not include KVCache compression (it full name should be W4A4KV16), and our task in this paper does not make any comparisons to KVCache.

Second, the purpose of our paper is to provide further insights into the rotational invariance proposed by QuaRot, to understand the differences in the effects caused by rotational invariance for different types of tokens (normal tokens and tokens with massive activation), and to further draw out the contributions of our paper.

Finally, if you read DuQuant's code carefully, you will realize that DuQuant did not investigate the compression of KVCache, they use KVCache with 16 bits, which is why we do not compare with them.

**Q3**: Are you aiming to highlight the limitations of your method?

**A3**: I agree that there are limitations in our work, for example, we only targeted $R_1$ in our optimization process, this is because

1. the optimization of $R_1$ is the most challenging of the rotational invariants because $R_1$ runs through the entire network, the rest of the optimization of $R_2$, $R_3$, and $R_4$ we can optimize by Block by Block; there is no optimization of $R_2$, $R_3$, and $R_4$ in our paper This is our future work, and we believe that the optimization of $R_1$ in our paper is sufficient to support the whole task; 
2. The massive activation mentioned in our paper only occurs in the network locations that $R_1$ can correspond to, and other locations will not be characterized by this massive activation, which means that this kind of token will not be encountered. the problems mentioned in the paper;
3. We have made some simplifications to our work in optimizing $R_1$, i.e., obtaining the calibration set, optimizing $R_1$. In fact, from the description above, since $R_1$ affects the quantization results of the preorder network, a better optimization of $R_1$ would be to alternate obtaining the calibration set-optimizing $R_1$, but this would inevitably lead to an increase in the overhead of optimizing $R_1$. The starting point of our paper is to optimize $R_1$ with the smallest possible code, and the alternating optimization mentioned here is also our future work (in progress), which we are reasonably confident will yield better results.

**Q4**: Your response appears to miss the mark, as it should provide a comparison between your method and other methods, rather than between QuaRot and other methods.

**A4**: That's a good question. 

The reason about why we do not compare with previous methods is based on our consideration of the experiments demonstrated by SpinQuant[1] in W4A4.

From the experiments we can recognize that previous scale-based (SmoothQuant, OS+, OminiQuant, AWQ, etc.) have invariably lost their competitiveness when the activation values are 4-bit, i.e., the performance of the model has completely collapsed. Currently, to the best of mt knowledge, at 4-bit per-token activation (no per-group quantization), **only methods based on rotational invariance are able to preserve the model's performance without a complete collapse**, even though a significant performance degradation still occurs compared to 8-bit activation values.

At the same time, only half a year has passed since the proposal of work based on rotational invariance in the quantization process, and currently the only method that can provide a fair comparison based on rotational invariance (specifically referring to models like QuaRot) is QuaRot.

These considerations combine to form the basis of our comparison with QuaRot alone in our paper.

**Q5**: Compare with SpinQuant.

**A5**: You can see it in Appendix as **Part-2 Q7** mentioned.
Response to additional question
Thank you for the authors' response. I would like to kindly address several claims that I find confusing.
1. "The introduction of $R_1$ doesn't introduce any additional computational overhead." Is this statement accurate?

2. "This leads to a large number of floating-point matrix multiplications, ..., which is not fully W4A4." From my perspective, QuaRot also introduces extra floating-point matrix multiplications. For instance, when multiplying K by V, we need to multiply K/V by a Hadamard matrix first. Is this operation fully W4A4? Additionally, if I'm correct, the QuaRot setting is K4V4Q16. Does this qualify as fully W4A4?

3. Regarding the second point you mentioned, are you aiming to highlight the limitations of your method?

4. Your response appears to miss the mark, as it should provide a comparison between your method and other methods, rather than between QuaRot and other methods.

5. Finally, while DuQuant was not compared, what about SpinQuant? I believe SpinQuant is more similar to your approach than DuQuant.
Response to Rebuttal: "Why DFRot Didn’t Compare It to DuQuant: They Are Completely Different"
**Q5**: How similar is the final matrix obtained by your method and the Hadamard matrix used for initialization?

**A5**: They're completely different. The rotation matrix R1 optimized by our method is more like a “random” orthogonal matrix because our rotation matrix is optimized for the features, adapts to the quantization error of the features, and achieves a better elimination of the overall quantization error. (Incidentally, if you look at SpinQuant's open source optimized rotation matrix, you will see that there is almost no significant difference between the SpinQuant trained matrix and the initial Hadamard matrix.)

**Q6**: Do different layers or projections share a common rotation matrix?

**A6**: We used the same $R_1$ for the different layers, which is the starting point of our work, as it is the only way we can reason without introducing any additional overhead, i.e., following the rotational invariance of the network as stated in QuaRot.

**Q7**: Compare with SpinQuant.

**A7**: **We have shown a comparison between DFRot and SpinQuant in the appendix and compared the differences in depth**. We would like the reviewer to understand that SpinQuant, which is based on the WikiText-2 training data and PPL for the rotation matrix, may lead to ''overfitting'' to the WikiText-2 data, so that although it can improve significantly on the WikiText-2 PPL, it does not produce significant results on the zero-shot task. In other words, we believe that the performance improvement on WikiText-2 PPL does not strictly demonstrate the capability of the quantized LLM, and it is important to investigate better evaluation metrics of the quantized model for the development of LLM quantization field.

**Q8**: On the visualization of quantization errors.

**A8**: We will add a visualization of the quantization error in a later release. But before that, I would like reviewers to read the previous questions carefully, which will help you to further understand the differences between QuaRot and DuQuant, as well as the motivation and contribution of our work.

Sincerely, the change in quantization error, while meaningful, is not a very fundamental difference. As in the case of cross-entropy loss when performing an image classification task, we can always tell that the loss goes down during the optimization process, but very little work has gone into showing the loss during the optimization process in papers.

**Q9**: Large LLM (e.g., 70B).

**A9**: We are very sorry that limited by computational resources, our lab is not equipped to explore larger models. However, we show further results of our method on QWen2-7B in the Appendix.

Meanwhile, based on the fact that Massive Activation is also present in larger LLMs (e.g., 70B) as found in Massive Activation [1], it is entirely reasonable to infer that our method is equally applicable to larger models.

[1] Sun, M., Chen, X., Kolter, J.Z. and Liu, Z., 2024. Massive activations in large language models. arXiv preprint arXiv:2402.17762.
Rebuttal for Reviewer 6qo1-Part2
**Q1**: The reviewer think our observation appears trivial and unworthy of further investigation.

**A1**: **We are completely unable to understand why the reviewer stated this opinion**. From our perspective, understanding why QuaRot's proposed approach to rotation improves the model's performance at quantization, especially under W4A4 quantization, can provide very valuable guidance for LLM's lower bit networks.

Our work starts from the rotational invariance mentioned in QuaRot, and from the perspective of quantization error, we have carried out a reasonable and rigorous visual analysis and experimental evidence, and obtained that Rh can significantly improve the quantization of LLM under W4A4 compared to RO rotation simply because RH can better achieve better quantization under the massive activation of LLM. error, for the vast majority of tokens (even more than 99%) the two are unable to produce a significant difference, which we believe is extremely valuable. Our work helps the community to further understand the results of evaluating LLM quantization based on rotational invariance, which helps the community to further promote the research on LLM based on rotational invariance.

At the same time, our method is identical to QuaRot in terms of inference speed because our method uses the same rotation matrix $R_1$ throughout the network as QuaRot, which allows $R_1$ to still be reparameterized into the weights of the network. Based on the fact that QuaRot is already supported by llama.cpp, we believe that our method only needs to provide the optimized rotation matrix (which often takes very little time to complete), and that our method can be seamlessly plugged into llama.cpp, as well as into any inference engine, as our method does not change the structure of the network and does not introduce any extra overhead (but DuQuant changes the structure and increases the floating point overhead of the network).

**Q2**: The novelty and effectiveness of the method seem limited.

**A2**: **With all due respect, I think we are working very efficiently and innovatively.** 

The overall lineage of our work is as follows: we have found that the reason for the huge performance difference between different rotation matrices (RO and RH) is the difference in the effect of Massive Activation, and have illustrated this through rigorous experiments, which are of great significance and which have helped to understand the specific behavior of rotational invariance in LLM quantization. To address this problem, we rationally abstract it to a long-tailed distribution and use the orthogonal Pluck transform to rationally solve the problem. Our approach is extremely rigorous and elegant in terms of experimental design and solution. At the same time, our method is efficient in the quantization process, and our method improves the modeling effect of QuaRot without introducing any additional computational overhead in inference, and it can be seamlessly inserted into llama.cpp, which is of significant application value.

**Q3**: The method does not include evaluations related to actual memory reduction or speedup.

**A3**: Our approach involves no modifications to the network structure, no kernel involvement, and DFRot and QuaRot have exactly the same overhead for inference, so we do not consider this to be necessary for the paper.

**Q4**: Is the rotation matrix a Diagnol Block matrix?

**A4**: No, $R_1$ is a dense $D\times D$ matrix, this is because as we know from the previous section, $R_1$ does not introduce any additional inference overhead, and we follow QuaRot's design of optimizing using Dense's rotation matrices. 

By the way, **I would again like the reviewers to carefully understand the differences between QuaRot and DuQuant as explained above**, and to understand that they are not at all the same type of work, and this will help you to understand why we are using Dense's rotation matrices instead of the block diagonal matrices mentioned in DuQuant.
Rebuttal for Reviewer 6qo1-Part1
Although I am very grateful to the reviewers for reading our work, I have to admit that you have completely missed the difference between QuaRot and DuQuant. Although both methods appear on the surface to use a rotation matrix to eliminate outliers in the activation values of an LLM, this **QuaRot and DuQuant have huge difference**:

1. **Computational overhead**: QuaRot uses the same rotation matrix $R_1$ for the global network, which, thanks to the nature of the RMSNorm, allows $R_1$ to be fully reparameterized into the model's weights, and the introduction of $R_1$ doesn't introduce any additional computational overhead; DuQuant uses a different rotation matrix for each Block, which leads to the fact that after each RMSNorm we DuQuant uses a different rotation matrix for each Block, which leads to an online rotation of the features after each RMSNorm, which leads to a large number of floating-point matrix multiplications, and therefore I believe that DuQuant is not fully W4A4, and the authors do not explicitly discuss the problems that these floating-point computations pose to the design of a W4A4 inference system in the paper.

2. **Optimization Difficulty**: QuaRot uses a single $R_1$ for the entire network, so optimizing $R_1$ will be very difficult because we can't optimize $R_1$ using the Block by Block approach (i.e., the quantization result of the previous Block will change while optimizing the rotation matrix $R_1$ for the next Block optimization); but DuQuant uses a different $R_1$ for each Block, in a way that would make QuaRot's optimization difficulty completely non-existent, by simply optimizing it using only the Block by Block optimization method.

**In summary, I don't think QuaRot and DuQuant can be compared**. This is because QuaRot and DuQuant are completely different from the point of view of inference reasoning and optimization, and DuQuant completely loses the advantage of QuaRot for network rotation invariance and introduces a lot of online computation in the design of the inference system.

After you understand the essential differences between QuaRot and DuQuant, we'll answer your question below about why we didn't compare with DuQuant.
Why DFRot didn't compare it to DuQuant, because they are completely different.
Yes
No ethics review needed.
1. Is the rotation matrix a Diagnol Block matrix, or a full $D\times D$ matrix, where $D$ is the hidden dimension size?
2. How similar is the final matrix obtained by your method and the Hadamard matrix used for initialization? If the similarity is small, it suggests that the method’s effectiveness may rely more on the Hadamard matrix than on your training approach.
3.It would be helpful if the authors provided visualizations comparing 1) original activation; 2) activation transformed by QuaRot; 3) activation transformed by their method, to better demonstrate the effectiveness of trained matrix.
5. Do different layers or projections share a common rotation matrix? Will this method lead to increased memory or time costs during the inference phase?
1. The paper is well-written and clearly structured, making it easy to follow.
2. The method’s motivation is substantiated by experimental results, lending it credibility.
The paper proposes a method, DFRot, designed to address outliers encountered when quantizing large language models (LLMs). The method initializes with a Hadamard matrix and adapts to activation outliers. It is then applied to LLaMA/Mistral models under W4A4 settings to verify its effectiveness.
1. The authors extensively emphasize that RH outperforms RO and effectively handles massive outliers, which RO cannot. However, this appears trivial and unworthy of further investigation. Firstly, it is evident that RH can distribute outliers evenly across channels, which RO cannot accomplish. Secondly, as already noted, numerous works, such as QuaRot [1], have employed Hadamard rotation matrices and confirmed the effectiveness of RH. Lastly, the paper lacks theoretical analysis on this point.
2. The novelty and effectiveness of the method seem limited. Numerous prior studies have explored learnable matrices, including SpinQuant [2] and DuQuant [3]. **There is a noticeable lack of comparison with these methods, both in terms of methodological differences and performance metrics.** From my perspective, the results in the paper do not outperform any of these works [2, 3]. Additionally, the model size used (≤13B) is relatively small, which limits the persuasiveness of the results.
3. The method does not include evaluations related to actual memory reduction or speedup.

[1] Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Jaggi, M., Alistarh, D., ... & Hensman, J. (2024). Quarot: Outlier-free 4-bit inference in rotated llms. arXiv preprint arXiv:2404.00456.

[2] Liu, Z., Zhao, C., Fedorov, I., Soran, B., Choudhary, D., Krishnamoorthi, R., ... & Blankevoort, T. (2024). SpinQuant--LLM quantization with learned rotations. arXiv preprint arXiv:2405.16406.

[3] Lin, H., Xu, H., Wu, Y., Cui, J., Zhang, Y., Mou, L., ... & Wei, Y. (2024). Duquant: Distributing outliers via dual transformation makes stronger quantized llms. arXiv preprint arXiv:2406.01721.
Yes
No ethics review needed.
- One suggestion: hadamard transformation is orthogonal, and therefore I think the contrast of hadamard vs orthogonal is not entirely accurate and a bit confusing. Perhaps the first time this comparison is made specify this difference, and then I think it's ok as a shorthand to say hadamard vs orthogonal.
- Per Section 3.1, why do the authors only target rotation matrices R1? and not R2,R3,R4?
- Also, R1 will be applied to the input activations, and therefore will need to be applied during inference. What are the computational costs?
- What exactly are the RO matrices? RH have been proposed by QuIP#, but it is unclear what the RO matrices are.
- Another reason why I think the naming of "random orthogonal" matrices is misleading: the authors are making claims about RH vs RO performance, but as far as I can tell "RO" is simply the matrices generated by the SpinQuant paper. It's more accurate to say that the rotations matrices from these two papers are being compared.
- The authors provide an empirical observation, that RH is better at reducing the quantization error for tokens with massive activations in 4 bit activation quantization. But why is this the case?
- what does the "time" row in Tab 2 mean? is this training or inference?
- Why don't the authors compare to SpinQuant? Comparing Llama2-7b 4-4-4 Tab 1 in the SpinQuant paper to Tab2 in this paper, appears that SpinQuant achieves a better wikitext 2 ppl of 5.9, vs DFRot which achieves 6.2. If I've made an incorrect comparison, please let me know. The baselines between the two papers looked to be comparable.
- I think looking at the massive activations is a sensible thing for investigating RH vs RO. 
- And I think that the resulting approach which takes advantage of these insights is interesting.
- Overall I do like the story of this paper, which I see as trying to gain understanding between 2 rotation matrices, and using this understanding to improve the method.
The authors study the difference between two rotation matrices used for post training quantization, and shows some empirical evidence explaining their difference in performance. Specifically, the random hadamard transformation is better at reducing quantization error for tokens with massive activations in 4 bit activation quantization. Using these insights, the authors take the framework for inserting rotations into an LLM architecture proposed by QuaRot, and learn a subset of those rotation matrices. Their optimization uses a weighted sum between the tokens with massive activations and those without. Experiments are provided on Llama2 7b, 13b, llama3 8b, and mistral 7b.
I have concerns about this paper. Overall, I think the writing could use work. It's also unclear why the authors only target the "R1" rotation matrices. But more significantly, I think they should include experiments comparing to SpinQuant. This is the more appropriate comparison, because this method also learns their rotation matrices. Finally, I think that there should be a inference time consequence to this method that I think the authors should make more clear. 

Some other comments:
- I find the difference between random hadamard and random orthogonal confusing, because hadamard transform is orthogonal. 
- I found the language describing the relative strengths/weaknesses of RO/RH (e.g. paragraph starting L77) to be confusing. 
- I think the writing in Section 3 could be improved. I believe that 3.1 describes how to integrate rotation matrices into the architecture. But I think readers unfamiliar with QuaRot can be confused. I think that some summary sentences describing the overall goal, and pointing readers to QuaRot for further details, would be helpful. I see that QuaRot is cited, but only for a RMSNorm equality, it is not clear from the text that QuaRot actually describes the overall approach of a way to integrate rotations. 
- I'm unconvinced by the line of argument in Section 3.2. The authors say that the activation quantization error in Fig 2 is not that different between "RH" and "RO", but the overall PPL from Table is significantly different. How Fig 2 is just from a single layer, and I honestly think it's hard to tell from the plots if there is a significant difference or not. Are there perhaps some summary statistics about activation errors that the authors can share?
Yes
No ethics review needed.
- What is the main difference between SpinQuant and Quip and your proposed method? Can you differentiate yourself from these methods?

- In Section 3.4, the authors mention that quantization is regarded as clustering, but the explanation is vague. Are the authors suggesting clustering the rows of matrix X? Each row is a high-dimensional vector, and clustering such vectors directly would likely yield random and meaningless results. Practical clustering-based quantization methods typically partition the coordinates of high-dimensional vectors into smaller blocks, applying clustering to these blocks. A more precise explanation here would improve clarity.

- The experiments are confusing. Which experiments focus on model quantization, and which are aimed at KV cache quantization?

- How does the proposed method perform in comparison to the aforementioned KV cache quantization methods?

Additionally, I would like to note: please try to address my concerns, and rest assured that I will consider raising my score if you provide sufficient results during the rebuttal.
Quantization is a crucial problem, as memory constraints significantly limit the scalability of large language models. This paper tackles an important and timely research question, making it a compelling contribution to the field.
The paper addresses the quantization problem for transformer architectures. It begins with an analysis of the impact of outliers—activations with unusually large values—on the performance of quantized models. The authors propose a commonly used approach of mitigate this issue: they rotate weight matrices and embedding vectors in the KV cache. The novelty of their approach is in using a learnable rotation matrix, designed to reduce quantization error specifically for outlier activations. This matrix is optimized via a cost function that prioritizes minimizing quantization error for these outliers. The authors introduce an iterative algorithm that appears computationally efficient for minimizing this cost function. The paper demonstrates the performance of the proposed method under 4-bit quantization in a series of experiments.
The presentation could be improved, as key details are missing that would help better position the work relative to existing methods and verify the authors' claims. For example, the paper does not adequately differentiate its method from SpinQuant and Quip, both of which also apply a learned rotation matrix to model weights before quantization.

Moreover, recent advancements in KV cache quantization are not sufficiently discussed. The practical performance of DFRot, for example, could have been compared to other recent approaches, such as:

- Liu, Zirui, et al. "Kivi: A tuning-free asymmetric 2bit quantization for kv cache." arXiv preprint arXiv:2402.02750 (2024).
- Zandieh, Amir, Majid Daliri, and Insu Han. "QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead." arXiv preprint arXiv:2406.03482 (2024).
Yes
No ethics review needed.
1. Clarity on the proposed method and experimental setup
  - When measuring the quantization error in Section 3.2, the l2-norm of the activation perturbation (Eq.(1)) was used? If yes, the reviewer thinks that the final task loss degradation, rather than the activation quantization error itself, needs to be used as a metric.
  - How to determine the clip ratio $\alpha$ and $\beta$ has not been mentioned in Section 3.4.
  - When optimizing the rotation matrix (with fixed quantization parameters), it seems that the proposed weighted loss function in Eq.(2) has NOT been used. The corresponding solutions in Eqs. (8) and (10) consider all activations equally, regardless of whether activations being normal activations or massive outliers. In such a case, the authors did not consider massive outliers as in existing works.
  - While the authors mentioned that they performed the iterative optimization process with just one round (line 351), the number of iterations has been specified as 100 in line 371.

2.  More experiments are needed.
  - The size of LLMs used in the experiments is larger than 7B for which the performance gap between the conventional and proposed methods is not that noticeable for W4A4. The reviewer recommends conducting experiments with OPT models whose size varies from 125M to 66B.
  - From Table 2, the performance gap between QuaRot and DFRot reduces greatly when the weight quantizer changes from RTN to GPTQ. The reviewer thinks that if a better weight quantizer (e.g., aespa [1]) has been used, the gap between QuaRot and DFRot is very marginal. Please show the results with better weight quantizers.
  - From Figures 9 and 10, when the activation clipping is used, the performance gap between DFRot and no rotation is reduced. The reviewer thinks that the gap between QuaRot and DFRot becomes smaller if the activation clipping is used. Please show the results when applying activation clipping to both QuaRot and DFRot.
  - While most conventional works consider the per-token dynamic activation quantization, such a setting is expensive to implement in real-world scenarios. Please show the results for the **per-tensor static** activation quantization.

[1] Towards next-level post-training quantization of hyper-scale transformers, arXiv 2024.
1. The authors analyzed why random Hadamard transforms lead to better performance compared to random orthogonal transforms.
2. The proposed weighted loss function is well-motivated and reasonable based on the above finding.
3. The claims presented by the authors are well supported by experimental results.
In this paper, the authors proposed a rotation-based post-training quantization (PTQ) scheme for the quantization of LLMs.
First, the authors empirically analyzed why random Hadamard transforms lead to better performance compared to random orthogonal transforms; the key reason is that random Hadamard transforms handle massive outliers much better than random orthogonal transforms. 
Second, the authors presented how to optimize the random matrix that leads to the effective quantization of massive outliers; alternating optimization of quantization parameters and the rotation matrix has been presented.
Finally, the efficacy of the proposed method has been demonstrated via extensive experiments.
1. Some statements contradict each other (see Questions below).
2. Experiments are limited to some specific cases.
  - The size of LLMs is larger than 7B for which the performance gap between the conventional and proposed methods is not that noticeable. 
  - The utilized weight quantizers (RTN and GPTQ) are not state-of-the-art.
  - The activation quantization configuration is dynamic per-token quantization, which is very difficult and expensive to implement in real hardware.
@inproceedings{
anonymous2024dfrot,
title={{DFR}ot: Achieving Outlier-Free and Massive Activation-Free for Rotated {LLM}s with Refined Rotation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=XtXa6hoNrU},
note={under review}
}
Using rotational invariance to eliminate outliers in large language models (LLMs) has recently gained considerable attention, especially in the context of quantization. Prior studies have shown that in low-precision quantization scenarios, such as 4-bit weights and 4-bit activations~(W4A4), randomized Hadamard transforms can achieve significantly higher accuracy than randomized orthogonal transforms. Notably, the reason behind this phenomena remains unknown. In this paper, we find that these transformations show substantial improvement in eliminating outliers for common tokens and achieve similar quantization error. The primary reason for the accuracy difference lies in the fact that randomized Hadamard transforms can slightly reduce the quantization error for tokens with massive activations while randomized orthogonal transforms increase the quantization error. Due to the extreme rarity of these tokens and their critical impact on model accuracy, we consider this a long-tail optimization problem, and therefore construct a simple yet effective method: a weighted loss function. Additionally, we propose an optimization strategy for the rotational matrix that involves alternating optimization of quantization parameters while employing orthogonal Procrustes transforms to refine the orthogonal matrix. This makes the distribution of the rotated activation values more conducive to quantization, especially for tokens with massive activations. Our method enhances the Rotated LLMs by achieving dual free, \textit{Outlier-Free} and \textit{Massive Activation-Free}, dubbed as DFRot. Extensive experiments demonstrate the effectiveness and efficiency of DFRot. By tuning the rotational matrix using just a single sample, DFRot achieves a perplexity improvement of 0.25 and 0.21 on W4A4KV4 and W4A4KV16, respectively, for LLaMA3-8B, a model known for its quantization challenges. Code is anonymously available at \url{https://anonymous.4open.science/r/DFRot-8FE3}.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
large language models rotational quantization orthogonal Procrustes transforms Outlier-Free and Massive Activation-Free
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/c988a9c4b915d90104e290ff5adb73c228930b65.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/f4a056a29927d11095dcfb677941231397dd010f.pdf
DFRot: Achieving Outlier-Free and Massive Activation-Free for Rotated LLMs with Refined Rotation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission