LoLCATs: On Low-Rank Linearizing of Large Language Models
Linear Attention;Linearizing Transformers;Low-rank Adaptation;Large Language Models;Architecture Distillation
Yes
No ethics review needed.
- How were the hyperparameters for the attention transfer and low-rank adaptation chosen?
- I believe this is a good work. The paper presents a novel method for linearizing LLMs that addresses key limitations of existing techniques. By focusing on approximating softmax attention outputs and using low-rank adjustments, the authors offer a fresh perspective on reducing computational complexity without sacrificing model quality.
- The LOLCATS method reduces the amount of training required, both in terms of model parameters updated and training data used. This efficiency makes the method practical for widespread use, especially in environments with limited computational resources.
- Demonstrating the ability to linearize large models up to 405B parameters is a notable achievement. The scalability of the method suggests it can be applied to future, even larger models.  The authors provide extensive experimental results, including comparisons with existing methods on multiple benchmarks. 
- The paper delves into the reasons why previous linear attentions struggled to approximate softmax attention effectively. By identifying issues like attention entropy and layer-wise errors, the authors provide valuable insights that inform their improved architecture.
This paper presents LoLCATs, a method for converting large language models (LLMs) with quadratic attention complexity into models with linear complexity while maintaining model quality. The key innovation is a two-step approach: (1) attention transfer - training linear attention layers to directly approximate the original softmax attention outputs, and (2) low-rank adaptation (LoRA) fine-tuning to adjust for approximation errors. The authors demonstrate that LoLCATs can effectively linearize models up to 405B parameters with significantly less compute and data compared to previous methods, while better preserving model capabilities.
- The introduction and analysis of the preliminaries are too lengthy, with the core improvements in LOLCATs not appearing until the seventh page, which makes the reading experience somewhat disjointed.
- The experimental results are promising given the training budgets. However, I notice that even though previous works perform significantly worse for more challenging benchmarks (like MMLU in the setting), LOLCAT still considerably underperforms the original models. What will happen when the benchmarks become more challenging? (e.g., complex reasoning).
- While the authors claim improvements in inference efficiency, quantitative metrics such as actual speedup factors, memory utilization during inference, or comparisons of throughput are not extensively reported.
Yes
No ethics review needed.
1. I would like the author to supplement and polish the article based on the weaknesses.

2. The authors mention that LoRA effectively reduces approximation errors. Have they considered conducting a specific error propagation analysis? This is crucial for understanding how low-rank adaptation accumulates errors across different layers in deeper models, as it is vital for controlling cumulative errors.

3. In the LoRA method, the choice of the rank parameter is critical for the model's approximation effectiveness. How have the authors taken into account the impact of low-rank parameters on performance across different layers of the model?

4. Does this approach extend to other variants of Linear Attention, such as Mamba 1/2 and Hgrn 1/2?
1. The LOLCATS approach significantly reduces computational and memory costs through a two-step process involving "attention transfer" and low-rank adaptation (LoRA), effectively lowering the complexity of large models. 

2. LOLCATS effectively retains the performance of the original self-attention model. Experimental results show that this method can recover much of the original model's quality after linearization, using only a small portion of the parameters and training data.

3. LOLCATS is the first to successfully apply linearization to large models with 70B and 405B parameters, expanding the applicability of linearization techniques.
The paper addresses the efficiency and scalability challenges of large language models (LLMs) caused by the quadratic complexity of traditional Transformer models. To overcome these limitations, it introduces LOLCATS (Low-rank Linear Conversion with Attention TranSfer), a novel method that linearizes attention mechanisms to reduce computational and memory demands. LOLCATS uses an “attention transfer” phase to approximate softmax attention efficiently and a low-rank adaptation (LoRA) to correct errors. This approach enables scalable training of LLMs with up to 405B parameters—50 times larger than previous models—while maintaining high performance. Experimental results show LOLCATS outperforms existing methods and opens new avenues for scaling LLMs further.
1. The paper mentions the differences in errors across layers and their impact on model performance, but it does not sufficiently discuss the underlying reasons, such as why lower soft-attention entropy leads to higher errors.

2. There is insufficient explanation for new terms, such as "attention transfer," which may lead to misunderstandings regarding specific implementation details and processes. Clearly defining key concepts within the paper would improve overall clarity.

3. The paper does not provide averages, variances, or confidence intervals from multiple experiments, making it difficult for readers to assess whether performance differences in the model are statistically significant and robust.

4. As indicated in [1], there is a notable difference between Linear Attention and Softmax Attention in retrieval tasks, particularly in "needle-in-a-haystack" scenarios. Therefore, assessing the performance of models on such tasks before and after the application of LoLCAT would provide a more comprehensive validation of the method's effectiveness.

[1] Xuyang Shen, Dong Li, Ruitao Leng, Zhen Qin, Weigao Sun, & Yiran Zhong. (2024). Scaling Laws for Linear Complexity Language Models.
Yes
No ethics review needed.
- Why was the proposed method not validated on a smaller model, such as llama1B?
- The method is shown to scale to unprecedentedly large models (70B and 405B parameters), which was not possible with previous linearization techniques.
- The method is applied to various models, showing its broad applicability and potential impact on the field of natural language processing.
The paper presents a novel method for linearizing large language models (LLMs) to make them more efficient in terms of memory and compute resources. The authors propose a method called Low-rank Linear Conversion via Attention Transfer (LOLCATS), which aims to replace the quadratic attention mechanisms in popular Transformer-based LLMs with subquadratic alternatives, such as linear attention. This approach avoids the expensive pretraining costs associated with traditional LLMs.
- There is a lack of an overall summary description of the LOLCATS method. An algorithm description or pseudo code can be added.
- There are some writing errors, such as Line 294: APP.???
Yes
No ethics review needed.
1. Can authors point out which line of appended code correspond to a. Eq 7?

2. I'd like to confirm the author is doing post-training approximation or training from scratch with this specialized architecture. (It's a bit unclear to me). And for all the baseline methods, are they doing training from scratch or post-training approximation.

3. My understanding is that actually there is no new technical thing proposed. Eq 5 is proposed by previous works, and sliding window attention was also proposed in the previous work. And it seems like combining this 2 idea is the only technical contribution. Is that correct? Or do you have other new proposed approximation that I overlooked? 

3. In that regard, adding a sliding window to any of the previous linearization techniques should improve the performance, and this has to be validated. If not, there is a need to investigate why synergy only happens here.

4. Apologize that I really don't get that's the purpose of Table 1, Figure 3 and Figure 5. What's the main point you want to convey here? 

5. Is there a study of different window size?

6. For ablation study I only see Table 5. Is there one with your proposed method? Or should I read it as Hedgehog == Eq 5 and your method is nothing butHedgehog + Sliding?  And somehow Hedgehog + sliding ==> 68.78 outperformed your proposed method a lot, which reads weird to me.

7. Also don't know why ablation is only done on MMLU, I am interested in seeing RCV-e and PiQA too.

8. I don't really see what's the help from the linearization to parameter-efficient finetuning. I still don't understand why PEFT can't work on other methods. In addition, full FT should still outweigh PEFT. I believe authors should also show their method with full parameter tuning to see what's the difference.
- The Introduction reads well (but unfortunately not the case for the rest)

- An interesting problem to study.

- Provide the code in the Appendix
Authors proposed a new method to approximate the quadratic attention operation to semi-linear ones (with a pre-specified quadratic window). The linearized approximation seems to work with PEFT so the computational burden is partially alleviated when accelerating the computation on large models.
- The way authour present their methods interleaves many previous works, and thus it's difficult to precisely pinpoint what's their contribution. 

- Some lacking experiments make me uncertain if the claimed effectiveness is true.

- The above two factors combined make me uncertain what's the real efficacy of the method.
@inproceedings{
anonymous2024lolcats,
title={Lo{LCAT}s: On Low-Rank Linearizing of Large Language Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=8VtGeyJyx9},
note={under review}
}
Recent works show we can linearize large language models (LLMs)---swapping the attentions of popular Transformer-based LLMs with subquadratic analogs---to create subquadratic LLMs at fractions of typical pretraining costs.  However, to adapt LLMs to the new layers, these approaches significantly degrade model quality, involve expensive full-model training over billions of tokens, and remain limited to smaller 1.3B to 7B LLMs. We thus propose Low-rank Linear Conversion via Attention Transfer (LoLCATs), a simple two-step method that improves LLM linearizing quality with magnitudes less memory and compute. We base these steps on two findings. First, rather than adapt LLMs to completely new layers, we find we can replace their softmax attentions with near-equivalent linear attentions, simply by training these layers to approximate their softmax counterparts ("attention transfer"). Then, this enables simply using low-rank adaptation (LoRA) to adjust for approximation errors, recovering quality in fully subquadratic LLMs. In experiments, LoLCATs significantly improves linearizing quality, training-efficiency, and scalability. First, by linearizing Llama 3 8B and Mistral 7B v0.1, LoLCATs produces state-of-the-art subquadratic LLMs that outperform both prior linearizing methods (SUPRA) and strong pretrained 7B Transformer alternatives (_e.g._, RWKV, Mamba, Griffin) by 2.9-8.0 points on popular zero-shot LM Evaluation Harness tasks (+20 points on 5-shot MMLU). Next, LoLCATs achieves the above with only 0.2\% of past methods' model parameters and 0.4\% of past methods' training tokens. Finally, we apply LOLCATS to create the first linearized 70B and 405B LLMs (50× larger than prior work). When compared with prior methods under the same compute budgets, LOLCATS significantly improves linearizing quality, closing the gap between linearized and original Llama 3.1 70B and 405B LLMs by 78.7% and 77.4% on 5-shot MMLU.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Linear Attention Linearizing Transformers Low-rank Adaptation Large Language Models Architecture Distillation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/85c6386aab13ea6b312cbd2e8527bc7ab8c11eca.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
LoLCATs: On Low-Rank Linearizing of Large Language Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission