SpaLLM: Unified Compressive Adaptation of Large Language Models with Sketching
large language model;sketching
Yes
No ethics review needed.
How to demonstrate the effectiveness of the sketching algorithm.
The paper presents an innovative solution to compressive adaptation by introducing SpaLLM, which avoids the traditional limitations of low-rank, two-tower architectures seen in methods like QLoRA. By employing a sketching-based parameter-sharing approach, it redefines how parameter compression and adaptation can be unified. This approach not only removes algebraic constraints but also opens new avenues for efficient fine-tuning, making SpaLLM a notable contribution that combines original ideas with practical advancements in LLM adaptation.

## Quality
The quality of the work is high, demonstrated through meticulous experimentation and comparison against established state-of-the-art methods. The experiments cover various LLM architectures and datasets, including LLaMA and CommonsenseQA benchmarks, highlighting the method’s accuracy and efficiency. The authors rigorously test SpaLLM’s performance under different bit compression rates, group sizes, and model configurations, showcasing the robustness of the approach and supporting the claims with thorough empirical evidence.

## Clarity
The paper is generally well-organized, with each section logically leading to the next. The authors provide clear definitions and motivations for concepts such as "two-tower architectures" and "parameter-sharing compression." Diagrams and tables, such as the illustration of row-wise parameter sketching and comparisons with baselines like LoftQ, further enhance clarity by visually presenting complex ideas. Although highly technical in parts, the paper effectively communicates its contributions and methodology.

## Significance
SpaLLM has high practical and theoretical significance, especially given the growing need for resource-efficient LLM adaptation techniques. By enabling adaptation directly on compressed parameters, SpaLLM reduces computational and memory overhead, which has profound implications for deploying LLMs on hardware with limited resources. Its flexibility in managing multiple adapters and dynamic task-switching makes it particularly valuable for multi-task learning scenarios, enhancing its potential for broader, real-world applications in natural language understanding and generation.

Overall, the paper’s strengths lie in its innovative approach to compressive adaptation, robust methodology, clear presentation, and significant contributions to scalable and efficient LLM adaptation.
SpaLLM (Sketched Parameter Adaptation of LLMs), a novel approach for compressive adaptation of large language models (LLMs) that simplifies the fine-tuning process while enhancing efficiency and accuracy. Unlike traditional methods such as QLoRA, which use a "two-tower" architecture with low-rank assumptions, SpaLLM employs a sketching-based parameter-sharing technique that avoids these constraints. This unified approach combines model compression and adaptation into a single, streamlined process.

Key contributions of the paper include:

* Single-Tower Architecture: SpaLLM eliminates the need for the two-tower structure, reducing computational overhead by requiring only one compressed matrix multiplication per layer during inference.
* Sketching-Based Compression: The method applies row-wise parameter sketching with optimized mappings, allowing direct adaptation of compressed parameters without relying on low-rank algebraic assumptions.
* Improved Efficiency and Accuracy: Experimental results show that SpaLLM outperforms state-of-the-art compressive adaptation methods in both natural language understanding and generation tasks, delivering significant efficiency gains in memory and inference speed.
* Task Flexibility: The approach supports dynamic maintenance of multiple adapters, allowing seamless task-switching and adaptability for multi-user serving environments.

SpaLLM offers a resource-efficient solution for fine-tuning large models, making it particularly suited for deployment on hardware with limited resources.
In general , I think this is a great work.

Some concerns are: 

* the success of SpaLLM relies heavily on the effectiveness of its sketching algorithm. If the sketching fails to capture essential details of the parameter distributions, it can lead to a loss in model accuracy or even failure to converge, particularly under aggressive compression settings.

* SpaLLM’s parameter-sharing approach could introduce challenges in tasks that require highly specialized or fine-grained learning (e.g., nuanced natural language understanding or highly specific domain tasks), as parameter-sharing might restrict the model’s flexibility.
Yes
No ethics review needed.
- Is the centroid learning process clustering?  are all of this Lloyd algorithm computation done during training? the overhead seems huge
- The one-hot encoding does not save memory, any compression of one-hot encoding?

- Line 196, what is the role of sketched parameters? is it similar to LoRA ranks? or just tuning more parameters? What are groups per row?

- Line 206, why do the sketch parameters become floating-point? which set of parameters are fixed-point?

- Line 215, no explanation of LUTs, how do you build it, how to maintain it in GPU memory? extra overhead?

- Line 241, why only the sketched parameters are stored and updated for each task, not the LUTs? Do you still need LUTs for inference?

- Line 255, how would compression not lose information? the identify matrix example is not convincing because real model weights are not identity.
- The paper presents an interesting PEFT method that adapts LLM model weights by sketching without relying on LoRA style low-rank adaption techniques.

- The experiments show competitive results than previous PEFT methods
The paper introduces a parameter-efficient finetuning (PEFT) method called SpaLLM for the compressive adaption of LLMs. It investigates the assumption and overhead issues of “two-tower” QLoRA-like PEFT methods. SpaLLM sketches the LLM weights into lookup tables and finetunes the table values without the low-rank assumption. The paper claims SpaLLM provides better accuracy and efficiency tradeoffs than previous methods.
- The paper is potentially flawed without further clarification on the sketching technique. The introduction and method sections are vague, unclear, and incoherent. 

Examples include:
1. The two-tower architecture is defined without any context, it normally means two branches of some encoder-like architecture, like in the Siamese network. From figure 2, the paper seems to indicate the weights getting finetuned in separate two steps and finetuning works like a two-twoer style. 
2. In the introduction, it is unclear how QLoRA causes a performance gap and how LoftQ reduces such a gap. And what are those performance gaps, task accuracy loss, or efficiency gaps? The introduction says the compression process is lossy (line 63), but why is it lossy? And why is the sketching via parameter sharing not lossy? Regarding difficulty in implementation, isn’t there a QLoRA library that makes it easy for people to use, the paper also claims QLoRA has additional overhead and requires significant system-level optimizations (line 73), but it does not get quantified. 
3. The presentation of SpaLLM and sketching ideas in the introduction are vague and redundant and do not contain much useful information. Line 78 says finetuning directly on compressed model parameters, how are the parameters compressed? Seems the parameters are quantized to low-bit precision and then applied to sketching. Sketching is parameter sharing and not contradictory compression. Line 85-94 then discuss the look-up tables (LUTs), how is sketching connected to these LUTs is unknown. How the LUTs are used also remains unclear. The introduction has no results or any comparison with previous methods.

4. Section 2.1 seems like a related work section but turns out not since it is mixed with existing work and the SpaLLM sketching-based parameter sharing. Line 113 - 122 says the two virtual and trainable parameter categorizations, which seems unnecessary because they are not referred to anywhere later in the paper. 

5. The Sketching idea is in fact parameter sharing from section 2.2, which is no new. The paper repurposes the parameter sharing for PEFT. However, it is unclear how the sketching works,  from lines 148 - 151, it consists of row-wise parameter sketching, weighted Lloyd optimization, and LUTs. It does not explain how these three components work together and why they are needed. The weighted Lloyd algorithm seems to avoid collision during hashing, so it is confusing to rebrand the hashing as sketching. 

- While the results are strong and much better than the compared baselines, without knowing the details of the SpaLLM method makes it much less convincing and difficult to evaluate the true improvements.
Yes
No ethics review needed.
1. The authors listed some compressive adapation methods like DoRA, OFT and BOFT. Did the author try to make exprimental comparison with these methods?
1. The method demonstrates precision improvements on sevaral LLM finetuning benchmarks, which is a hot topic in the literature.
2. The approach simplifies LLM's compressive adaptation workflow.
The paper presents a unified approach to compressive adaptation of LLMs, namely SpaLLM. The method allows adaptation without the strict low-rank matrix assumption. The authors claim that the method is more efficient and have better performance than methods like QLoRA. A row-wise parameter sketching method is introduced to skech the parameters for fine-tuning. It supports unified compressive adaptation with a single-tower architecture, task-specific compression and inherently supoorts the dynamic management of multiple adapters. The method is compared with several popular compressive LoRA methods and demonstrate comparable efficiency and accuracy.
1. The technical contribution seems limited and is not well elaborated. I expect some formal mathematical specification of the sketching method.The authors try to elaborate a lot on the use cases of the mehtod, however, the detailed description on how to implement those ideas are missing;
2. The evaluation in figure 3 shows that accuracy on llama2 7B have better accuracy when GPR=8, however, in other evaluations, GPR seems to be randomly setted to 1，4 ，8.  It makes the readers confused on the selection of GPR even with an ablation study. Moreover, the efficiency is only compared when GPR=1 in Table 5, which may lead to poteintial decay on the model performance.
3.On the settings of experiments, it seems that the authors carefully selects some good results to present without introducing the intuition on the selection of certain parameters.
Yes
No ethics review needed.
Please refer to the weakness.
1. The paper proposed sketched parameters, an interesting idea for compressive adaptation. 

2. The paper is well written. the proposed approach is simple and clear.
The paper introduced a novel compressive adaptation approach (SpaLLM) for LLMs that effectively addresses the limitations of existing methods QLoRA. Spallm maintains high performance while reducing computational and memory usage.
1. In section 2.3. how did you fix the mapping introduced by the sketching matrix?
2. in all the experimental settings, you choose the Lora rank equal to 64. I am not sure this comparison is fair. Could you show the results when Lora's rank equals 4 or 1? 
3. Why did you choose 5 common sense QA datasets in Table 3 while 7 common sense qa datasets in Table 4? Could you make it consistent?
4. in Figure 3, the accuracy is still improving as the increase of training parameters. Could you show more training parameters? in other words, is it scaling to more training parameters?

5. There is no code for the paper.

I promise I would like to raise my score if these questions can be solved.
@inproceedings{
anonymous2024spallm,
title={Spa{LLM}: Unified Compressive Adaptation of Large Language Models with Sketching},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=tTOo7r4G9j},
note={under review}
}
Compressive adaptation approaches, such as QLoRA, are widely popular alternatives for reducing memory requirements during fine-tuning of large language models (LLMs) while producing models capable of handling various downstream tasks. The key idea is to employ a “two-tower” architecture: compressing pretrained LLM parameters into compact representations and fine-tuning the additive full-precision adapter, which typically has few tunable parameters in low-rank format. However, the strict algebraic assumptions, such as low-rank assumption, and the complexity of composing two-tower architectures are some of the known shortcomings, resulting in a poor accuracy-efficiency trade-off. In response to these known limitations, we propose SpaLLM (Sketched Parameter Adaptation of LLMs), a novel compressive adaptation approach for LLMs. This method is also the first to illustrate parameter-sharing compression methods for LLM fine-tuning, which, unlike QLoRA, are free from strict low-rank algebraic assumptions on adapters. Furthermore, our proposal unifies model compression and adaptation into a single, streamlined process, eliminating the need for two-tower architectures. SpaLLM sketches pre-trained LLM weights into lookup tables and directly fine-tunes the values in these tables. This approach simplifies LLMs’ compressive adaptation workflow, potentially improves multi-user serving efficiency, and delivers significantly better accuracy for both natural language understanding and generation tasks. Moreover, by avoiding the “two-tower” architecture, our framework only requires one compressed matrix multiplication per layer during inference, demonstrating superior inference efficiency compared to previous methods.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
large language model sketching
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/6d895491f3d7dd0b55c7ecca40f5ed75ca356bd2.pdf
other topics in machine learning (i.e., none of the above)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
SpaLLM: Unified Compressive Adaptation of Large Language Models with Sketching
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission