KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models
Large Language Models;Parameter-efficient Fine-tuning;Singular Value Decomposition
Yes
No ethics review needed.
It would be great to justify why only MRPC, CoLA and RTE are chosen for the ablation studies.
* The evaluated tasks are comprehensive, including NLU, NLG and instruction-following.
* The ablation study provides evidence to demonstrate the effectiveness of all four design choices on selected NLP tasks.
This paper presents a novel parameter-efficient learning algorithm for LLMs. It addresses two issues of the prior research: i) a subset of model updates are associated with the noise encoded in the base model; and ii) the low-rank matrices A and B are initialized using the knowledge from the base model that is not relevant to the downstream tasks. The new method tackles the two issues with four components: knowledgebased SVD truncation, knowledge-aware singular value adaptation, singular value regularization L2, and orthogonal regularization. The ablation studies show that all four of them are effective for the selected evaluation tasks.
* The improvement of the proposed method over the baselines is marginal. The variances and the results of the significance tests may help justify the improvement.
* It is unclear why GPT-2 is used for NLG tasks, while Gemma, Mistral and LlaMA3 are used for instruction following tasks. Hence, the experimental design is not systematic.
Yes
No ethics review needed.
1. What exactly is the reader supposed to take away from the visualizations in Figure 4? It seems somewhat unclear. It might be helpful to have a comparison with and without KaSA

2. Why scale between 0 and 100 when using GPT4o as a judge? It seems a bit arbitrary/unnecessary. Do you have strong motivation for using this scaling?

Some small nits:
* Typo: line 043 “gets popular”
* Typo: line 426 “four synthetic datasets using GPT-4”; is it GPT-4 or GPT4o?
KaSA seems like a promising technique, and is competitive with LoRA and SVD style techniques like PiSSA and MiLoRA. The authors do extensive experiments and benchmarking, and the paper is very well written.

Minor notes:
* The paper has extensive citations that are very helpful to the reader
* Figure 2 with “ablations” is very informative - the trends are quite clear
* The “no robots” dataset is a good choice for seeding synthetic data, since it was generated by human annotators.
This study proposes a new technique for low-rank finetuning and inference. “KaSA” reparameterizes finetuned weights in two steps: task-specific SVD truncation, and “knowledge-aware singular- value adaptation” that selects particular singular values based on their relevance to particular tasks.

This study shows that KaSA is competitive with popular low rank approaches such as LoRA across multiple models and tasks. In particular, they report extensive results for RoBERTa evaluated on the GLUE tasks, GPT2 evaluated on the E2E NLG challenge, and three 7-8B LLMs evaluated on four custom synthetic instruction following datasets.
The experimental results are overall quite strong. The main weakness of the manuscript is that it lacks a straightforward explanation of the method. I’m not sure I fully understand the KaSA method here based on the description, and the diagram is slightly confusing as well. It would be very helpful to have a pseudocode block/section.

1. My current understanding is that the method requires (1) LoRA finetuning followed by (2) SVD truncation followed by (3) “knowledge aware singular-value adaptation.” Is this correct? If so, doesn’t this mean that it will require more training/compute than vanilla LoRA? Can you do a direct comparison of training compute/FLOPs for LoRA vs. KaSA?

2. “KaSA employs a small set of parameters Ψ to reparameterize the task-specific update △Θ” is the task specific update full finetuning or LoRA? I assume the task specific update is vanilla LoRA.

3. How do you calculate $W_{fft}$ for $\mathcal{L}_2$? Do you have to do full finetuning? I assume not…

There are a few smaller details that I think are worth mentioning:

1. What is the rank for instruction following? I don’t see it in the manuscript

2. It feels like important baselines are missing from Table 4 - how well do the models do out of the box? Since these datasets are created specifically for this study, it is hard to get a sense of task difficulty or to compare to other papers

3. The authors claim no extra inference latency due to the technique. It would be helpful to show these numbers. If I understand correctly, however, there will be extra _training_ latency. Can you expand on this in the paper? 

4. The most similar methods to KaSA are PiSSA and MiLoRA. In Table 4 it could be helpful to do a comparison with PiSSA and/or MiLoRA
Yes
No ethics review needed.
The SVD truncation stage may discard knowledge components deemed noisy or irrelevant. However, there’s a risk that some truncated components might contain task-relevant information. Did you experiment with different truncation thresholds, or analyze the potential trade-offs of truncation in various tasks? A discussion here would enhance understanding of KaSA’s robustness.

The SVD rank and regularization coefficients play an important role in KaSA’s performance. Could you provide practical guidelines or a sensitivity analysis to help practitioners choose these values for new tasks or datasets? This could make the method more accessible to those less familiar with PEFT or SVD.
KaSA combines SVD truncation with a knowledge-aware adaptation process, enabling it to achieve impressive performance on various NLP tasks. KaSA often outperforms other PEFT methods on benchmarks, showing its effectiveness in balancing performance and efficiency. The experiments are thorough, covering natural language understanding, generation, and instruction-following.
The ablation studies break down how each component (like the knowledge-aware adaptation) contributes to the overall performance, giving readers a clearer picture of why KaSA works so well.
The paper introduces KaSA (Knowledge-aware Singular-value Adaptation), a parameter-efficient fine-tuning (PEFT) method that adapts large language models (LLMs) using knowledge-aware singular values derived from singular value decomposition (SVD). KaSA aims to selectively activate task-relevant knowledge, discarding irrelevant or noisy information to improve model performance on specific tasks. The method consists of knowledge-based SVD truncation to filter noise and knowledge-aware singular value adaptation to dynamically manage task-specific knowledge activation. Experiments across natural language understanding, generation, and instruction-following tasks demonstrate that KaSA consistently outperforms other PEFT approaches, including LoRA and its variants.
The method adds some complexity, especially with the dynamic singular value adaptation, which might make it harder to implement compared to simpler PEFT methods. A potential limitation is the risk that SVD truncation might discard useful knowledge, especially if it’s not fully adapted to a task. It would also help to see this method applied to a wider range of LLM architectures to confirm its generalizability. While KaSA does well on the tested models, applying it to newer transformer architectures or other types of large models could demonstrate its versatility even more.
Yes
No ethics review needed.
- The PEFT methods are sensitive to hyperparameters like the learning rate. How does the paper ensure a fair comparison with the baseline models? For instance, in Section 4.2, a hyperparameter search is conducted for KaSA for each dataset. Is a similar hyperparameter search performed for all the baseline methods? For Section 4.3 and 4.4, do KaSA and baselines share the same hyperparameters?
- The setup for the instruction-following experiments is questionable. Given the availability of numerous instruction tuning datasets, why not conduct experiments using these existing datasets? Since other researchers may not have access to the synthetic dataset, reproducing the results could be challenging.
- The paper is well written and easy to follow. 
- The performance is good.
This paper proposes a novel PEFT method called KaSA. The main idea is to leverage singularvalue decomposition with knowledge-aware singular values to dynamically activate knowledge based on its relevance to the task at hand. Experiments on NLU, NLG and instruction tuning tasks demonstrate the effectiveness of the proposed method.
- The proposed method has a more complex training objective compared to baselines like LoRA, PiSSA, and MiLoRA. Consequently, the training time for KaSA may be longer than that of the baselines. However, the paper does not provide detailed information on this.
- The training objective introduces new hyperparameters, including \beta and \gamma, but there is no analysis regarding their impact.
- How does KaSA perform at different ranks? Can it consistently exceed the baselines across various ranks? This aspect also needs to be validated.
- Some important baselines are missing, like [1]

[1] CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning
Yes
No ethics review needed.
How well does KaSA scale to even larger models?
1. The method is novel, integrating SVD with knowledge-aware singular values. This method allows for the dynamic activation of relevant knowledge, which is a significant advancement over existing PEFT techniques.
2. Extensive experiments (various tasks with different LLMs, a wide range of baselines) consistently validate the efficacy and adaptability of the proposed method.
3. The paper is well written with good clarity.
The paper presents Knowledge-aware Singular-Value Adaptation (KaSA), an innovative PEFT method aimed at overcoming the computational and memory challenges of adapting LLMs to specific tasks. KaSA employs singular value decomposition with knowledge-aware singular values to dynamically activate relevant parametric knowledge for downstream tasks. Through extensive experiments, the authors show that KaSA consistently outperforms full fine-tuning and other popular PEFT methods across a range of tasks, including NLU, NLG, and instruction following.
My main concern -- most experiments, except experiments in Table 4 on instruction following, only use a small model (less than 1B). Need more experiments and fair comparison with the baselines on larger models. And the improvements on instruction following evaluation results are not significant, and there are even drops in performance on LLaMA3 8B.
@inproceedings{
anonymous2024kasa,
title={Ka{SA}: Knowledge-Aware Singular-Value Adaptation of Large Language Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=OQqNieeivq},
note={under review}
}
The increasing sizes of large language models (LLMs) result in significant computational overhead and memory usage when adapting these models to specific tasks or domains. Various parameter-efficient fine-tuning (PEFT) methods have been devised to mitigate these challenges by training a small set of parameters for the task-specific updates of the model weights. Among PEFT methods, LoRA stands out for its simplicity and efficiency, inspiring the development of a series of variants. However, LoRA and its successors disregard the knowledge that is noisy or irrelevant to the targeted task, detrimentally impacting model performance and leading to suboptimality. To address this limitation, we introduce Knowledge-aware Singular-value Adaptation (KaSA), a PEFT method that leverages singular value decomposition (SVD) with knowledge-aware singular values to dynamically activate knowledge based on its relevance to the task at hand. We conduct extensive experiments across a range of LLMs on tasks spanning natural language understanding (NLU), generation (NLG), and instruction following. The experimental results demonstrate that KaSA consistently outperforms FFT and other popular PEFT baselines across 7 benchmarks and 4 synthetic datasets, underscoring our method's efficacy and adaptability.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Large Language Models Parameter-efficient Fine-tuning Singular Value Decomposition
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/f81d5fc476e8cab122a91cfa8c3cdec340e549ee.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission