SPARQ: Outlier-free SpeechLM with Fast Adaptation and Robust Quantization
Multi-modal foundation model; Speech Language Model; Low-rank adaptation; Post-training quantization
Yes
No ethics review needed.
- What is meant by "Given this challenge, we would like to develop a transformer architecture that can efficiently solve
the challenge in Equation (3.1). "
Good empirical performance and reasonable model structure. 
Theoretical analysis presented in the appendices is also appreciated, even though most are from previous papers. But for completeness it is good to have them there.
Authors propose a way to integrate text and speech into a one single model where outliers do not have a significant impact on the modeling.
I am puzzled but this paper. It is well written, experimented and mathematized. But the central claim about the outliers is not clearly presented and it is not theoretically pursued! As an example, I would have expected that at least following things would be easy to find in the paper, taking into account that the keyword "outlier" is in the title: 
1. Definition of the outlier in the context of the paper. Especially considering that classical definition of outlier in statistics and ML is that observation is _not_ from the distribution that is being modeled. 
2. Explaining theoretically why Eq. (3.2) is resistant to outliers as defined in the 1. 
3. Show in experiments explicitly that this indeed happens in practice also. Noting that this is different to showing general performance of the model in some downstream tasks.
Yes
No ethics review needed.
Given the computational limitations that prevented testing with larger models, how might SpARQ perform when scaled up to models with billions of parameters, such as OPT-6.7B? Are there specific architectural adjustments needed to maintain stability and performance at this scale?

Several stabilization methods are compared in the paper, with max-shift normalization performing best. What were the primary reasons for choosing max-shift, and are there conditions under which other methods (like L1 or mean-centering) might be more effective?
Innovative Approach: SpARQ introduces an outlier-free layer within the SpeechLM architecture, a novel solution to the inefficiencies caused by outliers that often hamper model performance and computational effectiveness.

Enhanced Performance: The framework brings notable improvements to Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) tasks, showing a remarkable 41% boost in cross-modal adaptation and a 45% gain in post-training quantization accuracy.

Training Efficiency: With a 1.33x increase in training speed, SpARQ is well-suited for environments with limited computational resources, making advanced applications more accessible.

Thorough Evaluation: The model’s effectiveness is rigorously tested across various quantization and adaptation techniques, establishing a strong empirical basis for its success.
The paper, "SpARQ: Outlier-Free SpeechLM with Fast Adaptation and Robust Quantization," tackles a major hurdle in Speech and Language Models (SpeechLMs): the disruptive effects of outliers in multi-modal settings. To address this, the authors present SpARQ, a framework that replaces traditional attention layers with a mechanism specifically designed to handle outliers. This innovation helps overcome challenges in cross-modal low-rank adaptation and post-training quantization, resulting in a more efficient and accurate approach to tasks like Automatic Speech Recognition (ASR) and Text-to-Speech (TTS). The goal is to streamline training while improving performance across these applications.
Testing Constraints: The paper notes that due to computational limitations, the model hasn’t been tested on extremely large models, such as OPT 6.7B. This restriction could affect the generalizability of results for larger-scale applications.

Architectural Complexity: The architecture includes several layers and a specialized stabilization method, which could make practical implementation challenging without advanced technical expertise.

Risk of Bias Amplification: Although not deeply explored, the framework’s emphasis on reducing computational demands might unintentionally increase biases inherited from pre-trained models, given the reduced resources available for extensive fine-tuning.
Yes
No ethics review needed.
1. Why does fully fine-tuned SpARQ generally show worse performance than fully fine-tuned Vanilla, despite the claim that multimodal samples have more outliers that degrade performance?
2. What is the number of trainable parameters for LoRA in the tables presented in the paper?
3. Do Table 1 and Table 2 reflect experiments with only PTQ and LoRA, respectively? If so, what are the results when applying PTQ on the LoRA-trained SpARQ? Including these results would clarify the model’s performance when combining the techniques.
1. The paper addresses an important challenge induced by the outliers in multimodal attention mechanisms, particularly in SpeechLMs.
2. The stabilized outlier-free layer demonstrates effective mitigation of outliers during cross-modal low-rank adaptation and quantization, showing promise for real-world SpeechLM applications.
The paper introduces SpARQ, an outlier-free framework designed to address performance degradation in SpeechLMs by mitigating outliers. SpARQ replaces the standard attention mechanism with a stabilized, outlier-free layer that improves the model's adaptability in cross-modal tasks and enhances post-training quantization robustness. Built on the OPT model, SpARQ demonstrates resilience to outliers and offers performance gains across several tasks, such as textLM, speechLM, ASR, and TTS.
1. The primary contribution of this paper is the integration of the Outlier-Efficient Hopfield Layer with max-shift normalization, which limits the originality of SpARQ. While this is the first application of outlier mitigation specifically within SpeechLMs, the Outlier-Efficient Hopfield Layer itself was previously introduced by Hu et al.
2. Techniques like Clipped Softmax and Gated Attention are potential alternatives but were not included in comparisons. Experimental results of adapting them to the SpeechLM setup would further validate SpARQ's relative effectiveness.
3. It remains unclear how much of SpARQ's improvement is attributable to the outlier-free Hopfield layer itself versus the max-shift stabilization. An additional experiment applying the stabilization module to a standard Transformer would help clarify the effectiveness of SpARQ's contributions.
4. The author states that outliers negatively impact LoRA and quantization performance. Yet, in some cases (e.g., With OPT-350m), QLoRA performs better than LoRA. Clarification on this result would improve understanding.
5. Previous studies (e.g., “Robust multimodal models have outlier features and encode more concepts” by Crabbe) mention that outlier features increase robustness for out-of-domain (OOD) samples, but OOD evaluations of SpARQ are missing. Demonstrating SpARQ’s performance on OOD data would add value to this paper.
6. There is insufficient analysis in the ablation study to explain why the max-shift stabilization uniquely works in SpARQ.
7. In Equation 3.2, the notation "S = S - max(S)" should clearly distinguish between the normalized and original versions of S.
Yes
No ethics review needed.
See weaknesses section. I'm happy to increase my ratings if my concerns can be addressed.
* Although the novelty is a combination of existing works (SpeechLM + outlier-free layer), the combination seems to be novel. The authors also provided systematic theory support for the proposed approach.
* The organization of paper is good. Paper writing is good. References are enough.
This paper proposes to use the outlier-free layer within the attention to reduces the occurrence of weight outliers for quantization and LoRA adaptations with a SpeechLM model. The outlier-free layer is constructed by replacing the first softmax layer with a layer that uses S = S - max(S) as the input, where S is the original input vector. The authors conducted three sets of experiments to validate the proposed approach: quantization, LoRA, and ablation studies / analyses.
* Quantization experiments: A common approach to remove outliers in quantization is to do clipping. For example, ignore the 5% tail weight values during computing the quantization scale. It’d be helpful to include this as a baseline to correctly validate the effectiveness of the proposed work.
* LoRA experiments: rank=128 seems to be large. Please add the percentage of adapter size compared to the full model.
* LoRA experiments: I would also suggest to separate the experiments to two fold, if LoRA+Quantization is one of the focus on this work. In a first experiment, we can investigate the impact of LoRA (with FP 32/16). In a second experiment, we investigated the confounding effect between LoRA and quantization (should be more aggressive than just INT8).
* Table 2: Why are the ASR WEB numbers of vanilla -125M model much better than 350B and 1.3B. Is it because the rank is set to 128 across the board? If so, the experimental setup is problematic. When applying LoRA, the rule of thumb is to control the adapter size to be under a certain percentage.
* Table 3: where does the training efficiency improvement come from? Does it require fewer steps to converge? Or is it quantization?
@inproceedings{
anonymous2024sparq,
title={{SPARQ}: Outlier-free Speech{LM} with Fast Adaptation and Robust Quantization},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Z2uhdwOrn0},
note={under review}
}
We propose SpARQ (outlier-free SpeechLM for Fast Adaptation and Robust Quantization) to address the outlier problem in Speech and Language multi-modal Models (SpeechLMs). Our primary observation is that outliers stemming from cross-modal (speech and text) low-rank adaptation and post-training quantization stages affect the performance of the current SpeechLMs. Methodologically, SpARQ leverages a pretrained language model as its foundation, substituting the traditional attention layer with a novel stabilized outlier-free layer. This modification eliminates outliers typically arising during cross-modal low-rank adaptation and post-training quantization. The model is then fine-tuned on multi-modal data using this outlier-free architecture, allowing it to handle textLM, speechLM, ASR, and TTS tasks through a unified interface while maintaining compatibility with parameters adapted from standard pretrained LLMs. Consequently, on the OPT-1.3b model, the proposed framework achieves relative performance improvements: 41\% in cross-modal low-rank adaptation and 45\% in post-training quantization, along with a 1.33x training speedup. We benchmark it against state-of-the-art low-rank adaptation and post-training quantization methods.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Multi-modal foundation model; Speech Language Model; Low-rank adaptation; Post-training quantization
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/0319232c40b2ee9eed756915b1e63a2ff7d7f33b.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
SPARQ: Outlier-free SpeechLM with Fast Adaptation and Robust Quantization
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission