Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablation
LLM Safety;Exaggerated Safety;Representation Editing
Thank you for the valuable feedback and for recognizing that our approach is “very justified”, the idea of fine-grained control is “quite interesting and very useful in addressing the subjectivity nature of safety alignment”, and the paper “does a good job demonstrating that quantitatively and quantitatively”. 

We address the concerns raised by the reviewer below:

### **Claim about the unchanged inference cost**

We first want to highlight that our method can be used either **during inference time (activation steering)** or **before inference time (model editing)**, as the vector we extracted is **fixed** and **independent of the input**. We can directly apply the vector ablation once on the weight matrix through linear transformation, mitigating the false refusal problem on the edited model, which requires no intervention during its inference time. As the new weight matrix through linear transformation still **keeps the original matrix size**, the **memory and inference time are kept**.

### We give a detailed explanation below:

In the Transformer model, each Attention (Att) and FNN block writes its output to the residual stream. To prevent the Att and FNN blocks from representing the vector  $\mathbf{r}$  we want to ablate, we can apply the following transformation during inference time:
$\mathbf{X}{output} \rightarrow \mathbf{X}{output} - \mathbf{r} \mathbf{r}^T \mathbf{X}{output}$

where  $\mathbf{X}{output}$  is the output from either the Attention or FNN block.

Given that the representation in each Attention/FNN layer passes through a weight matrix  $\mathbf{W}$  before being written to the residual stream: $\mathbf{X}{output} = \mathbf{W} \mathbf{X}{pre}$,
ablating on $\mathbf{X}{output}$ is equivalent to directly ablating the weight matrix  $\mathbf{W}$:

\begin{equation}
\mathbf{X}{output} - \mathbf{r} \mathbf{r}^T \mathbf{X}{output}=  \mathbf{W} \mathbf{X}{pre} - \mathbf{r} \mathbf{r}^T (\mathbf{W} \mathbf{X}{pre}) = ( \mathbf{W}- \mathbf{r} \mathbf{r}^T \mathbf{W}) \mathbf{X}{pre} = \mathbf{W}^{\prime} \mathbf{X}{pre}
\end{equation}

, where $\mathbf{X}{pre}$ is the representation before the final linear layer in the Att/FNN block.

The new matrix  $\mathbf{W}^{\prime}$  has the same dimensions as the original matrix $\mathbf{W}$, resulting in no additional memory or inference cost.

We also refer the reviewer to Appendix E in [1], where the author also provides detailed proof of why it is equal to weight linear transformation.

Sorry for causing the confusion and we will add more explanation in the final version to make the claim clear.


### **Requiring access to model internals limits its applicability**
We kindly disagree that requiring access to model internals should be considered as a limitation since we provide a solution which can **also be adopted by the developers of the close-sourced models**. Looking at model internals also provides more benefits in interpreting and understanding the refusal mechanism inside the “black box” of the language model. A solution on top of the proprietary models also faces the **issue of reproducibility** and a **lack of model-internal understanding** of the “false refusal” problem.  	
Our work aims to provide a general solution to the transformer language model and gives insights that the “true” and “false refusal” can be partially separated in the representation space of the model.	

### **Comparison to training-based baseline**
Thank you for suggesting comparing to a training-based baseline. We agree adding such results provides more understanding of the gap. As discussed in the paper SCAN (our training-free baseline), they showed that training-based methods show a high refusal rate (low compliance rate) on pseudo-harmful data due to the scarcity of the data.

We compare our method with a training-based baseline DRO[2], which is also chosen as the only training-based baseline in SCAN. The Llama2-7B-Chat compliance rate result is shown below. The results of DRO are directly taken from the SCAN paper.
|  | Category | XSTest-S ↑ | XSTest-U ↓ |
| :--- | :--- | :--- | :--- |
| DRO | Training-based | 58.5 | 1.5 |
| SCAN | Training-free | **91.8** | 6.5 |
| Ours | Training-free | 85.2 | **0.0** |

The training-based method DRO performs significantly worse on pseudo-harmful data (XSTest-Safe). Our method outperforms DRO on both Safe and Unsafe data, showing the effectiveness of our method in separating the “false refusal” and “true refusal” features.
We will add training-based baseline results in our final version.

Thanks again for the valuable suggestions. We hope this can address your concern adequately. If you find the revisions and clarifications satisfactory, we would appreciate your consideration in re-evaluating the manuscript.
```
[1]: Arditi, Andy et al. “Refusal in Language Models Is Mediated by a Single Direction.” ArXiv abs/2406.11717 (2024)
[2]: Zheng, Chujie et al. “On Prompt-Driven Safeguarding for Large Language Models.” ICML 2024.
```
Reply to Reviewer JoyR
Yes
No ethics review needed.
I am mostly puzzled by the claim about the unchanged inference time cost. Please provide some numbers if you have any to make that claim more accurate.
1. The approach is simple and very justified

2. The idea of fine-grained control via partial orthogonalization is quite interesting and can be very useful for addressing the subjectivity nature of safety alignment. The paper does a good job demonstrating that quantitatively and quantitatively as well.

3. The paper presents an interesting set of experiments that demonstrate the effectiveness of the approach.
The paper considers the problem of false refusals in LLMs, e.g., refusing to answer "How do I kill a python process?".  The paper takes a "model internals steerability" approach and builds on previous work that extracts a refusal vector from the model's internal representations and ablate it from all layers at inference time. The paper modifies that method by focusing on false refusals instead of general refusals. The paper also presents a method that allows for numerically controlling how conservative the model should be which can be very useful with subjective cases. The paper presents results and analysis that demonstrates the effectiveness of the presented approach in comparison to existing methods.
1. The method requires access to model internals which limits its applicability to a certain extent. 

2. The paper claims that the inference cost does not change which does not seem accurate as far as I can tell. The operation in eq. 4 is applied at all layers of the model. The paper needs to report inference time numbers as well as memory consumptions in table 3 instead of claiming they are "unchanged". 

3. While the paper provides some argument against training-based methods, it'd still be valuable (and needed in my opinion) that the paper compares to such methods as a baseline to provide some understanding of the gap in results.
Yes
No ethics review needed.
The paper presents a novel method for mitigating false refusals in language models. The method's model-agnostic nature, training-free implementation, and ability to provide fine-grained control make it valuable for enhancing the usability and safety of LLMs. The experimental validation convincingly demonstrates the effectiveness of the proposed technique.

Here are some questions I want to ask:

1. Are there any theoretical guarantees or bounds that support the effectiveness of your method in distinguishing between false and true refusal behaviors?

2. How sensitive is the proposed method to the selection of pseudo-harmful queries used for extracting the false refusal vector?

3. Your experiments are conducted on specific models like GEMMA-7B-IT and various LLAMA models. How does your method scale to larger models (e.g., 175B parameters) or different architectures beyond the ones tested?
1. Addressing an Important Problem: Mitigating false refusals in language models is a relevant and significant issue, as it directly impacts the usability and reliability of LLMs in real-world applications.

2. Novel Approach: The idea of using single vector ablation and orthogonalization to disentangle false refusal behaviors from true refusal mechanisms is innovative and contributes to the existing body of work on LLMs safety.

3. Comprehensive Experiments: The paper conducts experiments across multiple models and datasets, providing a broad evaluation of the proposed method's effectiveness and generalizability.

4. Fine-Grained Control via Partial Orthogonalization: The ability to adjust the orthogonalization coefficient (λ) for fine-tuning the model's sensitivity to ambiguous queries is a valuable feature. It allows users to tailor the model's behavior to specific application requirements, balancing between over-restrictiveness and permissiveness.
The paper presents a method for reducing false refusals in language models by ablating a single vector from the model's activation stream. The proposed approach involves extracting a false refusal vector using pseudo-harmful queries and orthogonalizing it with the true refusal vector to minimize unintended impacts on the model's safety and general capabilities. The authors claim that their method is training-free, model-agnostic, and enables fine-grained calibration of model safety without additional inference costs. Experimental results are provided on several datasets using different language models to demonstrate the effectiveness of the proposed method.
1. Insufficient Theoretical Justification: The paper lacks a robust theoretical framework explaining why single vector ablation and orthogonalization effectively mitigate false refusals without adversely affecting true refusals or general capabilities. A deeper theoretical analysis or justification is necessary to understand the underlying mechanisms and guarantees of the proposed method.

2. Only partial Mitigation of False Refusals: While the proposed method significantly reduces false refusal rates, it does not entirely eliminate the issue. Further research is needed to fully address the underlying causes of false refusals in language models.
Yes
No ethics review needed.
Refer to the weakness part.
1.	The paper proposes a simple and surgical method for mitigating false refusal in LLMs via single vector ablation. The method is training-free and model-free, requiring no extra computation resource or memory during inference.

2.	The idea of separating refusal related features from false refusal vectors by orthogonalization is novel and interesting. Partial orthogonalization also provides fine-grained control of model safety and helpfulness.

3.	The paper conducted comprehensive experiments to demonstrate the effectiveness of the method, and enhance the understanding of different factors in the method. The method can effectively reduce false refusal while maintaining the performance of general tasks.
This paper focuses on the false refusal problem in the safety scenario of LLMs, i.e., LLMs tend to refuse safe requests that superficially resemble unsafe ones (e.g. “how do I kill a Python process?”). Specifically, the paper proposes a simple and surgical method for mitigating false refusal in LLMs via single vector ablation. For a given LLM, they extract a false refusal vector based on the activations of LLM’s layers using harmful, pseudo harmful, and harmless datasets. Then, they demonstrate that ablating this vector reduces false refusal rate without reducing model safety and general model capabilities. The partial orthogonalization also enables fine-grained calibration of model safety.
1.	While the method is described quite clear in the current form, adding a workflow figure to demonstrate the process of extracting (false) refusal vectors and ablating them can make it easier for understanding.

2.	In section 4, the paper uses greedy decoding for text generation, which however is not the common choice in practice. It would be interesting to see how sampling decoding affects the effectiveness of the method.

3.	In section 4, the samples of harmful, pseudo harmful, and harmless come from different sources. Will the difference of data content or domain affect the method? BTW, there are some typos of the symbols for datasets in line 182. 

4.	In section 5.1, why would Llama2-7B-Chat and Llama3-8B-Chat perform worse when adding the system prompt of Llama2 models?
Yes
No ethics review needed.
1. I may have missed the points: 

a. Do a harmful and a harmless query differ by a single token or multiple tokens, i.e., does Equation (1) assume both queries are of the same length?

b. In the case where the tokens in both queries vary a lot and/or query lengths are different, what is the physical meaning of a refusal vector?

c. Is the [/INST] token the only token position chosen for representing a refusal vector?

d. Just to clarify, is Equation (5) directly related to the proposed method in reducing false refusals?

e. How exactly is vector ablation applied to general tasks, such as MMLU?

f. Why does general capability, as measured by acc. or ppl., still drop after vector ablation, as shown in Table 2 (given the significant CR improvements in false refusal benchmarks)?

2. Pseudo-harmful queries are essentially harmless. Are we assuming that all pseudo-harmful queries are potentially treated as harmful by the models used in this work? I am curious that, for pseudo-harmful queries that are potentially treated as harmless by the model, would the difference between a truly harmful query and a pseudo-harmful query create a meaningful direction for ablation (as a false refusal vector)?

3. Can you discuss if the proposed method can be extended to, or compared with other intervention techniques? For example, I am curious about any similarities/differences, pros and cons, between the proposed method and causal interventions, such as [1].

[1] K. Meng, D. Bau, A. Andonian, and Y. Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359–17372, 2022.
Mitigating false refusals is important in enabling chat models with more satisfying responses. The idea of orthogonalization on a true refusal vector and a false refusal vector is interesting and useful.

In particular, the authors identify that the true refusal vector and false refusal vector are not independent of each other. Therefore, they propose to apply orthogonalization between the candidate false refusal vectors and the true refusal vector, resulting in an orthogonalized false refusal vector for ablation.

The proposed method achieves increased compliance rates (CR) on false refusal datasets, including ORB-H, XSTest-S(H), and OKTest. The authors also illustrate that the proposed vector ablation method removes false refusal on safe queries while maintaining true refusal on unsafe queries.
This paper proposes an orthogonalization based vector ablation method to mitigate LLM false refusals, which ablates a false refusal vector extracted from the diff-in-means vectors obtained by prompting with pseudo-harmful and harmless queries. A main novelty is to orthogonalize the false refusal vector and the true refusal vector (where the true refusal vector is based on truly harmful and harmless queries) to avoid harming the true refusal ability. The proposed vector ablation method helps remove false refusals on safe queries while maintaining true refusals on unsafe queries.
The presentation could use clearer definitions and equations, and it might be better to provide an intuitive illustration showing example queries in an example transformer network targeting at a layer and a token position. 

For example, in Section 2.1, line 60-62, it is not very clear how the harmless query and harmful query look like, so I am a little confused by the physical meaning of taking the average output over all queries at a token position. 

In addition, in Equation (3), I am not sure why the definition for pt is omitted (according to the context, $p_t$ is token t’s probability at the first token position in the model’s response).

Other possible typos: in line 116, wrong subscript in Equation (7)? In line 182, three repeated $D^{train}_{harmful}$; in line 469, BLUE score or BLEU score?
We propose a surgical and flexible approach to mitigate the false refusal in LLMs with minimal effect on performance and inference cost.
@inproceedings{
anonymous2024surgical,
title={Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=SCBn8MCLwc},
note={under review}
}
Training a language model to be both helpful and harmless requires careful calibration of refusal behaviours: Models should refuse to follow malicious instructions or give harmful advice (e.g. “how do I kill someone?”), but they should not refuse safe requests, even if they superficially resemble unsafe ones (e.g. “how do I kill a Python process?”). Avoiding such false refusal, as prior work has shown, is challenging even for highly-capable language models. In this paper, we propose a simple and surgical method for mitigating false refusal in language models via single vector ablation. For a given model, we extract a false refusal vector and show that ablating this vector reduces false refusal rate without negatively impacting model safety and general model capabilities. We also show that our approach can be used for fine-grained calibration of model safety. Our approach is training-free and model-agnostic, making it useful for mitigating the problem of false refusal in current and future language models.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
LLM Safety Exaggerated Safety Representation Editing
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/c9c09365703c0edce19e1a6b0330f076b7c63c2a.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission