Towards Comprehensive and Efficient Post Safety Alignment of Large Language Models via Safety Patching
Post Safety Alignment;Large Language Models;Jailbreak Defense;Over-Safety Mitigation
Yes
No ethics review needed.
1. How does the use of gradient ascent and descent for patch derivation differ from recent work in unlearning?
1. The problem addressed—post-hoc safety alignment—is important for ensuring that LLMs behave safely in real-world applications.
2. The empirical evaluation and ablations are fairly comprehensive across different LLM backbones and benchmarks.
3. The method shows some promise in balancing safety with utility preservation compared to existing baselines.
The paper presents a method called SafePatching for post safety alignment (PSA) of large language models (LLMs). The authors claim that SafePatching addresses three PSA objectives: safety enhancement, over-safety mitigation, and utility preservation.
1. The proposed approach seems to be largely composed of a series of straightforward adaptations or incremental improvements on recent work. For instance, the use of gradient ascent and descent techniques for deriving safety and over-safety patches is largely an adaptation of existing machine unlearning methods described in the paper, rather than a truly novel contribution. The concept of patching the difference set of important parameters between safety and over-safety patches is perhaps the most novel aspect. However, it's still a relatively straightforward extension of existing ideas in parameter importance and model merging.
2. While the proposed approach does demonstrate that it is the only one to improve safety, over-safety, and utility over the backbone, in many cases, it performs significantly worse than the baselines for a particular safety or over-safety benchmark. Moreover, the safety and over-safety improvements over the backbone model are quite marginal in some cases. This highlights that there is more work to be done in effectively controlling the balance between safety enhancement and over-safety mitigation than the approach in its current state.
Yes
No ethics review needed.
Would you please address the concerns in weakness?
•	The idea is straightforward.
•	The experiments are extensive.
This paper proposes a post safety alignment method which merges two models post-trained on harmful data with gradient ascent and descent respectively. The post-trained and merged model preserves a balance on safety, over-safety mitigation, and utility preservation.
•	The paper lacks the comparision of external safeguards methods such as OpenChatKit and NeMo guardrails that are known to handle over-safety issues. Would these external safeguards methods also achieve the three objectives proposed in the paper?
•	There are a few hyperparameters in equation 7&8, such as a, b, \alpha, \beta. How you set these parameters? In Table 3, merging methods like the task arithmetic and TIES-merging do not have big differences compared to the intersect patch. Would the benefit comes from your hyperparameter selection?
Yes
Yes, Discrimination / bias / fairness concerns Yes, Privacy, security and safety
* Given that only the AdvBench dataset was used to evaluate SAFEPATCHING, how does the method perform across other safety-related datasets? Could testing with a broader range of harmful data enhance our understanding of its transferability to diverse safety scenarios?
* Since the authors did not specify whether they directly used the fine-tuned Longformer model from Wang et al. or performed additional fine-tuning, what impact might this setup have on the accuracy and reliability of the judgment model in this experiment?
* Could a deeper explanation of these aspects clarify the novelty and rigor of the proposed approach in Section 3.3?
* The paper proposes a new method named SAFEPATCHING to address the limitations of existing methods on post-safety alignment for LLMs, such as over-safety issues and high cost.
* The paper presents experimental results and comparisons with state-of-the-art methods to demonstrate the effectiveness of SAFEPATCHING and uses multiple open-source datasets on safety, over-safety, and utility for a comprehensive evaluation. Besides, this paper has interesting findings on the distribution of the most important parameters for safety and over-safety, providing future research directions for the community.
This paper proposes a novel post-safety alignment (PSA) method, called SAFEPATCHING, which aims to address safety, over-safety, and utility issues in large language models (LLMs). In this paper, the authors develop a two-stage PSA framework, which applies distinct safety patches to the backbone LLM based on harmful data to improve safety and reduce over-safety, meanwhile, maintaining the utility capability of the LLM. The experiment shows that SAFEPATCHING achieves more effective and efficient PSA compared to baseline methods across four aligned LLMs
* Lack of justification in Sec. 3.3 controllable patching. The authors may want to highlight the novelty of their tool and the rigor of their method. Currently, it appears that the approach relies on the SNIP score proposed by Lee et al., as well as model merging methods by Yu et al. and Hui et al., without a thorough explanation of the unique contributions or advancements made in this work.
* Although the authors conducted an excessive experiment to show the effectiveness of SAFEPATCHING, several concerns existed in the settings.
   * The study evaluates SAFEPATCHING using only a single harmful dataset, AdvBench, which may not adequately demonstrate the method's transferability across different safety scenarios. Given the extensive range of safety categories and perspectives, it's essential to assess whether a backbone LLM patched using AdvBench can maintain its effectiveness on other datasets representing diverse types of harmful content.
   * The authors did not specify how they fine-tuned the Longformer-based judger. Wang et al. used annotated data generated through human labor to fine-tune their Longformer model. It remains unclear whether the fine-tuned model from Wang et al.'s work was directly utilized in this experiment or if further adjustments were made. Clarification on this point would provide a better understanding of the model’s setup and any adaptations relevant to this study.

Yu, Le, et al. "Language models are super mario: Absorbing abilities from homologous models as a free lunch." Forty-first International Conference on Machine Learning. 2024.
        
Hui, Tingfeng, et al. "HFT: Half Fine-Tuning for Large Language Models." 
        
Wang, Yuxia, et al. "Do-not-answer: A dataset for evaluating safeguards in llms."
Yes
- Potential for Misuse and Safety Bypass

The SafePatching framework’s dual-patch approach is designed to mitigate over-safety, allowing the model to respond to benign prompts with sensitive keywords. However, this opens up a risk of misuse if bad actors attempt to exploit this flexibility to bypass safety mechanisms deliberately.
Yes, Privacy, security and safety Yes, Potentially harmful insights, methodologies and applications
In the SafePatching framework, Eq (1) and Eq (2) are designed to achieve two opposing objective by applying gradient-based updates in opposite directions on the same harmful dataset. Could you please clarify and elaborate how they are implemented given a harmful dataset?


In SafePatching, what requirements should a harmful dataset fulfill? For example, are there specific expectations concerning its size, diversity, or other characteristics? Additionally, are these requirements realistic for SafePatching's application in real-world scenarios?
+ The proposed approach is easy to understand, logical, and appears to be effective.

+ It addresses a significant and timely problem.

+ The paper is overall well-written.

+ The unlearning and fine-tuning techniques used in SafePatch are not new, the originality comes from considering dual patching at the same time.

+ The paper includes an extensive set of experiments.
This paper introduces a SafePatching framework to improve the safety of large language models (LLMs) while maintaining their utility. The major contribution is two types of safety patches. 
- The safety enhancement patch utilizes gradient ascent on harmful data to train the model to avoid generating unsafe responses. It effectively helps the model "unlearn" unsafe behaviors by adjusting the model parameters to minimize the risk of producing harmful content. 
- The over-safety mitigation patch, developed through gradient descent, is designed to prevent the model from being overly cautious. It fine-tunes the model to ensure it does not overly restrict or reject benign inputs that might superficially appear sensitive or risky. 

 The approach is tested across multiple LLMs, showing better performance in reducing harmful outputs, handling over-safety, and preserving utility compared to several existing methods.
- Limited Novelty in Core Techniques

While the dual-patching approach is innovative in combining safety enhancement with over-safety mitigation, the core methods (e.g., gradient ascent and descent on harmful data) rely heavily on existing unlearning and fine-tuning techniques.

- Clarity on Practical Deployment

The paper would benefit from more actionable details regarding the real-world deployment of SafePatching, especially the requirements on the harmful data set.

-  Stability of SafePatching Approach

SafePatching's dual-patch integration requires careful parameter tuning, especially with the two gradient-based patches potentially introducing conflicts within the model. The process of managing these interactions, although effective, may lack robustness or generalizability across different architectures or types of prompts.
@inproceedings{
anonymous2024towards,
title={Towards Comprehensive and Efficient Post Safety Alignment of Large Language Models via Safety Patching},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=09JVxsEZPf},
note={under review}
}
Safety alignment of large language models (LLMs) has been gaining increasing attention. However, current safety-aligned LLMs suffer from the fragile and imbalanced safety mechanisms, which can still be induced to generate unsafe responses, exhibit over-safety by rejecting safe user inputs, and fail to preserve general utility after safety alignment. To this end, we propose a novel post safety alignment (PSA) method to address these inherent and emerging safety challenges, including safety enhancement, over-safety mitigation, and utility preservation. In specific, we introduce \textsc{SafePatching}, a novel framework for comprehensive and efficient PSA, where two distinct safety patches are developed on the harmful data to enhance safety and mitigate over-safety concerns, and then seamlessly integrated into the target LLM backbone without compromising its utility.  Extensive experiments on four representative aligned LLMs, including LLaMA-2/3, Gemma and Mistral, show that \textsc{SafePatching} achieves a more comprehensive and efficient PSA than baseline methods. It even enhances the utility of the backbone, further optimizing the balance between being helpful and harmless in current aligned LLMs. Also, \textsc{SafePatching} demonstrates its superiority in continual PSA scenarios. \textcolor{red}{WARNING: This paper may contain content that is offensive and harmful.}
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Post Safety Alignment Large Language Models Jailbreak Defense Over-Safety Mitigation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/b28f0d1191a8411bdc4d03df6e9a2f113daa2a14.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/5eec85122a6ffebd9e8aacea99e1f803e3556094.zip
Towards Comprehensive and Efficient Post Safety Alignment of Large Language Models via Safety Patching
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission