AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs
adversarial attacks;prompt optimization;red-teaming LLMs
Yes
No ethics review needed.
1. Necessity of alternated update: Is it necessary to put the AdvPrompter in the loop of suffix generation (Figure 1 bottom right)? My understanding is that, the purpose of including it is to generate topk candidate tokens for the suffix. I am curious whether the author has tried to use a separate LLM to do this? The upside is that the dataset used to finetune the AdvPrompter can generated offline (without alternatively updating AdvPrompter); The downside is that the generated most likely tokens will not be adaptive.
1. Exploring the leanring-based paradigm for generating adversarial prompts is novel and relevant, to the best of the reviewer’s knowledge.
2. Compared with search-based paradigm, methods along the learning-based paradigm, like this one, naturally enjoy the benefit of inference-time efficiency.
3. Experiment results suggest that the proposed method surpasses previous method in terms of transferrability to black-box models, which is arguably a more practical scenario than white-box attack.
This paper studies how efficiently generate transferrable and interpretable surfix for jailbreak. Unlike previous white-box attack methods that adopt search-based optimization, the authors propose a learning-based method, i.e. finetuning a LLM to generate adversarial prompts using annotated harmful QA data. A major benefit of this approach is inference-time efficiency. To train the LLM, the authors propose an alternated optimization by first searching for the best surfix that prompts the target LLM to answer harmful queries, and subsequently use it to finetune the Prompter LLM. Experiments are mainly conducted by comparing with some white-box attacks, and both direct search and transfer settings are considered. The result show that a major inference efficiency boost, with mixed results in terms of ASR. The proposed method also exhibits stronger transferability to close-source proprietary models than baselines.
1. [major] It seems that, intuitively, the solution of equation 1 depends on the targetLLM. i.e. the optimal surfix that triggers a LLM to output the target response (e.g. “Sure, here are detailed xxx”) might be different. I’d imagine that this would hurt the transferrability in theory. It does appear that the transferability of AdvPrompter is at least better than early white-box attackers, but it might be due to poor transferability of white-box attackers in the first place.
2. [major] Following 1, I have some doubt about the practicality of jailbreak methods that requires transferrability in general. I would suggest comparing with SOTA blackbox methods on Figure 2. While I acknowledge that it is debatable whether such comparison is academically fair, the general practice usually guides us towards using whichever that is most effective. But I am happy to hear the author’s rebuttal and take them into consideration.
3. [minor] Learning-based paradigm, compared with search-based ones, incurs high training cost.
Yes
The paper is written in a technical way. Personally, I don't think the paper has ethical issues. However, the paper itself is about jailbreaking LLMs, a fairly sensitive topic, might benefit from an additional layer of caution.
Yes, Privacy, security and safety
1. Does the method require gradient during training? (i.e., does the method have to be trained with white-box LLM?) if that's the case, then that is another point needs to be made clear. If not, results showing how the trained advprompter from highly aligned models such as GPT then applied to less aligned models will be interesting. 

2. Training time and requirement of computing might also need to be discussed, although less important. 

3. The authors might need to compare their results in Sec. 4.3 with other jailbreak defensive methods for LLMs.
1. The idea of using a pretrained model to directly generate jailbreak as next token prediction is interesting. 

2. The propose method can generate jailbreak much faster than existing methods, especially since most existing methods are optimizing for every sample.
The paper introduces a new method, and potentially a new perspective, for jailbreak prompting, the main contribution is that the prompter is itself a trained model, thus it can generate the jailbreak prompts extremely efficiently. The authors also mentioned several other properties such as human-readable or gradient-free, but these properties have been well discussed before.
1. the idea of training a generation model for jailbreak has many natural limitations, such as the evaluation will require a different split of data, need to validating that the model can generalize across different target LLMs, and also different benchmark datasets. The authors didn't fully address these. 

     -. 1.1 For example, since the method does not require any gradient of the target LLMs, the authors need to report more results on the commerlized LLMs where the strengths might be more obvious (referring to Table 2). 

    -. 1.2. The evaluation is only limited to AdvBench, the empirical scope is too small. Some other choices are like HarmBench or JAMBench. 

    -. 1.3. The evaluation needs to demonstrate the power of the advPrompter while the model is trained on AdvBench, and tested on other benchmarks such as HarmBench or JamBench. This is very important to show the advantages of proposed methods over the per-sample optimization method. 

    -. 1.4. Similarly, the authors might want to offer more detailed and comprehensive discussions on the differences between the targetLLM during training vs. during testing, although a gentle discussion has been offered in 4.2. 

2. The empirical scope is also fairly limited in terms of the methods being compared. Newer methods in jailbreaks, even just the ones published and presented in recent conferences (excluding arXiv ones), need to be discussed and compared. There are more methods that can deliver human-readable jailbreaks. (Although another method that can simultaneously fulfill all the properties in Table 1 might not exist). 

3. While the authors present a unique method, and might be the only one at this moment can achieve all the properties in Table 1, the performance is unfortunately achieved by trade-offs. For example, in Table 2, the proposed method is not necessarily always the best performing method in ASR. The perplexity is always the lowest, but comparison to newer method might be needed, e.g., [1]. This is important because in AI security research, ASR and perplexity are probably more important factors than generation time. Authors might need to offer more convincing discussions why the method is favored although being lower in ASR in certain cases. 


[1]. Role-playing to Generate Natural-language Jailbreakings to Test Guideline Adherence of LLMs
Yes
No ethics review needed.
See Weaknesses.
* The proposed method enables the fast generation of specific adversarial suffixes for individual harmful requests.
* The experimental results show that the proposed method achieves great performance.
This paper proposes a novel method to enhance jailbreaking attacks on safety-aligned large language models (LLMs). The proposed method involves constructing a framework that fine-tunes an LLM from a base model by encouraging it to generate human-readable adversarial suffixes for harmful requests. Extensive experimental results demonstrate that the AdvPrompter can produce low-perplexity adversarial suffixes and achieve performance comparable to two baseline methods, i.e., GCG and AutoDAN.
Overall, I think the method is sound, but there are a few concerns.

* The advantages of AdvPrompter mentioned in lines 108-136 should be specified under certain comparative conditions. For example, the "adaptivity to input" should be highlighted in the context of generating at a low time cost, as both GCG-individual and AutoDAN-individual are also adaptive to input. The "fast generation" compared to GCG and AutoDAN should be specified in the context of generating individual adversarial suffixes, since GCG-universal and AutoDAN-universal are ready to be used once obtained.

* Since both AutoDAN-universal and AdvPrompter generate human-readable adversarial suffixes quickly, it would be beneficial to discuss their performance in more detail, especially in Table 3.

* I think the "Gradient-free TargetLLM" is not a significant advantage, and it is unnecessary to emphasize the "gray-box TargetLLM" since it is actually a "white-box" model.

* More comparisons to existing methods, such as TAP and PAP, should be included. These methods also generate human-readable adversarial prompts.
Yes
No ethics review needed.
I encourage the authors to address the points raised in the weaknesses section and to conduct additional experiments where further investigation is required.
1. **Clarity**: The paper is well-structured and provides clear explanations of its methodology, backed by comprehensive experimental results and ablation studies. The use of figures and tables, such as Table 1, aids in understanding the comparative advantages of the method. The training algorithm and attack framework are concisely explained, which improves accessibility.

2. **Significance**: AdvPrompter presents significant contributions to the area of LLM robustness and safety, offering a highly scalable solution to automatic red-teaming. Its gradient-free approach makes it applicable in both white-box and black-box settings, which broadens its impact for securing deployed LLM systems. The paper’s findings on fine-tuning models with adversarial data to improve safety also open up new avenues for automated adversarial training in LLMs.
The paper titled "AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs" presents a novel approach to generating adversarial prompts that jailbreak large language models (LLMs), enabling the generation of harmful or inappropriate content. Traditional methods for adversarial prompt generation, such as manual red-teaming or optimization-based methods, can be slow, inefficient, and prone to generating semantically meaningless attacks. In contrast, the authors propose AdvPrompter, an LLM trained using a novel algorithm to rapidly generate human-readable adversarial prompts without requiring gradient information from the target LLM.

The core innovation of the paper lies in its alternating training method, AdvPrompterTrain, which alternates between generating adversarial suffixes and fine-tuning the AdvPrompter model. The resulting adversarial prompts are highly effective, achieving state-of-the-art results on the AdvBench and HarmBench datasets, with improved attack success rates, faster generation times, and strong transferability to black-box LLMs. Additionally, the paper demonstrates that by fine-tuning LLMs on datasets generated by AdvPrompter, models can become more robust against jailbreaking attacks while maintaining high performance on benchmarks like MMLU and MT-bench.
1. **Lack of Comparison with BEAST**: One notable omission is the lack of detailed comparison with BEAST (introduced in "Fast Adversarial Attacks on Language Models in One GPU Minute")​. BEAST also focuses on gradient-free attacks and is highly efficient, achieving impressive success rates within one GPU minute, making it an essential baseline. The authors of AdvPrompter reference BEAST, but they fail to provide head-to-head benchmarks, especially in terms of speed and success rates. This limits the ability to assess whether AdvPrompter's claim of "fast" generation holds up against a method already proven to be both rapid and effective.

2. "Unclear Computational Efficiency": While AdvPrompter claims faster generation of adversarial prompts compared to gradient-based methods, the paper does not include detailed benchmarks or profiling to demonstrate computational efficiency on a per-prompt basis. For instance, BEAST reports precise GPU utilization metrics and compares the attack time per prompt across different models. AdvPrompter lacks such concrete data, making its claims of speed improvements less convincing. Without these comparisons, it is unclear whether the method is truly fast or simply optimized for a limited set of tasks.

Incomplete Discussion of Attack Readability: While AdvPrompter claims to generate human-readable adversarial prompts, there is limited qualitative analysis of the readability or coherence of these prompts. I would like to see more analysis on this.
A novel method for generating human-readable adversarial prompts in seconds for attacking and red-teaming LLMs.
@inproceedings{
anonymous2024advprompter,
title={AdvPrompter: Fast Adaptive Adversarial Prompting for {LLM}s},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=E9GakjQype},
note={under review}
}
While recently Large Language Models (LLMs) have achieved remarkable successes, they are vulnerable to certain `jailbreaking attacks` that lead to generation of inappropriate or harmful content. Manual red-teaming requires finding adversarial prompts that cause such jailbreaking, e.g. by appending a suffix to a given instruction, which is inefficient and time-consuming. 
On the other hand, automatic adversarial prompt generation often leads to semantically meaningless attacks that can easily be detected by perplexity-based filters, may require gradient information from the TargetLLM, or do not scale well due to time-consuming discrete optimization processes over the token space. In this paper, we present a novel method that uses another LLM, called the `AdvPrompter`, to generate human-readable adversarial prompts in seconds, $\sim800\times$ faster than existing optimization-based approaches.
We train the AdvPrompter using a novel algorithm that `does not require gradients` of the TargetLLM. This process alternates between two steps: (1) generating high-quality target adversarial suffixes by optimizing the AdvPrompter predictions, and (2) fine-tuning of the AdvPrompter with the generated adversarial suffixes. The trained AdvPrompter generates suffixes that veil the input instruction without changing its meaning, such that the TargetLLM is lured to give a harmful response. Experimental results on popular open source TargetLLMs show state-of-the-art results on the AdvBench dataset, that also transfer to closed-source black-box LLM APIs. Further, we demonstrate that by fine-tuning on a synthetic dataset generated by AdvPrompter, LLMs can be made more robust against jailbreaking attacks while maintaining performance, i.e. high MMLU scores.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
adversarial attacks prompt optimization red-teaming LLMs
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/55df61f364a7b2a5b048e31a26a1c4a977b7f9e7.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission