Learning Diverse Attacks on Large Language Models for Robust Red-Teaming and Safety Tuning
red-teaming;LLM;diversity
Yes
No ethics review needed.
- Where does the diversity come from (besides the ablation on $\beta$ values)? $\gamma$? sampling temperature $\tau$? replay buffer / "off-policy-ness"? The entropy/diversity of $p_{ref}$? The mix between replay buffer and online sampling in each iteration? Given that this is a main focus of the paper I would have been excited about more ablations / investigations in this direction.
- The results for REINFORCE, ICL, SFT seem mostly consistent across the different red-teamed models. PPO+Novelty results are less consistent and so is GFlowNet - is this due to hyper-parameter sensitivity? Or variance between runs? GFlowNet+MLE looks more consistently strong, so the core results of the paper are not impacted by this.
- How strong was the toxicity classifier, how often did the method "hack" shortcomings of the classifier rather than finding actual toxic outputs? This was hinted at in the discussion of setting $\beta$ too low (high weight of R1), but wondering if there are some more concrete results on this?
- What are sensible values for $r1, r2$?
- Clearly strong results and exciting possibilities for improving automated red-teaming and finding diverse attacks with strong motivations as one of the core challenges.
- The paper is well written and easy to follow. Experiments and ablations are documented well and replication of the results seems straightforward.
- I appreciate comparing to multiple RL-based baselines, as well as red-teaming multiple models. This gives confidence that the results will hold up in a wider range of settings.
This paper applies GFlowNet to the problem of automated red-teaming, achieving a favorable balance between attack success rate and attack diversity. A model is trained to sample attacks with probability proportionally to their reward, followed by a cheaper second stage of training a smoother version of the model on rollouts of the trained model. The method achieves high levels of diversity combined with strong attack success rates, and, when including the smoothing step, consistently strong results across different hyper-parameters and different target models.
- Given the focus on learning diverse attacks, there could have been more in-depth ablations and experiments to investigate which parts of the method most strongly influence diversity (also see Questions section).
- It would be nice to also plot error bars, at least on a version in the appendix. It was not clear to me if the various plots are also based on multiple random seeds (as table 2 is).
Yes
No ethics review needed.
- What are the models you evaluating?
- Have you considered evaluating the method against more methods?
- It’s good to have a method that produces diverse and effective red-teaming attacks
- Prompt injection is tricky and it’d be good to have a method to red-team for it.
The paper introduces a two step automatic red-teaming process to produce effective and diverse attacks. In particular, this is used for automated red-teaming of jailbreaks and injection prompts.
The first step consists of generating a diverse set of instructions and criteria both from data and from using a rule-based reward.
In the second step an LLM red-teamer is trained using multi-step reinforcement learning on the instructions and criteria collected at step 1. The reward includes attack success, similarity and a length penalty.
The red-teaming method is tested one state-of-the-art model and one small model (that is not mentioned in the text)
- Some typos throughout the text
- The section about AutoRBR should include more technical details. It’s not clear what is the role of the rule-based reward for the first step of the method
- The baselines should include other red-teaming methods, not just mainly variations of the proposed method
- The method is evaluated on two models that are not mentioned because of concerns about double blind reviews, but it’s not clear why
- Plots in figure 4 and 5 are a bit small and are not clear. Which model is scored in these plots? What do the “crosses” represent?
Yes
No ethics concerns
No ethics review needed.
I'm curious to know whether the method scales with stronger base model for the attacker model.
1. The proposed method could sample diverse prompts and appear more efficient than the baseline methods. 
2. The main claims are well-supported by the experiments.
3. The writing is clear and the paper is easy to follow.
The paper proposes a two-stage approach combining GFlowNet fine-tuning with MLE smoothing to automatically generate diverse and effective adversarial prompts for testing language model safety, demonstrating good performance over a selection of baseline red teaming methods, balancing attack success rate and prompt diversity while enabling reasonable transfer to new target models.
1. There's a lack of evaluation against stronger attacks and defenses. The paper did not consider many of the non-optimization-based gray /black box attacks, which might perform better at lower computation budget. The paper also did not consider robustified versions of the models such as models that underwent circuit breaking or (latent) adversarial training.
2. It's unclear how to adapt this for targeted attack against specific harmful requests and how well that would work.
Yes
No ethics review needed.
On linea 305-309, it is not clear to me what dataset is being used. Can you be more explicit if you are adopting the datasets for all methods listed on these lines? Or do you use a different dataset for your fine tuning?
On a standardized dataset and with standardized evaluations, and within the class of reinforcement learning methods that train an attacker model to produce a single adversarial prompt, this paper proposes a method that does achieve better diversity and stronger attack success rate. It is important for the community to be aware that this reinforcement learning method can produce attacker models with stronger adversarial “power” and diversity. The discussion of the adjustments needed to the GFlowNet method is also important.
The paper studies methods of training a language model to generate adversarial prompts that, when provided to more standard language models (such as GPT, Gemma, or Llama), produce responses deemed to be violating by a safety classifier. The main contribution of the paper is to apply the GFlowNet reinforcement learning method for this fine tuning of the attacker model. The paper produces attacker models that generate prompts of better diversity and higher attack success rate than other methods that take the same approach to red teaming.
I would say that this paper suffers from weaknesses that the whole field of automated jailbreaking is currently subject to. I do not consider this to be a reason for rejection but I also think it is important to record this critique for future iterations of automated red teaming work.
To be more precise, it is not clear to me that the majority of methods in this field – this work included – discover ways to elicit meaningfully harmful responses from models. It appears to me that the responses provided in the Appendix are all generally informational. To put this another way, this paper – along with most works who take AdvBench off the shelf as the goal set and slap some classifier on the responses – sidestep a robust definition of what is and is not considered harmful. This is not necessarily a problem of the work itself and rather a shortcoming of the yardsticks (harmful prompt datasets and classifiers) that have been widely adopted. 

However, what is important for the authors of this work, is that the end result of this leads to methods that generate adversarial prompts that likely exploit the model’s willingness to be generically helpful. In particular, the prompts listed in tables B.5 and B.6 are themselves very generic. For example, it is not clear why or how “records in the financial database” are being manipulated. Is this necessarily some harmful or unauthorized manipulation? The model response itself assumes that you would be doing this with a properly authorized user. This is likely because the prompt leaves enough vagueness in it to be interpreted as asking for something that is perfectly legal and acceptable (help with interacting with a database). 

Thus, I believe methods in this space in the future should also consider specificity of the harmful request as another axis of desirable properties, in addition to diversity and attack success rate judged by a classifier. So instead of ending up with prompts that dance around a vaguely specified line, methods should a) be explicit about the line and b) make sure that their attacks clearly cross it. It would be interesting if GFlowNet adversarial methods can help elicit specific and truly harmful responses from language models.
Yes
No ethics review needed.
Could you please elaborate on how you expect your method to fare against current SOTA guardrails and the challenges you see in overcoming those with attacks using your method?
The method is well motivated, presented in an easy to understand way, and backed by a rigorous experimental setup. Especially the performance in the transfer setting is impressive, as most other methods completely fail at this task. This paper advances the state of the art in a significant way and addresses a crucial problem in AI security.
This paper introduces GFlowNet fine-tuning plus a follow-up smoothing phase for attack generation in LLM red teaming. Their approach overcomes the typical problems of lacking diversity, effectiveness and model collapse that arise in RL-based automated red teaming. The authors not only show the effectiveness of their method for red teaming (toxicity focus) but also that their method can be used to generate highly effective fine-tuning data for safety tuning.
Experiments on the performance against LLM guardrails would have been of interest, as real-life deployments will always include input and output guardrail models. Given the strong performance of the method in transfer settings, this could also prove to be another potential strongpoint of this method.
Generating diverse, effective, and transferable prompts for red-teaming black-box large language models.
@inproceedings{
anonymous2024learning,
title={Learning Diverse Attacks on Large Language Models for Robust Red-Teaming and Safety Tuning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=1mXufFuv95},
note={under review}
}
Red-teaming, or identifying prompts that elicit harmful responses, is a critical step in ensuring the safe and responsible deployment of large language models (LLMs). Developing effective protection against many modes of attack prompts requires discovering diverse attacks. Automated red-teaming typically uses reinforcement learning to fine-tune an attacker language model to generate prompts that elicit undesirable responses from a target LLM, as measured, for example, by an auxiliary toxicity classifier. We show that even with explicit regularization to favor novelty and diversity, existing approaches suffer from mode collapse or fail to generate effective attacks. As a flexible and probabilistically principled alternative, we propose to use GFlowNet fine-tuning, followed by a secondary smoothing phase, to train the attacker model to generate *diverse* and *effective* attack prompts. We find that the attacks generated by our method are effective against a wide range of target LLMs, both with and without safety tuning, and transfer well between target LLMs. Finally, we demonstrate that models safety-tuned using a dataset of red-teaming prompts generated by our method are robust to attacks from other RL-based red-teaming approaches.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
red-teaming LLM diversity
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/a885e83ff10f0c57dbc3a397450915954e78dcb9.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/eaaf23af6fac0703db7d5de23ca74e0e61b59798.zip
Learning Diverse Attacks on Large Language Models for Robust Red-Teaming and Safety Tuning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission