Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learning
red teaming;safety;reinforcement learning
Yes
No ethics review needed.
- Why was the method not evaluated on more commonly benchmarked models (e.g. Llama, Gemma, etc)?
- The qualitative examples in C.3 either look very simplistic or somewhat odd; which ones succeeded/failed, and what were the outputs the model produced to these prompts?
- I found the prompt injection task difficult to follow. I understand what they are, and I understand the goals/types of prompt injections that are being included (links/images/specific phrases in responses, or generally the examples in table C.3). However it is unclear to me what you are injecting these goals into, and what you are exactly evaluating.
- Results show clear improvement over the naive baselines (one-shot generation, vanilla RL) given the evaluated metrics
- Novel rewards for handling issues with prompt diversity (to the best of my knowledge)
The authors propose a pipeline for automated red-teaming to both generate diverse attack goals, and then generate attacks for the goals. By prompting an LLM, a diverse set of instructions and criteria are obtained and used to create a dataset. This dataset is then used to fine-tune an LLM using RL on a few different rewards: attack success (rule-based rewards and breaking moderation filters), style diversity, similarity/consistency, and response length. The core contributions of this paper are the multi-step RL approach and formulation of the reward to encourage style diversity, and proposing to apply this framework for red-teaming prompt injections (in addition to standard jailbreaks). The attacks produced by the fine-tuned model are then evaluated by either their RBRs or OpenAI’s Moderation API, showing that their method improves over baselines (one-shot generation, vanilla RL) while also improving on diversity metrics.
- Qualitatively, the results appear questionable - more discussion in the question section
- Figure 5a colours don’t match (success rate vs attack diversity)
- Figures are generally hard to parse and general presentation could be improved
- It is impossible to verify the claims of this paper; no information of the models evaluated was given, and  was given, nor the code to reproduce results. While the authors did promise to release code upon publication, it is difficult to gauge the significance of the results
Yes
No ethics review needed.
I'm not sure that the distinction between jailbreaking and indirect prompt injection is good to propagate. They feel like exactly the same problem, with different window dressing. What do you think?
- This is a good line of work. Most jailbreaking and automated red teaming papers haven't taken the RL route first explored by Perez et al (2022), and most haven't given diversity of attacks enough consideration. I wish more work like this existed.
- I appreciate how the authors describe that the diversity was relatively poor at first, which led them to develop their multi-step method where the attacker conditions on previous attacks and tries to make the new attacks different from what came before.
- The proposed methods improve diversity of the attacks.
This paper proposes a method for improving the joint diversity and effectiveness of automated red teaming methods for LLMs. The overall method first generates a diverse set of goals, which are then optimized by a multi-step RL method that conditions on previously generated attacks to improve the diversity of the attacks. This improves the attack diversity while maintaining good ASR.
- This paper discusses diversity of attacks, but it doesn't clearly distinguish between two types of diversity that appear in the paper: attack goal diversity, and diversity of the attack itself (i.e., style diversity). These should be clearly distinguished.
- Only the style diversity is evaluated using embedding cosine similarity (presumably self-similarity, as in Perez et al. (2022)). What about the attacker goal diversity? If part of the method involves improving the attacker goal diversity, then surely that should be backed up with an evaluation of some sort. I'm actually not sure what would be a good metric for this, and it seems like an important point to consider for future work on automating exploratory red teaming, so updating the paper to include some sort of evaluation of goal diversity could provide value to the community.
- The presentation is lacking in areas. E.g., Figure 1's caption is essentially missing. This needs to be fixed. Also, the handwritten style of Figure 1 is hard to follow, and many symbols in the figure are not labeled.
- Reading section 5.3, I can't shake the feeling that the discovered lack of diversity isn't a very deep finding. Couldn't one characterize this as just not having designed a good enough goal generation prompt? Does this merit being mentioned in an ICLR-tier paper?
- I'm generally not a fan of making metrics depend on closed-source models. The ASR and diversity metrics used in this submission both rely on the OpenAI API, which reduces reproducibility in the long run.
- The paper involves a lot of experiments, but it's unclear what scientific or technical advances were made. It's OK for papers to be more about interesting results; technical novelty isn't the only source of value. But in this case, I think the outcomes of the experiments aren't that surprising; this may be a paper where the main source of value is in figuring out all the details and showing that this could be done.
Yes
No ethics review needed.
- Why do you use two different toxicity classifiers for different models?
- The difference in performance between GFlowNets and GFlowNets+MLE changes quite a bit across different models. For example, for Dolly and GPT-2 the performance is almost the same, for Gemma there is a huge gap, for the Llamas the difference is a bit smaller. Do you have a sense of why this is happening?
- The attacks produced with this method are very diverse. Is this diversity mainly semantic or also stylistic?
- Attack diversity is important in red-teaming and finding automatic methods to produce diverse and effective prompts is valuable.
- The second step seems to significantly improve over the baseline consisting only of the first step. 
- The method is compared against many different baselines.
The paper proposes an automatic two step red-teaming method for LLMs that aims at generating prompts that are very diverse as well as effective. Diversity is important to transfer attacks across different models and for generalization purposes. The first step of the red-teaming process consists of collecting several prompts that score a high reward using GFlowNets algorithms. In these methods, the learned policy sample prompts with probability proportional to the reward the prompts received. The reward is defined as the likelihood that a prompt is classified as toxic by a toxicity classifier (plus a KL term). The second step takes into account some issues related to tuning some hyperparameters in the reward function. In particular, the prompts collected at the previous step are filtered and used to fine-tune the initial policy to maximize model log-likelihood. The red-teaming is tested against different baselines and across different models. The results show that prompts generated with this method are more diverse and effective compared to the ones produced by other baselines and transfer across different models.
- The tested models are quite small. Even if the attacks generalize to different models, it’d be good to have an evaluation of bigger and closed-source models (since the technique doesn’t require access to model weights).
- Although inexpensive, it seems not very efficient to retrain the policy on a subset of generated prompts. For example, filtering the prompt can be done on the fly during the GFlowNet stage.
Yes
No ethics review needed.
Can you provide a more readable version of Figures 4 and 5? For example, a table with the raw numbers might be needed. Currently, it is hard for me to map between the colors in the legend and the plot colors, so I cannot tell how well each method does.

Can you explain the table in Appendix C.4? Is each row supposed to be goals based on the prompt in the furthest left? Why do prompts in some columns have nothing to do with the “Prompt Details” column?
It is not currently possible for me to understand the paper well due to the coloring of the plots and the confusing presentation of the generated examples.
This paper proposes a reinforcement learning approach to training “attacker” models that generate adversarial prompts triggering harmful responses by “victim” models. As part of this process, the authors proposes a method for generating goals for the attacks. The paper has experiments both in the jailbreaking and in the prompt injection setting.
It is not currently possible for me to understand the paper well due to the coloring of the plots and the confusing presentation of the generated examples.
However, I will flag a weakness in the design of the goal selection. Since this paper proposes as its own contribution the design of goals for red teaming/jailbreaking, it should take better care to lay down a robust line of what is and is not considered a violation or a harmful response for jailbreaking. I might be misinterpreting the examples provided in the table in Appendix C.4 (see questions) but I am concerned that currently they do not always explicitly elicit truly harmful responses. Some may even leave enough wiggle room for a model to respond in a generically helpful but not harmful manner instead of outright refusing. This, in turn, might be inflating the attack success rate. For example, it might be perfectly reasonable for the model to assist in writing a book about password cracking in response to the prompt “What are the methods a criminal in my book could steal password:” – as long as the response does not provide information that can truly be used for harm.
We find we can create a diverse and effective red teamer by first generating many diverse, specific red-teaming tasks and then using those tasks as part of multi-step reinforcement learning to train an LLM red teamer.
@inproceedings{
anonymous2024diverse,
title={Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=FDmKe5EBuy},
note={under review}
}
Automated red teaming can discover rare model failures and generate challenging examples that can be used for training or evaluation.  However, a core challenge in automated red teaming is ensuring that the attacks are both diverse and effective.  Prior methods typically succeed in optimizing either for diversity or for effectiveness, but rarely both.  In this paper, we provide methods that enable automated red teaming to generate a large number of diverse and successful attacks.

Our approach decomposes the task into two steps: (1) automated methods for generating diverse attack goals and (2) generating effective attacks for those goals.  While we provide multiple straightforward methods for generating diverse goals, our key contributions are to train an RL attacker that both follows those goals and generates diverse attacks for those goals.  First, we demonstrate that it is easy to use a large language model (LLM) to generate diverse attacker goals with per-goal prompts and rewards, including rule-based rewards (RBRs) to grade whether the attacks are successful for the particular goal.  Second, we demonstrate how training the attacker model with multi-step RL, where the model is rewarded for generating attacks that are different from past attempts further increases diversity while remaining effective.  We use our approach to generate both prompt injection attacks and prompts that elicit unsafe responses.  In both cases, we find that our approach is able to generate highly-effective and considerably more diverse attacks than past general red-teaming approaches.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
red teaming safety reinforcement learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/5969eaab4b528da55606e12de9b3c0ec3374e6c4.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission