Universal Adversarial Triggers Are Not Universal
Nicholas Meade Arkil Patel Siva Reddy


[
    
Damien Tourret^1,*, 
Rouhollah Tavakoli^1,2, 
Adrian D. Boccardo^1, 
Ahmed K. Boukellal^1,3, 
Muzi Li^1
 and 
Jon Molina-Aldareguia^1,4

    April 29, 2024
=============================================================================================================================================




    Recent work has developed optimization procedures to find token sequences, called adversarial triggers, which can elicit unsafe responses from aligned language models. 
    These triggers are believed to be universally transferable, i.e., a trigger optimized on one model can jailbreak other models.
    In this paper, we concretely show that such adversarial triggers are not universal. 
    We extensively investigate trigger transfer amongst 13 open models and observe inconsistent transfer. 
    Our experiments further reveal a significant difference in robustness to adversarial triggers between models Aligned by Preference Optimization (APO) and models Aligned by Fine-Tuning (AFT). 
    We find that APO models are extremely hard to jailbreak even when the trigger is optimized directly on the model. 
    On the other hand, while AFT models may appear safe on the surface, exhibiting refusals to a range of unsafe instructions, we show that they are highly susceptible to adversarial triggers. 
    Lastly, we observe that most triggers optimized on AFT models also generalize to new unsafe instructions from five diverse domains, further emphasizing their vulnerability. 
    Overall, our work highlights the need for more comprehensive safety evaluations for aligned language models.[Code, data, and models for reproducing our study are available here: <https://github.com/McGill-NLP/AdversarialTriggers>.]

    

        Warning: This paper contains examples that may be offensive or upsetting.
    





§ INTRODUCTION

Large scale pre-training (; inter alia) alongside instruction-tuning on high-quality data <cit.> has given rise to Large Language Models (LLMs) which can follow instructions.
These LLMs are capable of providing useful responses to a range of requests, including those with malicious intent.
Because of this, significant work has investigated methods for aligning LLMs with human values <cit.>.
Using techniques such as Reinforcement Learning from Human Feedback (RLHF; ), LLMs can be optimized to refuse such harmful requests while maintaining helpfulness.
Despite these efforts, recent work <cit.> has found adversarial attacks capable of circumventing these safeguards (also known as jailbreaking).

One popular attack <cit.> uses a gradient-guided search to find token sequences, referred to as adversarial triggers, which can elicit harmful responses when appended to user inputs. 
This attack is particularly interesting for three reasons.
First, it can be used with any Transformer-based <cit.> model irrespective of size, architecture, and safety optimizations.
Second, triggers obtained through this optimization are often unnatural (i.e., nonsensical) making them difficult to identify via human red-teaming.
And finally, results from <cit.> have led to the belief that these triggers transfer universally across models.
More concretely, this means that triggers optimized on one model can be used to jailbreak a different model.
Implications of such universal transfer would be substantial.
For instance, this would suggest a relatively small amount of compute can be used to create highly versatile attacks for both open and proprietary models.

In this work, we investigate adversarial trigger transfer using 13 open models.
We provide strong empirical evidence that triggers do not transfer consistently across models.
Concretely, we first optimize triggers using the setup of <cit.>.
We demonstrate that when transferred, these triggers fail to jailbreak many of the models in our study.
In particular, we observe that open models Aligned by Preference Optimization (APO), using approaches such as RLHF, are extremely robust to trigger transfer.
For instance, we find that even within the same model family, triggers do not reliably transfer (e.g., Llama2-7B-Chat triggers fail to transfer to Llama2-13B-Chat and vice versa).
Additionally, we observe that even optimizing triggers directly on such rigorously aligned models fails to find triggers that can successfully transfer to other models. 

Through our investigation, we notice that models Aligned by Fine-Tuning (AFT; ) are particularly susceptible to adversarial triggers. 
While these models may appear safe, generating refusals to a range of unsafe instructions, they lack the robustness against adversarial triggers exhibited by APO models.
For a range of different AFT models, we show that triggers can be optimized faster (i.e., in fewer steps) and more frequently elicit harmful responses compared to APO models.
Additionally, we show that it is comparatively easier to transfer triggers to AFT models from various different source ensembles.

Finally, to further expose how susceptible AFT models are to adversarial triggers, we investigate generalization to new unsafe instructions. 
We experiment with five diverse safety benchmarks and find that a trigger that succeeds in jailbreaking these models over seen instructions, i.e., instructions the trigger was optimized on, generalizes surprisingly well to unseen instructions, including those from different domains.





§ RELATED WORK



  
Adversarial Attacks on LLMs.
<cit.> originally proposed a gradient-guided procedure for finding token sequences which elicit desired predictions from LLMs.
<cit.> adapted the procedure of <cit.> to find adversarial triggers for aligned LLMs.
Following <cit.>, many additional attacks have been investigated.
Proposed attacks have used genetic algorithms <cit.>, prompting <cit.>, low-resource languages and ciphers <cit.> in-context learning <cit.> and exploitation of generation hyperparameters <cit.>.
One particularly interesting aspect of <cit.>'s method was that triggers optimized on one model could be used to jailbreak another model.
Based on their results, <cit.> claim about the reliability of their approach in creating universally transferable triggers across LLMs.
In this work, we show that their approach does not consistently produce triggers transferable across a range of LLMs.



  
Defending LLMs Against Adversarial Attacks.
Given the development of adversarial attacks for LLMs, a number of works have investigated attack defenses.
Since many attacks produce unnatural inputs, several works have proposed using input pre-processing or filtering for defense <cit.>.
<cit.> found that many attacks are sensitive to character-level perturbations and proposed an input perturbation-based defense.
Other work has proposed using in-context learning <cit.>, prompting <cit.>, and discrete optimization <cit.> for defense.
In our work, we do not investigate how attack defenses impact trigger transferability.



  
Alignment and Supervised Fine-Tuning.
Recent work has investigated whether alignment can be obtained, removed, or circumvented using supervised fine-tuning.
<cit.> fine-tuned a 65B model on 1000 high-quality examples and found it performed competitively with GPT-4 in human evaluation.
With more focus on safety, other work <cit.> has argued that model harmfulness can be reduced via supervised fine-tuning on safety data.
Conversely, other work has shown a small amount of fine-tuning can be used to increase response harmfulness <cit.> or insert exploitable backdoors into a model <cit.>.
In this work, we argue that while fine-tuning enables models to learn to refuse unsafe instructions, it does little to improve a model's adversarial robustness.



§ EXPERIMENTAL SETUP



 §.§ Trigger Optimization



  
Method.
We use Greedy Coordinate Gradient (GCG; ) for attacking aligned LLMs.
GCG is a white-box attack which uses the gradient to find token sequences (i.e., adversarial triggers) to jailbreak LLMs.
GCG initializes the trigger as a fixed length sequence of tokens (e.g., ) and then optimizes it using a dataset of harmful instructions (e.g., ).
Concretely, the trigger is iteratively updated to minimize the cross-entropy loss of affirmative responses (e.g., ) to these harmful instructions.
Since this optimization is over discrete token sequences, a greedy coordinate gradient-based search is used. 
At each step, a set of new candidate triggers are formed by using the gradient to perform token substitutions on the current trigger.
The candidate trigger obtaining the lowest cross-entropy loss on the harmful instructions is then greedily selected.



  
Dataset. 
We use examples from  <cit.> for trigger optimization and evaluation.
Each  example consists of an unsafe instruction and an affirmative response.
As previous work <cit.> has raised concerns about instruction diversity within ,[For instance, 25 of the original instructions include the term bomb.] we use the 50 example subset from <cit.> for our experiments. 
We randomly select 25 examples from this subset for optimizing the triggers in all our experiments.



  
Trigger Optimization and Generation Parameters.
Following <cit.>, we optimize triggers with 20 tokens using a top-K of 256 and a batch size of 512.
We run all trigger optimizations for a maximum of 24 hours with early stopping using a loss threshold.
During evaluation, we sample responses using greedy decoding with a maximum length of 64 tokens.



 §.§ Evaluation Metrics

Given a collection of unsafe instructions and an adversarial trigger, we need to evaluate whether an LLM's responses are harmful.
Following previous work <cit.>, we define the Attack Success Rate (ASR) as the proportion of instructions a model responds to harmfully. 
Throughout our work, we report the ΔASR, which we define as the difference in ASRs obtained with and without appending the trigger to the input (clamping negative values to 0). 
ΔASR gives us a single value to faithfully represent the impact of a trigger in jailbreaking the model.
Solely reporting ASRs does not indicate if an attack is effective.

For detecting whether triggers jailbreak models, we use Llama-Guard <cit.> as opposed to the brittle string-based metric used in previous work <cit.>.
<cit.> demonstrated Llama-Guard roughly matches the performance of proprietary systems for detecting harmful content, such as PerspectiveAPI, while being openly available <cit.>.
For a given instruction and response, we use Llama-Guard to label the response as harmful or harmless.



 §.§ Models

We use 13 open models in our work: Gemma (Instruct-2B and Instruct-7B; ), Guanaco (7B and 13B; ), Llama2 (7B-Chat and 13B-Chat; ), MPT-7B-Chat <cit.>, OpenChat-3.5-7B <cit.>, Starling-7B (α and β; ), Vicuna (7B and 13B; ), and Koala-7B <cit.>.
Of these 13 models, only the Gemma, Llama2, and Starling models have been trained for safety using RLHF or DPO. 
We will refer to these as Aligned by Preference Optimization (APO) models. We report results for proprietary models GPT-3.5-Turbo <cit.> and GPT-4 <cit.> in Figures <ref> and <ref> in <ref> for completeness as well as consistency with past work. However, we do not extensively discuss results over such proprietary models for two reasons.
First, it is possible that these models have been patched against such adversarial triggers without our knowledge. 
Second, we have no understanding of their deployment pipeline, i.e., the input and output filters used, system prompts, etc. 
These factors make it difficult to reliably derive conclusions about adversarial trigger transfer for these models.



§ ADVERSARIAL TRIGGERS DO NOT TRANSFER UNIVERSALLY


In this section, we investigate how well adversarial triggers transfer across models.
We begin by showing that the best performing ensembles from <cit.> transfer inconsistently across models (<ref>). 
Since we observe that the transfer is particularly poor for the open APO models, we then show that even when we optimize triggers over these APO models themselves, we fail to see reliable transfer to other models (<ref>).



 §.§ Examining the Universal Transfer Claim of Previous Work





  
Reproducing Previous Work. It is important to note that our results are not contradictory to the results provided by <cit.>. In <ref> and <ref>, we are able to replicate a subset of their results. However, we wish to investigate their claim of universal transfer by more extensive experimentation.



  
Setup.
We evaluate three model ensembles that <cit.> showed could produce triggers transferable to GPT-3.5, GPT-4, and other models: 1) Vicuna-7B; 2) Vicuna-7B/13B; and 3) Guanaco-7B/13B + Vicuna-7B/13B.[A comprehensive list of all ensembles used in this work and the short names used to denote them is provided in <ref> in the Appendix.]
We optimize three triggers for each ensemble.
For each optimization, we use the trigger from the step with the lowest loss.
We then generate responses to the  instructions (that the triggers were optimized on) with and without the optimized triggers and report the ΔASRs.



  
Differences with Previous Work. We note three differences between our setup and that of <cit.>.
First, we use a curated subset of  <cit.> to ensure example diversity for our experiments.
Second, we use Llama-Guard as opposed to the string-based metric used by <cit.> for evaluating whether triggers jailbreak models. 
And finally, we evaluate transfer to open models which have been optimized for safety using RLHF or DPO. For consistency with past work, we also provide results using the string-based jailbreak metric in <ref>.











  
Results.
In <ref>, we report ΔASRs for triggers optimized using the three ensembles (see <ref> in <ref> for results with additional models).
Overall, we find that triggers optimized on each ensemble transfer inconsistently across LLMs. 
For two of the three ensembles, we observe some transfer to MPT-7B-Chat. 
For the Vicuna-only ensembles, we see transfer to Guanaco-7B.
Although transfer to MPT-7B-Chat and Guanaco-7B is interesting, we note that neither of them is an APO model.
While we find evidence of trigger transfer sometimes failing even on models which are not APO, such as OpenChat-3.5-7B, the transfer results are particularly poor for the three families of APO models: Llama2, Gemma, and Starling.
Notably, we obtain a mean ΔASR of zero for Gemma-7B-Chat, Llama2-7B-Chat, and Starling-7B-β.

[24]R0.48
    
    < g r a p h i c s >

    ΔASRs for triggers optimized on APO models (source) transferred to different models. We report the mean ΔASR over three independently optimized triggers.
    




 §.§ Investigating Trigger Transfer Amongst APO Models




  
Setup.
Here, we investigate whether triggers that are optimized over APO models themselves can transfer consistently to new models.
Concretely, using our six APO models, we form six single model ensembles and three two model ensembles,[Since Gemma and Starling do not use Llama2's tokenizer, we can not combine them with any of the other APO models.] giving a total of nine ensembles.
We then optimize three triggers using each ensemble.
Importantly, for each ensemble, we evaluate against both the seen models (i.e., the models on which the trigger has been optimized) and unseen models. For all target models, we report the ΔASR.



  
Results.
We present results for the six individual models in <ref> (see <ref> in <ref> for results with the three two model ensembles).
We first note that even when applying triggers to the model they are optimized on, low ΔASRs are sometimes obtained.
For instance, on Llama2-13B-Chat, we obtain an average ΔASR of 4.0 across three triggers.
Across all nine ensembles, we again observe no consistent transfer to the other models.
For example, we find that no ensemble produces triggers transferable to Llama2-7B/13B-Chat (outside of optimizing directly on these models).
It is remarkable that triggers do not reliably transfer even within models families (e.g., Llama2-7B-Chat and Llama2-13B-Chat).



 §.§ Discussion


While triggers transfer to many non-APO models, such as MPT-7B-Chat and Guanaco-7B, exceptions such as OpenChat-3.5-7B exist. 
Moreover, it is clear that triggers do not transfer to APO models, such as Llama2 and Gemma. 
We find that optimizing triggers on APO models themselves also does not aid in transfer to other APO models. 
Surprisingly, these APO models are so robust that even triggers that are optimized on them fail at consistently eliciting harmful responses.

It is interesting that we do not observe trigger transfer even within model families (e.g., Llama2-7B-Chat triggers do not transfer to Llama2-13B-Chat and vice versa). 
Our results indicate a large amount of variability in trigger optimization, particularly when optimizing on multiple APO models.
For instance, two of the three triggers optimized on Llama2-7B/13B-Chat fail to elicit any harmful responses from Llama2-7B/13B-Chat on .
Altogether, our results provide concrete evidence refuting previous claims about reliable universal transfer across models for adversarial triggers obtained using GCG.

Experimenting with proprietary APO models, <cit.> showed that triggers transferred to GPT-3.5-Turbo, and we are also able to reproduce those results (see <ref> in <ref>). 
On the other hand, we observe little transfer to GPT-4.

We wish to emphasize that we do not claim non-existence of triggers that transfer to APO models. 
However, our extensive empirical results highlight the difficulty in reliably[By unreliability, we mean that we are unable to find good triggers in the majority of GCG runs.] finding triggers using GCG that transfer consistently[By inconsistency of transfer, we mean that a trigger optimized over a single source ensemble does not transfer to a majority of other models.] across models.
We advocate for future work to clearly present the effectiveness of an attack across an explicit number of runs evaluated across an extensive set of models.





§ ALIGNMENT BY FINE-TUNING IS NOT ROBUST AGAINST ADVERSARIAL TRIGGERS

In the previous section, we observed that Alignment by Preference Optimization, i.e., APO models seem robust against adversarial triggers. 
Recently, many methods have been proposed to align models for safety with just supervised fine-tuning. 
In this section, we investigate whether such Alignment by Fine-Tuning (AFT) models are also robust against adversarial triggers.
We begin by describing AFT methods proposed in existing works (<ref>).
We then investigate their performance on standard safety benchmarks (<ref>).
Finally, we investigate the robustness of these methods against adversarial triggers (<ref>).



 §.§ Alignment by Fine-Tuning (AFT) Methods


We investigate three methods for aligning LLMs using supervised fine-tuning.[We also investigated LIMA <cit.>, but found fine-tuning on their data did little to improve model safety. See <ref> for results.]
We detail these methods below.
We consider the APO models Llama2-7B-Chat and Starling-7B-β as baselines to contextualize the robustness of these AFT methods.



  
Model Imitation.
Recent work found that some capabilities of aligned models, including refusing harmful queries, can be inherited via fine-tuning on their responses <cit.>. 
We experiment with two models distilled from larger aligned models: Koala-7B <cit.> and Vicuna-7B <cit.>. 
Additionally, we create our own imitation model by fine-tuning Llama2-7B-Base on single-turn conversations from ShareGPT augmented with Llama2-7B-Chat responses. 
We wished to investigate whether fine-tuning a model on responses from Llama2-7B-Chat makes it inherit its robustness against adversarial triggers.



  
Saferpaca.
<cit.> found that fine-tuning LLaMA-7B on Saferpaca, which is an augmented version of the Alpaca dataset <cit.> consisting of a few hundred safety demonstrations, decreased response harmfulness. 
We investigate whether fine-tuning Llama2-7B-Base on Saferpaca can make it robust against adversarial triggers.



  
SelfAlign.
<cit.> propose a procedure for obtaining synthetic alignment data from base LLMs.
Similar to <cit.>, SelfAlign uses a small number of human written guidelines to generate demonstrations using prompting and in-context learning.
We use data obtained using SelfAlign to fine-tune Llama2-7B-Base and call the model SelfAlign-7B.





 §.§ Do AFT Models Produce Safe Responses?


We begin by investigating the performance of AFT models on four diverse safety benchmarks apart from . 
We list these benchmarks below:


    
    
  *  <cit.>. 100 unsafe instructions across ten malicious intents (e.g., theft, tax fraud, and hacking).
    
  *  <cit.>. 40 unsafe instructions related to controversial topics (e.g., vaccination and immigration).
    
  *  <cit.>. 100 unsafe instructions drawn from the Anthropic HH-RLHF dataset <cit.>.
    
  *  <cit.>. 178 unsafe instructions about hate-speech generation.


For each benchmark, we generate responses using each of the models.
We score the resulting responses for harmfulness using Llama-Guard and report the percentage of harmful responses for each model and benchmark.

We present our safety results in <ref>.
We observe that most of the AFT models correctly refuse unsafe instructions across different domains. 
DistilLlama2-7B, Saferpaca-7B, and SelfAlign-7B appear to be almost as safe as the APO models. 
In fact, we observe that DistilLlama2-7B obtains a lower average score than Starling-7B-β across the benchmarks.
While Vicuna-7B is significantly less safe, it is still able to refrain from generating unsafe responses for most of the instructions in the benchmarks.
We provide additional safety results in <ref> in <ref>.



 §.§ Are AFT Models Robust to Adversarial Triggers?


We now investigate how robust AFT models are to triggers optimized using GCG.



  
AFT models converge faster than APO models. 
For all models, we compute the mean ΔASR at each step of optimization on  over three independent triggers.
The results are shown in <ref>.
We find that the AFT models DistilLlama2-7B, Saferpaca-7B, SelfAlign-7B,  and Vicuna-7B obtain high ΔASRs within the first few optimization steps. 
In contrast, the APO models Llama2-7B-Chat and Starling-7B-β are much harder and slower to optimize.
Overall, we note that the AFT models jailbreak faster than the APO models.



  
AFT models are more susceptible to adversarial triggers. 
In <ref>, we report the mean ΔASR for triggers optimized directly on APO and AFT models. 
We find that the two APO models have a low ΔASR and high variance.
In contrast, we observe that the AFT models all obtain higher mean ΔASRs than the APO models, highlighting their susceptibility to adversarial triggers.



  
Triggers transfer more to AFT models.
We experiment with transferring triggers from 21 diverse source ensembles (see <ref> in Appendix for the full list) to APO and AFT models. 
For each model, we plot the ΔASR of the ensemble from which the best transfer is observed (excluding optimizing on the model itself) in <ref>. 
As noted earlier in <ref>, we observe almost no transfer for both APO models Llama2-7B-Chat and Starling-7B-β. 
However, we observe a high amount of trigger transfer to various AFT models, such as Koala-7B and Saferpaca-7B. 
The overall trend we observed is that triggers transfer more onto AFT models compared to APO models.



 §.§ Discussion

We investigated the adversarial robustness of Alignment by Fine-Tuning, i.e., AFT models.
We found that while superficially these models may appear safe by refusing to answer unsafe instructions from multiple safety benchmarks, they lack the adversarial robustness of APO models.
Concretely, we empirically showed that AFT models are jailbroken in fewer optimization steps compared to APO models when using GCG. 
We further found that when directly optimizing triggers and applying them to the same model, triggers optimized on AFT models more successfully elicit harmful responses compared to triggers optimized on APO models.
Finally, for some AFT models, we also found triggers optimized on other models transferred more frequently to them compared to APO models. 
It is important to note that an ΔASR of ∼20% for a transfer experiment is quite significant considering that most triggers when optimized and tested on the same model yield an ΔASR of ∼50% or less (as seen from <ref> in the <ref>).



§ TRIGGERS OPTIMIZED ON MODELS ALIGNED BY FINE-TUNING GENERALIZE TO UNSEEN INSTRUCTIONS


In the previous section, we showed that AFT models are extremely susceptible to adversarial triggers optimized using GCG.
We now investigate how well these triggers generalize to new unsafe instructions.
Concretely, we evaluate how often triggers elicit harmful responses when applied to instructions they were not optimized on.





  
Setup.
We optimize triggers on the 25 instructions in , hereafter referred to as , for 13 different models.
We then evaluate how well these triggers generalize to the 25 unseen examples in  and four diverse safety benchmarks introduced in <ref>.
Importantly, these benchmarks include instructions from domains substantially different from  (e.g., hate-speech generation in ). 
For all models, we report the ΔASR.



  
Results.
We present the ΔASRs across the five safety benchmarks for four AFT models in <ref>. 
The results for other models are provided in <ref> in <ref>. 
We first note that triggers generalize well to .
Interestingly, we see that all 4 models have a higher ΔASR on  compared to .
We also note that triggers generalize well and almost to a similar extent for , , and . Lastly, while we observe the generalization to  to be significantly lower than the other datasets, it is still non-trivial.
Overall, we observe that when triggers successfully elicit harmful responses on , they also work across a variety of other instructions for AFT models.



  
Discussion.
Our results imply that triggers which successfully jailbreak AFT models on instructions that they were optimized on also generalize surprisingly well to new unsafe instructions. Surprisingly, we find this generalization ability to be domain-agnostic.
For example, we found that triggers generalize well to instructions related to discussion of controversial topics () and hate-speech generation (), despite not being optimized on instructions in these domains.
One limitation of these benchmarks is that all of the instructions are of similar length.
It is therefore possible that triggers transfer less when added to longer instructions. 
We leave it to future work to investigate generalization across unsafe instructions of different lengths.



§ CONCLUSION

We investigated whether adversarial triggers obtained using GCG transfer across models.
We found, particularly amongst models aligned by preference optimization, triggers do not reliably transfer. 
We then demonstrated that models that have been aligned for safety using supervised fine-tuning are extremely susceptible to adversarial triggers.
Our results show that while models can easily learn answer refusal, adversarial robustness is not as easily obtained.
We caution the community on making claims about how model safety can be improved using minimal interventions (i.e., fine-tuning on a small number of safety demonstrations). 
In line with other recent work <cit.>, we advocate for the use of automatic red-teaming for evaluating model robustness, alongside other traditional safety evaluations.




§ LIMITATIONS

We discuss three main limitations of our work.



  
1) We only investigate a single method for adversarially attacking LLMs. 
In our work, we only investigate using GCG for attacking LLMs however, many other attacks have been proposed (; inter alia).
We focus our study on GCG due to its popularity and flexibility.
In addition, GCG is also interesting as it often finds unnatural (i.e., nonsensical) triggers which are difficult to identify via human red-teaming.
While we demonstrated that AFT models are particularly susceptible to GCG attacks, we hypothesize they may be vulnerable to other attacks.
Future work can investigate the robustness of AFT models using additional attacks.



  
2) We do not investigate the relevance or helpfulness of generated responses.
We only evaluate the harmfulness of generated responses and not whether they provide information relevant to the instruction.
For instance, while an adversarial trigger may elicit harmful responses, they may not be helpful or related.



  
3) Not all triggers converge within the 24 hour optimization process.
In our work, due to resource constraints, we optimized all triggers for 24 hours with early stopping.
We observed that when optimizing triggers on APO models, many ensembles failed to elicit harmful responses for any  examples after 24 hours (e.g., two of the three triggers optimized on Llama2-13B-Chat).
In any case, we argue that the amount of compute used for trigger optimization in this work exceeds that available to an average attacker.



§ ACKNOWLEDGEMENTS

We thank the members of SR's research group for helpful comments throughout the duration of this project.
NM is supported by a Canada Graduate Scholarship (CGS-D) funded by the Natural Sciences and Engineering Research Council (NSERC).
SR is supported by the Canada CIFAR AI Chairs program and the NSERC Discovery Grant program.
AP is supported by a Canada Graduate Scholarship (CGS-M) funded by NSERC.


colm2024_conference





    Supplementary Information



appendixsection[0em]
    

    2.3em
    
    *[1pc].

0




§ ADDITIONAL EXPERIMENTAL DETAILS

In this section, we provide additional details and examples of our experimental setup.



 §.§ Models


We use PyTorch <cit.> and the Hugging Face Transformers <cit.> implementations of all models studied in our work.
These models are available for download from the Hugging Face Hub.
We list the identifier for each model on the Hugging Face Hub in <ref>.





 §.§ Prompts


In <ref>, we list the system messages for the models used in our work.
In <ref>, we list the chat templates for the models used in our work.







 §.§ Safety Benchmarks


In <ref>, we provide example instructions from the five safety benchmarks used in our work.



 §.§ Fine-Tuning Hyperparameters


For training DistilLlama2-7B, Lima-7B, SelfAlign-7B, and Saferpaca-7B we use the following hyperparameters:

    
    
  * Learning rate: 2e-5
    
  * Batch size: 64
    
  * Scheduler: Cosine
    
  * Precision: Brain  (i.e., )
    
  * Optimizer: AdamW <cit.>

We train Lima-7B for three epochs following <cit.>, Saferpaca-7B for three epochs following <cit.>, SelfAlign-7B for one epoch, and DistilLlama2-7B for one epoch.
We train all four of these models using the Llama2 chat template (see <ref>) and Llama2 system message (see <ref>).

<cit.> created an augmented version of the  dataset by adding safety-related instructions.
Concretely, they randomly sampled 20000 instructions from the  dataset and added either 100, 300, 500, 1000, 1500, or 2000 safety instructions.[The  dataset is available for download here: <https://github.com/vinid/safety-tuned-llamas>.]
For our experiments, we use the version with 2000 safety instructions.

We use DeepSpeed ZeRO Stage 2 <cit.> optimizations for fine-tuning all of our models.





 §.§ Ensembles


We list all ensembles used for trigger optimization in <ref>.





§ ADDITIONAL RESULTS





 §.§ Investigating Universal Transfer of Adversarial Triggers



In <ref>, we report results for all 13 models for our transfer experiments detailed in <ref>. 
We observe that triggers do not transfer consistently across all models. 
We again note that trigger transfer is particularly absent for the APO models.



 §.§ Investigating Trigger Transfer Amongst APO Models



<ref> shows results for the six safety trained models and the three ensembles for the transfer experiments detailed in <ref>. 
We find that triggers trained on ensembles of APO models do not transfer to other models that are not present in the ensemble.



 §.§ Generalization to Unseen Examples



<ref> shows how well each model generalizes to unseen instructions from the 5 safety benchmarks under the experimental setup outlined in <ref>. 
For most models, when triggers successfully jailbreak the model on the seen examples, we also observe generalization to new unsafe instructions.



 §.§ Detailed Results for All Models On Each Dataset



We provide complete results for , , , , , and  in Figures <ref>, <ref>, <ref>, <ref>, <ref>, and <ref>, respectively.





















 §.§ Detailed Safety Results



<ref> shows safety results for all models in our study.





 §.§ String-based Metric Results



We present results with the string-based metric used in <cit.>.
This metric checks if generated responses contain any strings indicative of an answer refusal.
For completeness, we list these strings below (delimited by ).



  
Jailbreak Strings.


In <ref>, we present complete results for  using the string-based metric.





§ EXAMPLE RESPONSES


In <ref> and <ref> we provide example responses from different models.







§ VERIFYING IMPLEMENTATION CORRECTNESS





 §.§ Reproducing Table 1 From <cit.>

We replicate a subset of the results shown in Table 1 from <cit.>.
For Vicuna-7B and Llama-2-7B-Chat, we randomly select 25  examples and optimize three triggers on each model.
As in <cit.>, we optimize 20 token triggers using a top-K of 256 and a batch size of 512 for 500 steps.
We then apply these triggers to the 25 seen (i.e., train) examples and 100 randomly sampled unseen (i.e., test)  examples.
During evaluation, we generate responses with a maximum of 64 tokens using greedy decoding.
We also use the string-based metric from <cit.> as opposed to Llama-Guard.

We present our results in <ref>.
For Vicuna-7B, we find we are closely able to match the results from <cit.>.
For Llama2-7B-Chat, we observe more substantial differences however, we also note the large variance across the three triggers.





 §.§ Reproducing Figure 3 From <cit.>

We replicate a subset of the results shown in Figure 3 from <cit.>.
We follow their setup and optimize three triggers on Guanaco-7B/13B + Vicuna-7B/13B on 25 random examples from .
Importantly, we do not use the 50 example  subset from <cit.> here.
As in <cit.>, we optimize 20 token triggers using a top-K of 256 and a batch size of 512 for 500 steps.
We evaluate the trigger corresponding to the optimization step with the lowest loss against 388 held-out instructions.[The 388 instructions are available here: <https://github.com/llm-attacks/llm-attacks>.]
During evaluation, we generate responses with a maximum of 64 tokens using greedy decoding.
Here, we use the string-based metric proposed in <cit.> for computing ASRs as opposed to Llama-Guard.

We present our results in <ref>.
We first note that without using our optimized triggers, we obtain highly similar results to <cit.> for three of the four models (<3 point absolute difference).
For Guanaco-7B, we see a more substantial difference in ASRs (9.6 point absolute difference).
For the GCG results, we obtain similar results to <cit.> for all models.
In some cases, we obtain a higher ASR than <cit.> (Guanaco-7B and MPT-7B-Chat).
We attribute differences in results here to the random selection of 25 examples for trigger optimization.



