# embedding_metadata

key: chroma:document
string_value:
## Usage

```bash
# 1. run directly
python examples/scripts/ppo.py
# 2. run via `accelerate` (recommended), enabling more features (e.g., multiple GPUs, deepspeed)
accelerate config # will prompt you to define the training configuration
accelerate launch examples/scripts/ppo.py # launches training
# 3. get help text and documentation
python examples/scripts/ppo.py --help
# 4. configure logging with wandb and, say, mini_batch_size=1 and gradient_accumulation_steps=16
python examples/scripts/ppo.py --log_with wandb --mini_batch_size 1 --gradient_accumulation_steps 16
```

Note: if you don't want to log with `wandb` remove `log_with="wandb"` in the scripts/notebooks. You can also replace it with your favourite experiment tracker that's [supported by `accelerate`](https://huggingface.co/docs/accelerate/usage_guides/tracking).


## Few notes on multi-GPU 

To run in multi-GPU setup with DDP (distributed Data Parallel) change the `device_map` value to `device_map={"": Accelerator().process_index}` and make sure to run your script with `accelerate launch yourscript.py`. If you want to apply naive pipeline parallelism you can use `device_map="auto"`.


## Benchmarks

Below are some benchmark results for `examples/scripts/ppo.py`. To reproduce locally, please check out the `--command` arguments below.

```bash
python benchmark/benchmark.py \
    --command "python examples/scripts/ppo.py --log_with wandb" \
    --num-seeds 5 \
    --start-seed 1 \
    --workers 10 \
    --slurm-nodes 1 \
    --slurm-gpus-per-task 1 \
    --slurm-ntasks 1 \
    --slurm-total-cpus 12 \
    --slurm-template-path benchmark/trl.slurm_template
```

![](https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/benchmark/v0.4.7-55-g110e672/sentiment.png)



## With and without gradient accumulation

```bash
python benchmark/benchmark.py \
    --command "python examples/scripts/ppo.py --exp_name sentiment_tuning_step_grad_accu --mini_batch_size 1 --gradient_accumulation_steps 128 --log_with wandb" \
    --num-seeds 5 \
    --start-seed 1 \
    --workers 10 \
    --slurm-nodes 1 \
    --slurm-gpus-per-task 1 \
    --slurm-ntasks 1 \
    --slurm-total-cpus 12 \
    --slurm-template-path benchmark/trl.slurm_template
```

![](https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/benchmark/v0.4.7-55-g110e672/gradient_accu.png)


key: _node_content
string_value:
{"id_": "60852c40-ee68-4ffc-8fde-6d3ca4432fc1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/sentiment_tuning", "title": "Sentiment Tuning Examples", "tokens": 1601, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "068713a8-03d9-4ee0-9160-1eddea0b396e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/sentiment_tuning", "title": "Sentiment Tuning Examples", "tokens": 1601, "retrieve_doc": true, "source": "trl"}, "hash": "79b3ef31caa859bcf817dbfbf8ebf948e5aed5c4469fc35486d3971cb19fea23", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "179b4a0c-d18f-4dc2-bd4e-e0757808623b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/sentiment_tuning", "title": "Sentiment Tuning Examples", "tokens": 1601, "retrieve_doc": true, "source": "trl"}, "hash": "da21ba8bf295d3f7ac50b0689621b78b901bd76b57373025b7884cfd761db295", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3068, "end_char_idx": 6218, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Benchmarks

Below are some benchmark results for `examples/scripts/ppo.py`. To reproduce locally, please check out the `--command` arguments below.

```bash
python benchmark/benchmark.py \
    --command "python examples/scripts/ppo.py --log_with wandb" \
    --num-seeds 5 \
    --start-seed 1 \
    --workers 10 \
    --slurm-nodes 1 \
    --slurm-gpus-per-task 1 \
    --slurm-ntasks 1 \
    --slurm-total-cpus 12 \
    --slurm-template-path benchmark/trl.slurm_template
```

![](https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/benchmark/v0.4.7-55-g110e672/sentiment.png)



## With and without gradient accumulation

```bash
python benchmark/benchmark.py \
    --command "python examples/scripts/ppo.py --exp_name sentiment_tuning_step_grad_accu --mini_batch_size 1 --gradient_accumulation_steps 128 --log_with wandb" \
    --num-seeds 5 \
    --start-seed 1 \
    --workers 10 \
    --slurm-nodes 1 \
    --slurm-gpus-per-task 1 \
    --slurm-ntasks 1 \
    --slurm-total-cpus 12 \
    --slurm-template-path benchmark/trl.slurm_template
```

![](https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/benchmark/v0.4.7-55-g110e672/gradient_accu.png)


## Comparing different models (gpt2, gpt2-xl, falcon, llama2)

```bash
python benchmark/benchmark.py \
    --command "python examples/scripts/ppo.py --exp_name sentiment_tuning_gpt2 --log_with wandb" \
    --num-seeds 5 \
    --start-seed 1 \
    --workers 10 \
    --slurm-nodes 1 \
    --slurm-gpus-per-task 1 \
    --slurm-ntasks 1 \
    --slurm-total-cpus 12 \
    --slurm-template-path benchmark/trl.slurm_template
python benchmark/benchmark.py \
    --command "python examples/scripts/ppo.py --exp_name sentiment_tuning_gpt2xl_grad_accu --model_name gpt2-xl --mini_batch_size 16 --gradient_accumulation_steps 8 --log_with wandb" \
    --num-seeds 5 \
    --start-seed 1 \
    --workers 10 \
    --slurm-nodes 1 \
    --slurm-gpus-per-task 1 \
    --slurm-ntasks 1 \
    --slurm-total-cpus 12 \
    --slurm-template-path benchmark/trl.slurm_template
python benchmark/benchmark.py \
    --command "python examples/scripts/ppo.py --exp_name sentiment_tuning_falcon_rw_1b --model_name tiiuae/falcon-rw-1b --log_with wandb" \
    --num-seeds 5 \
    --start-seed 1 \
    --workers 10 \
    --slurm-nodes 1 \
    --slurm-gpus-per-task 1 \
    --slurm-ntasks 1 \
    --slurm-total-cpus 12 \
    --slurm-template-path benchmark/trl.slurm_template
```

![](https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/benchmark/v0.4.7-55-g110e672/different_models.png)

## With and without PEFT

```
python benchmark/benchmark.py \
    --command "python examples/scripts/ppo.py --exp_name sentiment_tuning_peft --use_peft --log_with wandb" \
    --num-seeds 5 \
    --start-seed 1 \
    --workers 10 \
    --slurm-nodes 1 \
    --slurm-gpus-per-task 1 \
    --slurm-ntasks 1 \
    --slurm-total-cpus 12 \
    --slurm-template-path benchmark/trl.slurm_template
```

![](https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/benchmark/v0.4.7-55-g110e672/peft.png)


key: _node_content
string_value:
{"id_": "f4faa8b7-d133-4c68-8ec6-97fe4a3028d5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/quickstart", "title": "Quickstart", "tokens": 902, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "405128a5-d00d-4adc-92ec-60fe87f9a3c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/quickstart", "title": "Quickstart", "tokens": 902, "retrieve_doc": true, "source": "trl"}, "hash": "8bfbd730047ec01b65ac7e0f6fadff824f713bf9cff56ce01daf95fd4d5fc15b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f2183629-2d1f-4a09-b744-b93a5ae205b0", "node_type": "1", "metadata": {}, "hash": "2969d01d4915ee549e7b3c50dcd22f3ed38801b803988640fa435bda14a574c2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1438, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Quickstart

## How does it work?

Fine-tuning a language model via PPO consists of roughly three steps:

1. **Rollout**: The language model generates a response or continuation based on a query which could be the start of a sentence.
2. **Evaluation**: The query and response are evaluated with a function, model, human feedback, or some combination of them. The important thing is that this process should yield a scalar value for each query/response pair. The optimization will aim at maximizing this value.
3. **Optimization**: This is the most complex part. In the optimisation step the query/response pairs are used to calculate the log-probabilities of the tokens in the sequences. This is done with the model that is trained and a reference model, which is usually the pre-trained model before fine-tuning. The KL-divergence between the two outputs is used as an additional reward signal to make sure the generated responses don't deviate too far from the reference language model. The active language model is then trained with PPO.

The full process is illustrated in the following figure:
<img src="https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/trl_overview.png"/>

## Minimal example

The following code illustrates the steps above. 

```python
# 0. imports
import torch
from transformers import GPT2Tokenizer

from trl import AutoModelForCausalLMWithValueHead, PPOConfig, PPOTrainer


key: _node_content
string_value:
{"id_": "f2183629-2d1f-4a09-b744-b93a5ae205b0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/quickstart", "title": "Quickstart", "tokens": 902, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "405128a5-d00d-4adc-92ec-60fe87f9a3c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/quickstart", "title": "Quickstart", "tokens": 902, "retrieve_doc": true, "source": "trl"}, "hash": "8bfbd730047ec01b65ac7e0f6fadff824f713bf9cff56ce01daf95fd4d5fc15b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f4faa8b7-d133-4c68-8ec6-97fe4a3028d5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/quickstart", "title": "Quickstart", "tokens": 902, "retrieve_doc": true, "source": "trl"}, "hash": "0745f8e1a2bf1a924bee522813462572f800ef07ceb588027ded3debd371aefd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3690, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Quickstart

## How does it work?

Fine-tuning a language model via PPO consists of roughly three steps:

1. **Rollout**: The language model generates a response or continuation based on a query which could be the start of a sentence.
2. **Evaluation**: The query and response are evaluated with a function, model, human feedback, or some combination of them. The important thing is that this process should yield a scalar value for each query/response pair. The optimization will aim at maximizing this value.
3. **Optimization**: This is the most complex part. In the optimisation step the query/response pairs are used to calculate the log-probabilities of the tokens in the sequences. This is done with the model that is trained and a reference model, which is usually the pre-trained model before fine-tuning. The KL-divergence between the two outputs is used as an additional reward signal to make sure the generated responses don't deviate too far from the reference language model. The active language model is then trained with PPO.

The full process is illustrated in the following figure:
<img src="https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/trl_overview.png"/>

## Minimal example

The following code illustrates the steps above. 

```python
# 0. imports
import torch
from transformers import GPT2Tokenizer

from trl import AutoModelForCausalLMWithValueHead, PPOConfig, PPOTrainer


# 1. load a pretrained model
model = AutoModelForCausalLMWithValueHead.from_pretrained("gpt2")
ref_model = AutoModelForCausalLMWithValueHead.from_pretrained("gpt2")
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
tokenizer.pad_token = tokenizer.eos_token

# 2. initialize trainer
ppo_config = {"mini_batch_size": 1, "batch_size": 1}
config = PPOConfig(**ppo_config)
ppo_trainer = PPOTrainer(config, model, ref_model, tokenizer)

# 3. encode a query
query_txt = "This morning I went to the "
query_tensor = tokenizer.encode(query_txt, return_tensors="pt").to(model.pretrained_model.device)

# 4. generate model response
generation_kwargs = {
    "min_length": -1,
    "top_k": 0.0,
    "top_p": 1.0,
    "do_sample": True,
    "pad_token_id": tokenizer.eos_token_id,
    "max_new_tokens": 20,
}
response_tensor = ppo_trainer.generate([item for item in query_tensor], return_prompt=False, **generation_kwargs)
response_txt = tokenizer.decode(response_tensor[0])

# 5. define a reward for response
# (this could be any reward such as human feedback or output from another model)
reward = [torch.tensor(1.0, device=model.pretrained_model.device)]

# 6. train model with ppo
train_stats = ppo_trainer.step([query_tensor[0]], [response_tensor[0]], reward)
```

In general, you would run steps 3-6 in a for-loop and run it on many diverse queries. You can find more realistic examples in the examples section. 

## How to use a trained model

After training a `AutoModelForCausalLMWithValueHead`, you can directly use the model in `transformers`.
```python

# .. Let's assume we have a trained model using `PPOTrainer` and `AutoModelForCausalLMWithValueHead`

# push the model on the Hub
model.push_to_hub("my-fine-tuned-model-ppo")

# or save it locally
model.save_pretrained("my-fine-tuned-model-ppo")

# load the model from the Hub
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("my-fine-tuned-model-ppo")
```

You can also load your model with `AutoModelForCausalLMWithValueHead` if you want to use the value head, for example to continue training.

```python
from trl.model import AutoModelForCausalLMWithValueHead

model = AutoModelForCausalLMWithValueHead.from_pretrained("my-fine-tuned-model-ppo")
```


key: _node_content
string_value:
{"id_": "d1fb3764-3e95-458d-8619-e8ed5596061c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/text_environments", "title": "Text Environments", "tokens": 2275, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1bd6dc94-02bd-4f50-b752-0cbe7044b926", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/text_environments", "title": "Text Environments", "tokens": 2275, "retrieve_doc": true, "source": "trl"}, "hash": "9cba40e36c2453aef7be446350c9a4c783340bf7cb3c68911d89d32228a01679", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d2fe3676-ee89-4ad4-a612-728e4afddaf8", "node_type": "1", "metadata": {}, "hash": "512df27de01bdf0c8fe468c0ef6659205df4695f2885cafec9562f39f057ec80", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3321, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Text Environments

Text environments provide a learning ground for language agents. It allows a language model to use tools to accomplish a task such as using a Python interpreter to answer math questions or using a search index for trivia questions. Having access to tools allows language models to solve tasks that would be very hard for the models itself but can be trivial for the appropriate tools. A good example is arithmetics of large numbers that become a simple copy-paste task once you have access to a calculator.

<div style="text-align: center">
<img src="https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/textenv.png">
</div>

Let's dive into how text environments work and start with tools!

## Tools

One of the core building blocks of text environments are tools that the model can use to solve tasks. In general tools can be any Python function that takes a string as input and returns string. The `TextEnvironment` offers two options for tools: either go with predefined tools from `transformers.Tool` or define your own function or class with `__call__` method. Let's have a look at both!

### `transformers.Tool`

Text environments fully support tools of the class `transformers.Tool`. The advantage of building tools in that framework is that they can easily be shared 

```Python
from transformers import load_tool

# simple calculator tool that runs +-/* operations
calc_tool = load_tool("ybelkada/simple-calculator")

# python interpreter that executes program and returns outputs
py_tool = load_tool("lvwerra/python-interpreter")

# wikipedia search index that returns best search match
wiki_tool = load_tool("vwxyzjn/pyserini-wikipedia-kilt-doc")
```

These tools are either loaded from the hub or from a local folder. Using the tool is as simple as calling them with a text query:

```Python
calc_tool("1/2")
>>> "0.5"
```

Note that both input and return values are strings to enable easy usage with a language model.

### Custom Tools

The following is an example of a tool that adds two integers:

```Python
def add(text):
    int_1, int_2 = text.split("+")
    result = int(int_1) + int(int_2)
    return str(result)

print(add("1+1"))
>>> "2"
```

We looked at basic examples such as a calculator but the principle holds for more complex tools as well such as a web search tool where you input the query and get the search results in return. Now let's look at how the model can use the tools with the call syntax.

### Call syntax

In order to have a unified way for the model to call a tool we created a simple syntax that looks as follows:

```python
"<request><TOOL_NAME>QUERY<call>TOOL_RESPONSE<response>"
```

There are a few special tokens involved so let's decompose it: First the model can signal that it wants to use a tool by emitting the `<request>` token. After that we want to know the name of the tool to call which is done by enclosing the tool name with `<>` brackets. Once we know which tool to call the tool query follows which is in free text form. The `<call>` tokens signifies the end of the query and stops the model generation. At this point the model output is parsed and the query sent to the tool. The environment appends the tool response to the string followed by the `<response>` token to show the end the tool output.


key: _node_content
string_value:
{"id_": "d2fe3676-ee89-4ad4-a612-728e4afddaf8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/text_environments", "title": "Text Environments", "tokens": 2275, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1bd6dc94-02bd-4f50-b752-0cbe7044b926", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/text_environments", "title": "Text Environments", "tokens": 2275, "retrieve_doc": true, "source": "trl"}, "hash": "9cba40e36c2453aef7be446350c9a4c783340bf7cb3c68911d89d32228a01679", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d1fb3764-3e95-458d-8619-e8ed5596061c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/text_environments", "title": "Text Environments", "tokens": 2275, "retrieve_doc": true, "source": "trl"}, "hash": "aa99e54e9b58af91518e6b62d07f498e5d865bd45d279b6525a6c1689efe5a72", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2856b7ac-e619-475a-a684-b08c3bc8f358", "node_type": "1", "metadata": {}, "hash": "bb1f706289acdc264259e5c8683af676cf5c1f1d5ebe5e9557cabab79d140b16", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1787, "end_char_idx": 3750, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Using the tool is as simple as calling them with a text query:

```Python
calc_tool("1/2")
>>> "0.5"
```

Note that both input and return values are strings to enable easy usage with a language model.

### Custom Tools

The following is an example of a tool that adds two integers:

```Python
def add(text):
    int_1, int_2 = text.split("+")
    result = int(int_1) + int(int_2)
    return str(result)

print(add("1+1"))
>>> "2"
```

We looked at basic examples such as a calculator but the principle holds for more complex tools as well such as a web search tool where you input the query and get the search results in return. Now let's look at how the model can use the tools with the call syntax.

### Call syntax

In order to have a unified way for the model to call a tool we created a simple syntax that looks as follows:

```python
"<request><TOOL_NAME>QUERY<call>TOOL_RESPONSE<response>"
```

There are a few special tokens involved so let's decompose it: First the model can signal that it wants to use a tool by emitting the `<request>` token. After that we want to know the name of the tool to call which is done by enclosing the tool name with `<>` brackets. Once we know which tool to call the tool query follows which is in free text form. The `<call>` tokens signifies the end of the query and stops the model generation. At this point the model output is parsed and the query sent to the tool. The environment appends the tool response to the string followed by the `<response>` token to show the end the tool output.

Let's look at the concrete example of the calculator and assume its name is `Calculator` (more on how the name of a tool is inferred later):

```python
"<request><Calculator>1/2<call>0.5<response>"
```

Finally, the episode is ended and generation stops when the model generates `<submit>` which marks the interaction as completed.

Now let's have a look how we can create a new text environment!

## Create a `TextEnvironment`


key: _node_content
string_value:
{"id_": "2856b7ac-e619-475a-a684-b08c3bc8f358", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/text_environments", "title": "Text Environments", "tokens": 2275, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1bd6dc94-02bd-4f50-b752-0cbe7044b926", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/text_environments", "title": "Text Environments", "tokens": 2275, "retrieve_doc": true, "source": "trl"}, "hash": "9cba40e36c2453aef7be446350c9a4c783340bf7cb3c68911d89d32228a01679", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d2fe3676-ee89-4ad4-a612-728e4afddaf8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/text_environments", "title": "Text Environments", "tokens": 2275, "retrieve_doc": true, "source": "trl"}, "hash": "68e7272dd4854a20ed5fe9561e08056f92e0709a78b56b2d560388e7dc10b8fd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0c060dcd-0fed-4dac-b806-a93a098942fa", "node_type": "1", "metadata": {}, "hash": "17b2ccbf885a65d2c032202b9c56c891124e402961945a79746989eeb9834b78", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2416, "end_char_idx": 5679, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Now let's look at how the model can use the tools with the call syntax.

### Call syntax

In order to have a unified way for the model to call a tool we created a simple syntax that looks as follows:

```python
"<request><TOOL_NAME>QUERY<call>TOOL_RESPONSE<response>"
```

There are a few special tokens involved so let's decompose it: First the model can signal that it wants to use a tool by emitting the `<request>` token. After that we want to know the name of the tool to call which is done by enclosing the tool name with `<>` brackets. Once we know which tool to call the tool query follows which is in free text form. The `<call>` tokens signifies the end of the query and stops the model generation. At this point the model output is parsed and the query sent to the tool. The environment appends the tool response to the string followed by the `<response>` token to show the end the tool output.

Let's look at the concrete example of the calculator and assume its name is `Calculator` (more on how the name of a tool is inferred later):

```python
"<request><Calculator>1/2<call>0.5<response>"
```

Finally, the episode is ended and generation stops when the model generates `<submit>` which marks the interaction as completed.

Now let's have a look how we can create a new text environment!

## Create a `TextEnvironment`


```python
prompt = """\
What is 13-3?
<request><SimpleCalculatorTool>13-3<call>10.0<response>
Result=10<submit>
"""

def reward_fn(result, answer):
    """Simplified reward function returning 1 if result matches answer and 0 otherwise."""
    result_parsed = result.split("=")[1].split("<")[0]
    return int(result_parsed==answer)

text_env = TextEnvironemnt(
    model=model, 
    tokenizer=tokenizer,
    tools= {"SimpleCalculatorTool": load_tool("ybelkada/simple-calculator")},
    reward_fn=exact_match_reward,
    prompt=prompt, 
    max_turns=1
    max_tool_response=100
    generation_kwargs={"do_sample": "true"}
)
```

Let's decompose the settings:

| Argument           | Description     |
|:-------------------|:----------------|
| `model`            | Language model to interact with the environment and generate requests. |
| `tokenizer`        | Tokenizer of language model handling tokenization of strings. |
| `tools`            | `list` of `dict` of tools. If former the name of the tool is inferred from class name and otherwise it's the keys of the dictionary.|
| `reward_fn`        | A function that takes a string as input and returns. Can have extra arguments that are passed to `.run()` such as ground truth.|
| `prompt`           | Prompt to prepend to every task. Usually a few examples to demonstrate to the model how to use the tools in a few-shot fashion. |
| `max_turns`        | Maximum number of interactions between model and tools before episode ends.|
| `max_tool_response`| The tool response is truncated to this number to avoid running out of model context.|
| `max_length`       |  The maximum number of tokens to allow in an episode. |
| `generation_kwargs`| Generation settings used by the language model. |

You can customize the environment to your needs and add custom tools and settings. Let's see how you can use the environment to have the model interact with the available tools!


key: _node_content
string_value:
{"id_": "0c060dcd-0fed-4dac-b806-a93a098942fa", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/text_environments", "title": "Text Environments", "tokens": 2275, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1bd6dc94-02bd-4f50-b752-0cbe7044b926", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/text_environments", "title": "Text Environments", "tokens": 2275, "retrieve_doc": true, "source": "trl"}, "hash": "9cba40e36c2453aef7be446350c9a4c783340bf7cb3c68911d89d32228a01679", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2856b7ac-e619-475a-a684-b08c3bc8f358", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/text_environments", "title": "Text Environments", "tokens": 2275, "retrieve_doc": true, "source": "trl"}, "hash": "d04dff1b262b088aaced77bf4ebd09333d20b652df91598b0acc2a3a0228503b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1956e205-aa6e-47aa-be20-621c4288f6ab", "node_type": "1", "metadata": {}, "hash": "28128b023640b5743d3b6eaec26b74aa0fe1279ea40e7133b9a2d07b085d2fb3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5682, "end_char_idx": 6957, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Run an Episode

To run a set of queries through the text environment one can simply use the `run` method.

```python
queries = ["What is 1/2?"]
answers = ["0.5"]

queries, responses, masks, rewards, histories = text_env.run(queries, answers=answers)
```

This will execute the model/tool feedback loop for each query until either no tool is called anymore, the maximum number of turns is reached or to maximum number of tokens in an episode is exceeded. The extra `kwargs` (e.g. `answers=answers` above) passed to `run` will be passed on to the reward function.

There are five objects that are returned by `run`: 

- `queries`: a list of the tokenized queries
- `responses`: all tokens that have been generated withing the environment including model and tool tokens
- `masks`: mask that indicates which tokens have been generated by the model and which tokens are generated by the tool
- `rewards`: a list of reward for each query/response
- `histories`: list of `TextHistory` objects, which are useful objects containing all the above and also the text equivalents

The masks are crucial for training as we don't want to optimize tokens that the model has not generated which are tokens produced by the tools.

Next, we'll train a PPO step with the generated responses!


key: _node_content
string_value:
{"id_": "1956e205-aa6e-47aa-be20-621c4288f6ab", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/text_environments", "title": "Text Environments", "tokens": 2275, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1bd6dc94-02bd-4f50-b752-0cbe7044b926", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/text_environments", "title": "Text Environments", "tokens": 2275, "retrieve_doc": true, "source": "trl"}, "hash": "9cba40e36c2453aef7be446350c9a4c783340bf7cb3c68911d89d32228a01679", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0c060dcd-0fed-4dac-b806-a93a098942fa", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/text_environments", "title": "Text Environments", "tokens": 2275, "retrieve_doc": true, "source": "trl"}, "hash": "d5b1553d3e64b05e600e681e80eec462106c731810d796427466b1b208c44875", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5682, "end_char_idx": 10343, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Run an Episode

To run a set of queries through the text environment one can simply use the `run` method.

```python
queries = ["What is 1/2?"]
answers = ["0.5"]

queries, responses, masks, rewards, histories = text_env.run(queries, answers=answers)
```

This will execute the model/tool feedback loop for each query until either no tool is called anymore, the maximum number of turns is reached or to maximum number of tokens in an episode is exceeded. The extra `kwargs` (e.g. `answers=answers` above) passed to `run` will be passed on to the reward function.

There are five objects that are returned by `run`: 

- `queries`: a list of the tokenized queries
- `responses`: all tokens that have been generated withing the environment including model and tool tokens
- `masks`: mask that indicates which tokens have been generated by the model and which tokens are generated by the tool
- `rewards`: a list of reward for each query/response
- `histories`: list of `TextHistory` objects, which are useful objects containing all the above and also the text equivalents

The masks are crucial for training as we don't want to optimize tokens that the model has not generated which are tokens produced by the tools.

Next, we'll train a PPO step with the generated responses!


### Train
Training on episodes from the `TextEnvironment` is straight forward and simply requires forwarding all the returned variables except the `TextHistory` objects to the `step` method:

```python
train_stats = ppo_trainer.step(queries, responses, rewards, masks)
```

## `TextHistory`

The `TextHistory` object stores the interactions between the model and the text environment. It stores tokens and text generated in each turn and their source in each turn (model or system) as well as rewards. Let's go through the class attributes and methods.

### Attributes

The following table summarises the available attributes of the `TextEnvironment` class:

| Attribute           | Description     |
|:-------------------|:----------------|
| `text`             | The full string of the text generated in the text environment with both model and system generated text. |
| `text_spans`       | A list of tuples with the spans for each model or system generated text segment. |
| `system_spans`     | A list of boolean values indicating if the segment is model or system generated. |
| `tokens`           | All tokens generated in text environment with both model and system generated tokens. |
| `token_spans`      | Similar to `text_spans` the `token_spans` indicate the boundaries of model andsystem generated tokens. |
| `token_masks`      | The token masks can be used to ignore system generated tokens by masking them. |
| `completed`        | Indicates if the interaction with the environment has completed. |
| `truncated`        | Indicates if the interaction with the environment has completed because max length was reached. |

With these attributes you can reconstruct every interaction of the model with the `TextEnvironment`. The `TextHistory` also lets you visualize the text history. Let's have a look!

### Visualization

When the model interacts inside the `TextEnvironment` it can be useful to visualize and separate which parts of the text outputs were generated by the model and which parts come from the system and tools. For that purpose there are the two methods [`TextHistory.show_text`] and [`TextHistory.show_tokens`]. They print the text and tokens respectively and highlight the various segments using the [`rich` libray](https://github.com/Textualize/rich) (make sure to install it before using these methods).

You can see that the prompt is highlighted in gray, whereas system segments such as query and tool responses are highlighted in green. All segments generated by the model are highlighted in blue and in addition to the pure text output the reward is displayed as additional text in plum. Here an example of `show_text`:

<div style="text-align: center">
<img src="https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/textenv_show_text.png" width=600>
</div>

Sometimes there can be tricky tokenization related issues that are hidden when showing the decoded text. Thus `TextHistory` also offers an option to display the same highlighting on the tokens directly with `show_tokens`:

<div style="text-align: center">
<img src="https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/textenv_show_tokens.png" width=800>
</div>

Note that you can turn on the colour legend by passing `show_legend=True`.

## API Documentation

[[autodoc]] TextEnvironment

[[autodoc]] TextHistory


key: _node_content
string_value:
{"id_": "66bb9ddb-cc2d-49bf-9dc9-be0ef6cf478c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2d44c2ff-d5ef-4567-ad9e-bb309738b311", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "hash": "c373a01b66f6d8dd0042e7ce819cfc28b3b25495657841605926550d68bd7330", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8cc6d408-461b-4860-b73e-384d51cbedcd", "node_type": "1", "metadata": {}, "hash": "3055cc8b574567cd6e227ac35bbc0f091e917f46598716fbdaf37b2dc5504e22", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2927, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# DPO Trainer

TRL supports the DPO Trainer for training language models from preference data, as described in the paper [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://huggingface.co/papers/2305.18290) by Rafailov et al., 2023. For a full example have a look at  [`examples/scripts/dpo.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/dpo.py).

The first step as always is to train your SFT model, to ensure the data we train on is in-distribution for the DPO algorithm.

## How DPO works

Fine-tuning a language model via DPO consists of two steps and is easier than PPO:

1. **Data collection**: Gather a preference dataset with positive and negative selected pairs of generation, given a prompt.
2. **Optimization**: Maximize the log-likelihood of the DPO loss directly.

DPO-compatible datasets can be found with [the tag `dpo` on Hugging Face Hub](https://huggingface.co/datasets?other=dpo). You can also explore the [librarian-bots/direct-preference-optimization-datasets](https://huggingface.co/collections/librarian-bots/direct-preference-optimization-datasets-66964b12835f46289b6ef2fc) Collection to identify datasets that are likely to support DPO training.

This process is illustrated in the sketch below (from [figure 1 of the original paper](https://huggingface.co/papers/2305.18290)):

<img width="835" alt="Screenshot 2024-03-19 at 12 39 41" src="https://github.com/huggingface/trl/assets/49240599/9150fac6-3d88-4ca2-8ec6-2a6f3473216d">

Read more about DPO algorithm in the [original paper](https://huggingface.co/papers/2305.18290).


## Expected dataset format

The DPO trainer expects a very specific format for the dataset. Since the model will be trained to directly optimize the preference of which sentence is the most relevant, given two sentences. We provide an example from the [`Anthropic/hh-rlhf`](https://huggingface.co/datasets/Anthropic/hh-rlhf) dataset below:

<div style="text-align: center">
<img src="https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/rlhf-antropic-example.png", width="50%">
</div>

Therefore the final dataset object should contain these 3 entries if you use the default [`DPODataCollatorWithPadding`] data collator. The entries should be named:

- `prompt`
- `chosen`
- `rejected`

for example:

```py
dpo_dataset_dict = {
    "prompt": [
        "hello",
        "how are you",
        "What is your name?",
        "What is your name?",
        "Which is the best programming language?",
        "Which is the best programming language?",
        "Which is the best programming language?",
    ],
    "chosen": [
        "hi nice to meet you",
        "I am fine",
        "My name is Mary",
        "My name is Mary",
        "Python",
        "Python",
        "Java",
    ],
    "rejected": [
        "leave me alone",
        "I am not fine",
        "Whats it to you?


key: _node_content
string_value:
{"id_": "8cc6d408-461b-4860-b73e-384d51cbedcd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2d44c2ff-d5ef-4567-ad9e-bb309738b311", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "hash": "c373a01b66f6d8dd0042e7ce819cfc28b3b25495657841605926550d68bd7330", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "66bb9ddb-cc2d-49bf-9dc9-be0ef6cf478c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "hash": "c80b0858683d9175d9f85f5fa0fe69dc92e1419c67c6388d982eb225ad8a9d12", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b30c1e2b-b3cc-465b-8b1f-b54217723fd2", "node_type": "1", "metadata": {}, "hash": "6dbd6c623818be1800f305ff4ba17f6cde5977acd03d69a1d989d8323fa09cc7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1613, "end_char_idx": 4799, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Expected dataset format

The DPO trainer expects a very specific format for the dataset. Since the model will be trained to directly optimize the preference of which sentence is the most relevant, given two sentences. We provide an example from the [`Anthropic/hh-rlhf`](https://huggingface.co/datasets/Anthropic/hh-rlhf) dataset below:

<div style="text-align: center">
<img src="https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/rlhf-antropic-example.png", width="50%">
</div>

Therefore the final dataset object should contain these 3 entries if you use the default [`DPODataCollatorWithPadding`] data collator. The entries should be named:

- `prompt`
- `chosen`
- `rejected`

for example:

```py
dpo_dataset_dict = {
    "prompt": [
        "hello",
        "how are you",
        "What is your name?",
        "What is your name?",
        "Which is the best programming language?",
        "Which is the best programming language?",
        "Which is the best programming language?",
    ],
    "chosen": [
        "hi nice to meet you",
        "I am fine",
        "My name is Mary",
        "My name is Mary",
        "Python",
        "Python",
        "Java",
    ],
    "rejected": [
        "leave me alone",
        "I am not fine",
        "Whats it to you?",
        "I dont have a name",
        "Javascript",
        "C++",
        "C++",
    ],
}
```

where the `prompt` contains the context inputs, `chosen` contains the corresponding chosen responses and `rejected` contains the corresponding negative (rejected) responses. As can be seen a prompt can have multiple responses and this is reflected in the entries being repeated in the dictionary's value arrays.

[`DPOTrainer`] can be used to fine-tune visual language models (VLMs). In this case, the dataset must also contain the key `images`, and the trainer's `tokenizer` is the VLM's `processor`. For example, for Idefics2, the processor expects the dataset to have the following format:

Note: Currently, VLM support is exclusive to Idefics2 and does not extend to other VLMs.

```py
dpo_dataset_dict = {
    'images': [
        [Image.open('beach.jpg')],
        [Image.open('street.jpg')],
    ],
    'prompt': [
        'The image <image> shows',
        '<image> The image depicts',
    ],
    'chosen': [
        'a sunny beach with palm trees.',
        'a busy street with several cars and buildings.',
    ],
    'rejected': [
        'a snowy mountain with skiers.',
        'a calm countryside with green fields.',
    ],
}
```

## Expected model format

The DPO trainer expects a model of `AutoModelForCausalLM` or `AutoModelForVision2Seq`, compared to PPO that expects `AutoModelForCausalLMWithValueHead` for the value function.

## Using the `DPOTrainer`

For a detailed example have a look at the `examples/scripts/dpo.py` script. At a high level we need to initialize the [`DPOTrainer`] with a `model` we wish to train, a reference `ref_model` which we will use to calculate the implicit rewards of the preferred and rejected response, the `beta` refers to the hyperparameter of the implicit reward, and the dataset contains the 3 entries listed above.


key: _node_content
string_value:
{"id_": "b30c1e2b-b3cc-465b-8b1f-b54217723fd2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2d44c2ff-d5ef-4567-ad9e-bb309738b311", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "hash": "c373a01b66f6d8dd0042e7ce819cfc28b3b25495657841605926550d68bd7330", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8cc6d408-461b-4860-b73e-384d51cbedcd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "hash": "59b83027224fb6a9d4919f76a6f90d07aed850acab571dd47da6b48099b15bed", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a6cd65e6-ecf9-4ada-a2ff-92ea04f473bc", "node_type": "1", "metadata": {}, "hash": "67c8c02a82997deb2bae081d05981eef1a1101fd136ba240759a38b830bbdc2a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3200, "end_char_idx": 6271, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
As can be seen a prompt can have multiple responses and this is reflected in the entries being repeated in the dictionary's value arrays.

[`DPOTrainer`] can be used to fine-tune visual language models (VLMs). In this case, the dataset must also contain the key `images`, and the trainer's `tokenizer` is the VLM's `processor`. For example, for Idefics2, the processor expects the dataset to have the following format:

Note: Currently, VLM support is exclusive to Idefics2 and does not extend to other VLMs.

```py
dpo_dataset_dict = {
    'images': [
        [Image.open('beach.jpg')],
        [Image.open('street.jpg')],
    ],
    'prompt': [
        'The image <image> shows',
        '<image> The image depicts',
    ],
    'chosen': [
        'a sunny beach with palm trees.',
        'a busy street with several cars and buildings.',
    ],
    'rejected': [
        'a snowy mountain with skiers.',
        'a calm countryside with green fields.',
    ],
}
```

## Expected model format

The DPO trainer expects a model of `AutoModelForCausalLM` or `AutoModelForVision2Seq`, compared to PPO that expects `AutoModelForCausalLMWithValueHead` for the value function.

## Using the `DPOTrainer`

For a detailed example have a look at the `examples/scripts/dpo.py` script. At a high level we need to initialize the [`DPOTrainer`] with a `model` we wish to train, a reference `ref_model` which we will use to calculate the implicit rewards of the preferred and rejected response, the `beta` refers to the hyperparameter of the implicit reward, and the dataset contains the 3 entries listed above. Note that the `model` and `ref_model` need to have the same architecture (ie decoder only or encoder-decoder).

```py
training_args = DPOConfig(
    beta=0.1,
)
dpo_trainer = DPOTrainer(
    model,
    ref_model,
    args=training_args,
    train_dataset=train_dataset,
    tokenizer=tokenizer,  # for visual language models, use tokenizer=processor instead
)
```

After this one can then call:

```py
dpo_trainer.train()
```

Note that the `beta` is the temperature parameter for the DPO loss, typically something in the range of `0.1` to `0.5`. We ignore the reference model as `beta` -> 0.

## Loss functions

Given the preference data, we can fit a binary classifier according to the Bradley-Terry model and in fact the [DPO](https://huggingface.co/papers/2305.18290) authors propose the sigmoid loss on the normalized likelihood via the `logsigmoid` to fit a logistic regression. To use this loss, set the `loss_type="sigmoid"` (default) in the [`DPOConfig`].

The [RSO](https://huggingface.co/papers/2309.06657) authors propose to use a hinge loss on the normalized likelihood from the [SLiC](https://huggingface.co/papers/2305.10425) paper. To use this loss, set the `loss_type="hinge"` in the [`DPOConfig`]. In this case, the `beta` is the reciprocal of the margin.

The [IPO](https://huggingface.co/papers/2310.12036) authors provide a deeper theoretical understanding of the DPO algorithms and identify an issue with overfitting and propose an alternative loss.


key: _node_content
string_value:
{"id_": "a6cd65e6-ecf9-4ada-a2ff-92ea04f473bc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2d44c2ff-d5ef-4567-ad9e-bb309738b311", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "hash": "c373a01b66f6d8dd0042e7ce819cfc28b3b25495657841605926550d68bd7330", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b30c1e2b-b3cc-465b-8b1f-b54217723fd2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "hash": "fbb68b773b357c06308b9ac19d421892f77f1eac8d8bdfba17c75faa6149afb8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9be6df82-e7f2-48ca-b8f9-ff3ef89132bd", "node_type": "1", "metadata": {}, "hash": "82690f79474a2cde7c54a0c4708715cbc7762a8aef3734a8e8ad1fc9d2dd0822", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4800, "end_char_idx": 7665, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note that the `model` and `ref_model` need to have the same architecture (ie decoder only or encoder-decoder).

```py
training_args = DPOConfig(
    beta=0.1,
)
dpo_trainer = DPOTrainer(
    model,
    ref_model,
    args=training_args,
    train_dataset=train_dataset,
    tokenizer=tokenizer,  # for visual language models, use tokenizer=processor instead
)
```

After this one can then call:

```py
dpo_trainer.train()
```

Note that the `beta` is the temperature parameter for the DPO loss, typically something in the range of `0.1` to `0.5`. We ignore the reference model as `beta` -> 0.

## Loss functions

Given the preference data, we can fit a binary classifier according to the Bradley-Terry model and in fact the [DPO](https://huggingface.co/papers/2305.18290) authors propose the sigmoid loss on the normalized likelihood via the `logsigmoid` to fit a logistic regression. To use this loss, set the `loss_type="sigmoid"` (default) in the [`DPOConfig`].

The [RSO](https://huggingface.co/papers/2309.06657) authors propose to use a hinge loss on the normalized likelihood from the [SLiC](https://huggingface.co/papers/2305.10425) paper. To use this loss, set the `loss_type="hinge"` in the [`DPOConfig`]. In this case, the `beta` is the reciprocal of the margin.

The [IPO](https://huggingface.co/papers/2310.12036) authors provide a deeper theoretical understanding of the DPO algorithms and identify an issue with overfitting and propose an alternative loss. To use the loss set the `loss_type="ipo"` in the [`DPOConfig`]. In this case, the `beta` is the reciprocal of the gap between the log-likelihood ratios of the chosen vs the rejected completion pair and thus the smaller the `beta` the larger this gaps is. As per the paper the loss is averaged over log-likelihoods of the completion (unlike DPO which is summed only). 

The [cDPO](https://ericmitchell.ai/cdpo.pdf) is a tweak on the DPO loss where we assume that the preference labels are noisy with some probability. In this approach, the `label_smoothing` parameter in the [`DPOConfig`] is used to model the probability of existing label noise. To apply this conservative loss, set `label_smoothing` to a value greater than 0.0 (between 0.0 and 0.5; the default is 0.0).

The [EXO](https://huggingface.co/papers/2402.00856) authors propose to minimize the reverse KL instead of the negative log-sigmoid loss of DPO which corresponds to forward KL. To use the loss set the `loss_type="exo_pair"` in the [`DPOConfig`]. Setting non-zero `label_smoothing` (default `1e-3`) leads to a simplified version of EXO on pair-wise preferences (see Eqn. (16) of the [EXO paper](https://huggingface.co/papers/2402.00856)). The full version of EXO uses `K>2` completions generated by the SFT policy, which becomes an unbiased estimator of the PPO objective (up to a constant) when `K` is sufficiently large.


key: _node_content
string_value:
{"id_": "9be6df82-e7f2-48ca-b8f9-ff3ef89132bd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2d44c2ff-d5ef-4567-ad9e-bb309738b311", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "hash": "c373a01b66f6d8dd0042e7ce819cfc28b3b25495657841605926550d68bd7330", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a6cd65e6-ecf9-4ada-a2ff-92ea04f473bc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "hash": "a29310d8d4624537b1ce53b409ea986fac6371c3c01bc6fd2d23122787c895c7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e0295bbb-5671-4100-963a-eb06ac47c473", "node_type": "1", "metadata": {}, "hash": "6165775d544b62425a9ae35d50b1c445960905a9a7baad8c1aca63e099cfae77", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6272, "end_char_idx": 9044, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
To use the loss set the `loss_type="ipo"` in the [`DPOConfig`]. In this case, the `beta` is the reciprocal of the gap between the log-likelihood ratios of the chosen vs the rejected completion pair and thus the smaller the `beta` the larger this gaps is. As per the paper the loss is averaged over log-likelihoods of the completion (unlike DPO which is summed only). 

The [cDPO](https://ericmitchell.ai/cdpo.pdf) is a tweak on the DPO loss where we assume that the preference labels are noisy with some probability. In this approach, the `label_smoothing` parameter in the [`DPOConfig`] is used to model the probability of existing label noise. To apply this conservative loss, set `label_smoothing` to a value greater than 0.0 (between 0.0 and 0.5; the default is 0.0).

The [EXO](https://huggingface.co/papers/2402.00856) authors propose to minimize the reverse KL instead of the negative log-sigmoid loss of DPO which corresponds to forward KL. To use the loss set the `loss_type="exo_pair"` in the [`DPOConfig`]. Setting non-zero `label_smoothing` (default `1e-3`) leads to a simplified version of EXO on pair-wise preferences (see Eqn. (16) of the [EXO paper](https://huggingface.co/papers/2402.00856)). The full version of EXO uses `K>2` completions generated by the SFT policy, which becomes an unbiased estimator of the PPO objective (up to a constant) when `K` is sufficiently large.

The [NCA](https://huggingface.co/papers/2402.05369) authors shows that NCA optimizes the absolute likelihood for each response rather than the relative likelihood. To use the loss set the `loss_type="nca_pair"` in the [`DPOConfig`].

The [Robust DPO](https://huggingface.co/papers/2403.00409) authors propose an unbiased estimate of the DPO loss that is robust to preference noise in the data. Like in cDPO, it assumes that the preference labels are noisy with some probability. In this approach, the `label_smoothing` parameter in the [`DPOConfig`] is used to model the probability of existing label noise. To apply this conservative loss, set `label_smoothing` to a value greater than 0.0 (between 0.0 and 0.5; the default is 0.0) and set the `loss_type="robust"` in the [`DPOConfig`].

The [BCO](https://huggingface.co/papers/2404.04656) authors train a binary classifier whose logit serves as a reward so that the classifier maps {prompt, chosen completion} pairs to 1 and {prompt, rejected completion} pairs to 0. To use this loss, set the `loss_type="bco_pair"` in the [`DPOConfig`].

The [TR-DPO](https://huggingface.co/papers/2404.09656) paper suggests syncing the reference model weights after every `ref_model_sync_steps` steps of SGD with weight `ref_model_mixup_alpha` during DPO training. To toggle this callback use the `sync_ref_model=True` in the [`DPOConfig`].


key: _node_content
string_value:
{"id_": "e0295bbb-5671-4100-963a-eb06ac47c473", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2d44c2ff-d5ef-4567-ad9e-bb309738b311", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "hash": "c373a01b66f6d8dd0042e7ce819cfc28b3b25495657841605926550d68bd7330", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9be6df82-e7f2-48ca-b8f9-ff3ef89132bd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "hash": "0888d7329bfbdd907dea665cf3eda8cc394a25365f23bece11966af45d0bb894", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "16464480-c44b-4186-a33e-fba2d5bb9b29", "node_type": "1", "metadata": {}, "hash": "0e864a59c0f692cb20940f93f6722b77fe3a23624e9d34e083886061861de940", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7667, "end_char_idx": 10633, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The [NCA](https://huggingface.co/papers/2402.05369) authors shows that NCA optimizes the absolute likelihood for each response rather than the relative likelihood. To use the loss set the `loss_type="nca_pair"` in the [`DPOConfig`].

The [Robust DPO](https://huggingface.co/papers/2403.00409) authors propose an unbiased estimate of the DPO loss that is robust to preference noise in the data. Like in cDPO, it assumes that the preference labels are noisy with some probability. In this approach, the `label_smoothing` parameter in the [`DPOConfig`] is used to model the probability of existing label noise. To apply this conservative loss, set `label_smoothing` to a value greater than 0.0 (between 0.0 and 0.5; the default is 0.0) and set the `loss_type="robust"` in the [`DPOConfig`].

The [BCO](https://huggingface.co/papers/2404.04656) authors train a binary classifier whose logit serves as a reward so that the classifier maps {prompt, chosen completion} pairs to 1 and {prompt, rejected completion} pairs to 0. To use this loss, set the `loss_type="bco_pair"` in the [`DPOConfig`].

The [TR-DPO](https://huggingface.co/papers/2404.09656) paper suggests syncing the reference model weights after every `ref_model_sync_steps` steps of SGD with weight `ref_model_mixup_alpha` during DPO training. To toggle this callback use the `sync_ref_model=True` in the [`DPOConfig`].

The [RPO](https://huggingface.co/papers/2404.19733) paper implements an iterative preference tuning algorithm using a loss related to the RPO loss in this [paper](https://huggingface.co/papers/2405.16436) that essentially consists of a weighted SFT loss on the chosen preferences together with the DPO loss. To use this loss, set the `rpo_alpha` in the [`DPOConfig`] to an appropriate value. The paper suggests setting this weight to 1.0.

The [SPPO](https://huggingface.co/papers/2405.00675) authors claim that SPPO is capable of solving the Nash equilibrium iteratively by pushing the chosen rewards to be as large as 1/2 and the rejected rewards to be as small as -1/2 and can alleviate data sparsity issues. The implementation approximates this algorithm by employing hard label probabilities, assigning 1 to the winner and 0 to the loser. To use this loss, set the `loss_type="sppo_hard"` in the [`DPOConfig`].

The [AOT](https://huggingface.co/papers/2406.05882) authors propose to use Distributional Preference Alignment Via Optimal Transport. Traditionally, the alignment algorithms use paired preferences at a sample level, which does not ensure alignment on the distributional level. AOT, on the other hand, can align LLMs on paired or unpaired preference data by making the reward distribution of the positive samples stochastically dominant in the first order on the distribution of negative samples. Specifically, `loss_type="aot"` is appropriate for paired datasets, where each prompt has both chosen and rejected responses; `loss_type="aot_pair"` is for unpaired datasets.


key: _node_content
string_value:
{"id_": "16464480-c44b-4186-a33e-fba2d5bb9b29", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2d44c2ff-d5ef-4567-ad9e-bb309738b311", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "hash": "c373a01b66f6d8dd0042e7ce819cfc28b3b25495657841605926550d68bd7330", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e0295bbb-5671-4100-963a-eb06ac47c473", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "hash": "0b0dddae0e106738f1d9aea42f3a15fff62119859c9060cf67191e77a50211af", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2de2d4ce-d247-400d-bef6-2523d2f567b1", "node_type": "1", "metadata": {}, "hash": "d2821d5317f04773f9937c9d4149af305e7bbd46700048b8d18b07eb8c4d4e89", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8969, "end_char_idx": 12339, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
To toggle this callback use the `sync_ref_model=True` in the [`DPOConfig`].

The [RPO](https://huggingface.co/papers/2404.19733) paper implements an iterative preference tuning algorithm using a loss related to the RPO loss in this [paper](https://huggingface.co/papers/2405.16436) that essentially consists of a weighted SFT loss on the chosen preferences together with the DPO loss. To use this loss, set the `rpo_alpha` in the [`DPOConfig`] to an appropriate value. The paper suggests setting this weight to 1.0.

The [SPPO](https://huggingface.co/papers/2405.00675) authors claim that SPPO is capable of solving the Nash equilibrium iteratively by pushing the chosen rewards to be as large as 1/2 and the rejected rewards to be as small as -1/2 and can alleviate data sparsity issues. The implementation approximates this algorithm by employing hard label probabilities, assigning 1 to the winner and 0 to the loser. To use this loss, set the `loss_type="sppo_hard"` in the [`DPOConfig`].

The [AOT](https://huggingface.co/papers/2406.05882) authors propose to use Distributional Preference Alignment Via Optimal Transport. Traditionally, the alignment algorithms use paired preferences at a sample level, which does not ensure alignment on the distributional level. AOT, on the other hand, can align LLMs on paired or unpaired preference data by making the reward distribution of the positive samples stochastically dominant in the first order on the distribution of negative samples. Specifically, `loss_type="aot"` is appropriate for paired datasets, where each prompt has both chosen and rejected responses; `loss_type="aot_pair"` is for unpaired datasets. In a nutshell, `loss_type="aot"` ensures that the log-likelihood ratio of chosen to rejected of the aligned model has higher quantiles than that ratio for the reference model. `loss_type="aot_pair"` ensures that the chosen reward is higher on all quantiles than the rejected reward. Note that in both cases quantiles are obtained via sorting. To fully leverage the advantages of the AOT algorithm, it is important to maximize the per-GPU batch size.

The [APO](https://huggingface.co/papers/2408.06266) method introduces an "anchored" version of the alignment objective. There are two variants: `apo_zero` and `apo_down`. The `apo_zero` loss increases the likelihood of winning outputs while decreasing the likelihood of losing outputs, making it suitable when the model is less performant than the winning outputs. On the other hand, `apo_down` decreases the likelihood of both winning and losing outputs, but with a stronger emphasis on reducing the likelihood of losing outputs. This variant is more effective when the model is better than the winning outputs. To use these losses, set `loss_type="apo_zero"` or `loss_type="apo_down"` in the [`DPOConfig`].

### For Mixture of Experts Models: Enabling the auxiliary loss

MOEs are the most efficient if the load is about equally distributed between experts.  
To ensure that we train MOEs similarly during preference-tuning, it is beneficial to add the auxiliary loss from the load balancer to the final loss.

This option is enabled by setting `output_router_logits=True` in the model config (e.g. MixtralConfig).  
To scale how much the auxiliary loss contributes to the total loss, use the hyperparameter `router_aux_loss_coef=...` (default: 0.001).


key: _node_content
string_value:
{"id_": "2de2d4ce-d247-400d-bef6-2523d2f567b1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2d44c2ff-d5ef-4567-ad9e-bb309738b311", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "hash": "c373a01b66f6d8dd0042e7ce819cfc28b3b25495657841605926550d68bd7330", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "16464480-c44b-4186-a33e-fba2d5bb9b29", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "hash": "a8584920f22208fb2ae62dfac90bf06f616d08faca29732aa0cdbe0c130cfd55", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0b1d4f77-1c8d-4136-8612-2b2a162b79b1", "node_type": "1", "metadata": {}, "hash": "2f2f3401303438286677321b849903805230026eabc072a3af5c8eae3b33010e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10634, "end_char_idx": 13802, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In a nutshell, `loss_type="aot"` ensures that the log-likelihood ratio of chosen to rejected of the aligned model has higher quantiles than that ratio for the reference model. `loss_type="aot_pair"` ensures that the chosen reward is higher on all quantiles than the rejected reward. Note that in both cases quantiles are obtained via sorting. To fully leverage the advantages of the AOT algorithm, it is important to maximize the per-GPU batch size.

The [APO](https://huggingface.co/papers/2408.06266) method introduces an "anchored" version of the alignment objective. There are two variants: `apo_zero` and `apo_down`. The `apo_zero` loss increases the likelihood of winning outputs while decreasing the likelihood of losing outputs, making it suitable when the model is less performant than the winning outputs. On the other hand, `apo_down` decreases the likelihood of both winning and losing outputs, but with a stronger emphasis on reducing the likelihood of losing outputs. This variant is more effective when the model is better than the winning outputs. To use these losses, set `loss_type="apo_zero"` or `loss_type="apo_down"` in the [`DPOConfig`].

### For Mixture of Experts Models: Enabling the auxiliary loss

MOEs are the most efficient if the load is about equally distributed between experts.  
To ensure that we train MOEs similarly during preference-tuning, it is beneficial to add the auxiliary loss from the load balancer to the final loss.

This option is enabled by setting `output_router_logits=True` in the model config (e.g. MixtralConfig).  
To scale how much the auxiliary loss contributes to the total loss, use the hyperparameter `router_aux_loss_coef=...` (default: 0.001).

## Logging

While training and evaluating we record the following reward metrics:

- `rewards/chosen`: the mean difference between the log probabilities of the policy model and the reference model for the chosen responses scaled by beta
- `rewards/rejected`: the mean difference between the log probabilities of the policy model and the reference model for the rejected responses scaled by beta
- `rewards/accuracies`: mean of how often the chosen rewards are > than the corresponding rejected rewards
- `rewards/margins`: the mean difference between the chosen and corresponding rejected rewards

## Accelerate DPO fine-tuning using `unsloth`

You can further accelerate QLoRA / LoRA (2x faster, 60% less memory) using the [`unsloth`](https://github.com/unslothai/unsloth) library that is fully compatible with `SFTTrainer`. Currently `unsloth` supports only Llama (Yi, TinyLlama, Qwen, Deepseek etc) and Mistral architectures. Some benchmarks for DPO listed below:

| GPU      | Model     | Dataset    | 🤗  | 🤗 + Flash Attention 2 | 🦥 Unsloth | 🦥 VRAM saved |
| -------- | --------- | ---------- | --- | ---------------------- | ---------- | ------------- |
| A100 40G | Zephyr 7b | Ultra Chat | 1x  | 1.24x                  | **1.88x**  | -11.6%        |
| Tesla T4 | Zephyr 7b | Ultra Chat | 1x  | 1.09x                  | **1.55x**  | -18.6%        |

First install `unsloth` according to the [official documentation](https://github.com/unslothai/unsloth).


key: _node_content
string_value:
{"id_": "0b1d4f77-1c8d-4136-8612-2b2a162b79b1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2d44c2ff-d5ef-4567-ad9e-bb309738b311", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "hash": "c373a01b66f6d8dd0042e7ce819cfc28b3b25495657841605926550d68bd7330", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2de2d4ce-d247-400d-bef6-2523d2f567b1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "hash": "46a187979e3e05eacf2266ab6080e919567d486e14f9a4652c55e550bcd67482", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2e66ffba-672a-4521-ae34-c5cc29e5e490", "node_type": "1", "metadata": {}, "hash": "2add5cf67e740489fa805be7428ece7290dd9a68b998f6f6fc21287fe82fcfd7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12341, "end_char_idx": 14494, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Logging

While training and evaluating we record the following reward metrics:

- `rewards/chosen`: the mean difference between the log probabilities of the policy model and the reference model for the chosen responses scaled by beta
- `rewards/rejected`: the mean difference between the log probabilities of the policy model and the reference model for the rejected responses scaled by beta
- `rewards/accuracies`: mean of how often the chosen rewards are > than the corresponding rejected rewards
- `rewards/margins`: the mean difference between the chosen and corresponding rejected rewards

## Accelerate DPO fine-tuning using `unsloth`

You can further accelerate QLoRA / LoRA (2x faster, 60% less memory) using the [`unsloth`](https://github.com/unslothai/unsloth) library that is fully compatible with `SFTTrainer`. Currently `unsloth` supports only Llama (Yi, TinyLlama, Qwen, Deepseek etc) and Mistral architectures. Some benchmarks for DPO listed below:

| GPU      | Model     | Dataset    | 🤗  | 🤗 + Flash Attention 2 | 🦥 Unsloth | 🦥 VRAM saved |
| -------- | --------- | ---------- | --- | ---------------------- | ---------- | ------------- |
| A100 40G | Zephyr 7b | Ultra Chat | 1x  | 1.24x                  | **1.88x**  | -11.6%        |
| Tesla T4 | Zephyr 7b | Ultra Chat | 1x  | 1.09x                  | **1.55x**  | -18.6%        |

First install `unsloth` according to the [official documentation](https://github.com/unslothai/unsloth). Once installed, you can incorporate unsloth into your workflow in a very simple manner; instead of loading `AutoModelForCausalLM`, you just need to load a `FastLanguageModel` as follows:

```python
import torch
from trl import DPOConfig, DPOTrainer
from unsloth import FastLanguageModel

max_seq_length = 2048 # Supports automatic RoPE Scaling, so choose any number.

# Load model
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/zephyr-sft",
    max_seq_length = max_seq_length,
    dtype = None, # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+
    load_in_4bit = True, # Use 4bit quantization to reduce memory usage. Can be False.


key: _node_content
string_value:
{"id_": "2e66ffba-672a-4521-ae34-c5cc29e5e490", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2d44c2ff-d5ef-4567-ad9e-bb309738b311", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "hash": "c373a01b66f6d8dd0042e7ce819cfc28b3b25495657841605926550d68bd7330", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0b1d4f77-1c8d-4136-8612-2b2a162b79b1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "hash": "d8038986dc82e6b321714c7783329bcc82c5fd1d2924e7669de4f866fad33a01", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fda47293-062a-4fff-be81-47b3ed3fa145", "node_type": "1", "metadata": {}, "hash": "a54522f0a723bb9802e1ae88e5f8fb638db2e79c6ef0de645c0681adc89407ee", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13167, "end_char_idx": 16032, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Currently `unsloth` supports only Llama (Yi, TinyLlama, Qwen, Deepseek etc) and Mistral architectures. Some benchmarks for DPO listed below:

| GPU      | Model     | Dataset    | 🤗  | 🤗 + Flash Attention 2 | 🦥 Unsloth | 🦥 VRAM saved |
| -------- | --------- | ---------- | --- | ---------------------- | ---------- | ------------- |
| A100 40G | Zephyr 7b | Ultra Chat | 1x  | 1.24x                  | **1.88x**  | -11.6%        |
| Tesla T4 | Zephyr 7b | Ultra Chat | 1x  | 1.09x                  | **1.55x**  | -18.6%        |

First install `unsloth` according to the [official documentation](https://github.com/unslothai/unsloth). Once installed, you can incorporate unsloth into your workflow in a very simple manner; instead of loading `AutoModelForCausalLM`, you just need to load a `FastLanguageModel` as follows:

```python
import torch
from trl import DPOConfig, DPOTrainer
from unsloth import FastLanguageModel

max_seq_length = 2048 # Supports automatic RoPE Scaling, so choose any number.

# Load model
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/zephyr-sft",
    max_seq_length = max_seq_length,
    dtype = None, # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+
    load_in_4bit = True, # Use 4bit quantization to reduce memory usage. Can be False.
    # token = "hf_...", # use one if using gated models like meta-llama/Llama-2-7b-hf
)

# Do model patching and add fast LoRA weights
model = FastLanguageModel.get_peft_model(
    model,
    r = 16,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj",],
    lora_alpha = 16,
    lora_dropout = 0, # Dropout = 0 is currently optimized
    bias = "none",    # Bias = "none" is currently optimized
    use_gradient_checkpointing = True,
    random_state = 3407,
)

training_args = DPOConfig(
    output_dir="./output",
    beta=0.1,
)

dpo_trainer = DPOTrainer(
    model,
    ref_model=None,
    args=training_args,
    train_dataset=train_dataset,
    tokenizer=tokenizer,
)
dpo_trainer.train()
```

The saved model is fully compatible with Hugging Face's transformers library. Learn more about unsloth in their [official repository](https://github.com/unslothai/unsloth).

## Reference model considerations with PEFT

You have three main options (plus several variants) for how the reference model works when using PEFT, assuming the model that you would like to further enhance with DPO was tuned using (Q)LoRA.

1. Simply create two instances of the model, each loading your adapter - works fine but is very inefficient.
2. Merge the adapter into the base model, create another adapter on top, then leave the `ref_model` param null, in which case DPOTrainer will unload the adapter for reference inference - efficient, but has potential downsides discussed below.
3.


key: _node_content
string_value:
{"id_": "fda47293-062a-4fff-be81-47b3ed3fa145", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2d44c2ff-d5ef-4567-ad9e-bb309738b311", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "hash": "c373a01b66f6d8dd0042e7ce819cfc28b3b25495657841605926550d68bd7330", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2e66ffba-672a-4521-ae34-c5cc29e5e490", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "hash": "7cd64588ccdacf377472a510bac96ff14fb87f3d42061aae8a3a53930f15e337", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "80a7228e-6438-4b6d-8a4f-4152ae941757", "node_type": "1", "metadata": {}, "hash": "9b1cd7d75a536a5d920eddefbbc8342059491d138841fb025a1a943dadca574b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14481, "end_char_idx": 17150, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Can be False.
    # token = "hf_...", # use one if using gated models like meta-llama/Llama-2-7b-hf
)

# Do model patching and add fast LoRA weights
model = FastLanguageModel.get_peft_model(
    model,
    r = 16,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj",],
    lora_alpha = 16,
    lora_dropout = 0, # Dropout = 0 is currently optimized
    bias = "none",    # Bias = "none" is currently optimized
    use_gradient_checkpointing = True,
    random_state = 3407,
)

training_args = DPOConfig(
    output_dir="./output",
    beta=0.1,
)

dpo_trainer = DPOTrainer(
    model,
    ref_model=None,
    args=training_args,
    train_dataset=train_dataset,
    tokenizer=tokenizer,
)
dpo_trainer.train()
```

The saved model is fully compatible with Hugging Face's transformers library. Learn more about unsloth in their [official repository](https://github.com/unslothai/unsloth).

## Reference model considerations with PEFT

You have three main options (plus several variants) for how the reference model works when using PEFT, assuming the model that you would like to further enhance with DPO was tuned using (Q)LoRA.

1. Simply create two instances of the model, each loading your adapter - works fine but is very inefficient.
2. Merge the adapter into the base model, create another adapter on top, then leave the `ref_model` param null, in which case DPOTrainer will unload the adapter for reference inference - efficient, but has potential downsides discussed below.
3. Load the adapter twice with different names, then use `set_adapter` during training to swap between the adapter being DPO'd and the reference adapter - slightly less efficient compared to 2 (~adapter size VRAM overhead), but avoids the pitfalls.

### Downsides to merging QLoRA before DPO (approach 2)

As suggested by [Benjamin Marie](https://medium.com/@bnjmn_marie/dont-merge-your-lora-adapter-into-a-4-bit-llm-65b6da287997), the best option for merging QLoRA adapters is to first dequantize the base model, then merge the adapter. Something similar to [this script](https://github.com/jondurbin/qlora/blob/main/qmerge.py).

However, after using this approach, you will have an unquantized base model. Therefore, to use QLoRA for DPO, you will need to re-quantize the merged model or use the unquantized merge (resulting in higher memory demand).

### Using option 3 - load the adapter twice

To avoid the downsides with option 2, you can load your fine-tuned adapter into the model twice, with different names, and set the model/ref adapter names in [`DPOTrainer`].

For example:

```python
# Load the base model.


key: _node_content
string_value:
{"id_": "80a7228e-6438-4b6d-8a4f-4152ae941757", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2d44c2ff-d5ef-4567-ad9e-bb309738b311", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "hash": "c373a01b66f6d8dd0042e7ce819cfc28b3b25495657841605926550d68bd7330", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fda47293-062a-4fff-be81-47b3ed3fa145", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/dpo_trainer", "title": "DPO Trainer", "tokens": 4570, "retrieve_doc": true, "source": "trl"}, "hash": "3cd8f5dd286ffb9f5ce25dce053fba94d925a283fda289f5a738faf9f88591e5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15678, "end_char_idx": 18248, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
1. Simply create two instances of the model, each loading your adapter - works fine but is very inefficient.
2. Merge the adapter into the base model, create another adapter on top, then leave the `ref_model` param null, in which case DPOTrainer will unload the adapter for reference inference - efficient, but has potential downsides discussed below.
3. Load the adapter twice with different names, then use `set_adapter` during training to swap between the adapter being DPO'd and the reference adapter - slightly less efficient compared to 2 (~adapter size VRAM overhead), but avoids the pitfalls.

### Downsides to merging QLoRA before DPO (approach 2)

As suggested by [Benjamin Marie](https://medium.com/@bnjmn_marie/dont-merge-your-lora-adapter-into-a-4-bit-llm-65b6da287997), the best option for merging QLoRA adapters is to first dequantize the base model, then merge the adapter. Something similar to [this script](https://github.com/jondurbin/qlora/blob/main/qmerge.py).

However, after using this approach, you will have an unquantized base model. Therefore, to use QLoRA for DPO, you will need to re-quantize the merged model or use the unquantized merge (resulting in higher memory demand).

### Using option 3 - load the adapter twice

To avoid the downsides with option 2, you can load your fine-tuned adapter into the model twice, with different names, and set the model/ref adapter names in [`DPOTrainer`].

For example:

```python
# Load the base model.
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    llm_int8_threshold=6.0,
    llm_int8_has_fp16_weight=False,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
)
model = AutoModelForCausalLM.from_pretrained(
    "mistralai/mixtral-8x7b-v0.1",
    load_in_4bit=True,
    quantization_config=bnb_config,
    attn_implementation="flash_attention_2",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.config.use_cache = False

# Load the adapter.
model = PeftModel.from_pretrained(
    model,
    "/path/to/peft",
    is_trainable=True,
    adapter_name="train",
)
# Load the adapter a second time, with a different name, which will be our reference model.
model.load_adapter("/path/to/peft", adapter_name="reference")

# Initialize the trainer, without a ref_model param.
training_args = DPOConfig(
    model_adapter_name="train",
    ref_adapter_name="reference",
)
dpo_trainer = DPOTrainer(
    model,
    args=training_args,
    ...
)
```

## DPOTrainer

[[autodoc]] DPOTrainer

## DPOConfig

[[autodoc]] DPOConfig


key: _node_content
string_value:
{"id_": "b7cf1519-3201-4669-bb10-ebc0dcab40fe", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/how_to_train", "title": "Training FAQ", "tokens": 1449, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "74f24977-4b7c-40b9-91e5-d7cf1f262af2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/how_to_train", "title": "Training FAQ", "tokens": 1449, "retrieve_doc": true, "source": "trl"}, "hash": "6ebac2186358630b480c3669534171db736f8a85e7bf06265b73412a24d6d236", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0f531903-5279-40ef-ba18-1ec86b774f91", "node_type": "1", "metadata": {}, "hash": "2ff3ed763d261bc0cd70d608b35e87663ad33bcc6e9fe5a18cb094b56372c19a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3653, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Training FAQ

## What Metrics Should I Look at?

When performing classical supervised fine-tuning of language models, the loss (especially the validation loss) serves as a good indicator of the training progress. However, in Reinforcement Learning (RL), the loss becomes less informative about the model's performance, and its value may fluctuate while the actual performance improves.

To address this, we recommend focusing on two key metrics first:

**Mean Reward**: The primary goal is to maximize the reward achieved by the model during RL training.
**Objective KL Divergence**: KL divergence (Kullback-Leibler divergence) measures the dissimilarity between two probability distributions. In the context of RL training, we use it to quantify the difference between the current model and a reference model. Ideally, we want to keep the KL divergence between 0 and 10 to ensure the model's generated text remains close to what the reference model produces.

However, there are more metrics that can be useful for debugging, checkout the [logging section](logging).

## Why Do We Use a Reference Model, and What's the Purpose of KL Divergence?

When training RL models, optimizing solely for reward may lead to unexpected behaviors, where the model exploits the environment in ways that don't align with good language generation. In the case of RLHF, we use a reward model trained to predict whether a generated text is highly ranked by humans.

However, the RL model being optimized against the reward model may learn patterns that yield high reward but do not represent good language. This can result in extreme cases where the model generates texts with excessive exclamation marks or emojis to maximize the reward. In some worst-case scenarios, the model may generate patterns completely unrelated to natural language yet receive high rewards, similar to adversarial attacks.

<div style="text-align: center">
<img src="https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/kl-example.png">
<p style="text-align: center;"> <b>Figure:</b> Samples without a KL penalty from <a href="https://huggingface.co/papers/1909.08593">https://huggingface.co/papers/1909.08593</a>. </p>
</div>

To address this issue, we add a penalty to the reward function based on the KL divergence between the current model and the reference model. By doing this, we encourage the model to stay close to what the reference model generates.

## What Is the Concern with Negative KL Divergence?

If you generate text by purely sampling from the model distribution things work fine in general. But when you use the `generate` method there are a few caveats because it does not always purely sample depending on the settings which can cause KL-divergence to go negative. Essentially when the active model achieves `log_p_token_active < log_p_token_ref` we get negative KL-div. This can happen in a several cases:

- **top-k sampling**: the model can smooth out the probability distribution causing the top-k tokens having a smaller probability than those of the reference model but they still are selected
- **min_length**: this ignores the EOS token until `min_length` is reached. thus the model can assign a very low log prob to the EOS token and very high probs to all others until min_length is reached

These are just a few examples. Why is negative KL an issue? The total reward `R` is computed `R = r - beta * KL` so if the model can learn how to drive KL-divergence negative it effectively gets a positive reward. In many cases it can be much easier to exploit such a bug in the generation than actually learning the reward function.


key: _node_content
string_value:
{"id_": "0f531903-5279-40ef-ba18-1ec86b774f91", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/how_to_train", "title": "Training FAQ", "tokens": 1449, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "74f24977-4b7c-40b9-91e5-d7cf1f262af2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/how_to_train", "title": "Training FAQ", "tokens": 1449, "retrieve_doc": true, "source": "trl"}, "hash": "6ebac2186358630b480c3669534171db736f8a85e7bf06265b73412a24d6d236", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b7cf1519-3201-4669-bb10-ebc0dcab40fe", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/how_to_train", "title": "Training FAQ", "tokens": 1449, "retrieve_doc": true, "source": "trl"}, "hash": "2f3dcf2a430912397784191bb5e2cb8dadcbbd0d00f5730b7fc235688aa1c0a7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b3caf674-aa69-4efb-9142-72f9e3ff2ed7", "node_type": "1", "metadata": {}, "hash": "3188cb470e03133d100f7841758739587a3d225e8ec70221ae6beec4ce40d0bf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2211, "end_char_idx": 5746, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</p>
</div>

To address this issue, we add a penalty to the reward function based on the KL divergence between the current model and the reference model. By doing this, we encourage the model to stay close to what the reference model generates.

## What Is the Concern with Negative KL Divergence?

If you generate text by purely sampling from the model distribution things work fine in general. But when you use the `generate` method there are a few caveats because it does not always purely sample depending on the settings which can cause KL-divergence to go negative. Essentially when the active model achieves `log_p_token_active < log_p_token_ref` we get negative KL-div. This can happen in a several cases:

- **top-k sampling**: the model can smooth out the probability distribution causing the top-k tokens having a smaller probability than those of the reference model but they still are selected
- **min_length**: this ignores the EOS token until `min_length` is reached. thus the model can assign a very low log prob to the EOS token and very high probs to all others until min_length is reached

These are just a few examples. Why is negative KL an issue? The total reward `R` is computed `R = r - beta * KL` so if the model can learn how to drive KL-divergence negative it effectively gets a positive reward. In many cases it can be much easier to exploit such a bug in the generation than actually learning the reward function. In addition the KL can become arbitrarily small thus the actual reward can be very small compared to it.

So how should you generate text for PPO training? Let's have a look!

## How to generate text for training?

In order to avoid the KL issues described above we recommend to use the following settings:

```python
generation_kwargs = {
    "min_length": -1, # don't ignore the EOS token (see above)
    "top_k": 0.0, # no top-k sampling
    "top_p": 1.0, # no nucleus sampling
    "do_sample": True, # yes, we want to sample
    "pad_token_id": tokenizer.eos_token_id, # most decoder models don't have a padding token - use EOS token instead
    "max_new_tokens": 32, # specify how many tokens you want to generate at most
}
```

With these settings we usually don't encounter any issues. You can also experiments with other settings but if you encounter issues with negative KL-divergence try to go back to these and see if they persist.

## How can debug your own use-case?

Debugging the RL pipeline can be challenging due to its complexity. Here are some tips and suggestions to make the process easier:

- **Start from a working example**: Begin with a working example from the trl repository and gradually modify it to fit your specific use-case. Changing everything at once can make it difficult to identify the source of potential issues. For example, you can start by replacing the model in the example and once you figure out the best hyperparameters try to switch to your dataset and reward model. If you change everything at once you won't know where a potential problem comes from.
- **Start small, scale later**: Training large models can be very slow and take several hours or days until you see any improvement. For debugging this is not a convenient timescale so try to use small model variants during the development phase and scale up once that works. That being said you sometimes have to be careful as small models might not have the capacity to solve a complicated task either.
- **Start simple**: Try to start with a minimal example and build complexity from there.


key: _node_content
string_value:
{"id_": "b3caf674-aa69-4efb-9142-72f9e3ff2ed7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/how_to_train", "title": "Training FAQ", "tokens": 1449, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "74f24977-4b7c-40b9-91e5-d7cf1f262af2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/how_to_train", "title": "Training FAQ", "tokens": 1449, "retrieve_doc": true, "source": "trl"}, "hash": "6ebac2186358630b480c3669534171db736f8a85e7bf06265b73412a24d6d236", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0f531903-5279-40ef-ba18-1ec86b774f91", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/how_to_train", "title": "Training FAQ", "tokens": 1449, "retrieve_doc": true, "source": "trl"}, "hash": "9f249c02283505b22b134f7d0581ccc88d22f8722150f29f97ce6e00971837aa", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4447, "end_char_idx": 6922, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can also experiments with other settings but if you encounter issues with negative KL-divergence try to go back to these and see if they persist.

## How can debug your own use-case?

Debugging the RL pipeline can be challenging due to its complexity. Here are some tips and suggestions to make the process easier:

- **Start from a working example**: Begin with a working example from the trl repository and gradually modify it to fit your specific use-case. Changing everything at once can make it difficult to identify the source of potential issues. For example, you can start by replacing the model in the example and once you figure out the best hyperparameters try to switch to your dataset and reward model. If you change everything at once you won't know where a potential problem comes from.
- **Start small, scale later**: Training large models can be very slow and take several hours or days until you see any improvement. For debugging this is not a convenient timescale so try to use small model variants during the development phase and scale up once that works. That being said you sometimes have to be careful as small models might not have the capacity to solve a complicated task either.
- **Start simple**: Try to start with a minimal example and build complexity from there. Your use-case might require for example a complicated reward function consisting of many different rewards - try to use one signal first and see if you can optimize that and then add more complexity after that.
- **Inspect the generations**: It's always a good idea to inspect what the model is generating. Maybe there is a bug in your post-processing or your prompt. Due to bad settings you might cut-off generations too soon. These things are very hard to see on the metrics but very obvious if you look at the generations.
- **Inspect the reward model**: If you reward is not improving over time maybe there's an issue with the reward model. You can look at extreme cases to see if it does what it should: e.g. in the sentiment case you can check if simple positive and negative examples really get different rewards. And you can look at the distribution of your dataset. Finally, maybe the reward is dominated by the query which the model can't affect so you might need to normalize this (e.g. reward of query+response minus reward of the query).

These are just a few tips that we find helpful - if you have more useful tricks feel free to open a PR to add them as well!


key: _node_content
string_value:
{"id_": "3843dcd8-b4d8-4e66-ad08-e458c86cf6a2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/ddpo_trainer", "title": "Denoising Diffusion Policy Optimization", "tokens": 1924, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e5bce9-55ed-41e1-8560-cc983baf3e9f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/ddpo_trainer", "title": "Denoising Diffusion Policy Optimization", "tokens": 1924, "retrieve_doc": true, "source": "trl"}, "hash": "b0634b27587b9aebba82d04ac07e3dc86d3aadb8d744491d5e0c1e44640d14c3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1fcb2598-9dbd-4487-b72e-00c275088ae3", "node_type": "1", "metadata": {}, "hash": "65e56be18eb8c8728163c2ae6f01d6205442084a879aea958e2e7b543d6de1d1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3533, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Denoising Diffusion Policy Optimization
## The why

| Before | After DDPO finetuning |
| --- | --- |
| <div style="text-align: center"><img src="https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/pre_squirrel.png"/></div> |  <div style="text-align: center"><img src="https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/post_squirrel.png"/></div> |
| <div style="text-align: center"><img src="https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/pre_crab.png"/></div> |  <div style="text-align: center"><img src="https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/post_crab.png"/></div> |
|  <div style="text-align: center"><img src="https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/pre_starfish.png"/></div> |  <div style="text-align: center"><img src="https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/post_starfish.png"/></div> |


## Getting started with Stable Diffusion finetuning with reinforcement learning

The machinery for finetuning of Stable Diffusion models with reinforcement learning makes heavy use of HuggingFace's `diffusers`
library. A reason for  stating this is that getting started requires a bit of familiarity with the `diffusers` library concepts, mainly two of them - pipelines and schedulers.
Right out of the box (`diffusers` library), there isn't a `Pipeline` nor a `Scheduler` instance that is suitable for finetuning with reinforcement learning. Some adjustments need to made. 

There is a pipeline interface that is provided by this library that is required to be implemented to be used with the `DDPOTrainer`, which is the main machinery for fine-tuning Stable Diffusion with reinforcement learning. **Note: Only the StableDiffusion architecture is supported at this point.**
There is a default implementation of this interface that you can use out of the box. Assuming the default implementation is sufficient and/or to get things moving, refer to the training example alongside this guide. 

The point of the interface is to fuse the pipeline and the scheduler into one object which allows for minimalness in terms of having the constraints all in one place. The interface was designed in hopes of catering to pipelines and schedulers beyond the examples in this repository and elsewhere at this time of writing. Also the scheduler step is a method of this pipeline interface and this may seem redundant given that the raw scheduler is accessible via the interface but this is the only way to constrain the scheduler step output to an output type befitting of the algorithm at hand (DDPO).

For a more detailed look into the interface and the associated default implementation, go [here](https://github.com/lvwerra/trl/tree/main/trl/models/modeling_sd_base.py)

Note that the default implementation has a LoRA implementation path and a non-LoRA based implementation path. The LoRA flag enabled by default and this can be turned off by passing in the flag to do so. LORA based training is faster and the LORA associated model hyperparameters responsible for model convergence aren't as finicky as non-LORA based training.

Also in addition, there is the expectation of providing a reward function and a prompt function. The reward function is used to evaluate the generated images  and the prompt function is used to generate the prompts that are used to generate the images.


key: _node_content
string_value:
{"id_": "1fcb2598-9dbd-4487-b72e-00c275088ae3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/ddpo_trainer", "title": "Denoising Diffusion Policy Optimization", "tokens": 1924, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e5bce9-55ed-41e1-8560-cc983baf3e9f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/ddpo_trainer", "title": "Denoising Diffusion Policy Optimization", "tokens": 1924, "retrieve_doc": true, "source": "trl"}, "hash": "b0634b27587b9aebba82d04ac07e3dc86d3aadb8d744491d5e0c1e44640d14c3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3843dcd8-b4d8-4e66-ad08-e458c86cf6a2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/ddpo_trainer", "title": "Denoising Diffusion Policy Optimization", "tokens": 1924, "retrieve_doc": true, "source": "trl"}, "hash": "c681e12a96ae01291fe53f9ee419d10f514490074d00ed510ad4d5d4de21a2eb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "aaeeb883-dc2d-4bda-a1c7-9de085a83a20", "node_type": "1", "metadata": {}, "hash": "7a0efa8b8570098adcfd37838526041a85b483fd033cfb26913426ea19d94b6f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1601, "end_char_idx": 4387, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Some adjustments need to made. 

There is a pipeline interface that is provided by this library that is required to be implemented to be used with the `DDPOTrainer`, which is the main machinery for fine-tuning Stable Diffusion with reinforcement learning. **Note: Only the StableDiffusion architecture is supported at this point.**
There is a default implementation of this interface that you can use out of the box. Assuming the default implementation is sufficient and/or to get things moving, refer to the training example alongside this guide. 

The point of the interface is to fuse the pipeline and the scheduler into one object which allows for minimalness in terms of having the constraints all in one place. The interface was designed in hopes of catering to pipelines and schedulers beyond the examples in this repository and elsewhere at this time of writing. Also the scheduler step is a method of this pipeline interface and this may seem redundant given that the raw scheduler is accessible via the interface but this is the only way to constrain the scheduler step output to an output type befitting of the algorithm at hand (DDPO).

For a more detailed look into the interface and the associated default implementation, go [here](https://github.com/lvwerra/trl/tree/main/trl/models/modeling_sd_base.py)

Note that the default implementation has a LoRA implementation path and a non-LoRA based implementation path. The LoRA flag enabled by default and this can be turned off by passing in the flag to do so. LORA based training is faster and the LORA associated model hyperparameters responsible for model convergence aren't as finicky as non-LORA based training.

Also in addition, there is the expectation of providing a reward function and a prompt function. The reward function is used to evaluate the generated images  and the prompt function is used to generate the prompts that are used to generate the images.

## Getting started with `examples/scripts/ddpo.py`

The `ddpo.py` script is a working example of using the `DDPO` trainer to finetune a Stable Diffusion model. This example explicitly configures a small subset of the overall parameters associated with the config object (`DDPOConfig`).

**Note:** one A100 GPU is recommended to get this running. Anything below a A100 will not be able to run this example script and even if it does via relatively smaller sized parameters, the results will most likely be poor.

Almost every configuration parameter has a default. There is only one commandline flag argument that is required of the user to get things up and running. The user is expected to have a [huggingface user access token](https://huggingface.co/docs/hub/security-tokens) that will be used to upload the model post finetuning to HuggingFace hub.


key: _node_content
string_value:
{"id_": "aaeeb883-dc2d-4bda-a1c7-9de085a83a20", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/ddpo_trainer", "title": "Denoising Diffusion Policy Optimization", "tokens": 1924, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e5bce9-55ed-41e1-8560-cc983baf3e9f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/ddpo_trainer", "title": "Denoising Diffusion Policy Optimization", "tokens": 1924, "retrieve_doc": true, "source": "trl"}, "hash": "b0634b27587b9aebba82d04ac07e3dc86d3aadb8d744491d5e0c1e44640d14c3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1fcb2598-9dbd-4487-b72e-00c275088ae3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/ddpo_trainer", "title": "Denoising Diffusion Policy Optimization", "tokens": 1924, "retrieve_doc": true, "source": "trl"}, "hash": "a0457c9eb2cc0e8f74a6f09f476f94cdd055d5add3fd2a2a42fa4fb08b1423c3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d1d404a4-7638-490f-a840-476202080415", "node_type": "1", "metadata": {}, "hash": "d3277d8f7da1d77ed4893ab8eb3e1225af4ccf9db0d2612e44555f83658cd032", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2750, "end_char_idx": 6334, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For a more detailed look into the interface and the associated default implementation, go [here](https://github.com/lvwerra/trl/tree/main/trl/models/modeling_sd_base.py)

Note that the default implementation has a LoRA implementation path and a non-LoRA based implementation path. The LoRA flag enabled by default and this can be turned off by passing in the flag to do so. LORA based training is faster and the LORA associated model hyperparameters responsible for model convergence aren't as finicky as non-LORA based training.

Also in addition, there is the expectation of providing a reward function and a prompt function. The reward function is used to evaluate the generated images  and the prompt function is used to generate the prompts that are used to generate the images.

## Getting started with `examples/scripts/ddpo.py`

The `ddpo.py` script is a working example of using the `DDPO` trainer to finetune a Stable Diffusion model. This example explicitly configures a small subset of the overall parameters associated with the config object (`DDPOConfig`).

**Note:** one A100 GPU is recommended to get this running. Anything below a A100 will not be able to run this example script and even if it does via relatively smaller sized parameters, the results will most likely be poor.

Almost every configuration parameter has a default. There is only one commandline flag argument that is required of the user to get things up and running. The user is expected to have a [huggingface user access token](https://huggingface.co/docs/hub/security-tokens) that will be used to upload the model post finetuning to HuggingFace hub. The following bash command is to be entered to get things running

```batch
python ddpo.py --hf_user_access_token <token>
```

To obtain the documentation of `stable_diffusion_tuning.py`, please run `python stable_diffusion_tuning.py --help`

The following are things to keep in mind (The code checks this for you as well) in general while configuring the trainer (beyond the use case of using the example script)

- The configurable sample batch size (`--ddpo_config.sample_batch_size=6`) should be greater than or equal to the configurable training batch size (`--ddpo_config.train_batch_size=3`)
- The configurable sample batch size (`--ddpo_config.sample_batch_size=6`) must be divisible by the configurable train batch size (`--ddpo_config.train_batch_size=3`)
- The configurable sample batch size (`--ddpo_config.sample_batch_size=6`) must be divisible by both the configurable gradient accumulation steps (`--ddpo_config.train_gradient_accumulation_steps=1`) and the configurable accelerator processes count 

## Setting up the image logging hook function

Expect the function to be given a list of lists of the form
```python
[[image, prompt, prompt_metadata, rewards, reward_metadata], ...]

```
and `image`, `prompt`, `prompt_metadata`, `rewards`, `reward_metadata` are batched.
The last list in the lists of lists represents the last sample batch. You are likely to want to log this one
While you are free to log however you want the use of `wandb` or `tensorboard` is recommended.

### Key terms

- `rewards` : The rewards/score is a numerical associated with the generated image and is key to steering the RL process
- `reward_metadata` : The reward metadata is the metadata associated with the reward. Think of this as extra information payload delivered alongside the reward
- `prompt` : The prompt is the text that is used to generate the image
- `prompt_metadata` : The prompt metadata is the metadata associated with the prompt.


key: _node_content
string_value:
{"id_": "d1d404a4-7638-490f-a840-476202080415", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/ddpo_trainer", "title": "Denoising Diffusion Policy Optimization", "tokens": 1924, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99e5bce9-55ed-41e1-8560-cc983baf3e9f", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/ddpo_trainer", "title": "Denoising Diffusion Policy Optimization", "tokens": 1924, "retrieve_doc": true, "source": "trl"}, "hash": "b0634b27587b9aebba82d04ac07e3dc86d3aadb8d744491d5e0c1e44640d14c3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "aaeeb883-dc2d-4bda-a1c7-9de085a83a20", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/ddpo_trainer", "title": "Denoising Diffusion Policy Optimization", "tokens": 1924, "retrieve_doc": true, "source": "trl"}, "hash": "8eb496b4b1fe88aa1ac2a81131ad36c47cb534dda0c56af5a10af5cf401f05b1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5677, "end_char_idx": 8514, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The last list in the lists of lists represents the last sample batch. You are likely to want to log this one
While you are free to log however you want the use of `wandb` or `tensorboard` is recommended.

### Key terms

- `rewards` : The rewards/score is a numerical associated with the generated image and is key to steering the RL process
- `reward_metadata` : The reward metadata is the metadata associated with the reward. Think of this as extra information payload delivered alongside the reward
- `prompt` : The prompt is the text that is used to generate the image
- `prompt_metadata` : The prompt metadata is the metadata associated with the prompt. A situation where this will not be empty is when the reward model comprises of a [`FLAVA`](https://huggingface.co/docs/transformers/model_doc/flava) setup where questions and ground answers (linked to the generated image) are expected with the generated image (See here: https://github.com/kvablack/ddpo-pytorch/blob/main/ddpo_pytorch/rewards.py#L45)
- `image` : The image generated by the Stable Diffusion model

Example code for logging sampled images with `wandb` is given below.

```python
# for logging these images to wandb

def image_outputs_hook(image_data, global_step, accelerate_logger):
    # For the sake of this example, we only care about the last batch
    # hence we extract the last element of the list
    result = {}
    images, prompts, _, rewards, _ = image_data[-1]
    for i, image in enumerate(images):
        pil = Image.fromarray(
            (image.cpu().numpy().transpose(1, 2, 0) * 255).astype(np.uint8)
        )
        pil = pil.resize((256, 256))
        result[f"{prompts[i]:.25} | {rewards[i]:.2f}"] = [pil]
    accelerate_logger.log_images(
        result,
        step=global_step,
    )

```

### Using the finetuned model

Assuming you've done with all the epochs and have pushed up your model to the hub, you can use the finetuned model as follows

```python

import torch
from trl import DefaultDDPOStableDiffusionPipeline

pipeline = DefaultDDPOStableDiffusionPipeline("metric-space/ddpo-finetuned-sd-model")

device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")

# memory optimization
pipeline.vae.to(device, torch.float16)
pipeline.text_encoder.to(device, torch.float16)
pipeline.unet.to(device, torch.float16)

prompts = ["squirrel", "crab", "starfish", "whale","sponge", "plankton"]
results = pipeline(prompts)

for prompt, image in zip(prompts,results.images):
    image.save(f"{prompt}.png")

```

## Credits

This work is heavily influenced by the repo [here](https://github.com/kvablack/ddpo-pytorch) and the associated paper [Training Diffusion Models
with Reinforcement Learning by Kevin Black, Michael Janner, Yilan Du, Ilya Kostrikov, Sergey Levine](https://huggingface.co/papers/2305.13301).


key: _node_content
string_value:
{"id_": "7a1029b0-cbdb-4daa-b83c-90ec8df17269", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/learning_tools", "title": "Learning Tools (Experimental \ud83e\uddea)", "tokens": 3184, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2831b0af-5745-4c78-b3ea-4ac5967d9d4c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/learning_tools", "title": "Learning Tools (Experimental \ud83e\uddea)", "tokens": 3184, "retrieve_doc": true, "source": "trl"}, "hash": "add7c31d2128098622a1fde39ebd09c9501bf5e0f8bb4fb626ae0619ea2aac21", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "de5f4bb2-1631-47c0-a0c5-c53d477a6a05", "node_type": "1", "metadata": {}, "hash": "8f70e66203c4f55e2d6ccad5b507d3e450f92f044fbb3136637ee0b28ceaf6d9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2901, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Learning Tools (Experimental 🧪)

Using Large Language Models (LLMs) with tools has been a popular topic recently with awesome works such as [ToolFormer](https://huggingface.co/papers/2302.04761) and [ToolBench](https://huggingface.co/papers/2305.16504). In TRL, we provide a simple example of how to teach LLM to use tools with reinforcement learning. 


Here's an overview of the scripts in the [trl repository](https://github.com/lvwerra/trl/tree/main/examples/research_projects/tools):

| File | Description | 
|---|---| 
| [`calculator.py`](https://github.com/lvwerra/trl/blob/main/examples/research_projects/tools/calculator.py) | Script to train LLM to use a calculator with reinforcement learning. |
| [`triviaqa.py`](https://github.com/lvwerra/trl/blob/main/examples/research_projects/tools/triviaqa.py) | Script to train LLM to use a wiki tool to answer questions. |
| [`python_interpreter.py`](https://github.com/lvwerra/trl/blob/main/examples/research_projects/tools/python_interpreter.py) | Script to train LLM to use python interpreter to solve math puzzles. |

<Tip warning={true}>

Note that the scripts above rely heavily on the `TextEnvironment` API which is still under active development. The API may change in the future. Please see [`TextEnvironment`](text_environment) for the related docs.
</Tip>


## Learning to Use a Calculator


The rough idea is as follows:

1. Load a tool such as [ybelkada/simple-calculator](https://huggingface.co/spaces/ybelkada/simple-calculator) that parse a text calculation like `"14 + 34"` and return the calulated number:
    ```python
    from transformers import AutoTokenizer, load_tool
    tool = load_tool("ybelkada/simple-calculator")
    tool_fn = lambda text: str(round(float(tool(text)), 2))  # rounding to 2 decimal places
    ```
1. Define a reward function that returns a positive reward if the tool returns the correct answer. In the script we create a dummy reward function like `reward_fn = lambda x: 1`, but we override the rewards directly later.
1. Create a prompt on how to use the tools
    ```python
    # system prompt
    prompt = """\
    What is 13.1-3?

    <request><SimpleCalculatorTool>13.1-3<call>10.1<response>

    Result=10.1<submit>

    What is 4*3?

    <request><SimpleCalculatorTool>4*3<call>12<response>

    Result=12<submit>

    What is 12.1+1?

    <request><SimpleCalculatorTool>12.1+1<call>13.1<response>

    Result=13.1<submit>

    What is 12.1-20?

    <request><SimpleCalculatorTool>12.1-20<call>-7.9<response>

    Result=-7.9<submit>"""
    ```
3. Create a `trl.TextEnvironment` with the model 
    ```python
    env = TextEnvironment(
        model,
        tokenizer,
        {"SimpleCalculatorTool": tool_fn},
        reward_fn,
        prompt,
        generation_kwargs=generation_kwargs,
    )
    ```
4. Then generate some data such as `tasks = ["\n\nWhat is 13.1-3?", "\n\nWhat is 4*3?


key: _node_content
string_value:
{"id_": "de5f4bb2-1631-47c0-a0c5-c53d477a6a05", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/learning_tools", "title": "Learning Tools (Experimental \ud83e\uddea)", "tokens": 3184, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2831b0af-5745-4c78-b3ea-4ac5967d9d4c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/learning_tools", "title": "Learning Tools (Experimental \ud83e\uddea)", "tokens": 3184, "retrieve_doc": true, "source": "trl"}, "hash": "add7c31d2128098622a1fde39ebd09c9501bf5e0f8bb4fb626ae0619ea2aac21", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7a1029b0-cbdb-4daa-b83c-90ec8df17269", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/learning_tools", "title": "Learning Tools (Experimental \ud83e\uddea)", "tokens": 3184, "retrieve_doc": true, "source": "trl"}, "hash": "c0190615f5aa7707d6a693419a261d1156c2c5497e0f9a6dda17a4b2a6f401ba", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "edb10951-3881-4ffb-93fe-9171d77bd9be", "node_type": "1", "metadata": {}, "hash": "b3b4053327fe804fe2a5f8ce897afbeaf26c2c5ae167e3cefc10d664e4dcfec1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1801, "end_char_idx": 4456, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Define a reward function that returns a positive reward if the tool returns the correct answer. In the script we create a dummy reward function like `reward_fn = lambda x: 1`, but we override the rewards directly later.
1. Create a prompt on how to use the tools
    ```python
    # system prompt
    prompt = """\
    What is 13.1-3?

    <request><SimpleCalculatorTool>13.1-3<call>10.1<response>

    Result=10.1<submit>

    What is 4*3?

    <request><SimpleCalculatorTool>4*3<call>12<response>

    Result=12<submit>

    What is 12.1+1?

    <request><SimpleCalculatorTool>12.1+1<call>13.1<response>

    Result=13.1<submit>

    What is 12.1-20?

    <request><SimpleCalculatorTool>12.1-20<call>-7.9<response>

    Result=-7.9<submit>"""
    ```
3. Create a `trl.TextEnvironment` with the model 
    ```python
    env = TextEnvironment(
        model,
        tokenizer,
        {"SimpleCalculatorTool": tool_fn},
        reward_fn,
        prompt,
        generation_kwargs=generation_kwargs,
    )
    ```
4. Then generate some data such as `tasks = ["\n\nWhat is 13.1-3?", "\n\nWhat is 4*3?"]` and run the environment with `queries, responses, masks, rewards, histories = env.run(tasks)`. The environment will look for the `<call>` token in the prompt and append the tool output to the response; it will also return the mask associated with the response. You can further use the `histories` to visualize the interaction between the model and the tool; `histories[0].show_text()` will show the text with color-coded tool output and `histories[0].show_tokens(tokenizer)` will show visualize the tokens.
    ![](https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/learning_tools.png)
1. Finally, we can train the model with `train_stats = ppo_trainer.step(queries, responses, rewards, masks)`. The trainer will use the mask to ignore the tool output when computing the loss, make sure to pass that argument to `step`.

## Experiment results

We trained a model with the above script for 10 random seeds. You can reproduce the run with the following command. Feel free to remove the `--slurm-*` arguments if you don't have access to a slurm cluster.

```
WANDB_TAGS="calculator_final" python benchmark/benchmark.py \
    --command "python examples/research_projects/tools/calculator.py" \
    --num-seeds 10 \
    --start-seed 1 \
    --workers 10 \
    --slurm-gpus-per-task 1 \
    --slurm-ntasks 1 \
    --slurm-total-cpus 8 \
    --slurm-template-path benchmark/trl.slurm_template
```

We can then use [`openrlbenchmark`](https://github.com/openrlbenchmark/openrlbenchmark) which generates the following plot.


key: _node_content
string_value:
{"id_": "edb10951-3881-4ffb-93fe-9171d77bd9be", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/learning_tools", "title": "Learning Tools (Experimental \ud83e\uddea)", "tokens": 3184, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2831b0af-5745-4c78-b3ea-4ac5967d9d4c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/learning_tools", "title": "Learning Tools (Experimental \ud83e\uddea)", "tokens": 3184, "retrieve_doc": true, "source": "trl"}, "hash": "add7c31d2128098622a1fde39ebd09c9501bf5e0f8bb4fb626ae0619ea2aac21", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "de5f4bb2-1631-47c0-a0c5-c53d477a6a05", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/learning_tools", "title": "Learning Tools (Experimental \ud83e\uddea)", "tokens": 3184, "retrieve_doc": true, "source": "trl"}, "hash": "57ec603ce3562e21eca1dafee59aa3a9c37c00050b932b09b8e37bf15db5a62a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fb852ffa-77c4-45ab-bdd7-1172f5e2c382", "node_type": "1", "metadata": {}, "hash": "8cc2eeed4f8793afccbe0f141d73bd25a6c9ac3de160d49414c7333338bdbac2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2881, "end_char_idx": 5886, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
", "\n\nWhat is 4*3?"]` and run the environment with `queries, responses, masks, rewards, histories = env.run(tasks)`. The environment will look for the `<call>` token in the prompt and append the tool output to the response; it will also return the mask associated with the response. You can further use the `histories` to visualize the interaction between the model and the tool; `histories[0].show_text()` will show the text with color-coded tool output and `histories[0].show_tokens(tokenizer)` will show visualize the tokens.
    ![](https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/learning_tools.png)
1. Finally, we can train the model with `train_stats = ppo_trainer.step(queries, responses, rewards, masks)`. The trainer will use the mask to ignore the tool output when computing the loss, make sure to pass that argument to `step`.

## Experiment results

We trained a model with the above script for 10 random seeds. You can reproduce the run with the following command. Feel free to remove the `--slurm-*` arguments if you don't have access to a slurm cluster.

```
WANDB_TAGS="calculator_final" python benchmark/benchmark.py \
    --command "python examples/research_projects/tools/calculator.py" \
    --num-seeds 10 \
    --start-seed 1 \
    --workers 10 \
    --slurm-gpus-per-task 1 \
    --slurm-ntasks 1 \
    --slurm-total-cpus 8 \
    --slurm-template-path benchmark/trl.slurm_template
```

We can then use [`openrlbenchmark`](https://github.com/openrlbenchmark/openrlbenchmark) which generates the following plot.
```
python -m openrlbenchmark.rlops_multi_metrics \
    --filters '?we=openrlbenchmark&wpn=trl&xaxis=_step&ceik=trl_ppo_trainer_config.value.tracker_project_name&cen=trl_ppo_trainer_config.value.log_with&metrics=env/reward_mean&metrics=objective/kl' \
        'wandb?tag=calculator_final&cl=calculator_mask' \
    --env-ids trl \
    --check-empty-runs \
    --pc.ncols 2 \
    --pc.ncols-legend 1 \
    --output-filename static/0compare \
    --scan-history
```

![](https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/learning_tools_chart.png)

As we can see, while 1-2 experiments crashed for some reason, most of the runs obtained near perfect proficiency in the calculator task.


## (Early Experiments 🧪): learning to use a wiki tool for question answering

In the [ToolFormer](https://huggingface.co/papers/2302.04761) paper, it shows an interesting use case that utilizes a Wikipedia Search tool to help answer questions. In this section, we attempt to perform similar experiments but uses RL instead to teach the model to use a wiki tool on the [TriviaQA](https://nlp.cs.washington.edu/triviaqa/) dataset.


<Tip warning={true}>

**Note that many settings are different so the results are not directly comparable.**
</Tip>




### Building a search index

Since [ToolFormer](https://huggingface.co/papers/2302.04761) did not open source, we needed to first replicate the search index.


key: _node_content
string_value:
{"id_": "fb852ffa-77c4-45ab-bdd7-1172f5e2c382", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/learning_tools", "title": "Learning Tools (Experimental \ud83e\uddea)", "tokens": 3184, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2831b0af-5745-4c78-b3ea-4ac5967d9d4c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/learning_tools", "title": "Learning Tools (Experimental \ud83e\uddea)", "tokens": 3184, "retrieve_doc": true, "source": "trl"}, "hash": "add7c31d2128098622a1fde39ebd09c9501bf5e0f8bb4fb626ae0619ea2aac21", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "edb10951-3881-4ffb-93fe-9171d77bd9be", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/learning_tools", "title": "Learning Tools (Experimental \ud83e\uddea)", "tokens": 3184, "retrieve_doc": true, "source": "trl"}, "hash": "ff0fa39dcb98df385ac1342432ce6a8a5e0f96e1f3d334e586c6bc54b2815199", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6f3ca07d-54c6-4969-ad96-882f684946d1", "node_type": "1", "metadata": {}, "hash": "f231ea22bd0e80c9cd33cb7f7b42805ec9332b189a8628d103cd6aee4eec512f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4457, "end_char_idx": 7723, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```
python -m openrlbenchmark.rlops_multi_metrics \
    --filters '?we=openrlbenchmark&wpn=trl&xaxis=_step&ceik=trl_ppo_trainer_config.value.tracker_project_name&cen=trl_ppo_trainer_config.value.log_with&metrics=env/reward_mean&metrics=objective/kl' \
        'wandb?tag=calculator_final&cl=calculator_mask' \
    --env-ids trl \
    --check-empty-runs \
    --pc.ncols 2 \
    --pc.ncols-legend 1 \
    --output-filename static/0compare \
    --scan-history
```

![](https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/learning_tools_chart.png)

As we can see, while 1-2 experiments crashed for some reason, most of the runs obtained near perfect proficiency in the calculator task.


## (Early Experiments 🧪): learning to use a wiki tool for question answering

In the [ToolFormer](https://huggingface.co/papers/2302.04761) paper, it shows an interesting use case that utilizes a Wikipedia Search tool to help answer questions. In this section, we attempt to perform similar experiments but uses RL instead to teach the model to use a wiki tool on the [TriviaQA](https://nlp.cs.washington.edu/triviaqa/) dataset.


<Tip warning={true}>

**Note that many settings are different so the results are not directly comparable.**
</Tip>




### Building a search index

Since [ToolFormer](https://huggingface.co/papers/2302.04761) did not open source, we needed to first replicate the search index. It is mentioned in their paper that the authors built the search index using a BM25 retriever that indexes the Wikipedia dump from [KILT](https://github.com/facebookresearch/KILT)

Fortunately, [`pyserini`](https://github.com/castorini/pyserini) already implements the BM25 retriever and provides a prebuilt index for the KILT Wikipedia dump. We can use the following code to search the index.

```python
from pyserini.search.lucene import LuceneSearcher
import json
searcher = LuceneSearcher.from_prebuilt_index('wikipedia-kilt-doc')
def search(query):
    hits = searcher.search(query, k=1)
    hit = hits[0]
    contents = json.loads(hit.raw)['contents']
    return contents
print(search("tennis racket"))
```
```
Racket (sports equipment)
A racket or racquet is a sports implement consisting of a handled frame with an open hoop across which a network of strings or catgut is stretched tightly. It is used for striking a ball or shuttlecock in games such as squash, tennis, racquetball, and badminton. Collectively, these games are known as racket sports. Racket design and manufacturing has changed considerably over the centuries.

The frame of rackets for all sports was traditionally made of solid wood (later laminated wood) and the strings of animal intestine known as catgut. The traditional racket size was limited by the strength and weight of the wooden frame which had to be strong enough to hold the strings and stiff enough to hit the ball or shuttle. Manufacturers started adding non-wood laminates to wood rackets to improve stiffness. Non-wood rackets were made first of steel, then of aluminum, and then carbon fiber composites. Wood is still used for real tennis, rackets, and xare. Most rackets are now made of composite materials including carbon fiber or fiberglass, metals such as titanium alloys, or ceramics.


key: _node_content
string_value:
{"id_": "6f3ca07d-54c6-4969-ad96-882f684946d1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/learning_tools", "title": "Learning Tools (Experimental \ud83e\uddea)", "tokens": 3184, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2831b0af-5745-4c78-b3ea-4ac5967d9d4c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/learning_tools", "title": "Learning Tools (Experimental \ud83e\uddea)", "tokens": 3184, "retrieve_doc": true, "source": "trl"}, "hash": "add7c31d2128098622a1fde39ebd09c9501bf5e0f8bb4fb626ae0619ea2aac21", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fb852ffa-77c4-45ab-bdd7-1172f5e2c382", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/learning_tools", "title": "Learning Tools (Experimental \ud83e\uddea)", "tokens": 3184, "retrieve_doc": true, "source": "trl"}, "hash": "4f9c40eec3f92adc63b369cdfd26d130e9771c688ce0e69c32faf0895316b0c5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5ecdf0a1-80be-4ce6-aa3b-b8331993d9a9", "node_type": "1", "metadata": {}, "hash": "55407e0b5cee7c6ce310ba81fc6e956cdc56fbcbfb362ff8b5b384ae9758fbd4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6230, "end_char_idx": 9273, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We can use the following code to search the index.

```python
from pyserini.search.lucene import LuceneSearcher
import json
searcher = LuceneSearcher.from_prebuilt_index('wikipedia-kilt-doc')
def search(query):
    hits = searcher.search(query, k=1)
    hit = hits[0]
    contents = json.loads(hit.raw)['contents']
    return contents
print(search("tennis racket"))
```
```
Racket (sports equipment)
A racket or racquet is a sports implement consisting of a handled frame with an open hoop across which a network of strings or catgut is stretched tightly. It is used for striking a ball or shuttlecock in games such as squash, tennis, racquetball, and badminton. Collectively, these games are known as racket sports. Racket design and manufacturing has changed considerably over the centuries.

The frame of rackets for all sports was traditionally made of solid wood (later laminated wood) and the strings of animal intestine known as catgut. The traditional racket size was limited by the strength and weight of the wooden frame which had to be strong enough to hold the strings and stiff enough to hit the ball or shuttle. Manufacturers started adding non-wood laminates to wood rackets to improve stiffness. Non-wood rackets were made first of steel, then of aluminum, and then carbon fiber composites. Wood is still used for real tennis, rackets, and xare. Most rackets are now made of composite materials including carbon fiber or fiberglass, metals such as titanium alloys, or ceramics.
...
```

We then basically deployed this snippet as a Hugging Face space [here](https://huggingface.co/spaces/vwxyzjn/pyserini-wikipedia-kilt-doc), so that we can use the space as a `transformers.Tool` later.

![](https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/pyserini.png)

### Experiment settings

We use the following settings:

* use the `bigcode/starcoderbase` model as the base model
* use the `pyserini-wikipedia-kilt-doc` space as the wiki tool and only uses the first paragrahs of the search result, allowing the `TextEnvironment` to obtain at most `max_tool_reponse=400` response tokens from the tool.
* test if the response contain the answer string, if so, give a reward of 1, otherwise, give a reward of 0.
    * notice this is a simplified evaluation criteria. In [ToolFormer](https://huggingface.co/papers/2302.04761), the authors checks if the first 20 words of the response contain the correct answer.
* used the following prompt that demonstrates the usage of the wiki tool.
```python
prompt = """\
Answer the following question:

Q: In which branch of the arts is Patricia Neary famous?
A: Ballets
A2: <request><Wiki>Patricia Neary<call>Patricia Neary (born October 27, 1942) is an American ballerina, choreographer and ballet director, who has been particularly active in Switzerland. She has also been a highly successful ambassador for the Balanchine Trust, bringing George Balanchine's ballets to 60 cities around the globe.<response>
Result=Ballets<submit>

Q: Who won Super Bowl XX?


key: _node_content
string_value:
{"id_": "5ecdf0a1-80be-4ce6-aa3b-b8331993d9a9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/learning_tools", "title": "Learning Tools (Experimental \ud83e\uddea)", "tokens": 3184, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2831b0af-5745-4c78-b3ea-4ac5967d9d4c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/learning_tools", "title": "Learning Tools (Experimental \ud83e\uddea)", "tokens": 3184, "retrieve_doc": true, "source": "trl"}, "hash": "add7c31d2128098622a1fde39ebd09c9501bf5e0f8bb4fb626ae0619ea2aac21", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6f3ca07d-54c6-4969-ad96-882f684946d1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/learning_tools", "title": "Learning Tools (Experimental \ud83e\uddea)", "tokens": 3184, "retrieve_doc": true, "source": "trl"}, "hash": "57024b26e3dd167952aec84abe5642db2c914b963ba35c05249bfc317493e14d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ce3d8779-7e92-4ed4-a0d7-da67bd20f093", "node_type": "1", "metadata": {}, "hash": "e6c11774c22a0a20eb72eec2ab2a3b95ea90e5c6392755b85cf818cf15920cd2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7724, "end_char_idx": 10387, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
...
```

We then basically deployed this snippet as a Hugging Face space [here](https://huggingface.co/spaces/vwxyzjn/pyserini-wikipedia-kilt-doc), so that we can use the space as a `transformers.Tool` later.

![](https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/pyserini.png)

### Experiment settings

We use the following settings:

* use the `bigcode/starcoderbase` model as the base model
* use the `pyserini-wikipedia-kilt-doc` space as the wiki tool and only uses the first paragrahs of the search result, allowing the `TextEnvironment` to obtain at most `max_tool_reponse=400` response tokens from the tool.
* test if the response contain the answer string, if so, give a reward of 1, otherwise, give a reward of 0.
    * notice this is a simplified evaluation criteria. In [ToolFormer](https://huggingface.co/papers/2302.04761), the authors checks if the first 20 words of the response contain the correct answer.
* used the following prompt that demonstrates the usage of the wiki tool.
```python
prompt = """\
Answer the following question:

Q: In which branch of the arts is Patricia Neary famous?
A: Ballets
A2: <request><Wiki>Patricia Neary<call>Patricia Neary (born October 27, 1942) is an American ballerina, choreographer and ballet director, who has been particularly active in Switzerland. She has also been a highly successful ambassador for the Balanchine Trust, bringing George Balanchine's ballets to 60 cities around the globe.<response>
Result=Ballets<submit>

Q: Who won Super Bowl XX?
A: Chicago Bears
A2: <request><Wiki>Super Bowl XX<call>Super Bowl XX was an American football game between the National Football Conference (NFC) champion Chicago Bears and the American Football Conference (AFC) champion New England Patriots to decide the National Football League (NFL) champion for the 1985 season. The Bears defeated the Patriots by the score of 46–10, capturing their first NFL championship (and Chicago's first overall sports victory) since 1963, three years prior to the birth of the Super Bowl. Super Bowl XX was played on January 26, 1986 at the Louisiana Superdome in New Orleans.<response>
Result=Chicago Bears<submit>

Q: """
```


### Result and Discussion


Our experiments show that the agent can learn to use the wiki tool to answer questions. The learning curves would go up mostly, but one of the experiment did crash.

![](https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/triviaqa_learning_curves.png)

Wandb report is [here](https://wandb.ai/costa-huang/cleanRL/reports/TriviaQA-Final-Experiments--Vmlldzo1MjY0ODk5) for further inspection.


key: _node_content
string_value:
{"id_": "ce3d8779-7e92-4ed4-a0d7-da67bd20f093", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/learning_tools", "title": "Learning Tools (Experimental \ud83e\uddea)", "tokens": 3184, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2831b0af-5745-4c78-b3ea-4ac5967d9d4c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/learning_tools", "title": "Learning Tools (Experimental \ud83e\uddea)", "tokens": 3184, "retrieve_doc": true, "source": "trl"}, "hash": "add7c31d2128098622a1fde39ebd09c9501bf5e0f8bb4fb626ae0619ea2aac21", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5ecdf0a1-80be-4ce6-aa3b-b8331993d9a9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/learning_tools", "title": "Learning Tools (Experimental \ud83e\uddea)", "tokens": 3184, "retrieve_doc": true, "source": "trl"}, "hash": "12f48a280adae35d76c7a9c91caa426788b0d4f48466e5405528f1cef4bc9582", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b1ac48ce-6e48-4e48-bac6-9915fe7c3598", "node_type": "1", "metadata": {}, "hash": "48bc5bd1ce601f78dd67718b4cbe607d608b5817c167686d43c4241dfe5f3721", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8759, "end_char_idx": 11178, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
prompt = """\
Answer the following question:

Q: In which branch of the arts is Patricia Neary famous?
A: Ballets
A2: <request><Wiki>Patricia Neary<call>Patricia Neary (born October 27, 1942) is an American ballerina, choreographer and ballet director, who has been particularly active in Switzerland. She has also been a highly successful ambassador for the Balanchine Trust, bringing George Balanchine's ballets to 60 cities around the globe.<response>
Result=Ballets<submit>

Q: Who won Super Bowl XX?
A: Chicago Bears
A2: <request><Wiki>Super Bowl XX<call>Super Bowl XX was an American football game between the National Football Conference (NFC) champion Chicago Bears and the American Football Conference (AFC) champion New England Patriots to decide the National Football League (NFL) champion for the 1985 season. The Bears defeated the Patriots by the score of 46–10, capturing their first NFL championship (and Chicago's first overall sports victory) since 1963, three years prior to the birth of the Super Bowl. Super Bowl XX was played on January 26, 1986 at the Louisiana Superdome in New Orleans.<response>
Result=Chicago Bears<submit>

Q: """
```


### Result and Discussion


Our experiments show that the agent can learn to use the wiki tool to answer questions. The learning curves would go up mostly, but one of the experiment did crash.

![](https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/triviaqa_learning_curves.png)

Wandb report is [here](https://wandb.ai/costa-huang/cleanRL/reports/TriviaQA-Final-Experiments--Vmlldzo1MjY0ODk5) for further inspection.


Note that the correct rate of the trained model is on the low end, which could be due to the following reasons:

* **incorrect searches:** When given the question `"What is Bruce Willis' real first name?"` if the model searches for `Bruce Willis`, our wiki tool returns "Patrick Poivey (born 18 February 1948) is a French actor. He is especially known for his voice: he is the French dub voice of Bruce Willis since 1988.` But a correct search should be `Walter Bruce Willis (born March 19, 1955) is an American former actor. He achieved fame with a leading role on the comedy-drama series Moonlighting (1985–1989) and appeared in over a hundred films, gaining recognition as an action hero after his portrayal of John McClane in the Die Hard franchise (1988–2013) and other roles.[1][2]"


key: _node_content
string_value:
{"id_": "b1ac48ce-6e48-4e48-bac6-9915fe7c3598", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/learning_tools", "title": "Learning Tools (Experimental \ud83e\uddea)", "tokens": 3184, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2831b0af-5745-4c78-b3ea-4ac5967d9d4c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/learning_tools", "title": "Learning Tools (Experimental \ud83e\uddea)", "tokens": 3184, "retrieve_doc": true, "source": "trl"}, "hash": "add7c31d2128098622a1fde39ebd09c9501bf5e0f8bb4fb626ae0619ea2aac21", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ce3d8779-7e92-4ed4-a0d7-da67bd20f093", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/learning_tools", "title": "Learning Tools (Experimental \ud83e\uddea)", "tokens": 3184, "retrieve_doc": true, "source": "trl"}, "hash": "4eb582ffccbe55eb5c19a312041f01b8062025ac8492336c8cab0313bf282cd7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1df4cdcd-abc2-4784-9b7a-ad7ce2eb6e86", "node_type": "1", "metadata": {}, "hash": "3fe5b2363d6d916a24180639b8ed9049b2e047ad7771d6bba1ea2bf9ad8ce7e8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9591, "end_char_idx": 12863, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The Bears defeated the Patriots by the score of 46–10, capturing their first NFL championship (and Chicago's first overall sports victory) since 1963, three years prior to the birth of the Super Bowl. Super Bowl XX was played on January 26, 1986 at the Louisiana Superdome in New Orleans.<response>
Result=Chicago Bears<submit>

Q: """
```


### Result and Discussion


Our experiments show that the agent can learn to use the wiki tool to answer questions. The learning curves would go up mostly, but one of the experiment did crash.

![](https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/triviaqa_learning_curves.png)

Wandb report is [here](https://wandb.ai/costa-huang/cleanRL/reports/TriviaQA-Final-Experiments--Vmlldzo1MjY0ODk5) for further inspection.


Note that the correct rate of the trained model is on the low end, which could be due to the following reasons:

* **incorrect searches:** When given the question `"What is Bruce Willis' real first name?"` if the model searches for `Bruce Willis`, our wiki tool returns "Patrick Poivey (born 18 February 1948) is a French actor. He is especially known for his voice: he is the French dub voice of Bruce Willis since 1988.` But a correct search should be `Walter Bruce Willis (born March 19, 1955) is an American former actor. He achieved fame with a leading role on the comedy-drama series Moonlighting (1985–1989) and appeared in over a hundred films, gaining recognition as an action hero after his portrayal of John McClane in the Die Hard franchise (1988–2013) and other roles.[1][2]"


    ![](https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/real_first_name.png)

* **unnecessarily long response**: The wiki tool by default sometimes output very long sequences. E.g., when the wiki tool searches for "Brown Act"
    * Our wiki tool returns "The Ralph M. Brown Act, located at California Government Code 54950 "et seq.", is an act of the California State Legislature, authored by Assemblymember Ralph M. Brown and passed in 1953, that guarantees the public's right to attend and participate in meetings of local legislative bodies."
    * [ToolFormer](https://huggingface.co/papers/2302.04761)'s wiki tool returns "The Ralph M. Brown Act is an act of the California State Legislature that guarantees the public's right to attend and participate in meetings of local legislative bodies." which is more succinct.

    ![](https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/brown_act.png)


## (Early Experiments 🧪): solving math puzzles with python interpreter

In this section, we attempt to teach the model to use a python interpreter to solve math puzzles. The rough idea is to give the agent a prompt like the following:

```python
prompt = """\
Example of using a Python API to solve math questions.

Q: Olivia has $23. She bought five bagels for $3 each. How much money does she have left?

<request><PythonInterpreter>
def solution():
    money_initial = 23
    bagels = 5
    bagel_cost = 3
    money_spent = bagels * bagel_cost
    money_left = money_initial - money_spent
    result = money_left
    return result
print(solution())
<call>72<response>

Result = 72 <submit>

Q: """
```


key: _node_content
string_value:
{"id_": "1df4cdcd-abc2-4784-9b7a-ad7ce2eb6e86", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/learning_tools", "title": "Learning Tools (Experimental \ud83e\uddea)", "tokens": 3184, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2831b0af-5745-4c78-b3ea-4ac5967d9d4c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/learning_tools", "title": "Learning Tools (Experimental \ud83e\uddea)", "tokens": 3184, "retrieve_doc": true, "source": "trl"}, "hash": "add7c31d2128098622a1fde39ebd09c9501bf5e0f8bb4fb626ae0619ea2aac21", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b1ac48ce-6e48-4e48-bac6-9915fe7c3598", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/learning_tools", "title": "Learning Tools (Experimental \ud83e\uddea)", "tokens": 3184, "retrieve_doc": true, "source": "trl"}, "hash": "b6fdb873d76949d196ee12ea1d87609158f4301018ef07fde1ff7a5f7b06b73f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11185, "end_char_idx": 13069, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
![](https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/real_first_name.png)

* **unnecessarily long response**: The wiki tool by default sometimes output very long sequences. E.g., when the wiki tool searches for "Brown Act"
    * Our wiki tool returns "The Ralph M. Brown Act, located at California Government Code 54950 "et seq.", is an act of the California State Legislature, authored by Assemblymember Ralph M. Brown and passed in 1953, that guarantees the public's right to attend and participate in meetings of local legislative bodies."
    * [ToolFormer](https://huggingface.co/papers/2302.04761)'s wiki tool returns "The Ralph M. Brown Act is an act of the California State Legislature that guarantees the public's right to attend and participate in meetings of local legislative bodies." which is more succinct.

    ![](https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/brown_act.png)


## (Early Experiments 🧪): solving math puzzles with python interpreter

In this section, we attempt to teach the model to use a python interpreter to solve math puzzles. The rough idea is to give the agent a prompt like the following:

```python
prompt = """\
Example of using a Python API to solve math questions.

Q: Olivia has $23. She bought five bagels for $3 each. How much money does she have left?

<request><PythonInterpreter>
def solution():
    money_initial = 23
    bagels = 5
    bagel_cost = 3
    money_spent = bagels * bagel_cost
    money_left = money_initial - money_spent
    result = money_left
    return result
print(solution())
<call>72<response>

Result = 72 <submit>

Q: """
```


Training experiment can be found at https://wandb.ai/lvwerra/trl-gsm8k/runs/a5odv01y

![](https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/images/gms8k_learning_curve.png)


key: _node_content
string_value:
{"id_": "83f7bd4b-be41-4745-8865-24b0c8860b80", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/example_overview", "title": "Examples", "tokens": 1870, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4395e4c9-16cd-4654-b349-69caa9a0f3bb", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/example_overview", "title": "Examples", "tokens": 1870, "retrieve_doc": true, "source": "trl"}, "hash": "70442542ee78bccf060063edae6104b824db40f531ba844b131f73c590a844fc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0e826995-dbff-41ec-9e66-7d873be183ab", "node_type": "1", "metadata": {}, "hash": "d96620a474131de3e63cec141f7688dc8156f79af08af4f007a604aee215355b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 4649, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Examples


## Introduction

The examples should work in any of the following settings (with the same script):
   - single GPU
   - multi GPUS (using PyTorch distributed mode)
   - multi GPUS (using DeepSpeed ZeRO-Offload stages 1, 2, & 3)
   - fp16 (mixed-precision), fp32 (normal precision), or bf16 (bfloat16 precision)

To run it in each of these various modes, first initialize the accelerate
configuration with `accelerate config`

**NOTE to train with a 4-bit or 8-bit model**, please run

```bash
pip install --upgrade trl[quantization]
```


## Accelerate Config
For all the examples, you'll need to generate a 🤗 Accelerate config file with:

```shell
accelerate config # will prompt you to define the training configuration
```

Then, it is encouraged to launch jobs with `accelerate launch`!


# Maintained Examples



| File                                                                                                                          | Description                                                                                                                                                                                                                                                                                                     |
| ----------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [`examples/scripts/alignprop.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/alignprop.py)                 | This script shows how to use the [`AlignPropTrainer`] to fine-tune a diffusion model.                                                                                                                                                                                                                           |
| [`examples/scripts/bco.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/bco.py)                             | This script shows how to use the [`KTOTrainer`] with the BCO loss to fine-tune a model to increase instruction-following, truthfulness, honesty and helpfulness using the [openbmb/UltraFeedback](https://huggingface.co/datasets/openbmb/UltraFeedback) dataset.                                               |
| [`examples/scripts/chat.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/chat.py)                           | This script allows you to load and use a model as a chatbot.                                                                                                                                                                                                                                                    |
| [`examples/scripts/cpo.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/cpo.py)                             | This script shows how to use the [`CPOTrainer`] to fine-tune a model to increase helpfulness and harmlessness using the [Anthropic/hh-rlhf](https://huggingface.co/datasets/Anthropic/hh-rlhf) dataset.                                                                                                         |
| [`examples/scripts/ddpo.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/ddpo.py)                           | This script shows how to use the [`DDPOTrainer`] to fine-tune a stable diffusion model using reinforcement learning.                                                                                                                                                                                            |
| [`examples/scripts/dpo_visual.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/dpo_visual.py)               | This script shows how to use the [`DPOTrainer`] to fine-tune a Vision Language Model to reduce hallucinations using the [openbmb/RLAIF-V-Dataset](https://huggingface.co/datasets/openbmb/RLAIF-V-Dataset) dataset.                                                                                             |
| [`examples/scripts/dpo.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/dpo.py)                             | This script shows how to use the [`DPOTrainer`] to fine-tune a stable to increase helpfulness and harmlessness using the [Anthropic/hh-rlhf](https://huggingface.co/datasets/Anthropic/hh-rlhf) dataset.


key: _node_content
string_value:
{"id_": "0e826995-dbff-41ec-9e66-7d873be183ab", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/example_overview", "title": "Examples", "tokens": 1870, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4395e4c9-16cd-4654-b349-69caa9a0f3bb", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/example_overview", "title": "Examples", "tokens": 1870, "retrieve_doc": true, "source": "trl"}, "hash": "70442542ee78bccf060063edae6104b824db40f531ba844b131f73c590a844fc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "83f7bd4b-be41-4745-8865-24b0c8860b80", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/example_overview", "title": "Examples", "tokens": 1870, "retrieve_doc": true, "source": "trl"}, "hash": "90b10d8adfedd9a59629d47d5014938330b04a649043ebe81b2aa386cdd2b83f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ee15f58a-8c16-4526-a544-3cd1aa661104", "node_type": "1", "metadata": {}, "hash": "78c4ab59806373fe09fa344d372c911b6b9c5ebe633baa222f86a85d4715aa63", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2573, "end_char_idx": 7168, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|
| [`examples/scripts/chat.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/chat.py)                           | This script allows you to load and use a model as a chatbot.                                                                                                                                                                                                                                                    |
| [`examples/scripts/cpo.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/cpo.py)                             | This script shows how to use the [`CPOTrainer`] to fine-tune a model to increase helpfulness and harmlessness using the [Anthropic/hh-rlhf](https://huggingface.co/datasets/Anthropic/hh-rlhf) dataset.                                                                                                         |
| [`examples/scripts/ddpo.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/ddpo.py)                           | This script shows how to use the [`DDPOTrainer`] to fine-tune a stable diffusion model using reinforcement learning.                                                                                                                                                                                            |
| [`examples/scripts/dpo_visual.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/dpo_visual.py)               | This script shows how to use the [`DPOTrainer`] to fine-tune a Vision Language Model to reduce hallucinations using the [openbmb/RLAIF-V-Dataset](https://huggingface.co/datasets/openbmb/RLAIF-V-Dataset) dataset.                                                                                             |
| [`examples/scripts/dpo.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/dpo.py)                             | This script shows how to use the [`DPOTrainer`] to fine-tune a stable to increase helpfulness and harmlessness using the [Anthropic/hh-rlhf](https://huggingface.co/datasets/Anthropic/hh-rlhf) dataset.                                                                                                        |
| [`examples/scripts/kto.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/kto.py)                             | This script shows how to use the [`KTOTrainer`] to fine-tune a model.                                                                                                                                                                                                                                           |
| [`examples/scripts/orpo.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/orpo.py)                           | This script shows how to use the [`ORPOTrainer`] to fine-tune a model to increase helpfulness and harmlessness using the [Anthropic/hh-rlhf](https://huggingface.co/datasets/Anthropic/hh-rlhf) dataset.                                                                                                        |
| [`examples/scripts/ppo_multi_adapter.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/ppo_multi_adapter.py) | This script shows how to use the [`PPOTrainer`] to train a single base model with multiple adapters. Requires you to run the example script with the reward model training beforehand.                                                                                                                          |
| [`examples/scripts/ppo.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/ppo.py)                             | This script shows how to use the [`PPOTrainer`] to fine-tune a sentiment analysis model using [IMDB dataset](https://huggingface.co/datasets/stanfordnlp/imdb).                                                                                                                                                 |
| [`examples/scripts/reward_modeling.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/reward_modeling.py)     | This script shows how to use the [`RewardTrainer`] to train a reward model on your own dataset.                                                                                                                                                                                                                 |
| [`examples/scripts/sft.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/sft.py)                             | This script shows how to use the [`SFTTrainer`] to fine-tune a model or adapters into a target dataset.


key: _node_content
string_value:
{"id_": "ee15f58a-8c16-4526-a544-3cd1aa661104", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/example_overview", "title": "Examples", "tokens": 1870, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4395e4c9-16cd-4654-b349-69caa9a0f3bb", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/example_overview", "title": "Examples", "tokens": 1870, "retrieve_doc": true, "source": "trl"}, "hash": "70442542ee78bccf060063edae6104b824db40f531ba844b131f73c590a844fc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0e826995-dbff-41ec-9e66-7d873be183ab", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/example_overview", "title": "Examples", "tokens": 1870, "retrieve_doc": true, "source": "trl"}, "hash": "aefa7ea1080dec8b8dd79164a08fa6375bfbae761e408e753d8b33f5509aaf2c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4173489b-0db5-489b-803a-85c96e4e8379", "node_type": "1", "metadata": {}, "hash": "e2cabe4b472fecae894dfca552da5299ddbcb8a1d668f34049b33b3a0c295cd7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4753, "end_char_idx": 9179, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|
| [`examples/scripts/kto.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/kto.py)                             | This script shows how to use the [`KTOTrainer`] to fine-tune a model.                                                                                                                                                                                                                                           |
| [`examples/scripts/orpo.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/orpo.py)                           | This script shows how to use the [`ORPOTrainer`] to fine-tune a model to increase helpfulness and harmlessness using the [Anthropic/hh-rlhf](https://huggingface.co/datasets/Anthropic/hh-rlhf) dataset.                                                                                                        |
| [`examples/scripts/ppo_multi_adapter.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/ppo_multi_adapter.py) | This script shows how to use the [`PPOTrainer`] to train a single base model with multiple adapters. Requires you to run the example script with the reward model training beforehand.                                                                                                                          |
| [`examples/scripts/ppo.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/ppo.py)                             | This script shows how to use the [`PPOTrainer`] to fine-tune a sentiment analysis model using [IMDB dataset](https://huggingface.co/datasets/stanfordnlp/imdb).                                                                                                                                                 |
| [`examples/scripts/reward_modeling.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/reward_modeling.py)     | This script shows how to use the [`RewardTrainer`] to train a reward model on your own dataset.                                                                                                                                                                                                                 |
| [`examples/scripts/sft.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/sft.py)                             | This script shows how to use the [`SFTTrainer`] to fine-tune a model or adapters into a target dataset.                                                                                                                                                                                                         |
| [`examples/scripts/vsft_llava.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/vsft_llava.py)               | This script shows how to use the [`SFTTrainer`] to fine-tune a Vision Language Model in a chat setting. The script has only been tested on a [LLaVA 1.5]([llava-hf/llava-1.5-7b-hf](https://huggingface.co/llava-hf/llava-1.5-7b-hf)) model so users may see unexpected behaviour in other model architectures. |

Here are also some easier-to-run colab notebooks that you can use to get started with TRL:

| File                                                                                                                              | Description                                                                                                             |
| --------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| [`examples/notebooks/best_of_n.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/best_of_n.ipynb)           | This notebook demonstrates how to use the "Best of N" sampling strategy using TRL when fine-tuning your model with PPO. |
| [`examples/notebooks/gpt2-sentiment.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/gpt2-sentiment.ipynb) | This notebook demonstrates how to reproduce the GPT2 imdb sentiment tuning example on a jupyter notebook.               |
| [`examples/notebooks/gpt2-control.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/gpt2-control.ipynb)     | This notebook demonstrates how to reproduce the GPT2 sentiment control example on a jupyter notebook.                   |


key: _node_content
string_value:
{"id_": "4173489b-0db5-489b-803a-85c96e4e8379", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/example_overview", "title": "Examples", "tokens": 1870, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4395e4c9-16cd-4654-b349-69caa9a0f3bb", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/example_overview", "title": "Examples", "tokens": 1870, "retrieve_doc": true, "source": "trl"}, "hash": "70442542ee78bccf060063edae6104b824db40f531ba844b131f73c590a844fc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ee15f58a-8c16-4526-a544-3cd1aa661104", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/example_overview", "title": "Examples", "tokens": 1870, "retrieve_doc": true, "source": "trl"}, "hash": "c19867aa21a4c093dd36d9af0f579807c2c5263a9052ab31b3b7580ed5b3e9cc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7369, "end_char_idx": 10800, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
|
| [`examples/scripts/vsft_llava.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/vsft_llava.py)               | This script shows how to use the [`SFTTrainer`] to fine-tune a Vision Language Model in a chat setting. The script has only been tested on a [LLaVA 1.5]([llava-hf/llava-1.5-7b-hf](https://huggingface.co/llava-hf/llava-1.5-7b-hf)) model so users may see unexpected behaviour in other model architectures. |

Here are also some easier-to-run colab notebooks that you can use to get started with TRL:

| File                                                                                                                              | Description                                                                                                             |
| --------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| [`examples/notebooks/best_of_n.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/best_of_n.ipynb)           | This notebook demonstrates how to use the "Best of N" sampling strategy using TRL when fine-tuning your model with PPO. |
| [`examples/notebooks/gpt2-sentiment.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/gpt2-sentiment.ipynb) | This notebook demonstrates how to reproduce the GPT2 imdb sentiment tuning example on a jupyter notebook.               |
| [`examples/notebooks/gpt2-control.ipynb`](https://github.com/huggingface/trl/tree/main/examples/notebooks/gpt2-control.ipynb)     | This notebook demonstrates how to reproduce the GPT2 sentiment control example on a jupyter notebook.                   |


We also have some other examples that are less maintained but can be used as a reference:
1. **[research_projects](https://github.com/huggingface/trl/tree/main/examples/research_projects)**: Check out this folder to find the scripts used for some research projects that used TRL (LM de-toxification, Stack-Llama, etc.)


## Distributed training

All of the scripts can be run on multiple GPUs by providing the path of an 🤗 Accelerate config file when calling `accelerate launch`. To launch one of them on one or multiple GPUs, run the following command (swapping `{NUM_GPUS}` with the number of GPUs in your machine and `--all_arguments_of_the_script` with your arguments.)

```shell
accelerate launch --config_file=examples/accelerate_configs/multi_gpu.yaml --num_processes {NUM_GPUS} path_to_script.py --all_arguments_of_the_script
```

You can also adjust the parameters of the 🤗 Accelerate config file to suit your needs (e.g. training in mixed precision).

### Distributed training with DeepSpeed

Most of the scripts can be run on multiple GPUs together with DeepSpeed ZeRO-{1,2,3} for efficient sharding of the optimizer states, gradients, and model weights. To do so, run following command (swapping `{NUM_GPUS}` with the number of GPUs in your machine, `--all_arguments_of_the_script` with your arguments, and `--deepspeed_config` with the path to the DeepSpeed config file such as `examples/deepspeed_configs/deepspeed_zero1.yaml`):

```shell
accelerate launch --config_file=examples/accelerate_configs/deepspeed_zero{1,2,3}.yaml --num_processes {NUM_GPUS} path_to_script.py --all_arguments_of_the_script
```


key: _node_content
string_value:
{"id_": "2c57f318-2dcd-4ba5-b408-1f09c2ebfdad", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/using_llama_models", "title": "Using LLaMA models with TRL", "tokens": 2435, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "93a491eb-cddf-48d0-82d2-52bc28a0bdce", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/using_llama_models", "title": "Using LLaMA models with TRL", "tokens": 2435, "retrieve_doc": true, "source": "trl"}, "hash": "259be3a8f639379b43879cb2f6ee34ff3313ac5605649be4013eb4b9921baa51", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "663c0a58-3176-4584-97e1-726d7fbb099a", "node_type": "1", "metadata": {}, "hash": "82b6f055b202a4c1e9c691617f9f1a07eb207cfc8e01147ec71ce5b624f84a39", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3106, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Using LLaMA models with TRL

We've begun rolling out examples to use Meta's LLaMA models in `trl` (see [Meta's LLaMA release](https://ai.facebook.com/blog/large-language-model-llama-meta-ai/) for the original LLaMA model).

## Efficient training strategies

Even training the smallest LLaMA model requires an enormous amount of memory. Some quick math: in bf16, every parameter uses 2 bytes (in fp32 4 bytes) in addition to 8 bytes used, e.g., in the Adam optimizer (see the [performance docs](https://huggingface.co/docs/transformers/perf_train_gpu_one#optimizer) in Transformers for more info). So a 7B parameter model would use `(2+8)*7B=70GB` just to fit in memory and would likely need more when you compute intermediate values such as attention scores. So you couldn’t train the model even on a single 80GB A100 like that. You can use some tricks, like more efficient optimizers of half-precision training, to squeeze a bit more into memory, but you’ll run out sooner or later.

Another option is to use Parameter-Efficient Fine-Tuning (PEFT) techniques, such as the [`peft`](https://github.com/huggingface/peft) library, which can perform low-rank adaptation (LoRA) on a model loaded in 8-bit.
For more on `peft` + `trl`, see the [docs](https://huggingface.co/docs/trl/sentiment_tuning_peft).

Loading the model in 8bit reduces the memory footprint drastically since you only need one byte per parameter for the weights (e.g. 7B LlaMa is 7GB in memory).
Instead of training the original weights directly, LoRA adds small adapter layers on top of some specific layers (usually the attention layers); thus, the number of trainable parameters is drastically reduced.

In this scenario, a rule of thumb is to allocate ~1.2-1.4GB per billion parameters (depending on the batch size and sequence length) to fit the entire fine-tuning setup.
This enables fine-tuning larger models (up to 50-60B scale models on a NVIDIA A100 80GB) at low cost.

Now we can fit very large models into a single GPU, but the training might still be very slow.
The simplest strategy in this scenario is data parallelism: we replicate the same training setup into separate GPUs and pass different batches to each GPU.
With this, you can parallelize the forward/backward passes of the model and scale with the number of GPUs.

![chapter10_ddp.png](https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/blog/stackllama/chapter10_ddp.png)

We use either the `transformers.Trainer` or `accelerate`, which both support data parallelism without any code changes, by simply passing arguments when calling the scripts with `torchrun` or `accelerate launch`. The following runs a training script with 8 GPUs on a single machine with `accelerate` and `torchrun`, respectively.

```bash
accelerate launch --multi_gpu --num_machines 1  --num_processes 8 my_accelerate_script.py
torchrun --nnodes 1  --nproc_per_node 8 my_torch_script.py
```

## Supervised fine-tuning

Before we start training reward models and tuning our model with RL, it helps if the model is already good in the domain we are interested in.


key: _node_content
string_value:
{"id_": "663c0a58-3176-4584-97e1-726d7fbb099a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/using_llama_models", "title": "Using LLaMA models with TRL", "tokens": 2435, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "93a491eb-cddf-48d0-82d2-52bc28a0bdce", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/using_llama_models", "title": "Using LLaMA models with TRL", "tokens": 2435, "retrieve_doc": true, "source": "trl"}, "hash": "259be3a8f639379b43879cb2f6ee34ff3313ac5605649be4013eb4b9921baa51", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2c57f318-2dcd-4ba5-b408-1f09c2ebfdad", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/using_llama_models", "title": "Using LLaMA models with TRL", "tokens": 2435, "retrieve_doc": true, "source": "trl"}, "hash": "de2be5ddfec77d28800086297f51cb99e85beed42c1cbe3894a73f9794b2cf7b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "99ec0bee-4a71-4412-8a22-52c68e33f7ca", "node_type": "1", "metadata": {}, "hash": "d85f798857dfff6e4be0f60acb308747f0486a6396ce818d6dc9e9acd9b70a61", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1435, "end_char_idx": 4853, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
7B LlaMa is 7GB in memory).
Instead of training the original weights directly, LoRA adds small adapter layers on top of some specific layers (usually the attention layers); thus, the number of trainable parameters is drastically reduced.

In this scenario, a rule of thumb is to allocate ~1.2-1.4GB per billion parameters (depending on the batch size and sequence length) to fit the entire fine-tuning setup.
This enables fine-tuning larger models (up to 50-60B scale models on a NVIDIA A100 80GB) at low cost.

Now we can fit very large models into a single GPU, but the training might still be very slow.
The simplest strategy in this scenario is data parallelism: we replicate the same training setup into separate GPUs and pass different batches to each GPU.
With this, you can parallelize the forward/backward passes of the model and scale with the number of GPUs.

![chapter10_ddp.png](https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/blog/stackllama/chapter10_ddp.png)

We use either the `transformers.Trainer` or `accelerate`, which both support data parallelism without any code changes, by simply passing arguments when calling the scripts with `torchrun` or `accelerate launch`. The following runs a training script with 8 GPUs on a single machine with `accelerate` and `torchrun`, respectively.

```bash
accelerate launch --multi_gpu --num_machines 1  --num_processes 8 my_accelerate_script.py
torchrun --nnodes 1  --nproc_per_node 8 my_torch_script.py
```

## Supervised fine-tuning

Before we start training reward models and tuning our model with RL, it helps if the model is already good in the domain we are interested in.
In our case, we want it to answer questions, while for other use cases, we might want it to follow instructions, in which case instruction tuning is a great idea.
The easiest way to achieve this is by continuing to train the language model with the language modeling objective on texts from the domain or task.
The [StackExchange dataset](https://huggingface.co/datasets/HuggingFaceH4/stack-exchange-preferences) is enormous (over 10 million instructions), so we can easily train the language model on a subset of it.

There is nothing special about fine-tuning the model before doing RLHF - it’s just the causal language modeling objective from pretraining that we apply here.
To use the data efficiently, we use a technique called packing: instead of having one text per sample in the batch and then padding to either the longest text or the maximal context of the model, we concatenate a lot of texts with a EOS token in between and cut chunks of the context size to fill the batch without any padding.

![chapter10_preprocessing-clm.png](https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/blog/stackllama/chapter10_preprocessing-clm.png)

With this approach the training is much more efficient as each token that is passed through the model is also trained in contrast to padding tokens which are usually masked from the loss.
If you don't have much data and are more concerned about occasionally cutting off some tokens that are overflowing the context you can also use a classical data loader.

The packing is handled by the `ConstantLengthDataset` and we can then use the `Trainer` after loading the model with `peft`. First, we load the model in int8, prepare it for training, and then add the LoRA adapters.


key: _node_content
string_value:
{"id_": "99ec0bee-4a71-4412-8a22-52c68e33f7ca", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/using_llama_models", "title": "Using LLaMA models with TRL", "tokens": 2435, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "93a491eb-cddf-48d0-82d2-52bc28a0bdce", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/using_llama_models", "title": "Using LLaMA models with TRL", "tokens": 2435, "retrieve_doc": true, "source": "trl"}, "hash": "259be3a8f639379b43879cb2f6ee34ff3313ac5605649be4013eb4b9921baa51", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "663c0a58-3176-4584-97e1-726d7fbb099a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/using_llama_models", "title": "Using LLaMA models with TRL", "tokens": 2435, "retrieve_doc": true, "source": "trl"}, "hash": "9509fc237237f3c3ade5ba77533858ca5f3b93aef3880ca159aaf2805cdab5c4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "926b4c30-c629-4484-a286-eae1c75e5491", "node_type": "1", "metadata": {}, "hash": "7f4f23ace1997b54897b81cdd5907dbd4d4f579deb2abf4ae745d5abe758e711", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3107, "end_char_idx": 6466, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In our case, we want it to answer questions, while for other use cases, we might want it to follow instructions, in which case instruction tuning is a great idea.
The easiest way to achieve this is by continuing to train the language model with the language modeling objective on texts from the domain or task.
The [StackExchange dataset](https://huggingface.co/datasets/HuggingFaceH4/stack-exchange-preferences) is enormous (over 10 million instructions), so we can easily train the language model on a subset of it.

There is nothing special about fine-tuning the model before doing RLHF - it’s just the causal language modeling objective from pretraining that we apply here.
To use the data efficiently, we use a technique called packing: instead of having one text per sample in the batch and then padding to either the longest text or the maximal context of the model, we concatenate a lot of texts with a EOS token in between and cut chunks of the context size to fill the batch without any padding.

![chapter10_preprocessing-clm.png](https://huggingface.co/datasets/trl-internal-testing/example-images/resolve/main/blog/stackllama/chapter10_preprocessing-clm.png)

With this approach the training is much more efficient as each token that is passed through the model is also trained in contrast to padding tokens which are usually masked from the loss.
If you don't have much data and are more concerned about occasionally cutting off some tokens that are overflowing the context you can also use a classical data loader.

The packing is handled by the `ConstantLengthDataset` and we can then use the `Trainer` after loading the model with `peft`. First, we load the model in int8, prepare it for training, and then add the LoRA adapters.

```python
# load model in 8bit
model = AutoModelForCausalLM.from_pretrained(
        args.model_path,
        load_in_8bit=True,
        device_map={"": Accelerator().local_process_index}
    )
model = prepare_model_for_kbit_training(model)

# add LoRA to model
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, config)
```

We train the model for a few thousand steps with the causal language modeling objective and save the model.
Since we will tune the model again with different objectives, we merge the adapter weights with the original model weights.

**Disclaimer:** due to LLaMA's license, we release only the adapter weights for this and the model checkpoints in the following sections.
You can apply for access to the base model's weights by filling out Meta AI's [form](https://docs.google.com/forms/d/e/1FAIpQLSfqNECQnMkycAp2jP4Z9TFX0cGR4uf7b_fBxjY_OjhJILlKGA/viewform) and then converting them to the 🤗 Transformers format by running this [script](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py).
Note that you'll also need to install 🤗 Transformers from source until the `v4.28` is released.

Now that we have fine-tuned the model for the task, we are ready to train a reward model.

## Reward modeling and human preferences

In principle, we could fine-tune the model using RLHF directly with the human annotations.
However, this would require us to send some samples to humans for rating after each optimization iteration.


key: _node_content
string_value:
{"id_": "926b4c30-c629-4484-a286-eae1c75e5491", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/using_llama_models", "title": "Using LLaMA models with TRL", "tokens": 2435, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "93a491eb-cddf-48d0-82d2-52bc28a0bdce", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/using_llama_models", "title": "Using LLaMA models with TRL", "tokens": 2435, "retrieve_doc": true, "source": "trl"}, "hash": "259be3a8f639379b43879cb2f6ee34ff3313ac5605649be4013eb4b9921baa51", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "99ec0bee-4a71-4412-8a22-52c68e33f7ca", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/using_llama_models", "title": "Using LLaMA models with TRL", "tokens": 2435, "retrieve_doc": true, "source": "trl"}, "hash": "c7427b83232903f619956a99f62afed5cc562a42d6d1be472142251a6ef48346", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7d3c76ad-24a6-48ac-906b-aec5aa369d67", "node_type": "1", "metadata": {}, "hash": "bd9bf0586e0f93dcb99d47d2b41f50a94bdb98c1af488f69360ae58f3c50f37a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5393, "end_char_idx": 8718, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Since we will tune the model again with different objectives, we merge the adapter weights with the original model weights.

**Disclaimer:** due to LLaMA's license, we release only the adapter weights for this and the model checkpoints in the following sections.
You can apply for access to the base model's weights by filling out Meta AI's [form](https://docs.google.com/forms/d/e/1FAIpQLSfqNECQnMkycAp2jP4Z9TFX0cGR4uf7b_fBxjY_OjhJILlKGA/viewform) and then converting them to the 🤗 Transformers format by running this [script](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py).
Note that you'll also need to install 🤗 Transformers from source until the `v4.28` is released.

Now that we have fine-tuned the model for the task, we are ready to train a reward model.

## Reward modeling and human preferences

In principle, we could fine-tune the model using RLHF directly with the human annotations.
However, this would require us to send some samples to humans for rating after each optimization iteration.
This is expensive and slow due to the number of training samples needed for convergence and the inherent latency of human reading and annotator speed.

A trick that works well instead of direct feedback is training a reward model on human annotations collected before the RL loop.
The goal of the reward model is to imitate how a human would rate a text. There are several possible strategies to build a reward model: the most straightforward way would be to predict the annotation (e.g. a rating score or a binary value for “good”/”bad”).
In practice, what works better is to predict the ranking of two examples, where the reward model is presented with two candidates `(y_k, y_j)` for a given prompt `x` and has to predict which one would be rated higher by a human annotator.

With the StackExchange dataset, we can infer which of the two answers was preferred by the users based on the score.
With that information and the loss defined above, we can then modify the `transformers.Trainer` by adding a custom loss function.

```python
class RewardTrainer(Trainer):
    def compute_loss(self, model, inputs, return_outputs=False):
        rewards_j = model(input_ids=inputs["input_ids_j"],  attention_mask=inputs["attention_mask_j"])[0]
        rewards_k = model(input_ids=inputs["input_ids_k"], attention_mask=inputs["attention_mask_k"])[0]
        loss = -nn.functional.logsigmoid(rewards_j - rewards_k).mean()
        if return_outputs:
            return loss, {"rewards_j": rewards_j, "rewards_k": rewards_k}
        return loss
```

We utilize a subset of a 100,000 pair of candidates and evaluate on a held-out set of 50,000. With a modest training batch size of 4, we train the Llama model using the LoRA `peft` adapter for a single epoch using the Adam optimizer with BF16 precision. Our LoRA configuration is:

```python
peft_config = LoraConfig(
    task_type=TaskType.SEQ_CLS,
    inference_mode=False,
    r=8,
    lora_alpha=32,
    lora_dropout=0.1,
)
```
As detailed in the next section, the resulting adapter can be merged into the frozen model and saved for further downstream use.

## Reinforcement Learning from Human Feedback

With the fine-tuned language model and the reward model at hand, we are now ready to run the RL loop.


key: _node_content
string_value:
{"id_": "7d3c76ad-24a6-48ac-906b-aec5aa369d67", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/using_llama_models", "title": "Using LLaMA models with TRL", "tokens": 2435, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "93a491eb-cddf-48d0-82d2-52bc28a0bdce", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/using_llama_models", "title": "Using LLaMA models with TRL", "tokens": 2435, "retrieve_doc": true, "source": "trl"}, "hash": "259be3a8f639379b43879cb2f6ee34ff3313ac5605649be4013eb4b9921baa51", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "926b4c30-c629-4484-a286-eae1c75e5491", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/using_llama_models", "title": "Using LLaMA models with TRL", "tokens": 2435, "retrieve_doc": true, "source": "trl"}, "hash": "b8b82bf0f5ebb3ff8c2585ac3349b3a1dccf60700b334c99e91001d612b776d5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7247, "end_char_idx": 10461, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
With the StackExchange dataset, we can infer which of the two answers was preferred by the users based on the score.
With that information and the loss defined above, we can then modify the `transformers.Trainer` by adding a custom loss function.

```python
class RewardTrainer(Trainer):
    def compute_loss(self, model, inputs, return_outputs=False):
        rewards_j = model(input_ids=inputs["input_ids_j"],  attention_mask=inputs["attention_mask_j"])[0]
        rewards_k = model(input_ids=inputs["input_ids_k"], attention_mask=inputs["attention_mask_k"])[0]
        loss = -nn.functional.logsigmoid(rewards_j - rewards_k).mean()
        if return_outputs:
            return loss, {"rewards_j": rewards_j, "rewards_k": rewards_k}
        return loss
```

We utilize a subset of a 100,000 pair of candidates and evaluate on a held-out set of 50,000. With a modest training batch size of 4, we train the Llama model using the LoRA `peft` adapter for a single epoch using the Adam optimizer with BF16 precision. Our LoRA configuration is:

```python
peft_config = LoraConfig(
    task_type=TaskType.SEQ_CLS,
    inference_mode=False,
    r=8,
    lora_alpha=32,
    lora_dropout=0.1,
)
```
As detailed in the next section, the resulting adapter can be merged into the frozen model and saved for further downstream use.

## Reinforcement Learning from Human Feedback

With the fine-tuned language model and the reward model at hand, we are now ready to run the RL loop. It follows roughly three steps:

1. Generate responses from prompts,
2. Rate the responses with the reward model,
3. Run a reinforcement learning policy-optimization step with the ratings.

The Query and Response prompts are templated as follows before being tokenized and passed to the model:

```bash
Question: <Query>

Answer: <Response>
```

The same template was used for SFT, RM and RLHF stages.
Once more, we utilize `peft` for memory-efficient training, which offers an extra advantage in the RLHF context.
Here, the reference model and policy share the same base, the SFT model, which we load in 8-bit and freeze during training.
We exclusively optimize the policy's LoRA weights using PPO while sharing the base model's weights.

```python
for epoch, batch in tqdm(enumerate(ppo_trainer.dataloader)):
    question_tensors = batch["input_ids"]

	# sample from the policy and to generate responses
    response_tensors = ppo_trainer.generate(
        question_tensors,
        return_prompt=False,
        length_sampler=output_length_sampler,
        **generation_kwargs,
    )
    batch["response"] = tokenizer.batch_decode(response_tensors, skip_special_tokens=True)

    # Compute sentiment score
    texts = [q + r for q, r in zip(batch["query"], batch["response"])]
    pipe_outputs = sentiment_pipe(texts, **sent_kwargs)
    rewards = [torch.tensor(output[0]["score"] - script_args.reward_baseline) for output in pipe_outputs]

    # Run PPO step
    stats = ppo_trainer.step(question_tensors, response_tensors, rewards)
	# Log stats to Wandb
    ppo_trainer.log_stats(stats, batch, rewards)
```

For the rest of the details and evaluation, please refer to our [blog post on StackLLaMA](https://huggingface.co/blog/stackllama).


key: _node_content
string_value:
{"id_": "45215e04-47d0-40ef-ae22-74b441a4bf5a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/orpo_trainer", "title": "ORPO Trainer", "tokens": 990, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a69e62f2-903a-4445-a88f-27029ab80188", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/orpo_trainer", "title": "ORPO Trainer", "tokens": 990, "retrieve_doc": true, "source": "trl"}, "hash": "0848c2f0388c0fdf544a01368ae241f8738c4e8a2d0a8be6c6d581963a0e5074", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c5aef82d-da6d-460d-a0a7-01095e38e68f", "node_type": "1", "metadata": {}, "hash": "c5dbd118570e6d541077e2b9abccded09ab4f0618bafa3351c352b5dcb7e02bb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3296, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# ORPO Trainer

[Odds Ratio Preference Optimization](https://huggingface.co/papers/2403.07691) (ORPO) by Jiwoo Hong, Noah Lee, and James Thorne studies the crucial role of SFT within the context of preference alignment. Using preference data the method posits that a minor penalty for the disfavored generation together with a strong adaption signal to the chosen response via a simple log odds ratio term appended to the NLL loss is sufficient for preference-aligned SFT.

Thus ORPO is a reference model-free preference optimization algorithm eliminating the necessity for an additional preference alignment phase thus saving compute and memory.

The official code can be found [xfactlab/orpo](https://github.com/xfactlab/orpo).

## Expected dataset format

The ORPO trainer expects a format identical to the DPO trainer, which should include three entries. These entries should be named as follows:

- `prompt`
- `chosen`
- `rejected`

for example:

```py
orpo_dataset_dict = {
    "prompt": [
        "hello",
        "how are you",
        "What is your name?",
        "What is your name?",
        "Which is the best programming language?",
        "Which is the best programming language?",
        "Which is the best programming language?",
    ],
    "chosen": [
        "hi nice to meet you",
        "I am fine",
        "My name is Mary",
        "My name is Mary",
        "Python",
        "Python",
        "Java",
    ],
    "rejected": [
        "leave me alone",
        "I am not fine",
        "Whats it to you?",
        "I dont have a name",
        "Javascript",
        "C++",
        "C++",
    ],
}
```
where the `prompt` contains the context inputs, `chosen` contains the corresponding chosen responses and `rejected` contains the corresponding negative (rejected) responses. Note that a prompt can have multiple responses and this is reflected in the entries being repeated in the dictionary's value arrays.

## Expected model format
The ORPO trainer expects a model of `AutoModelForCausalLM`, compared to PPO that expects `AutoModelForCausalLMWithValueHead` for the value function.

## Using the `ORPOTrainer`
For a detailed example have a look at the `examples/scripts/orpo.py` script. At a high level we need to initialize the `ORPOTrainer` with a `model` we wish to train. **Note that ORPOTrainer eliminates the need to use the reference model, simplifying the optimization process.** The `beta` refers to the hyperparameter `lambda` in eq. (6) of the paper and refers to the weighting of the relative odd ratio loss in the standard cross-entropy loss used for SFT.

```py
orpo_config = ORPOConfig(
    beta=0.1, # the lambda/alpha hyperparameter in the paper/code
)

orpo_trainer = ORPOTrainer(
    model,
    args=orpo_config,
    train_dataset=train_dataset,
    tokenizer=tokenizer,
)
```
After this one can then call:

```py
orpo_trainer.train()
```

### For Mixture of Experts Models: Enabling the auxiliary loss

MOEs are the most efficient if the load is about equally distributed between experts.  
To ensure that we train MOEs similarly during preference-tuning, it is beneficial to add the auxiliary loss from the load balancer to the final loss.  

This option is enabled by setting `output_router_logits=True` in the model config (e.g. MixtralConfig).


key: _node_content
string_value:
{"id_": "c5aef82d-da6d-460d-a0a7-01095e38e68f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/orpo_trainer", "title": "ORPO Trainer", "tokens": 990, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a69e62f2-903a-4445-a88f-27029ab80188", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/orpo_trainer", "title": "ORPO Trainer", "tokens": 990, "retrieve_doc": true, "source": "trl"}, "hash": "0848c2f0388c0fdf544a01368ae241f8738c4e8a2d0a8be6c6d581963a0e5074", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "45215e04-47d0-40ef-ae22-74b441a4bf5a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/orpo_trainer", "title": "ORPO Trainer", "tokens": 990, "retrieve_doc": true, "source": "trl"}, "hash": "8bcf67deb1513035cdd218bdbfebe9fb007dcf41dcfba64b475f0c483c24fff5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1803, "end_char_idx": 4292, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note that a prompt can have multiple responses and this is reflected in the entries being repeated in the dictionary's value arrays.

## Expected model format
The ORPO trainer expects a model of `AutoModelForCausalLM`, compared to PPO that expects `AutoModelForCausalLMWithValueHead` for the value function.

## Using the `ORPOTrainer`
For a detailed example have a look at the `examples/scripts/orpo.py` script. At a high level we need to initialize the `ORPOTrainer` with a `model` we wish to train. **Note that ORPOTrainer eliminates the need to use the reference model, simplifying the optimization process.** The `beta` refers to the hyperparameter `lambda` in eq. (6) of the paper and refers to the weighting of the relative odd ratio loss in the standard cross-entropy loss used for SFT.

```py
orpo_config = ORPOConfig(
    beta=0.1, # the lambda/alpha hyperparameter in the paper/code
)

orpo_trainer = ORPOTrainer(
    model,
    args=orpo_config,
    train_dataset=train_dataset,
    tokenizer=tokenizer,
)
```
After this one can then call:

```py
orpo_trainer.train()
```

### For Mixture of Experts Models: Enabling the auxiliary loss

MOEs are the most efficient if the load is about equally distributed between experts.  
To ensure that we train MOEs similarly during preference-tuning, it is beneficial to add the auxiliary loss from the load balancer to the final loss.  

This option is enabled by setting `output_router_logits=True` in the model config (e.g. MixtralConfig).  
To scale how much the auxiliary loss contributes to the total loss, use the hyperparameter `router_aux_loss_coef=...` (default: 0.001).

## Logging

While training and evaluating we record the following reward metrics:

* `rewards/chosen`: the mean log probabilities of the policy model for the chosen responses scaled by beta
* `rewards/rejected`: the mean log probabilities of the policy model for the rejected responses scaled by beta
* `rewards/accuracies`: mean of how often the chosen rewards are > than the corresponding rejected rewards
* `rewards/margins`: the mean difference between the chosen and corresponding rejected rewards

* `log_odds_chosen`: the mean log odds ratio of the chosen responses over the rejected responses

* `log_odds_ratio`: the mean of the `log(sigmoid(log_odds_chosen))`

* `nll_loss`: the mean negative log likelihood loss from the SFT part of the loss over chosen responses
 
## ORPOTrainer

[[autodoc]] ORPOTrainer


## ORPOConfig

[[autodoc]] ORPOConfig


key: _node_content
string_value:
{"id_": "82907709-23e9-4779-a60a-13ab92c8749f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/cpo_trainer", "title": "CPO Trainer", "tokens": 1413, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "18aa29a8-b595-4c57-b0c7-b462e1ff90ca", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/cpo_trainer", "title": "CPO Trainer", "tokens": 1413, "retrieve_doc": true, "source": "trl"}, "hash": "3c5ddf63cd64bb51aab8396cb67938996154764122c4eec95a7ea0c7144808b4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c470726f-f414-4d01-950c-7f7e3f3c0581", "node_type": "1", "metadata": {}, "hash": "5bdef4a7c1a944c265cc6e98acab01055a99a872e3d952acf76fb41de40c1d58", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3220, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# CPO Trainer

Contrastive Preference Optimization (CPO) as introduced in the paper [Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation](https://huggingface.co/papers/2401.08417) by Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. At a high-level, CPO  trains models to
avoid generating adequate, but not perfect translations in Machine Translation (MT) tasks. However, CPO is a general approximation to the DPO loss and can be applied to other domains like chat.

CPO aims to mitigate two fundamental shortcomings of SFT. First, SFT’s methodology of minimizing the discrepancy between predicted outputs and gold-standard references inherently caps model performance at the quality level of the training data. Secondly, SFT lacks a mechanism to prevent the model from rejecting mistakes in translations. The CPO objective is derived from the DPO objective.

## SimPO
The [SimPO](https://huggingface.co/papers/2405.14734) method is also implemented in the `CPOTrainer`. SimPO is an alternative loss that adds a reward margin, allows for length normalization, and does not use BC regularization. To use this loss, we can use SimPO easily by turning on `loss_type="simpo"` and `cpo_alpha=0` in the `CPOConfig`.

## CPO-SimPO
We also offer the combined use of CPO and SimPO, which enables more stable training and improved performance. Learn more details at [CPO-SimPO Github](https://github.com/fe1ixxu/CPO_SIMPO). To use this method, simply enable SimPO by setting `loss_type="simpo"` and a non-zero `cpo_alpha` in the CPOConfig.

## Expected dataset format

The CPO trainer expects a format identical to the DPO trainer, which should include three entries. These entries should be named as follows:

- `prompt`
- `chosen`
- `rejected`

for example:

```py
cpo_dataset_dict = {
    "prompt": [
        "hello",
        "how are you",
        "What is your name?",
        "What is your name?",
        "Which is the best programming language?",
        "Which is the best programming language?",
        "Which is the best programming language?",
    ],
    "chosen": [
        "hi nice to meet you",
        "I am fine",
        "My name is Mary",
        "My name is Mary",
        "Python",
        "Python",
        "Java",
    ],
    "rejected": [
        "leave me alone",
        "I am not fine",
        "Whats it to you?",
        "I dont have a name",
        "Javascript",
        "C++",
        "C++",
    ],
}
```
where the `prompt` contains the context inputs, `chosen` contains the corresponding chosen responses and `rejected` contains the corresponding negative (rejected) responses. As can be seen a prompt can have multiple responses and this is reflected in the entries being repeated in the dictionary's value arrays.

## Expected model format
The CPO trainer expects a model of `AutoModelForCausalLM`, compared to PPO that expects `AutoModelForCausalLMWithValueHead` for the value function.

## Using the `CPOTrainer`
For a detailed example have a look at the `examples/scripts/cpo.py` script. At a high level we need to initialize the `CPOTrainer` with a `model` we wish to train.


key: _node_content
string_value:
{"id_": "c470726f-f414-4d01-950c-7f7e3f3c0581", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/cpo_trainer", "title": "CPO Trainer", "tokens": 1413, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "18aa29a8-b595-4c57-b0c7-b462e1ff90ca", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/cpo_trainer", "title": "CPO Trainer", "tokens": 1413, "retrieve_doc": true, "source": "trl"}, "hash": "3c5ddf63cd64bb51aab8396cb67938996154764122c4eec95a7ea0c7144808b4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "82907709-23e9-4779-a60a-13ab92c8749f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/cpo_trainer", "title": "CPO Trainer", "tokens": 1413, "retrieve_doc": true, "source": "trl"}, "hash": "583e153fe9a4edcb07f0e9d404e5129080c30f1f075d03666807a1f25a78fade", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "133146a5-95fb-43e2-9b9a-1c503f7ab994", "node_type": "1", "metadata": {}, "hash": "a4a643c321e03dbbea85d874c6790d571b1247ead0a31626af7f478c0a51519d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1648, "end_char_idx": 4748, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Expected dataset format

The CPO trainer expects a format identical to the DPO trainer, which should include three entries. These entries should be named as follows:

- `prompt`
- `chosen`
- `rejected`

for example:

```py
cpo_dataset_dict = {
    "prompt": [
        "hello",
        "how are you",
        "What is your name?",
        "What is your name?",
        "Which is the best programming language?",
        "Which is the best programming language?",
        "Which is the best programming language?",
    ],
    "chosen": [
        "hi nice to meet you",
        "I am fine",
        "My name is Mary",
        "My name is Mary",
        "Python",
        "Python",
        "Java",
    ],
    "rejected": [
        "leave me alone",
        "I am not fine",
        "Whats it to you?",
        "I dont have a name",
        "Javascript",
        "C++",
        "C++",
    ],
}
```
where the `prompt` contains the context inputs, `chosen` contains the corresponding chosen responses and `rejected` contains the corresponding negative (rejected) responses. As can be seen a prompt can have multiple responses and this is reflected in the entries being repeated in the dictionary's value arrays.

## Expected model format
The CPO trainer expects a model of `AutoModelForCausalLM`, compared to PPO that expects `AutoModelForCausalLMWithValueHead` for the value function.

## Using the `CPOTrainer`
For a detailed example have a look at the `examples/scripts/cpo.py` script. At a high level we need to initialize the `CPOTrainer` with a `model` we wish to train. **Note that CPOTrainer eliminates the need to use the reference model, simplifying the optimization process.** The `beta` refers to the hyperparameter of the implicit reward, and the dataset contains the 3 entries listed above.

```py
cpo_config = CPOConfig(
    beta=0.1,
)

cpo_trainer = CPOTrainer(
    model,
    args=cpo_config,
    train_dataset=train_dataset,
    tokenizer=tokenizer,
)
```
After this one can then call:

```py
cpo_trainer.train()
```

## Loss functions

Given the preference data, the `CPOTrainer` uses the sigmoid loss on the normalized likelihood via the `logsigmoid` to fit a logistic regression.

The [RSO](https://huggingface.co/papers/2309.06657) authors propose to use a hinge loss on the normalized likelihood from the [SLiC](https://huggingface.co/papers/2305.10425) paper. The `CPOTrainer` can be switched to this loss via the `loss_type="hinge"` argument and the `beta` in this case is the reciprocal of the margin.

The [IPO](https://huggingface.co/papers/2310.12036) authors provide a deeper theoretical understanding of the CPO algorithms and identify an issue with overfitting and propose an alternative loss which can be used via the `loss_type="ipo"` argument to the trainer. Note that the `beta`  parameter is the reciprocal of the gap between the log-likelihood ratios of the chosen vs the rejected completion pair and thus the smaller the `beta` the larger this gaps is. As per the paper the loss is averaged over log-likelihoods of the completion (unlike CPO which is summed only).


key: _node_content
string_value:
{"id_": "133146a5-95fb-43e2-9b9a-1c503f7ab994", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/cpo_trainer", "title": "CPO Trainer", "tokens": 1413, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "18aa29a8-b595-4c57-b0c7-b462e1ff90ca", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/cpo_trainer", "title": "CPO Trainer", "tokens": 1413, "retrieve_doc": true, "source": "trl"}, "hash": "3c5ddf63cd64bb51aab8396cb67938996154764122c4eec95a7ea0c7144808b4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c470726f-f414-4d01-950c-7f7e3f3c0581", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/trl/cpo_trainer", "title": "CPO Trainer", "tokens": 1413, "retrieve_doc": true, "source": "trl"}, "hash": "5b6e93d26174dc3de4af13eac33f897bbc1b1f9422d8c5c0c64f0e10f5cd0712", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3221, "end_char_idx": 5973, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
**Note that CPOTrainer eliminates the need to use the reference model, simplifying the optimization process.** The `beta` refers to the hyperparameter of the implicit reward, and the dataset contains the 3 entries listed above.

```py
cpo_config = CPOConfig(
    beta=0.1,
)

cpo_trainer = CPOTrainer(
    model,
    args=cpo_config,
    train_dataset=train_dataset,
    tokenizer=tokenizer,
)
```
After this one can then call:

```py
cpo_trainer.train()
```

## Loss functions

Given the preference data, the `CPOTrainer` uses the sigmoid loss on the normalized likelihood via the `logsigmoid` to fit a logistic regression.

The [RSO](https://huggingface.co/papers/2309.06657) authors propose to use a hinge loss on the normalized likelihood from the [SLiC](https://huggingface.co/papers/2305.10425) paper. The `CPOTrainer` can be switched to this loss via the `loss_type="hinge"` argument and the `beta` in this case is the reciprocal of the margin.

The [IPO](https://huggingface.co/papers/2310.12036) authors provide a deeper theoretical understanding of the CPO algorithms and identify an issue with overfitting and propose an alternative loss which can be used via the `loss_type="ipo"` argument to the trainer. Note that the `beta`  parameter is the reciprocal of the gap between the log-likelihood ratios of the chosen vs the rejected completion pair and thus the smaller the `beta` the larger this gaps is. As per the paper the loss is averaged over log-likelihoods of the completion (unlike CPO which is summed only).

### For Mixture of Experts Models: Enabling the auxiliary loss

MOEs are the most efficient if the load is about equally distributed between experts.  
To ensure that we train MOEs similarly during preference-tuning, it is beneficial to add the auxiliary loss from the load balancer to the final loss.  

This option is enabled by setting `output_router_logits=True` in the model config (e.g. MixtralConfig).  
To scale how much the auxiliary loss contributes to the total loss, use the hyperparameter `router_aux_loss_coef=...` (default: 0.001).

## Logging

While training and evaluating we record the following reward metrics:

* `rewards/chosen`: the mean log probabilities of the policy model for the chosen responses scaled by beta
* `rewards/rejected`: the mean log probabilities of the policy model for the rejected responses scaled by beta
* `rewards/accuracies`: mean of how often the chosen rewards are > than the corresponding rejected rewards
* `rewards/margins`: the mean difference between the chosen and corresponding rejected rewards
* `nll_loss`: the mean negative log likelihood loss of the policy model for the chosen responses

## CPOTrainer

[[autodoc]] CPOTrainer

## CPOConfig

[[autodoc]] CPOConfig


key: _node_content
string_value:
{"id_": "1dc69009-12f9-4ee4-a6d5-f096f370a2cd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/trl/installation", "title": "Installation", "tokens": 118, "retrieve_doc": true, "source": "trl"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "76223a48-8c3a-479d-8f7d-e10365954ad9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/trl/installation", "title": "Installation", "tokens": 118, "retrieve_doc": true, "source": "trl"}, "hash": "b9e9be9347caebecb81b0e038597148804af5127cf1f4791b1517de2d7f8d736", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 476, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Installation
You can install TRL either from pypi or from source:

## pypi
Install the library with pip:

```bash
pip install trl
```

### Source
You can also install the latest version from source. First clone the repo and then run the installation with `pip`:

```bash
git clone https://github.com/huggingface/trl.git
cd trl/
pip install -e .
```

If you want the development install you can replace the pip install with the following:

```bash
pip install -e ".[dev]"
```


