# embedding_metadata

key: chroma:document
string_value:
```py
>>> import numpy as np
>>> from datasets import Dataset


>>> seq_len, dataset_size = 512, 512
>>> dummy_data = {
...     "input_ids": np.random.randint(100, 30000, (dataset_size, seq_len)),
...     "labels": np.random.randint(0, 1, (dataset_size)),
... }
>>> ds = Dataset.from_dict(dummy_data)
>>> ds.set_format("pt")
```

To print summary statistics for the GPU utilization and the training run with the [`Trainer`] we define two helper functions:

```py
>>> from pynvml import *


>>> def print_gpu_utilization():
...     nvmlInit()
...     handle = nvmlDeviceGetHandleByIndex(0)
...     info = nvmlDeviceGetMemoryInfo(handle)
...     print(f"GPU memory occupied: {info.used//1024**2} MB.")


>>> def print_summary(result):
...     print(f"Time: {result.metrics['train_runtime']:.2f}")
...     print(f"Samples/second: {result.metrics['train_samples_per_second']:.2f}")
...     print_gpu_utilization()
```

Let's verify that we start with a free GPU memory:

```py
>>> print_gpu_utilization()
GPU memory occupied: 0 MB.
```

That looks good: the GPU memory is not occupied as we would expect before we load any models. If that's not the case on 
your machine make sure to stop all processes that are using GPU memory. However, not all free GPU memory can be used by 
the user. When a model is loaded to the GPU the kernels are also loaded, which can take up 1-2GB of memory. To see how 
much it is we load a tiny tensor into the GPU which triggers the kernels to be loaded as well.

```py
>>> import torch


>>> torch.ones((1, 1)).to("cuda")
>>> print_gpu_utilization()
GPU memory occupied: 1343 MB.
```

We see that the kernels alone take up 1.3GB of GPU memory. Now let's see how much space the model uses.

## Load Model

First, we load the `google-bert/bert-large-uncased` model. We load the model weights directly to the GPU so that we can check 
how much space just the weights use.


```py
>>> from transformers import AutoModelForSequenceClassification


>>> model = AutoModelForSequenceClassification.from_pretrained("google-bert/bert-large-uncased").to("cuda")
>>> print_gpu_utilization()
GPU memory occupied: 2631 MB.
```

We can see that the model weights alone take up 1.3 GB of GPU memory. The exact number depends on the specific 
GPU you are using. Note that on newer GPUs a model can sometimes take up more space since the weights are loaded in an 
optimized fashion that speeds up the usage of the model. Now we can also quickly check if we get the same result 
as with `nvidia-smi` CLI:


key: _node_content
string_value:
{"id_": "0da429cd-a87a-4558-8c3d-17f03c667a16", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_memory_anatomy/", "title": "Model training anatomy", "tokens": 2516, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3e29d51-de20-43e1-b322-7a9ffc9da2ae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_memory_anatomy/", "title": "Model training anatomy", "tokens": 2516, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "af98e245cd52e022658b1ded0421a8a18e6d596e2b2c6f0e334aacbaea2d3e2e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "21e9e3d9-9939-47d8-9fb1-10563beb12a8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_memory_anatomy/", "title": "Model training anatomy", "tokens": 2516, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "47941543f58de8358d0a43ccbe07411851019ecd3db5c0b386d89c3176ac455b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4313671b-1b5e-4916-b798-97d54926e373", "node_type": "1", "metadata": {}, "hash": "1566cacb2f23c92f1dfe7fbc0bf6ef2aa26ed03b0885c23d2bfbf941e4e279d4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3123, "end_char_idx": 6477, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> torch.ones((1, 1)).to("cuda")
>>> print_gpu_utilization()
GPU memory occupied: 1343 MB.
```

We see that the kernels alone take up 1.3GB of GPU memory. Now let's see how much space the model uses.

## Load Model

First, we load the `google-bert/bert-large-uncased` model. We load the model weights directly to the GPU so that we can check 
how much space just the weights use.


```py
>>> from transformers import AutoModelForSequenceClassification


>>> model = AutoModelForSequenceClassification.from_pretrained("google-bert/bert-large-uncased").to("cuda")
>>> print_gpu_utilization()
GPU memory occupied: 2631 MB.
```

We can see that the model weights alone take up 1.3 GB of GPU memory. The exact number depends on the specific 
GPU you are using. Note that on newer GPUs a model can sometimes take up more space since the weights are loaded in an 
optimized fashion that speeds up the usage of the model. Now we can also quickly check if we get the same result 
as with `nvidia-smi` CLI:


```bash
nvidia-smi
```

```bash
Tue Jan 11 08:58:05 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.91.03    Driver Version: 460.91.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:00:04.0 Off |                    0 |
| N/A   37C    P0    39W / 300W |   2631MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      3721      C   ...nvs/codeparrot/bin/python     2629MiB |
+-----------------------------------------------------------------------------+
```

We get the same number as before and you can also see that we are using a V100 GPU with 16GB of memory. So now we can 
start training the model and see how the GPU memory consumption changes. First, we set up a few standard training 
arguments:

```py
default_args = {
    "output_dir": "tmp",
    "eval_strategy": "steps",
    "num_train_epochs": 1,
    "log_level": "error",
    "report_to": "none",
}
```

<Tip>

 If you plan to run multiple experiments, in order to properly clear the memory between experiments, restart the Python 
 kernel between experiments.

</Tip>

## Memory utilization at vanilla training

Let's use the [`Trainer`] and train the model without using any GPU performance optimization techniques and a batch size of 4:

```py
>>> from transformers import TrainingArguments, Trainer, logging

>>> logging.set_verbosity_error()


key: _node_content
string_value:
{"id_": "4313671b-1b5e-4916-b798-97d54926e373", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_memory_anatomy/", "title": "Model training anatomy", "tokens": 2516, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3e29d51-de20-43e1-b322-7a9ffc9da2ae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_memory_anatomy/", "title": "Model training anatomy", "tokens": 2516, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "af98e245cd52e022658b1ded0421a8a18e6d596e2b2c6f0e334aacbaea2d3e2e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0da429cd-a87a-4558-8c3d-17f03c667a16", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_memory_anatomy/", "title": "Model training anatomy", "tokens": 2516, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "688603282557f815dd9e70269eb13f98a572b604e13b50c73bbd67f92928a68c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2a26e8ee-87b3-4689-beeb-2117a6b2bc41", "node_type": "1", "metadata": {}, "hash": "ce3bd0a4349584d623b0611e962a92bb7ab63b06c0f5212df274e862e9a4bcaf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6480, "end_char_idx": 8220, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> training_args = TrainingArguments(per_device_train_batch_size=4, **default_args)
>>> trainer = Trainer(model=model, args=training_args, train_dataset=ds)
>>> result = trainer.train()
>>> print_summary(result)
```

```
Time: 57.82
Samples/second: 8.86
GPU memory occupied: 14949 MB.
```

We see that already a relatively small batch size almost fills up our GPU's entire memory. However, a larger batch size 
can often result in faster model convergence or better end performance. So ideally we want to tune the batch size to our
model's needs and not to the GPU limitations. What's interesting is that we use much more memory than the size of the model. 
To understand a bit better why this is the case let's have a look at a model's operations and memory needs.

## Anatomy of Model's Operations

Transformers architecture includes 3 main groups of operations grouped below by compute-intensity.

1. **Tensor Contractions**

    Linear layers and components of Multi-Head Attention all do batched **matrix-matrix multiplications**. These operations are the most compute-intensive part of training a transformer.

2. **Statistical Normalizations**

    Softmax and layer normalization are less compute-intensive than tensor contractions, and involve one or more **reduction operations**, the result of which is then applied via a map.

3. **Element-wise Operators**

    These are the remaining operators: **biases, dropout, activations, and residual connections**. These are the least compute-intensive operations.

This knowledge can be helpful to know when analyzing performance bottlenecks.

This summary is derived from [Data Movement Is All You Need: A Case Study on Optimizing Transformers 2020](https://arxiv.org/abs/2007.00072)


key: _node_content
string_value:
{"id_": "2a26e8ee-87b3-4689-beeb-2117a6b2bc41", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_memory_anatomy/", "title": "Model training anatomy", "tokens": 2516, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3e29d51-de20-43e1-b322-7a9ffc9da2ae", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_memory_anatomy/", "title": "Model training anatomy", "tokens": 2516, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "af98e245cd52e022658b1ded0421a8a18e6d596e2b2c6f0e334aacbaea2d3e2e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4313671b-1b5e-4916-b798-97d54926e373", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_memory_anatomy/", "title": "Model training anatomy", "tokens": 2516, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5032b1817e7ffc92a19f97808a4f156f79c77b93b745572847a50b8e0332ec14", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6480, "end_char_idx": 11390, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> training_args = TrainingArguments(per_device_train_batch_size=4, **default_args)
>>> trainer = Trainer(model=model, args=training_args, train_dataset=ds)
>>> result = trainer.train()
>>> print_summary(result)
```

```
Time: 57.82
Samples/second: 8.86
GPU memory occupied: 14949 MB.
```

We see that already a relatively small batch size almost fills up our GPU's entire memory. However, a larger batch size 
can often result in faster model convergence or better end performance. So ideally we want to tune the batch size to our
model's needs and not to the GPU limitations. What's interesting is that we use much more memory than the size of the model. 
To understand a bit better why this is the case let's have a look at a model's operations and memory needs.

## Anatomy of Model's Operations

Transformers architecture includes 3 main groups of operations grouped below by compute-intensity.

1. **Tensor Contractions**

    Linear layers and components of Multi-Head Attention all do batched **matrix-matrix multiplications**. These operations are the most compute-intensive part of training a transformer.

2. **Statistical Normalizations**

    Softmax and layer normalization are less compute-intensive than tensor contractions, and involve one or more **reduction operations**, the result of which is then applied via a map.

3. **Element-wise Operators**

    These are the remaining operators: **biases, dropout, activations, and residual connections**. These are the least compute-intensive operations.

This knowledge can be helpful to know when analyzing performance bottlenecks.

This summary is derived from [Data Movement Is All You Need: A Case Study on Optimizing Transformers 2020](https://arxiv.org/abs/2007.00072)


## Anatomy of Model's Memory

We've seen that training the model uses much more memory than just putting the model on the GPU. This is because there 
are many components during training that use GPU memory. The components on GPU memory are the following:

1. model weights
2. optimizer states
3. gradients
4. forward activations saved for gradient computation
5. temporary buffers
6. functionality-specific memory

A typical model trained in mixed precision with AdamW requires 18 bytes per model parameter plus activation memory. For 
inference there are no optimizer states and gradients, so we can subtract those. And thus we end up with 6 bytes per 
model parameter for mixed precision inference, plus activation memory.

Let's look at the details.

**Model Weights:**

- 4 bytes * number of parameters for fp32 training
- 6 bytes * number of parameters for mixed precision training (maintains a model in fp32 and one in fp16 in memory)

**Optimizer States:**

- 8 bytes * number of parameters for normal AdamW (maintains 2 states)
- 2 bytes * number of parameters for 8-bit AdamW optimizers like [bitsandbytes](https://github.com/TimDettmers/bitsandbytes)
- 4 bytes * number of parameters for optimizers like SGD with momentum (maintains only 1 state)

**Gradients**

- 4 bytes * number of parameters for either fp32 or mixed precision training (gradients are always kept in fp32)

**Forward Activations**

- size depends on many factors, the key ones being sequence length, hidden size and batch size.

There are the input and output that are being passed and returned by the forward and the backward functions and the 
forward activations saved for gradient computation.

**Temporary Memory**

Additionally, there are all kinds of temporary variables which get released once the calculation is done, but in the 
moment these could require additional memory and could push to OOM. Therefore, when coding it's crucial to think 
strategically about such temporary variables and sometimes to explicitly free those as soon as they are no longer needed.

**Functionality-specific memory**

Then, your software could have special memory needs. For example, when generating text using beam search, the software 
needs to maintain multiple copies of inputs and outputs.

**`forward` vs `backward` Execution Speed**

For convolutions and linear layers there are 2x flops in the backward compared to the forward, which generally translates 
into ~2x slower (sometimes more, because sizes in the backward tend to be more awkward). Activations are usually 
bandwidth-limited, and it’s typical for an activation to have to read more data in the backward than in the forward 
(e.g. activation forward reads once, writes once, activation backward reads twice, gradOutput and output of the forward, 
and writes once, gradInput).

As you can see, there are potentially a few places where we could save GPU memory or speed up operations. 
Now that you understand what affects GPU utilization and computation speed, refer to 
the [Methods and tools for efficient training on a single GPU](perf_train_gpu_one) documentation page to learn about 
performance optimization techniques.


key: _node_content
string_value:
{"id_": "ff12b2ba-bd86-46e9-8a27-411436951892", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perplexity/", "title": "Perplexity of fixed-length models", "tokens": 1903, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4e163143-a8a7-4875-baa9-8b5d88b64d8c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perplexity/", "title": "Perplexity of fixed-length models", "tokens": 1903, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "40fcfb6382209cd90cf1c475de1da34148c7faf496db241970152bbd476296a5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0019bc88-0095-4cd4-a574-e742c84d1983", "node_type": "1", "metadata": {}, "hash": "5a4be51c678960f3274143c73cd6a39eff61d94b93e6f7072c9863b1901f3ea7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2776, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Perplexity of fixed-length models

[[open-in-colab]]

Perplexity (PPL) is one of the most common metrics for evaluating language models. Before diving in, we should note
that the metric applies specifically to classical language models (sometimes called autoregressive or causal language
models) and is not well defined for masked language models like BERT (see [summary of the models](model_summary)).

Perplexity is defined as the exponentiated average negative log-likelihood of a sequence. If we have a tokenized
sequence \\(X = (x_0, x_1, \dots, x_t)\\), then the perplexity of \\(X\\) is,

$$\text{PPL}(X) = \exp \left\{ {-\frac{1}{t}\sum_i^t \log p_\theta (x_i|x_{<i}) } \right\}$$

where \\(\log p_\theta (x_i|x_{<i})\\) is the log-likelihood of the ith token conditioned on the preceding tokens \\(x_{<i}\\) according to our model. Intuitively, it can be thought of as an evaluation of the model's ability to predict uniformly among the set of specified tokens in a corpus. Importantly, this means that the tokenization procedure has a direct impact on a model's perplexity which should always be taken into consideration when comparing different models.

This is also equivalent to the exponentiation of the cross-entropy between the data and model predictions. For more
intuition about perplexity and its relationship to Bits Per Character (BPC) and data compression, check out this
[fantastic blog post on The Gradient](https://thegradient.pub/understanding-evaluation-metrics-for-language-models/).

## Calculating PPL with fixed-length models

If we weren't limited by a model's context size, we would evaluate the model's perplexity by autoregressively
factorizing a sequence and conditioning on the entire preceding subsequence at each step, as shown below.

<img width="600" alt="Full decomposition of a sequence with unlimited context length" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/ppl_full.gif"/>

When working with approximate models, however, we typically have a constraint on the number of tokens the model can
process. The largest version of [GPT-2](model_doc/gpt2), for example, has a fixed length of 1024 tokens, so we
cannot calculate \\(p_\theta(x_t|x_{<t})\\) directly when \\(t\\) is greater than 1024.

Instead, the sequence is typically broken into subsequences equal to the model's maximum input size. If a model's max
input size is \\(k\\), we then approximate the likelihood of a token \\(x_t\\) by conditioning only on the
\\(k-1\\) tokens that precede it rather than the entire context. When evaluating the model's perplexity of a
sequence, a tempting but suboptimal approach is to break the sequence into disjoint chunks and add up the decomposed
log-likelihoods of each segment independently.


key: _node_content
string_value:
{"id_": "0019bc88-0095-4cd4-a574-e742c84d1983", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perplexity/", "title": "Perplexity of fixed-length models", "tokens": 1903, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4e163143-a8a7-4875-baa9-8b5d88b64d8c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perplexity/", "title": "Perplexity of fixed-length models", "tokens": 1903, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "40fcfb6382209cd90cf1c475de1da34148c7faf496db241970152bbd476296a5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ff12b2ba-bd86-46e9-8a27-411436951892", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perplexity/", "title": "Perplexity of fixed-length models", "tokens": 1903, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c1f31e78853834bab076d9d68abe8b43cd6fd0531e2c281b28b057c9d1d026f5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "eaee76e7-a2c8-45fc-85f1-35aaddd9d27c", "node_type": "1", "metadata": {}, "hash": "51226f61f00b351858a011cc07bf1c9ba4934db2e78c50cf9b5b71c0bb2f2ba3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1167, "end_char_idx": 4615, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This is also equivalent to the exponentiation of the cross-entropy between the data and model predictions. For more
intuition about perplexity and its relationship to Bits Per Character (BPC) and data compression, check out this
[fantastic blog post on The Gradient](https://thegradient.pub/understanding-evaluation-metrics-for-language-models/).

## Calculating PPL with fixed-length models

If we weren't limited by a model's context size, we would evaluate the model's perplexity by autoregressively
factorizing a sequence and conditioning on the entire preceding subsequence at each step, as shown below.

<img width="600" alt="Full decomposition of a sequence with unlimited context length" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/ppl_full.gif"/>

When working with approximate models, however, we typically have a constraint on the number of tokens the model can
process. The largest version of [GPT-2](model_doc/gpt2), for example, has a fixed length of 1024 tokens, so we
cannot calculate \\(p_\theta(x_t|x_{<t})\\) directly when \\(t\\) is greater than 1024.

Instead, the sequence is typically broken into subsequences equal to the model's maximum input size. If a model's max
input size is \\(k\\), we then approximate the likelihood of a token \\(x_t\\) by conditioning only on the
\\(k-1\\) tokens that precede it rather than the entire context. When evaluating the model's perplexity of a
sequence, a tempting but suboptimal approach is to break the sequence into disjoint chunks and add up the decomposed
log-likelihoods of each segment independently.

<img width="600" alt="Suboptimal PPL not taking advantage of full available context" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/ppl_chunked.gif"/>

This is quick to compute since the perplexity of each segment can be computed in one forward pass, but serves as a poor
approximation of the fully-factorized perplexity and will typically yield a higher (worse) PPL because the model will
have less context at most of the prediction steps.

Instead, the PPL of fixed-length models should be evaluated with a sliding-window strategy. This involves repeatedly
sliding the context window so that the model has more context when making each prediction.

<img width="600" alt="Sliding window PPL taking advantage of all available context" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/ppl_sliding.gif"/>

This is a closer approximation to the true decomposition of the sequence probability and will typically yield a more
favorable score. The downside is that it requires a separate forward pass for each token in the corpus. A good
practical compromise is to employ a strided sliding window, moving the context by larger strides rather than sliding by
1 token a time. This allows computation to proceed much faster while still giving the model a large context to make
predictions at each step.

## Example: Calculating perplexity with GPT-2 in 🤗 Transformers

Let's demonstrate this process with GPT-2.

```python
from transformers import GPT2LMHeadModel, GPT2TokenizerFast

device = "cuda"
model_id = "openai-community/gpt2-large"
model = GPT2LMHeadModel.from_pretrained(model_id).to(device)
tokenizer = GPT2TokenizerFast.from_pretrained(model_id)
```

We'll load in the WikiText-2 dataset and evaluate the perplexity using a few different sliding-window strategies.


key: _node_content
string_value:
{"id_": "eaee76e7-a2c8-45fc-85f1-35aaddd9d27c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perplexity/", "title": "Perplexity of fixed-length models", "tokens": 1903, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4e163143-a8a7-4875-baa9-8b5d88b64d8c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perplexity/", "title": "Perplexity of fixed-length models", "tokens": 1903, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "40fcfb6382209cd90cf1c475de1da34148c7faf496db241970152bbd476296a5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0019bc88-0095-4cd4-a574-e742c84d1983", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perplexity/", "title": "Perplexity of fixed-length models", "tokens": 1903, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a6da897b30ddd56c24d4cb8a5967a633d19bd0e0aa4b6c1063bad3bc3ee2616a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9fcf9849-0f51-4caf-a067-016aea7b56d4", "node_type": "1", "metadata": {}, "hash": "cc7119436ec9342d105aafdbd37a8a900fc506df9e37bc54541fe10956c0782b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3256, "end_char_idx": 6335, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Instead, the PPL of fixed-length models should be evaluated with a sliding-window strategy. This involves repeatedly
sliding the context window so that the model has more context when making each prediction.

<img width="600" alt="Sliding window PPL taking advantage of all available context" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/ppl_sliding.gif"/>

This is a closer approximation to the true decomposition of the sequence probability and will typically yield a more
favorable score. The downside is that it requires a separate forward pass for each token in the corpus. A good
practical compromise is to employ a strided sliding window, moving the context by larger strides rather than sliding by
1 token a time. This allows computation to proceed much faster while still giving the model a large context to make
predictions at each step.

## Example: Calculating perplexity with GPT-2 in 🤗 Transformers

Let's demonstrate this process with GPT-2.

```python
from transformers import GPT2LMHeadModel, GPT2TokenizerFast

device = "cuda"
model_id = "openai-community/gpt2-large"
model = GPT2LMHeadModel.from_pretrained(model_id).to(device)
tokenizer = GPT2TokenizerFast.from_pretrained(model_id)
```

We'll load in the WikiText-2 dataset and evaluate the perplexity using a few different sliding-window strategies. Since
this dataset is small and we're just doing one forward pass over the set, we can just load and encode the entire
dataset in memory.

```python
from datasets import load_dataset

test = load_dataset("wikitext", "wikitext-2-raw-v1", split="test")
encodings = tokenizer("\n\n".join(test["text"]), return_tensors="pt")
```

With 🤗 Transformers, we can simply pass the `input_ids` as the `labels` to our model, and the average negative
log-likelihood for each token is returned as the loss. With our sliding window approach, however, there is overlap in
the tokens we pass to the model at each iteration. We don't want the log-likelihood for the tokens we're just treating
as context to be included in our loss, so we can set these targets to `-100` so that they are ignored. The following
is an example of how we could do this with a stride of `512`. This means that the model will have at least 512 tokens
for context when calculating the conditional likelihood of any one token (provided there are 512 preceding tokens
available to condition on).

```python
import torch
from tqdm import tqdm

max_length = model.config.n_positions
stride = 512
seq_len = encodings.input_ids.size(1)

nlls = []
prev_end_loc = 0
for begin_loc in tqdm(range(0, seq_len, stride)):
    end_loc = min(begin_loc + max_length, seq_len)
    trg_len = end_loc - prev_end_loc  # may be different from stride on last loop
    input_ids = encodings.input_ids[:, begin_loc:end_loc].to(device)
    target_ids = input_ids.clone()
    target_ids[:, :-trg_len] = -100

    with torch.no_grad():
        outputs = model(input_ids, labels=target_ids)

        # loss is calculated using CrossEntropyLoss which averages over valid labels
        # N.B.


key: _node_content
string_value:
{"id_": "9fcf9849-0f51-4caf-a067-016aea7b56d4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perplexity/", "title": "Perplexity of fixed-length models", "tokens": 1903, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4e163143-a8a7-4875-baa9-8b5d88b64d8c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perplexity/", "title": "Perplexity of fixed-length models", "tokens": 1903, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "40fcfb6382209cd90cf1c475de1da34148c7faf496db241970152bbd476296a5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "eaee76e7-a2c8-45fc-85f1-35aaddd9d27c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perplexity/", "title": "Perplexity of fixed-length models", "tokens": 1903, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c36f4372a723506dece7f3b01446d1adc4dfa72106396eec7d74c7512603ea8f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4755, "end_char_idx": 7362, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from datasets import load_dataset

test = load_dataset("wikitext", "wikitext-2-raw-v1", split="test")
encodings = tokenizer("\n\n".join(test["text"]), return_tensors="pt")
```

With 🤗 Transformers, we can simply pass the `input_ids` as the `labels` to our model, and the average negative
log-likelihood for each token is returned as the loss. With our sliding window approach, however, there is overlap in
the tokens we pass to the model at each iteration. We don't want the log-likelihood for the tokens we're just treating
as context to be included in our loss, so we can set these targets to `-100` so that they are ignored. The following
is an example of how we could do this with a stride of `512`. This means that the model will have at least 512 tokens
for context when calculating the conditional likelihood of any one token (provided there are 512 preceding tokens
available to condition on).

```python
import torch
from tqdm import tqdm

max_length = model.config.n_positions
stride = 512
seq_len = encodings.input_ids.size(1)

nlls = []
prev_end_loc = 0
for begin_loc in tqdm(range(0, seq_len, stride)):
    end_loc = min(begin_loc + max_length, seq_len)
    trg_len = end_loc - prev_end_loc  # may be different from stride on last loop
    input_ids = encodings.input_ids[:, begin_loc:end_loc].to(device)
    target_ids = input_ids.clone()
    target_ids[:, :-trg_len] = -100

    with torch.no_grad():
        outputs = model(input_ids, labels=target_ids)

        # loss is calculated using CrossEntropyLoss which averages over valid labels
        # N.B. the model only calculates loss over trg_len - 1 labels, because it internally shifts the labels
        # to the left by 1.
        neg_log_likelihood = outputs.loss

    nlls.append(neg_log_likelihood)

    prev_end_loc = end_loc
    if end_loc == seq_len:
        break

ppl = torch.exp(torch.stack(nlls).mean())
```

Running this with the stride length equal to the max input length is equivalent to the suboptimal, non-sliding-window
strategy we discussed above. The smaller the stride, the more context the model will have in making each prediction,
and the better the reported perplexity will typically be.

When we run the above with `stride = 1024`, i.e. no overlap, the resulting PPL is `19.44`, which is about the same
as the `19.93` reported in the GPT-2 paper. By using `stride = 512` and thereby employing our striding window
strategy, this jumps down to `16.45`. This is not only a more favorable score, but is calculated in a way that is
closer to the true autoregressive decomposition of a sequence likelihood.


key: _node_content
string_value:
{"id_": "56c01f87-71cb-4c8e-9f72-8ba5ef77919b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "000629d8-e2ab-478e-a3f9-03b91a734c74", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "62ce954f331567f9b9f6455ed5413d798ada5cbd9b93211b12cbc7d1cef70b45", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2b9d391b-e34e-4ec8-9f71-71d9da21c0ff", "node_type": "1", "metadata": {}, "hash": "e086b5bc17fe6ebc4a01b3ace6dd5d430f698c2cd575cf6fd8f41de6f639bb9f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3420, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Agents and tools

[[open-in-colab]]

### What is an agent?

Large Language Models (LLMs) trained to perform [causal language modeling](./tasks/language_modeling.) can tackle a wide range of tasks, but they often struggle with basic tasks like logic, calculation, and search. When prompted in domains in which they do not perform well, they often fail to generate the answer we expect them to.

One approach to overcome this weakness is to create an *agent*.

An agent is a system that uses an LLM as its engine, and it has access to functions called *tools*.

These *tools* are functions for performing a task, and they contain all necessary description for the agent to properly use them.

The agent can be programmed to:
- devise a series of actions/tools and run them all at once like the [`CodeAgent`] for example
- plan and execute actions/tools one by one and wait for the outcome of each action before launching the next one like the [`ReactJsonAgent`] for example

### Types of agents

#### Code agent

This agent has a planning step, then generates python code to execute all its actions at once. It natively handles different input and output types for its tools, thus it is the recommended choice for multimodal tasks.

#### React agents

This is the go-to agent to solve reasoning tasks, since the ReAct framework ([Yao et al., 2022](https://huggingface.co/papers/2210.03629)) makes it really efficient to think on the basis of its previous observations.

We implement two versions of ReactJsonAgent: 
- [`ReactJsonAgent`] generates tool calls as a JSON in its output.
- [`ReactCodeAgent`] is a new type of ReactJsonAgent that generates its tool calls as blobs of code, which works really well for LLMs that have strong coding performance.

> [!TIP]
> Read [Open-source LLMs as LangChain Agents](https://huggingface.co/blog/open-source-llms-as-agents) blog post to learn more the ReAct agent.

![Framework of a React Agent](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/open-source-llms-as-agents/ReAct.png)

For example, here is how a ReAct Code agent would work its way through the following question.

```py3
>>> agent.run(
...     "How many more blocks (also denoted as layers) in BERT base encoder than the encoder from the architecture proposed in Attention is All You Need?",
... )
=====New task=====
How many more blocks (also denoted as layers) in BERT base encoder than the encoder from the architecture proposed in Attention is All You Need?
====Agent is executing the code below:
bert_blocks = search(query="number of blocks in BERT base encoder")
print("BERT blocks:", bert_blocks)
====
Print outputs:
BERT blocks: twelve encoder blocks

====Agent is executing the code below:
attention_layer = search(query="number of layers in Attention is All You Need")
print("Attention layers:", attention_layer)
====
Print outputs:
Attention layers: Encoder: The encoder is composed of a stack of N = 6 identical layers. Each layer has two sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, position- 2 Page 3 Figure 1: The Transformer - model architecture.

====Agent is executing the code below:
bert_blocks = 12
attention_layers = 6
diff = bert_blocks - attention_layers
print("Difference in blocks:", diff)
final_answer(diff)
====

Print outputs:
Difference in blocks: 6

Final answer: 6
```

### How can I build an agent?


key: _node_content
string_value:
{"id_": "2b9d391b-e34e-4ec8-9f71-71d9da21c0ff", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "000629d8-e2ab-478e-a3f9-03b91a734c74", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "62ce954f331567f9b9f6455ed5413d798ada5cbd9b93211b12cbc7d1cef70b45", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "56c01f87-71cb-4c8e-9f72-8ba5ef77919b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0788594d3b2441f0409483fcf17cf404d1312f4fdc92e2ae70573a1b78fd659b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "125b1eaf-01ef-424c-9fff-0d16ab17fb1a", "node_type": "1", "metadata": {}, "hash": "fc951d84db93366d6ab24e514d6bf5545e055b80ffccdc97c958c79c021e5344", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1755, "end_char_idx": 4397, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
> [!TIP]
> Read [Open-source LLMs as LangChain Agents](https://huggingface.co/blog/open-source-llms-as-agents) blog post to learn more the ReAct agent.

![Framework of a React Agent](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/open-source-llms-as-agents/ReAct.png)

For example, here is how a ReAct Code agent would work its way through the following question.

```py3
>>> agent.run(
...     "How many more blocks (also denoted as layers) in BERT base encoder than the encoder from the architecture proposed in Attention is All You Need?",
... )
=====New task=====
How many more blocks (also denoted as layers) in BERT base encoder than the encoder from the architecture proposed in Attention is All You Need?
====Agent is executing the code below:
bert_blocks = search(query="number of blocks in BERT base encoder")
print("BERT blocks:", bert_blocks)
====
Print outputs:
BERT blocks: twelve encoder blocks

====Agent is executing the code below:
attention_layer = search(query="number of layers in Attention is All You Need")
print("Attention layers:", attention_layer)
====
Print outputs:
Attention layers: Encoder: The encoder is composed of a stack of N = 6 identical layers. Each layer has two sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, position- 2 Page 3 Figure 1: The Transformer - model architecture.

====Agent is executing the code below:
bert_blocks = 12
attention_layers = 6
diff = bert_blocks - attention_layers
print("Difference in blocks:", diff)
final_answer(diff)
====

Print outputs:
Difference in blocks: 6

Final answer: 6
```

### How can I build an agent?

To initialize an agent, you need these arguments:

- an LLM to power your agent - the agent is not exactly the LLM, it’s more like the agent is a program that uses an LLM as its engine.
- a system prompt: what the LLM engine will be prompted with to generate its output
- a toolbox from which the agent pick tools to execute
- a parser to extract from the LLM output which tools are to call and with which arguments

Upon initialization of the agent system, the tool attributes are used to generate a tool description, then baked into the agent’s `system_prompt` to let it know which tools it can use and why.

To start with, please install the `agents` extras in order to install all default dependencies.

```bash
pip install transformers[agents]
```

Build your LLM engine by defining a `llm_engine` method which accepts a list of [messages](./chat_templating.) and returns text. This callable also needs to accept a `stop` argument that indicates when to stop generating.


key: _node_content
string_value:
{"id_": "125b1eaf-01ef-424c-9fff-0d16ab17fb1a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "000629d8-e2ab-478e-a3f9-03b91a734c74", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "62ce954f331567f9b9f6455ed5413d798ada5cbd9b93211b12cbc7d1cef70b45", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2b9d391b-e34e-4ec8-9f71-71d9da21c0ff", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "dffa71bbc3ccf4b37ce5ec84c9f7b1635410d445ae303b011ada1d434a0bdd50", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bb84a138-90d0-43ed-8f6e-f6636473b832", "node_type": "1", "metadata": {}, "hash": "c71e4e63eee97a0b82502febc8a219f72109dc51917715da03fbeedfd1bf69ff", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2974, "end_char_idx": 6175, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Each layer has two sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, position- 2 Page 3 Figure 1: The Transformer - model architecture.

====Agent is executing the code below:
bert_blocks = 12
attention_layers = 6
diff = bert_blocks - attention_layers
print("Difference in blocks:", diff)
final_answer(diff)
====

Print outputs:
Difference in blocks: 6

Final answer: 6
```

### How can I build an agent?

To initialize an agent, you need these arguments:

- an LLM to power your agent - the agent is not exactly the LLM, it’s more like the agent is a program that uses an LLM as its engine.
- a system prompt: what the LLM engine will be prompted with to generate its output
- a toolbox from which the agent pick tools to execute
- a parser to extract from the LLM output which tools are to call and with which arguments

Upon initialization of the agent system, the tool attributes are used to generate a tool description, then baked into the agent’s `system_prompt` to let it know which tools it can use and why.

To start with, please install the `agents` extras in order to install all default dependencies.

```bash
pip install transformers[agents]
```

Build your LLM engine by defining a `llm_engine` method which accepts a list of [messages](./chat_templating.) and returns text. This callable also needs to accept a `stop` argument that indicates when to stop generating.

```python
from huggingface_hub import login, InferenceClient

login("<YOUR_HUGGINGFACEHUB_API_TOKEN>")

client = InferenceClient(model="meta-llama/Meta-Llama-3-70B-Instruct")

def llm_engine(messages, stop_sequences=["Task"]) -> str:
    response = client.chat_completion(messages, stop=stop_sequences, max_tokens=1000)
    answer = response.choices[0].message.content
    return answer
```

You could use any `llm_engine` method as long as:
1. it follows the [messages format](./chat_templating.md) for its input (`List[Dict[str, str]]`) and returns a `str`
2. it stops generating outputs at the sequences passed in the argument `stop`

You also need a `tools` argument which accepts a list of `Tools`. You can provide an empty list for `tools`, but use the default toolbox with the optional argument `add_base_tools=True`.

Now you can create an agent, like [`CodeAgent`], and run it. For convenience, we also provide the [`HfEngine`] class that uses `huggingface_hub.InferenceClient` under the hood.

```python
from transformers import CodeAgent, HfEngine

llm_engine = HfEngine(model="meta-llama/Meta-Llama-3-70B-Instruct")
agent = CodeAgent(tools=[], llm_engine=llm_engine, add_base_tools=True)

agent.run(
    "Could you translate this sentence from French, say it out loud and return the audio.",
    sentence="Où est la boulangerie la plus proche?",
)
```

This will be handy in case of emergency baguette need!
You can even leave the argument `llm_engine` undefined, and an [`HfEngine`] will be created by default.

```python
from transformers import CodeAgent

agent = CodeAgent(tools=[], add_base_tools=True)

agent.run(
    "Could you translate this sentence from French, say it out loud and give me the audio.",
    sentence="Où est la boulangerie la plus proche?


key: _node_content
string_value:
{"id_": "bb84a138-90d0-43ed-8f6e-f6636473b832", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "000629d8-e2ab-478e-a3f9-03b91a734c74", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "62ce954f331567f9b9f6455ed5413d798ada5cbd9b93211b12cbc7d1cef70b45", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "125b1eaf-01ef-424c-9fff-0d16ab17fb1a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "bb573d03cebc54fd6280710715bc65365ae5f870d70e6d88f51c40c02da9069e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "67c5c644-26c8-4632-bf67-5c3b729886c1", "node_type": "1", "metadata": {}, "hash": "9cb72d14aab88e4e921c05b8a7b4acf7cba7fc2f892cfcb4ebc14d941ac930a0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5103, "end_char_idx": 8473, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can provide an empty list for `tools`, but use the default toolbox with the optional argument `add_base_tools=True`.

Now you can create an agent, like [`CodeAgent`], and run it. For convenience, we also provide the [`HfEngine`] class that uses `huggingface_hub.InferenceClient` under the hood.

```python
from transformers import CodeAgent, HfEngine

llm_engine = HfEngine(model="meta-llama/Meta-Llama-3-70B-Instruct")
agent = CodeAgent(tools=[], llm_engine=llm_engine, add_base_tools=True)

agent.run(
    "Could you translate this sentence from French, say it out loud and return the audio.",
    sentence="Où est la boulangerie la plus proche?",
)
```

This will be handy in case of emergency baguette need!
You can even leave the argument `llm_engine` undefined, and an [`HfEngine`] will be created by default.

```python
from transformers import CodeAgent

agent = CodeAgent(tools=[], add_base_tools=True)

agent.run(
    "Could you translate this sentence from French, say it out loud and give me the audio.",
    sentence="Où est la boulangerie la plus proche?",
)
```

Note that we used an additional `sentence` argument: you can pass text as additional arguments to the model.

You can also use this to indicate the path to local or remote files for the model to use:

```py
from transformers import ReactCodeAgent

agent = ReactCodeAgent(tools=[], llm_engine=llm_engine, add_base_tools=True)

agent.run("Why does Mike not know many people in New York?", audio="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/recording.mp3")
```


The prompt and output parser were automatically defined, but you can easily inspect them by calling the `system_prompt_template` on your agent.

```python
print(agent.system_prompt_template)
```

It's important to explain as clearly as possible the task you want to perform.
Every [`~Agent.run`] operation is independent, and since an agent is powered by an LLM, minor variations in your prompt might yield completely different results.
You can also run an agent consecutively for different tasks: each time the attributes `agent.task` and `agent.logs` will be re-initialized.


#### Code execution

A Python interpreter executes the code on a set of inputs passed along with your tools.
This should be safe because the only functions that can be called are the tools you provided (especially if it's only tools by Hugging Face) and the print function, so you're already limited in what can be executed.

The Python interpreter also doesn't allow imports by default outside of a safe list, so all the most obvious attacks shouldn't be an issue.
You can still authorize additional imports by passing the authorized modules as a list of strings in argument `additional_authorized_imports` upon initialization of your [`ReactCodeAgent`] or [`CodeAgent`]:

```py
>>> from transformers import ReactCodeAgent

>>> agent = ReactCodeAgent(tools=[], additional_authorized_imports=['requests', 'bs4'])
>>> agent.run("Could you get me the title of the page at url 'https://huggingface.co/blog'?")

(...)
'Hugging Face – Blog'
```

The execution will stop at any code trying to perform an illegal operation or if there is a regular Python error with the code generated by the agent.

> [!WARNING]
> The LLM can generate arbitrary code that will then be executed: do not add any unsafe imports!


key: _node_content
string_value:
{"id_": "67c5c644-26c8-4632-bf67-5c3b729886c1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "000629d8-e2ab-478e-a3f9-03b91a734c74", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "62ce954f331567f9b9f6455ed5413d798ada5cbd9b93211b12cbc7d1cef70b45", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bb84a138-90d0-43ed-8f6e-f6636473b832", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "1eae44bee1da093226ba1c06aaa633ff5822e553202af6861f059b7b90d8b749", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "45ec37a1-3860-4abd-b025-cbf3821e90cb", "node_type": "1", "metadata": {}, "hash": "2e8a143dd2627f08f892417ad8e81b2516a532a0eb698ea901adfb8a63245691", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6692, "end_char_idx": 10347, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The prompt and output parser were automatically defined, but you can easily inspect them by calling the `system_prompt_template` on your agent.

```python
print(agent.system_prompt_template)
```

It's important to explain as clearly as possible the task you want to perform.
Every [`~Agent.run`] operation is independent, and since an agent is powered by an LLM, minor variations in your prompt might yield completely different results.
You can also run an agent consecutively for different tasks: each time the attributes `agent.task` and `agent.logs` will be re-initialized.


#### Code execution

A Python interpreter executes the code on a set of inputs passed along with your tools.
This should be safe because the only functions that can be called are the tools you provided (especially if it's only tools by Hugging Face) and the print function, so you're already limited in what can be executed.

The Python interpreter also doesn't allow imports by default outside of a safe list, so all the most obvious attacks shouldn't be an issue.
You can still authorize additional imports by passing the authorized modules as a list of strings in argument `additional_authorized_imports` upon initialization of your [`ReactCodeAgent`] or [`CodeAgent`]:

```py
>>> from transformers import ReactCodeAgent

>>> agent = ReactCodeAgent(tools=[], additional_authorized_imports=['requests', 'bs4'])
>>> agent.run("Could you get me the title of the page at url 'https://huggingface.co/blog'?")

(...)
'Hugging Face – Blog'
```

The execution will stop at any code trying to perform an illegal operation or if there is a regular Python error with the code generated by the agent.

> [!WARNING]
> The LLM can generate arbitrary code that will then be executed: do not add any unsafe imports!

### The system prompt

An agent, or rather the LLM that drives the agent, generates an output based on the system prompt. The system prompt can be customized and tailored to the intended task. For example, check the system prompt for the [`ReactCodeAgent`] (below version is slightly simplified).

```text
You will be given a task to solve as best you can.
You have access to the following tools:
<<tool_descriptions>>

To solve the task, you must plan forward to proceed in a series of steps, in a cycle of 'Thought:', 'Code:', and 'Observation:' sequences.

At each step, in the 'Thought:' sequence, you should first explain your reasoning towards solving the task, then the tools that you want to use.
Then in the 'Code:' sequence, you shold write the code in simple Python. The code sequence must end with '/End code' sequence.
During each intermediate step, you can use 'print()' to save whatever important information you will then need.
These print outputs will then be available in the 'Observation:' field, for using this information as input for the next step.

In the end you have to return a final answer using the `final_answer` tool.

Here are a few examples using notional tools:
---
{examples}

Above example were using notional tools that might not exist for you. You only have acces to those tools:
<<tool_names>>
You also can perform computations in the python code you generate.

Always provide a 'Thought:' and a 'Code:\n```py' sequence ending with '```<end_code>' sequence. You MUST provide at least the 'Code:' sequence to move forward.

Remember to not perform too many operations in a single code block! You should split the task into intermediate code blocks.
Print results at the end of each step to save the intermediate results. Then use final_answer() to return the final result.

Remember to make sure that variables you use are all defined.


key: _node_content
string_value:
{"id_": "45ec37a1-3860-4abd-b025-cbf3821e90cb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "000629d8-e2ab-478e-a3f9-03b91a734c74", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "62ce954f331567f9b9f6455ed5413d798ada5cbd9b93211b12cbc7d1cef70b45", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "67c5c644-26c8-4632-bf67-5c3b729886c1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "fafe7f1efa159f8d791055deab34219a688dcdc69c009a8eadd37647421389ef", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7f10a7ec-0027-47d6-b229-a6ef483e5ee5", "node_type": "1", "metadata": {}, "hash": "910a6b9dabc332606eed91decb9a5bb857238f301fb1013279bc3652aa737a24", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8597, "end_char_idx": 11445, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The system prompt can be customized and tailored to the intended task. For example, check the system prompt for the [`ReactCodeAgent`] (below version is slightly simplified).

```text
You will be given a task to solve as best you can.
You have access to the following tools:
<<tool_descriptions>>

To solve the task, you must plan forward to proceed in a series of steps, in a cycle of 'Thought:', 'Code:', and 'Observation:' sequences.

At each step, in the 'Thought:' sequence, you should first explain your reasoning towards solving the task, then the tools that you want to use.
Then in the 'Code:' sequence, you shold write the code in simple Python. The code sequence must end with '/End code' sequence.
During each intermediate step, you can use 'print()' to save whatever important information you will then need.
These print outputs will then be available in the 'Observation:' field, for using this information as input for the next step.

In the end you have to return a final answer using the `final_answer` tool.

Here are a few examples using notional tools:
---
{examples}

Above example were using notional tools that might not exist for you. You only have acces to those tools:
<<tool_names>>
You also can perform computations in the python code you generate.

Always provide a 'Thought:' and a 'Code:\n```py' sequence ending with '```<end_code>' sequence. You MUST provide at least the 'Code:' sequence to move forward.

Remember to not perform too many operations in a single code block! You should split the task into intermediate code blocks.
Print results at the end of each step to save the intermediate results. Then use final_answer() to return the final result.

Remember to make sure that variables you use are all defined.

Now Begin!
```

The system prompt includes:
- An *introduction* that explains how the agent should behave and what tools are.
- A description of all the tools that is defined by a `<<tool_descriptions>>` token that is dynamically replaced at runtime with the tools defined/chosen by the user.
    - The tool description comes from the tool attributes, `name`, `description`, `inputs` and `output_type`,  and a simple `jinja2` template that you can refine.
- The expected output format.

You could improve the system prompt, for example, by adding an explanation of the output format.

For maximum flexibility, you can overwrite the whole system prompt template by passing your custom prompt as an argument to the `system_prompt` parameter.

```python
from transformers import ReactJsonAgent
from transformers.agents import PythonInterpreterTool

agent = ReactJsonAgent(tools=[PythonInterpreterTool()], system_prompt="{your_custom_prompt}")
```

> [!WARNING]
> Please make sure to define the `<<tool_descriptions>>` string somewhere in the `template` so the agent is aware 
of the available tools.


key: _node_content
string_value:
{"id_": "7f10a7ec-0027-47d6-b229-a6ef483e5ee5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "000629d8-e2ab-478e-a3f9-03b91a734c74", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "62ce954f331567f9b9f6455ed5413d798ada5cbd9b93211b12cbc7d1cef70b45", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "45ec37a1-3860-4abd-b025-cbf3821e90cb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e72ea8d51dac96d8dc0d580c1bf5194fde7c1fc26f0a765a33ed1ead4816ec7e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b464c7d6-a5c7-4741-8807-f7f0bf8f0119", "node_type": "1", "metadata": {}, "hash": "529c68a103c37035bd58003dba137a840a96e7af9fc865412e7a0b90fdfc313e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9624, "end_char_idx": 13661, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Here are a few examples using notional tools:
---
{examples}

Above example were using notional tools that might not exist for you. You only have acces to those tools:
<<tool_names>>
You also can perform computations in the python code you generate.

Always provide a 'Thought:' and a 'Code:\n```py' sequence ending with '```<end_code>' sequence. You MUST provide at least the 'Code:' sequence to move forward.

Remember to not perform too many operations in a single code block! You should split the task into intermediate code blocks.
Print results at the end of each step to save the intermediate results. Then use final_answer() to return the final result.

Remember to make sure that variables you use are all defined.

Now Begin!
```

The system prompt includes:
- An *introduction* that explains how the agent should behave and what tools are.
- A description of all the tools that is defined by a `<<tool_descriptions>>` token that is dynamically replaced at runtime with the tools defined/chosen by the user.
    - The tool description comes from the tool attributes, `name`, `description`, `inputs` and `output_type`,  and a simple `jinja2` template that you can refine.
- The expected output format.

You could improve the system prompt, for example, by adding an explanation of the output format.

For maximum flexibility, you can overwrite the whole system prompt template by passing your custom prompt as an argument to the `system_prompt` parameter.

```python
from transformers import ReactJsonAgent
from transformers.agents import PythonInterpreterTool

agent = ReactJsonAgent(tools=[PythonInterpreterTool()], system_prompt="{your_custom_prompt}")
```

> [!WARNING]
> Please make sure to define the `<<tool_descriptions>>` string somewhere in the `template` so the agent is aware 
of the available tools.


### Inspecting an agent run

Here are a few useful attributes to inspect what happened after a run:
- `agent.logs` stores the fine-grained logs of the agent. At every step of the agent's run, everything gets stored in a dictionary that then is appended to `agent.logs`.
- Running `agent.write_inner_memory_from_logs()` creates an inner memory of the agent's logs for the LLM to view, as a list of chat messages. This method goes over each step of the log and only stores what it's interested in as a message: for instance, it will save the system prompt and task in separate messages, then for each step it will store the LLM output as a message, and the tool call output as another message. Use this if you want a higher-level view of what has happened - but not every log will be transcripted by this method.

## Tools

A tool is an atomic function to be used by an agent.

You can for instance check the [`PythonInterpreterTool`]: it has a name, a description, input descriptions, an output type, and a `__call__` method to perform the action.

When the agent is initialized, the tool attributes are used to generate a tool description which is baked into the agent's system prompt. This lets the agent know which tools it can use and why.

### Default toolbox

Transformers comes with a default toolbox for empowering agents, that you can add to your agent upon initialization with argument `add_base_tools = True`:

- **Document question answering**: given a document (such as a PDF) in image format, answer a question on this document ([Donut](./model_doc/donut))
- **Image question answering**: given an image, answer a question on this image ([VILT](./model_doc/vilt))
- **Speech to text**: given an audio recording of a person talking, transcribe the speech into text ([Whisper](./model_doc/whisper))
- **Text to speech**: convert text to speech ([SpeechT5](./model_doc/speecht5))
- **Translation**: translates a given sentence from source language to target language.
- **Python code interpreter**: runs your the LLM generated Python code in a secure environment. This tool will only be added to [`ReactJsonAgent`] if you use `add_base_tools=True`, since code-based tools can already execute Python code


key: _node_content
string_value:
{"id_": "b464c7d6-a5c7-4741-8807-f7f0bf8f0119", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "000629d8-e2ab-478e-a3f9-03b91a734c74", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "62ce954f331567f9b9f6455ed5413d798ada5cbd9b93211b12cbc7d1cef70b45", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7f10a7ec-0027-47d6-b229-a6ef483e5ee5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e650a405f0c6a52dc878c9de183e4d81262af53643290962e5999e9d50f8f730", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "41816db0-a138-4dca-88ee-7bdf6c3c6171", "node_type": "1", "metadata": {}, "hash": "88476ff75c52c41a13390e4d1645b6bf50bc75b63a8097f11c4754d11ced2406", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13664, "end_char_idx": 15877, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can manually use a tool by calling the [`load_tool`] function and a task to perform.


```python
from transformers import load_tool

tool = load_tool("text-to-speech")
audio = tool("This is a text to speech tool")
```


### Create a new tool

You can create your own tool for use cases not covered by the default tools from Hugging Face.
For example, let's create a tool that returns the most downloaded model for a given task from the Hub.

You'll start with the code below.

```python
from huggingface_hub import list_models

task = "text-classification"

model = next(iter(list_models(filter=task, sort="downloads", direction=-1)))
print(model.id)
```

This code can be converted into a class that inherits from the [`Tool`] superclass.


The custom tool needs:
- An attribute `name`, which corresponds to the name of the tool itself. The name usually describes what the tool does. Since the code returns the model with the most downloads for a task, let's name is `model_download_counter`.
- An attribute `description` is used to populate the agent's system prompt.
- An `inputs` attribute, which is a dictionary with keys `"type"` and `"description"`. It contains information that helps the Python interpreter make educated choices about the input.
- An `output_type` attribute, which specifies the output type.
- A `forward` method which contains the inference code to be executed.


```python
from transformers import Tool
from huggingface_hub import list_models

class HFModelDownloadsTool(Tool):
    name = "model_download_counter"
    description = (
        "This is a tool that returns the most downloaded model of a given task on the Hugging Face Hub. "
        "It returns the name of the checkpoint."
    )

    inputs = {
        "task": {
            "type": "text",
            "description": "the task category (such as text-classification, depth-estimation, etc)",
        }
    }
    output_type = "text"

    def forward(self, task: str):
        model = next(iter(list_models(filter=task, sort="downloads", direction=-1)))
        return model.id
```

Now that the custom `HfModelDownloadsTool` class is ready, you can save it to a file named `model_downloads.py` and import it for use.


key: _node_content
string_value:
{"id_": "41816db0-a138-4dca-88ee-7bdf6c3c6171", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "000629d8-e2ab-478e-a3f9-03b91a734c74", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "62ce954f331567f9b9f6455ed5413d798ada5cbd9b93211b12cbc7d1cef70b45", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b464c7d6-a5c7-4741-8807-f7f0bf8f0119", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4a488a7907b0181d688e743b261d8616d35b477be2256f1d04a5d9dfa303f538", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2526c759-82bb-45fb-b0fb-cef654cdd7ef", "node_type": "1", "metadata": {}, "hash": "b135bc356bee18b528c1d4046da4f95d0702bda3910087f66f1b092548c14379", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14410, "end_char_idx": 17169, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The custom tool needs:
- An attribute `name`, which corresponds to the name of the tool itself. The name usually describes what the tool does. Since the code returns the model with the most downloads for a task, let's name is `model_download_counter`.
- An attribute `description` is used to populate the agent's system prompt.
- An `inputs` attribute, which is a dictionary with keys `"type"` and `"description"`. It contains information that helps the Python interpreter make educated choices about the input.
- An `output_type` attribute, which specifies the output type.
- A `forward` method which contains the inference code to be executed.


```python
from transformers import Tool
from huggingface_hub import list_models

class HFModelDownloadsTool(Tool):
    name = "model_download_counter"
    description = (
        "This is a tool that returns the most downloaded model of a given task on the Hugging Face Hub. "
        "It returns the name of the checkpoint."
    )

    inputs = {
        "task": {
            "type": "text",
            "description": "the task category (such as text-classification, depth-estimation, etc)",
        }
    }
    output_type = "text"

    def forward(self, task: str):
        model = next(iter(list_models(filter=task, sort="downloads", direction=-1)))
        return model.id
```

Now that the custom `HfModelDownloadsTool` class is ready, you can save it to a file named `model_downloads.py` and import it for use.


```python
from model_downloads import HFModelDownloadsTool

tool = HFModelDownloadsTool()
```

You can also share your custom tool to the Hub by calling [`~Tool.push_to_hub`] on the tool. Make sure you've created a repository for it on the Hub and are using a token with read access.

```python
tool.push_to_hub("{your_username}/hf-model-downloads")
```

Load the tool with the [`~Tool.load_tool`] function and pass it to the `tools` parameter in your agent.

```python
from transformers import load_tool, CodeAgent

model_download_tool = load_tool("m-ric/hf-model-downloads")
agent = CodeAgent(tools=[model_download_tool], llm_engine=llm_engine)
agent.run(
    "Can you give me the name of the model that has the most downloads in the 'text-to-video' task on the Hugging Face Hub?"
)
```

You get the following:
```text
======== New task ========
Can you give me the name of the model that has the most downloads in the 'text-to-video' task on the Hugging Face Hub?
==== Agent is executing the code below:
most_downloaded_model = model_download_counter(task="text-to-video")
print(f"The most downloaded model for the 'text-to-video' task is {most_downloaded_model}.")
====
```

And the output:
`"The most downloaded model for the 'text-to-video' task is ByteDance/AnimateDiff-Lightning."`


key: _node_content
string_value:
{"id_": "2526c759-82bb-45fb-b0fb-cef654cdd7ef", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "000629d8-e2ab-478e-a3f9-03b91a734c74", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "62ce954f331567f9b9f6455ed5413d798ada5cbd9b93211b12cbc7d1cef70b45", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "41816db0-a138-4dca-88ee-7bdf6c3c6171", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "42f5bcba51a29db13310382f4ab038060ae6dc43be9754878ccd2ac5c4c81c49", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f8f07093-4af1-4e24-8cf7-5fac7aa63077", "node_type": "1", "metadata": {}, "hash": "2c8c16a8a1470e35526a92a91bcbb35c86b892c3738273e1aecd3a7ec652c881", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15880, "end_char_idx": 18993, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from model_downloads import HFModelDownloadsTool

tool = HFModelDownloadsTool()
```

You can also share your custom tool to the Hub by calling [`~Tool.push_to_hub`] on the tool. Make sure you've created a repository for it on the Hub and are using a token with read access.

```python
tool.push_to_hub("{your_username}/hf-model-downloads")
```

Load the tool with the [`~Tool.load_tool`] function and pass it to the `tools` parameter in your agent.

```python
from transformers import load_tool, CodeAgent

model_download_tool = load_tool("m-ric/hf-model-downloads")
agent = CodeAgent(tools=[model_download_tool], llm_engine=llm_engine)
agent.run(
    "Can you give me the name of the model that has the most downloads in the 'text-to-video' task on the Hugging Face Hub?"
)
```

You get the following:
```text
======== New task ========
Can you give me the name of the model that has the most downloads in the 'text-to-video' task on the Hugging Face Hub?
==== Agent is executing the code below:
most_downloaded_model = model_download_counter(task="text-to-video")
print(f"The most downloaded model for the 'text-to-video' task is {most_downloaded_model}.")
====
```

And the output:
`"The most downloaded model for the 'text-to-video' task is ByteDance/AnimateDiff-Lightning."`


### Manage your agent's toolbox

If you have already initialized an agent, it is inconvenient to reinitialize it from scratch with a tool you want to use. With Transformers, you can manage an agent's toolbox by adding or replacing a tool.

Let's add the `model_download_tool` to an existing agent initialized with only the default toolbox.

```python
from transformers import CodeAgent

agent = CodeAgent(tools=[], llm_engine=llm_engine, add_base_tools=True)
agent.toolbox.add_tool(model_download_tool)
```
Now we can leverage both the new tool and the previous text-to-speech tool:

```python
agent.run(
    "Can you read out loud the name of the model that has the most downloads in the 'text-to-video' task on the Hugging Face Hub and return the audio?"
)
```


| **Audio**                                                                                                                                            |
|------------------------------------------------------------------------------------------------------------------------------------------------------|
| <audio controls><source src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/damo.wav" type="audio/wav"/> |


> [!WARNING]
> Beware when adding tools to an agent that already works well because it can bias selection towards your tool or select another tool other than the one already defined.


Use the `agent.toolbox.update_tool()` method to replace an existing tool in the agent's toolbox.
This is useful if your new tool is a one-to-one replacement of the existing tool because the agent already knows how to perform that specific task.
Just make sure the new tool follows the same API as the replaced tool or adapt the system prompt template to ensure all examples using the replaced tool are updated.


key: _node_content
string_value:
{"id_": "f8f07093-4af1-4e24-8cf7-5fac7aa63077", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "000629d8-e2ab-478e-a3f9-03b91a734c74", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "62ce954f331567f9b9f6455ed5413d798ada5cbd9b93211b12cbc7d1cef70b45", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2526c759-82bb-45fb-b0fb-cef654cdd7ef", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e816f944c7def1e3f6652b5f2a01798554526f7793044778cff250693496e324", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2d33e369-7aed-45b5-b19e-c1a0ccf37edb", "node_type": "1", "metadata": {}, "hash": "b62fb82e1bdbd6ed7f0ddd97846387ecc4e088ee0ea82873ccaf71b598459f8e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17172, "end_char_idx": 19766, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Manage your agent's toolbox

If you have already initialized an agent, it is inconvenient to reinitialize it from scratch with a tool you want to use. With Transformers, you can manage an agent's toolbox by adding or replacing a tool.

Let's add the `model_download_tool` to an existing agent initialized with only the default toolbox.

```python
from transformers import CodeAgent

agent = CodeAgent(tools=[], llm_engine=llm_engine, add_base_tools=True)
agent.toolbox.add_tool(model_download_tool)
```
Now we can leverage both the new tool and the previous text-to-speech tool:

```python
agent.run(
    "Can you read out loud the name of the model that has the most downloads in the 'text-to-video' task on the Hugging Face Hub and return the audio?"
)
```


| **Audio**                                                                                                                                            |
|------------------------------------------------------------------------------------------------------------------------------------------------------|
| <audio controls><source src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/damo.wav" type="audio/wav"/> |


> [!WARNING]
> Beware when adding tools to an agent that already works well because it can bias selection towards your tool or select another tool other than the one already defined.


Use the `agent.toolbox.update_tool()` method to replace an existing tool in the agent's toolbox.
This is useful if your new tool is a one-to-one replacement of the existing tool because the agent already knows how to perform that specific task.
Just make sure the new tool follows the same API as the replaced tool or adapt the system prompt template to ensure all examples using the replaced tool are updated.


### Use a collection of tools

You can leverage tool collections by using the ToolCollection object, with the slug of the collection you want to use.
Then pass them as a list to initialize you agent, and start using them!

```py
from transformers import ToolCollection, ReactCodeAgent

image_tool_collection = ToolCollection(collection_slug="huggingface-tools/diffusion-tools-6630bb19a942c2306a2cdb6f")
agent = ReactCodeAgent(tools=[*image_tool_collection.tools], add_base_tools=True)

agent.run("Please draw me a picture of rivers and lakes.")
```

To speed up the start, tools are loaded only if called by the agent.

This gets you this image:

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/rivers_and_lakes.png">


key: _node_content
string_value:
{"id_": "2d33e369-7aed-45b5-b19e-c1a0ccf37edb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "000629d8-e2ab-478e-a3f9-03b91a734c74", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "62ce954f331567f9b9f6455ed5413d798ada5cbd9b93211b12cbc7d1cef70b45", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f8f07093-4af1-4e24-8cf7-5fac7aa63077", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "ed155f18601321ec16db69582beeac4b6aec8785b8e7b843c57bf4da3cc43178", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0e5c4e3f-47ca-4868-b245-782bab3df77a", "node_type": "1", "metadata": {}, "hash": "37fb43898d0c221117e7f563ec97a41da207080bed7328dedf0a62e450175a68", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17937, "end_char_idx": 21751, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
| **Audio**                                                                                                                                            |
|------------------------------------------------------------------------------------------------------------------------------------------------------|
| <audio controls><source src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/damo.wav" type="audio/wav"/> |


> [!WARNING]
> Beware when adding tools to an agent that already works well because it can bias selection towards your tool or select another tool other than the one already defined.


Use the `agent.toolbox.update_tool()` method to replace an existing tool in the agent's toolbox.
This is useful if your new tool is a one-to-one replacement of the existing tool because the agent already knows how to perform that specific task.
Just make sure the new tool follows the same API as the replaced tool or adapt the system prompt template to ensure all examples using the replaced tool are updated.


### Use a collection of tools

You can leverage tool collections by using the ToolCollection object, with the slug of the collection you want to use.
Then pass them as a list to initialize you agent, and start using them!

```py
from transformers import ToolCollection, ReactCodeAgent

image_tool_collection = ToolCollection(collection_slug="huggingface-tools/diffusion-tools-6630bb19a942c2306a2cdb6f")
agent = ReactCodeAgent(tools=[*image_tool_collection.tools], add_base_tools=True)

agent.run("Please draw me a picture of rivers and lakes.")
```

To speed up the start, tools are loaded only if called by the agent.

This gets you this image:

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/rivers_and_lakes.png">


### Use gradio-tools

[gradio-tools](https://github.com/freddyaboulton/gradio-tools) is a powerful library that allows using Hugging
Face Spaces as tools. It supports many existing Spaces as well as custom Spaces.

Transformers supports `gradio_tools` with the [`Tool.from_gradio`] method. For example, let's use the [`StableDiffusionPromptGeneratorTool`](https://github.com/freddyaboulton/gradio-tools/blob/main/gradio_tools/tools/prompt_generator.py) from `gradio-tools` toolkit for improving prompts to generate better images.

Import and instantiate the tool, then pass it to the `Tool.from_gradio` method:

```python
from gradio_tools import StableDiffusionPromptGeneratorTool
from transformers import Tool, load_tool, CodeAgent

gradio_prompt_generator_tool = StableDiffusionPromptGeneratorTool()
prompt_generator_tool = Tool.from_gradio(gradio_prompt_generator_tool)
```

Now you can use it just like any other tool. For example, let's improve the prompt  `a rabbit wearing a space suit`.

```python
image_generation_tool = load_tool('huggingface-tools/text-to-image')
agent = CodeAgent(tools=[prompt_generator_tool, image_generation_tool], llm_engine=llm_engine)

agent.run(
    "Improve this prompt, then generate an image of it.", prompt='A rabbit wearing a space suit'
)
```

The model adequately leverages the tool:
```text
======== New task ========
Improve this prompt, then generate an image of it.
You have been provided with these initial arguments: {'prompt': 'A rabbit wearing a space suit'}.
==== Agent is executing the code below:
improved_prompt = StableDiffusionPromptGenerator(query=prompt)
while improved_prompt == "QUEUE_FULL":
    improved_prompt = StableDiffusionPromptGenerator(query=prompt)
print(f"The improved prompt is {improved_prompt}.")
image = image_generator(prompt=improved_prompt)
====
```

Before finally generating the image:

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/rabbit.png">


key: _node_content
string_value:
{"id_": "0e5c4e3f-47ca-4868-b245-782bab3df77a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "000629d8-e2ab-478e-a3f9-03b91a734c74", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "62ce954f331567f9b9f6455ed5413d798ada5cbd9b93211b12cbc7d1cef70b45", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2d33e369-7aed-45b5-b19e-c1a0ccf37edb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/agents/", "title": "Agents and tools", "tokens": 5169, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e6e09d144bb8e2bc49cc700838575a72238556fb6f97a20039c336efabeaf149", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21754, "end_char_idx": 22587, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
> [!WARNING]
> gradio-tools require *textual* inputs and outputs even when working with different modalities like image and audio objects. Image and audio inputs and outputs are currently incompatible.

### Use LangChain tools

We love Langchain and think it has a very compelling suite of tools.
To import a tool from LangChain, use the `from_langchain()` method.

Here is how you can use it to recreate the intro's search result using a LangChain web search tool.

```python
from langchain.agents import load_tools
from transformers import Tool, ReactCodeAgent

search_tool = Tool.from_langchain(load_tools(["serpapi"])[0])

agent = ReactCodeAgent(tools=[search_tool])

agent.run("How many more blocks (also denoted as layers) in BERT base encoder than the encoder from the architecture proposed in Attention is All You Need?")
```


key: _node_content
string_value:
{"id_": "26b3b099-b8ab-44e3-b759-b6d6a9180907", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/philosophy/", "title": "Philosophy", "tokens": 1289, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "606a80cc-db89-4dbf-a360-e69f5a574e8a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/philosophy/", "title": "Philosophy", "tokens": 1289, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "41ec03cf8834625cfaae3f2d8e11e3019326ce30589043df7822883c4be50558", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "320e8764-f9df-4d8b-b3e1-db7617fd5ea5", "node_type": "1", "metadata": {}, "hash": "6b282abe1fdaeab27847be3c03cf5c9f5af417d1ff98b5cc7d7500ce718b05e3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3305, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Philosophy

🤗 Transformers is an opinionated library built for:

- machine learning researchers and educators seeking to use, study or extend large-scale Transformers models.
- hands-on practitioners who want to fine-tune those models or serve them in production, or both.
- engineers who just want to download a pretrained model and use it to solve a given machine learning task.

The library was designed with two strong goals in mind:

1. Be as easy and fast to use as possible:

  - We strongly limited the number of user-facing abstractions to learn, in fact, there are almost no abstractions,
    just three standard classes required to use each model: [configuration](main_classes/configuration),
    [models](main_classes/model), and a preprocessing class ([tokenizer](main_classes/tokenizer) for NLP, [image processor](main_classes/image_processor) for vision, [feature extractor](main_classes/feature_extractor) for audio, and [processor](main_classes/processors) for multimodal inputs).
  - All of these classes can be initialized in a simple and unified way from pretrained instances by using a common
    `from_pretrained()` method which downloads (if needed), caches and
    loads the related class instance and associated data (configurations' hyperparameters, tokenizers' vocabulary,
    and models' weights) from a pretrained checkpoint provided on [Hugging Face Hub](https://huggingface.co/models) or your own saved checkpoint.
  - On top of those three base classes, the library provides two APIs: [`pipeline`] for quickly
    using a model for inference on a given task and [`Trainer`] to quickly train or fine-tune a PyTorch model (all TensorFlow models are compatible with `Keras.fit`).
  - As a consequence, this library is NOT a modular toolbox of building blocks for neural nets. If you want to
    extend or build upon the library, just use regular Python, PyTorch, TensorFlow, Keras modules and inherit from the base
    classes of the library to reuse functionalities like model loading and saving. If you'd like to learn more about our coding philosophy for models, check out our [Repeat Yourself](https://huggingface.co/blog/transformers-design-philosophy) blog post.

2. Provide state-of-the-art models with performances as close as possible to the original models:

  - We provide at least one example for each architecture which reproduces a result provided by the official authors
    of said architecture.
  - The code is usually as close to the original code base as possible which means some PyTorch code may be not as
    *pytorchic* as it could be as a result of being converted TensorFlow code and vice versa.

A few other goals:

- Expose the models' internals as consistently as possible:

  - We give access, using a single API, to the full hidden-states and attention weights.
  - The preprocessing classes and base model APIs are standardized to easily switch between models.

- Incorporate a subjective selection of promising tools for fine-tuning and investigating these models:

  - A simple and consistent way to add new tokens to the vocabulary and embeddings for fine-tuning.
  - Simple ways to mask and prune Transformer heads.

- Easily switch between PyTorch, TensorFlow 2.0 and Flax, allowing training with one framework and inference with another.


key: _node_content
string_value:
{"id_": "320e8764-f9df-4d8b-b3e1-db7617fd5ea5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/philosophy/", "title": "Philosophy", "tokens": 1289, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "606a80cc-db89-4dbf-a360-e69f5a574e8a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/philosophy/", "title": "Philosophy", "tokens": 1289, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "41ec03cf8834625cfaae3f2d8e11e3019326ce30589043df7822883c4be50558", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "26b3b099-b8ab-44e3-b759-b6d6a9180907", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/philosophy/", "title": "Philosophy", "tokens": 1289, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b10bbe2979959463ae8e9b172351f77643071b787df46584dfea81735176dfe4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "670d5306-525b-44ee-ac8e-d228d92eed91", "node_type": "1", "metadata": {}, "hash": "b5e72fff8aaaa7dee0ec177f6f1691305ae94d5e91286918e78e74a15032e1ec", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1450, "end_char_idx": 5068, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- On top of those three base classes, the library provides two APIs: [`pipeline`] for quickly
    using a model for inference on a given task and [`Trainer`] to quickly train or fine-tune a PyTorch model (all TensorFlow models are compatible with `Keras.fit`).
  - As a consequence, this library is NOT a modular toolbox of building blocks for neural nets. If you want to
    extend or build upon the library, just use regular Python, PyTorch, TensorFlow, Keras modules and inherit from the base
    classes of the library to reuse functionalities like model loading and saving. If you'd like to learn more about our coding philosophy for models, check out our [Repeat Yourself](https://huggingface.co/blog/transformers-design-philosophy) blog post.

2. Provide state-of-the-art models with performances as close as possible to the original models:

  - We provide at least one example for each architecture which reproduces a result provided by the official authors
    of said architecture.
  - The code is usually as close to the original code base as possible which means some PyTorch code may be not as
    *pytorchic* as it could be as a result of being converted TensorFlow code and vice versa.

A few other goals:

- Expose the models' internals as consistently as possible:

  - We give access, using a single API, to the full hidden-states and attention weights.
  - The preprocessing classes and base model APIs are standardized to easily switch between models.

- Incorporate a subjective selection of promising tools for fine-tuning and investigating these models:

  - A simple and consistent way to add new tokens to the vocabulary and embeddings for fine-tuning.
  - Simple ways to mask and prune Transformer heads.

- Easily switch between PyTorch, TensorFlow 2.0 and Flax, allowing training with one framework and inference with another.

## Main concepts

The library is built around three types of classes for each model:

- **Model classes** can be PyTorch models ([torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module)), Keras models ([tf.keras.Model](https://www.tensorflow.org/api_docs/python/tf/keras/Model)) or JAX/Flax models ([flax.linen.Module](https://flax.readthedocs.io/en/latest/api_reference/flax.linen/module.html)) that work with the pretrained weights provided in the library.
- **Configuration classes** store the hyperparameters required to build a model (such as the number of layers and hidden size). You don't always need to instantiate these yourself. In particular, if you are using a pretrained model without any modification, creating the model will automatically take care of instantiating the configuration (which is part of the model).
- **Preprocessing classes** convert the raw data into a format accepted by the model. A [tokenizer](main_classes/tokenizer) stores the vocabulary for each model and provide methods for encoding and decoding strings in a list of token embedding indices to be fed to a model. [Image processors](main_classes/image_processor) preprocess vision inputs, [feature extractors](main_classes/feature_extractor) preprocess audio inputs, and a [processor](main_classes/processors) handles multimodal inputs.

All these classes can be instantiated from pretrained instances, saved locally, and shared on the Hub with three methods:

- `from_pretrained()` lets you instantiate a model, configuration, and preprocessing class from a pretrained version either
  provided by the library itself (the supported models can be found on the [Model Hub](https://huggingface.co/models)) or
  stored locally (or on a server) by the user.


key: _node_content
string_value:
{"id_": "670d5306-525b-44ee-ac8e-d228d92eed91", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/philosophy/", "title": "Philosophy", "tokens": 1289, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "606a80cc-db89-4dbf-a360-e69f5a574e8a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/philosophy/", "title": "Philosophy", "tokens": 1289, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "41ec03cf8834625cfaae3f2d8e11e3019326ce30589043df7822883c4be50558", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "320e8764-f9df-4d8b-b3e1-db7617fd5ea5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/philosophy/", "title": "Philosophy", "tokens": 1289, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5c80a3a43916fa0e185828513749d8a365da2d5ddcd326bde55101c384716305", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3307, "end_char_idx": 5351, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Main concepts

The library is built around three types of classes for each model:

- **Model classes** can be PyTorch models ([torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module)), Keras models ([tf.keras.Model](https://www.tensorflow.org/api_docs/python/tf/keras/Model)) or JAX/Flax models ([flax.linen.Module](https://flax.readthedocs.io/en/latest/api_reference/flax.linen/module.html)) that work with the pretrained weights provided in the library.
- **Configuration classes** store the hyperparameters required to build a model (such as the number of layers and hidden size). You don't always need to instantiate these yourself. In particular, if you are using a pretrained model without any modification, creating the model will automatically take care of instantiating the configuration (which is part of the model).
- **Preprocessing classes** convert the raw data into a format accepted by the model. A [tokenizer](main_classes/tokenizer) stores the vocabulary for each model and provide methods for encoding and decoding strings in a list of token embedding indices to be fed to a model. [Image processors](main_classes/image_processor) preprocess vision inputs, [feature extractors](main_classes/feature_extractor) preprocess audio inputs, and a [processor](main_classes/processors) handles multimodal inputs.

All these classes can be instantiated from pretrained instances, saved locally, and shared on the Hub with three methods:

- `from_pretrained()` lets you instantiate a model, configuration, and preprocessing class from a pretrained version either
  provided by the library itself (the supported models can be found on the [Model Hub](https://huggingface.co/models)) or
  stored locally (or on a server) by the user.
- `save_pretrained()` lets you save a model, configuration, and preprocessing class locally so that it can be reloaded using
  `from_pretrained()`.
- `push_to_hub()` lets you share a model, configuration, and a preprocessing class to the Hub, so it is easily accessible to everyone.


key: _node_content
string_value:
{"id_": "1b836d2c-f10d-4734-8c02-02ed49bc5930", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_pipeline/", "title": "How to create a custom pipeline?", "tokens": 2459, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10317c3d-d1c5-4dcf-9036-94b2aee5ddbd", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_pipeline/", "title": "How to create a custom pipeline?", "tokens": 2459, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e4596e53352771f7f90ea172257d520a7e2b7f6c92eb19179e3ed287fddd29de", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "35da5b2c-6377-4668-806a-0ec4c2404e05", "node_type": "1", "metadata": {}, "hash": "4df29ee53ebefe7dd4e6d4cfe2a526e8f13c5186113c82817fdfa82629878812", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 891, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# How to create a custom pipeline?

In this guide, we will see how to create a custom pipeline and share it on the [Hub](https://hf.co/models) or add it to the
🤗 Transformers library.

First and foremost, you need to decide the raw entries the pipeline will be able to take. It can be strings, raw bytes,
dictionaries or whatever seems to be the most likely desired input. Try to keep these inputs as pure Python as possible
as it makes compatibility easier (even through other languages via JSON). Those will be the `inputs` of the
pipeline (`preprocess`).

Then define the `outputs`. Same policy as the `inputs`. The simpler, the better. Those will be the outputs of
`postprocess` method.

Start by inheriting the base class `Pipeline` with the 4 methods needed to implement `preprocess`,
`_forward`, `postprocess`, and `_sanitize_parameters`.


```python
from transformers import Pipeline


key: _node_content
string_value:
{"id_": "35da5b2c-6377-4668-806a-0ec4c2404e05", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_pipeline/", "title": "How to create a custom pipeline?", "tokens": 2459, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10317c3d-d1c5-4dcf-9036-94b2aee5ddbd", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_pipeline/", "title": "How to create a custom pipeline?", "tokens": 2459, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e4596e53352771f7f90ea172257d520a7e2b7f6c92eb19179e3ed287fddd29de", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1b836d2c-f10d-4734-8c02-02ed49bc5930", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_pipeline/", "title": "How to create a custom pipeline?", "tokens": 2459, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "86f7a70e93afb584e90a6f56cd2e00ddd0f24298afd8f3a511b0dde31fe214cb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5d06495f-89ea-495a-800f-b89f82139829", "node_type": "1", "metadata": {}, "hash": "3ad1db1736d739b5537e5f409aee2129ce00d29d385af562a6e308d11cfa206b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3544, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# How to create a custom pipeline?

In this guide, we will see how to create a custom pipeline and share it on the [Hub](https://hf.co/models) or add it to the
🤗 Transformers library.

First and foremost, you need to decide the raw entries the pipeline will be able to take. It can be strings, raw bytes,
dictionaries or whatever seems to be the most likely desired input. Try to keep these inputs as pure Python as possible
as it makes compatibility easier (even through other languages via JSON). Those will be the `inputs` of the
pipeline (`preprocess`).

Then define the `outputs`. Same policy as the `inputs`. The simpler, the better. Those will be the outputs of
`postprocess` method.

Start by inheriting the base class `Pipeline` with the 4 methods needed to implement `preprocess`,
`_forward`, `postprocess`, and `_sanitize_parameters`.


```python
from transformers import Pipeline


class MyPipeline(Pipeline):
    def _sanitize_parameters(self, **kwargs):
        preprocess_kwargs = {}
        if "maybe_arg" in kwargs:
            preprocess_kwargs["maybe_arg"] = kwargs["maybe_arg"]
        return preprocess_kwargs, {}, {}

    def preprocess(self, inputs, maybe_arg=2):
        model_input = Tensor(inputs["input_ids"])
        return {"model_input": model_input}

    def _forward(self, model_inputs):
        # model_inputs == {"model_input": model_input}
        outputs = self.model(**model_inputs)
        # Maybe {"logits": Tensor(...)}
        return outputs

    def postprocess(self, model_outputs):
        best_class = model_outputs["logits"].softmax(-1)
        return best_class
```

The structure of this breakdown is to support relatively seamless support for CPU/GPU, while supporting doing
pre/postprocessing on the CPU on different threads

`preprocess` will take the originally defined inputs, and turn them into something feedable to the model. It might
contain more information and is usually a `Dict`.

`_forward` is the implementation detail and is not meant to be called directly. `forward` is the preferred
called method as it contains safeguards to make sure everything is working on the expected device. If anything is
linked to a real model it belongs in the `_forward` method, anything else is in the preprocess/postprocess.

`postprocess` methods will take the output of `_forward` and turn it into the final output that was decided
earlier.

`_sanitize_parameters` exists to allow users to pass any parameters whenever they wish, be it at initialization
time `pipeline(...., maybe_arg=4)` or at call time `pipe = pipeline(...); output = pipe(...., maybe_arg=4)`.

The returns of `_sanitize_parameters` are the 3 dicts of kwargs that will be passed directly to `preprocess`,
`_forward`, and `postprocess`. Don't fill anything if the caller didn't call with any extra parameter. That
allows to keep the default arguments in the function definition which is always more "natural".

A classic example would be a `top_k` argument in the post processing in classification tasks.

```python
>>> pipe = pipeline("my-new-task")
>>> pipe("This is a test")
[{"label": "1-star", "score": 0.8}, {"label": "2-star", "score": 0.1}, {"label": "3-star", "score": 0.05}
{"label": "4-star", "score": 0.025}, {"label": "5-star", "score": 0.025}]

>>> pipe("This is a test", top_k=2)
[{"label": "1-star", "score": 0.8}, {"label": "2-star", "score": 0.1}]
```

In order to achieve that, we'll update our `postprocess` method with a default parameter to `5`. and edit
`_sanitize_parameters` to allow this new parameter.


key: _node_content
string_value:
{"id_": "5d06495f-89ea-495a-800f-b89f82139829", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_pipeline/", "title": "How to create a custom pipeline?", "tokens": 2459, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10317c3d-d1c5-4dcf-9036-94b2aee5ddbd", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_pipeline/", "title": "How to create a custom pipeline?", "tokens": 2459, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e4596e53352771f7f90ea172257d520a7e2b7f6c92eb19179e3ed287fddd29de", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "35da5b2c-6377-4668-806a-0ec4c2404e05", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_pipeline/", "title": "How to create a custom pipeline?", "tokens": 2459, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "78c6b17dca77486c5379a65a02f5a1331b7bf712dccb827e2f017089a380c392", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a3056edb-ebc9-4968-84f3-db00bad93c89", "node_type": "1", "metadata": {}, "hash": "645742bf4165fdef73410a9af72f609f053bf3891fc91390c816e280edcd69fa", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3547, "end_char_idx": 6671, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def postprocess(self, model_outputs, top_k=5):
    best_class = model_outputs["logits"].softmax(-1)
    # Add logic to handle top_k
    return best_class


def _sanitize_parameters(self, **kwargs):
    preprocess_kwargs = {}
    if "maybe_arg" in kwargs:
        preprocess_kwargs["maybe_arg"] = kwargs["maybe_arg"]

    postprocess_kwargs = {}
    if "top_k" in kwargs:
        postprocess_kwargs["top_k"] = kwargs["top_k"]
    return preprocess_kwargs, {}, postprocess_kwargs
```

Try to keep the inputs/outputs very simple and ideally JSON-serializable as it makes the pipeline usage very easy
without requiring users to understand new kinds of objects. It's also relatively common to support many different types
of arguments for ease of use (audio files, which can be filenames, URLs or pure bytes)



## Adding it to the list of supported tasks

To register your `new-task` to the list of supported tasks, you have to add it to the `PIPELINE_REGISTRY`:

```python
from transformers.pipelines import PIPELINE_REGISTRY

PIPELINE_REGISTRY.register_pipeline(
    "new-task",
    pipeline_class=MyPipeline,
    pt_model=AutoModelForSequenceClassification,
)
```

You can specify a default model if you want, in which case it should come with a specific revision (which can be the name of a branch or a commit hash, here we took `"abcdef"`) as well as the type:

```python
PIPELINE_REGISTRY.register_pipeline(
    "new-task",
    pipeline_class=MyPipeline,
    pt_model=AutoModelForSequenceClassification,
    default={"pt": ("user/awesome_model", "abcdef")},
    type="text",  # current support type: text, audio, image, multimodal
)
```

## Share your pipeline on the Hub

To share your custom pipeline on the Hub, you just have to save the custom code of your `Pipeline` subclass in a
python file. For instance, let's say we want to use a custom pipeline for sentence pair classification like this:

```py
import numpy as np

from transformers import Pipeline


def softmax(outputs):
    maxes = np.max(outputs, axis=-1, keepdims=True)
    shifted_exp = np.exp(outputs - maxes)
    return shifted_exp / shifted_exp.sum(axis=-1, keepdims=True)


class PairClassificationPipeline(Pipeline):
    def _sanitize_parameters(self, **kwargs):
        preprocess_kwargs = {}
        if "second_text" in kwargs:
            preprocess_kwargs["second_text"] = kwargs["second_text"]
        return preprocess_kwargs, {}, {}

    def preprocess(self, text, second_text=None):
        return self.tokenizer(text, text_pair=second_text, return_tensors=self.framework)

    def _forward(self, model_inputs):
        return self.model(**model_inputs)

    def postprocess(self, model_outputs):
        logits = model_outputs.logits[0].numpy()
        probabilities = softmax(logits)

        best_class = np.argmax(probabilities)
        label = self.model.config.id2label[best_class]
        score = probabilities[best_class].item()
        logits = logits.tolist()
        return {"label": label, "score": score, "logits": logits}
```

The implementation is framework agnostic, and will work for PyTorch and TensorFlow models.


key: _node_content
string_value:
{"id_": "a3056edb-ebc9-4968-84f3-db00bad93c89", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_pipeline/", "title": "How to create a custom pipeline?", "tokens": 2459, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10317c3d-d1c5-4dcf-9036-94b2aee5ddbd", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_pipeline/", "title": "How to create a custom pipeline?", "tokens": 2459, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e4596e53352771f7f90ea172257d520a7e2b7f6c92eb19179e3ed287fddd29de", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5d06495f-89ea-495a-800f-b89f82139829", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_pipeline/", "title": "How to create a custom pipeline?", "tokens": 2459, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "dbd8988e59f0c64f7839aca10f88759d16fa5bcf13b7466dd88b742967b58b1f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4998ce5a-b315-4a89-907e-cbe45d282ab6", "node_type": "1", "metadata": {}, "hash": "0109fa6cdc0dc26448456cb225bc2491261baa4645be00eadc10cd95393cd269", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5522, "end_char_idx": 8897, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
def softmax(outputs):
    maxes = np.max(outputs, axis=-1, keepdims=True)
    shifted_exp = np.exp(outputs - maxes)
    return shifted_exp / shifted_exp.sum(axis=-1, keepdims=True)


class PairClassificationPipeline(Pipeline):
    def _sanitize_parameters(self, **kwargs):
        preprocess_kwargs = {}
        if "second_text" in kwargs:
            preprocess_kwargs["second_text"] = kwargs["second_text"]
        return preprocess_kwargs, {}, {}

    def preprocess(self, text, second_text=None):
        return self.tokenizer(text, text_pair=second_text, return_tensors=self.framework)

    def _forward(self, model_inputs):
        return self.model(**model_inputs)

    def postprocess(self, model_outputs):
        logits = model_outputs.logits[0].numpy()
        probabilities = softmax(logits)

        best_class = np.argmax(probabilities)
        label = self.model.config.id2label[best_class]
        score = probabilities[best_class].item()
        logits = logits.tolist()
        return {"label": label, "score": score, "logits": logits}
```

The implementation is framework agnostic, and will work for PyTorch and TensorFlow models. If we have saved this in
a file named `pair_classification.py`, we can then import it and register it like this:

```py
from pair_classification import PairClassificationPipeline
from transformers.pipelines import PIPELINE_REGISTRY
from transformers import AutoModelForSequenceClassification, TFAutoModelForSequenceClassification

PIPELINE_REGISTRY.register_pipeline(
    "pair-classification",
    pipeline_class=PairClassificationPipeline,
    pt_model=AutoModelForSequenceClassification,
    tf_model=TFAutoModelForSequenceClassification,
)
```

Once this is done, we can use it with a pretrained model. For instance `sgugger/finetuned-bert-mrpc` has been
fine-tuned on the MRPC dataset, which classifies pairs of sentences as paraphrases or not.

```py
from transformers import pipeline

classifier = pipeline("pair-classification", model="sgugger/finetuned-bert-mrpc")
```

Then we can share it on the Hub by using the `push_to_hub` method:

```py
classifier.push_to_hub("test-dynamic-pipeline")
```

This will copy the file where you defined `PairClassificationPipeline` inside the folder `"test-dynamic-pipeline"`,
along with saving the model and tokenizer of the pipeline, before pushing everything into the repository
`{your_username}/test-dynamic-pipeline`. After that, anyone can use it as long as they provide the option
`trust_remote_code=True`:

```py
from transformers import pipeline

classifier = pipeline(model="{your_username}/test-dynamic-pipeline", trust_remote_code=True)
```

## Add the pipeline to 🤗 Transformers

If you want to contribute your pipeline to 🤗 Transformers, you will need to add a new module in the `pipelines` submodule
with the code of your pipeline, then add it to the list of tasks defined in `pipelines/__init__.py`.

Then you will need to add tests. Create a new file `tests/test_pipelines_MY_PIPELINE.py` with examples of the other tests.

The `run_pipeline_test` function will be very generic and run on small random models on every possible
architecture as defined by `model_mapping` and `tf_model_mapping`.

This is very important to test future compatibility, meaning if someone adds a new model for
`XXXForQuestionAnswering` then the pipeline test will attempt to run on it.


key: _node_content
string_value:
{"id_": "4998ce5a-b315-4a89-907e-cbe45d282ab6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_pipeline/", "title": "How to create a custom pipeline?", "tokens": 2459, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "10317c3d-d1c5-4dcf-9036-94b2aee5ddbd", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_pipeline/", "title": "How to create a custom pipeline?", "tokens": 2459, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e4596e53352771f7f90ea172257d520a7e2b7f6c92eb19179e3ed287fddd29de", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a3056edb-ebc9-4968-84f3-db00bad93c89", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/add_new_pipeline/", "title": "How to create a custom pipeline?", "tokens": 2459, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6acc6bc2597e0c4d3d181b939bb477fb9010a4ad306e8a99c546bbca06ec8630", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7279, "end_char_idx": 10081, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For instance `sgugger/finetuned-bert-mrpc` has been
fine-tuned on the MRPC dataset, which classifies pairs of sentences as paraphrases or not.

```py
from transformers import pipeline

classifier = pipeline("pair-classification", model="sgugger/finetuned-bert-mrpc")
```

Then we can share it on the Hub by using the `push_to_hub` method:

```py
classifier.push_to_hub("test-dynamic-pipeline")
```

This will copy the file where you defined `PairClassificationPipeline` inside the folder `"test-dynamic-pipeline"`,
along with saving the model and tokenizer of the pipeline, before pushing everything into the repository
`{your_username}/test-dynamic-pipeline`. After that, anyone can use it as long as they provide the option
`trust_remote_code=True`:

```py
from transformers import pipeline

classifier = pipeline(model="{your_username}/test-dynamic-pipeline", trust_remote_code=True)
```

## Add the pipeline to 🤗 Transformers

If you want to contribute your pipeline to 🤗 Transformers, you will need to add a new module in the `pipelines` submodule
with the code of your pipeline, then add it to the list of tasks defined in `pipelines/__init__.py`.

Then you will need to add tests. Create a new file `tests/test_pipelines_MY_PIPELINE.py` with examples of the other tests.

The `run_pipeline_test` function will be very generic and run on small random models on every possible
architecture as defined by `model_mapping` and `tf_model_mapping`.

This is very important to test future compatibility, meaning if someone adds a new model for
`XXXForQuestionAnswering` then the pipeline test will attempt to run on it. Because the models are random it's
impossible to check for actual values, that's why there is a helper `ANY` that will simply attempt to match the
output of the pipeline TYPE.

You also *need* to implement 2 (ideally 4) tests.

- `test_small_model_pt` : Define 1 small model for this pipeline (doesn't matter if the results don't make sense)
  and test the pipeline outputs. The results should be the same as `test_small_model_tf`.
- `test_small_model_tf` : Define 1 small model for this pipeline (doesn't matter if the results don't make sense)
  and test the pipeline outputs. The results should be the same as `test_small_model_pt`.
- `test_large_model_pt` (`optional`): Tests the pipeline on a real pipeline where the results are supposed to
  make sense. These tests are slow and should be marked as such. Here the goal is to showcase the pipeline and to make
  sure there is no drift in future releases.
- `test_large_model_tf` (`optional`): Tests the pipeline on a real pipeline where the results are supposed to
  make sense. These tests are slow and should be marked as such. Here the goal is to showcase the pipeline and to make
  sure there is no drift in future releases.


key: _node_content
string_value:
{"id_": "75d2de3a-fda5-47f9-8398-e198a674c7e2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/multilingual/", "title": "Multilingual models for inference", "tokens": 2131, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5a015970-bcf8-4baf-8cf0-f1ff3c80cf9c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/multilingual/", "title": "Multilingual models for inference", "tokens": 2131, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4923899e6781666bba557b0a4f2ca8e1c475029a627a1da6de67d84bae3b2a98", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3523574d-9d3c-441f-9242-d281c6b0739c", "node_type": "1", "metadata": {}, "hash": "6746c3bd46cebe250934c963231c302122487af67cac19d1d09d2848ba1ac019", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2495, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Multilingual models for inference

[[open-in-colab]]

There are several multilingual models in 🤗 Transformers, and their inference usage differs from monolingual models. Not *all* multilingual model usage is different though. Some models, like [google-bert/bert-base-multilingual-uncased](https://huggingface.co/google-bert/bert-base-multilingual-uncased), can be used just like a monolingual model. This guide will show you how to use multilingual models whose usage differs for inference.

## XLM

XLM has ten different checkpoints, only one of which is monolingual. The nine remaining model checkpoints can be split into two categories: the checkpoints that use language embeddings and those that don't.

### XLM with language embeddings

The following XLM models use language embeddings to specify the language used at inference:

- `FacebookAI/xlm-mlm-ende-1024` (Masked language modeling, English-German)
- `FacebookAI/xlm-mlm-enfr-1024` (Masked language modeling, English-French)
- `FacebookAI/xlm-mlm-enro-1024` (Masked language modeling, English-Romanian)
- `FacebookAI/xlm-mlm-xnli15-1024` (Masked language modeling, XNLI languages)
- `FacebookAI/xlm-mlm-tlm-xnli15-1024` (Masked language modeling + translation, XNLI languages)
- `FacebookAI/xlm-clm-enfr-1024` (Causal language modeling, English-French)
- `FacebookAI/xlm-clm-ende-1024` (Causal language modeling, English-German)

Language embeddings are represented as a tensor of the same shape as the `input_ids` passed to the model. The values in these tensors depend on the language used and are identified by the tokenizer's `lang2id` and `id2lang` attributes.

In this example, load the `FacebookAI/xlm-clm-enfr-1024` checkpoint (Causal language modeling, English-French):

```py
>>> import torch
>>> from transformers import XLMTokenizer, XLMWithLMHeadModel

>>> tokenizer = XLMTokenizer.from_pretrained("FacebookAI/xlm-clm-enfr-1024")
>>> model = XLMWithLMHeadModel.from_pretrained("FacebookAI/xlm-clm-enfr-1024")
```

The `lang2id` attribute of the tokenizer displays this model's languages and their ids:

```py
>>> print(tokenizer.lang2id)
{'en': 0, 'fr': 1}
```

Next, create an example input:

```py
>>> input_ids = torch.tensor([tokenizer.encode("Wikipedia was used to")])  # batch size of 1
```

Set the language id as `"en"` and use it to define the language embedding. The language embedding is a tensor filled with `0` since that is the language id for English. This tensor should be the same size as `input_ids`.


key: _node_content
string_value:
{"id_": "3523574d-9d3c-441f-9242-d281c6b0739c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/multilingual/", "title": "Multilingual models for inference", "tokens": 2131, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5a015970-bcf8-4baf-8cf0-f1ff3c80cf9c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/multilingual/", "title": "Multilingual models for inference", "tokens": 2131, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4923899e6781666bba557b0a4f2ca8e1c475029a627a1da6de67d84bae3b2a98", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "75d2de3a-fda5-47f9-8398-e198a674c7e2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/multilingual/", "title": "Multilingual models for inference", "tokens": 2131, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e8faf26d438b7f0a23e347d66363c99214ecefa3c97b6a6ce32a851311db6616", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8508ec70-07d9-4bf9-a4b2-a38c79ff77c0", "node_type": "1", "metadata": {}, "hash": "c88e540e08777fdc3c8b653f88e59d7067e87e01d6bdabaafe984397c2c4285d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1500, "end_char_idx": 4464, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The values in these tensors depend on the language used and are identified by the tokenizer's `lang2id` and `id2lang` attributes.

In this example, load the `FacebookAI/xlm-clm-enfr-1024` checkpoint (Causal language modeling, English-French):

```py
>>> import torch
>>> from transformers import XLMTokenizer, XLMWithLMHeadModel

>>> tokenizer = XLMTokenizer.from_pretrained("FacebookAI/xlm-clm-enfr-1024")
>>> model = XLMWithLMHeadModel.from_pretrained("FacebookAI/xlm-clm-enfr-1024")
```

The `lang2id` attribute of the tokenizer displays this model's languages and their ids:

```py
>>> print(tokenizer.lang2id)
{'en': 0, 'fr': 1}
```

Next, create an example input:

```py
>>> input_ids = torch.tensor([tokenizer.encode("Wikipedia was used to")])  # batch size of 1
```

Set the language id as `"en"` and use it to define the language embedding. The language embedding is a tensor filled with `0` since that is the language id for English. This tensor should be the same size as `input_ids`. 

```py
>>> language_id = tokenizer.lang2id["en"]  # 0
>>> langs = torch.tensor([language_id] * input_ids.shape[1])  # torch.tensor([0, 0, 0, ..., 0])

>>> # We reshape it to be of size (batch_size, sequence_length)
>>> langs = langs.view(1, -1)  # is now of shape [1, sequence_length] (we have a batch size of 1)
```

Now you can pass the `input_ids` and language embedding to the model:

```py
>>> outputs = model(input_ids, langs=langs)
```

The [run_generation.py](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-generation/run_generation.py) script can generate text with language embeddings using the `xlm-clm` checkpoints.

### XLM without language embeddings

The following XLM models do not require language embeddings during inference:

- `FacebookAI/xlm-mlm-17-1280` (Masked language modeling, 17 languages)
- `FacebookAI/xlm-mlm-100-1280` (Masked language modeling, 100 languages)

These models are used for generic sentence representations, unlike the previous XLM checkpoints.

## BERT

The following BERT models can be used for multilingual tasks:

- `google-bert/bert-base-multilingual-uncased` (Masked language modeling + Next sentence prediction, 102 languages)
- `google-bert/bert-base-multilingual-cased` (Masked language modeling + Next sentence prediction, 104 languages)

These models do not require language embeddings during inference. They should identify the language from the
context and infer accordingly.

## XLM-RoBERTa

The following XLM-RoBERTa models can be used for multilingual tasks:

- `FacebookAI/xlm-roberta-base` (Masked language modeling, 100 languages)
- `FacebookAI/xlm-roberta-large` (Masked language modeling, 100 languages)

XLM-RoBERTa was trained on 2.5TB of newly created and cleaned CommonCrawl data in 100 languages. It provides strong gains over previously released multilingual models like mBERT or XLM on downstream tasks like classification, sequence labeling, and question answering.


key: _node_content
string_value:
{"id_": "8508ec70-07d9-4bf9-a4b2-a38c79ff77c0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/multilingual/", "title": "Multilingual models for inference", "tokens": 2131, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5a015970-bcf8-4baf-8cf0-f1ff3c80cf9c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/multilingual/", "title": "Multilingual models for inference", "tokens": 2131, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4923899e6781666bba557b0a4f2ca8e1c475029a627a1da6de67d84bae3b2a98", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3523574d-9d3c-441f-9242-d281c6b0739c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/multilingual/", "title": "Multilingual models for inference", "tokens": 2131, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "325fb38ff9cba578389bab1a28cd5a26f81e6ce0258aafb5a2ad908242ef7044", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "29ab9778-f91f-47eb-8b07-0ad3113d6f9b", "node_type": "1", "metadata": {}, "hash": "f2b7cf5c8aae23948aac1ebb6029475bfe9f3b753b870c1e307871e701573ee0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3155, "end_char_idx": 5747, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### XLM without language embeddings

The following XLM models do not require language embeddings during inference:

- `FacebookAI/xlm-mlm-17-1280` (Masked language modeling, 17 languages)
- `FacebookAI/xlm-mlm-100-1280` (Masked language modeling, 100 languages)

These models are used for generic sentence representations, unlike the previous XLM checkpoints.

## BERT

The following BERT models can be used for multilingual tasks:

- `google-bert/bert-base-multilingual-uncased` (Masked language modeling + Next sentence prediction, 102 languages)
- `google-bert/bert-base-multilingual-cased` (Masked language modeling + Next sentence prediction, 104 languages)

These models do not require language embeddings during inference. They should identify the language from the
context and infer accordingly.

## XLM-RoBERTa

The following XLM-RoBERTa models can be used for multilingual tasks:

- `FacebookAI/xlm-roberta-base` (Masked language modeling, 100 languages)
- `FacebookAI/xlm-roberta-large` (Masked language modeling, 100 languages)

XLM-RoBERTa was trained on 2.5TB of newly created and cleaned CommonCrawl data in 100 languages. It provides strong gains over previously released multilingual models like mBERT or XLM on downstream tasks like classification, sequence labeling, and question answering.

## M2M100

The following M2M100 models can be used for multilingual translation:

- `facebook/m2m100_418M` (Translation)
- `facebook/m2m100_1.2B` (Translation)

In this example, load the `facebook/m2m100_418M` checkpoint to translate from Chinese to English. You can set the source language in the tokenizer:

```py
>>> from transformers import M2M100ForConditionalGeneration, M2M100Tokenizer

>>> en_text = "Do not meddle in the affairs of wizards, for they are subtle and quick to anger."
>>> chinese_text = "不要插手巫師的事務, 因為他們是微妙的, 很快就會發怒."

>>> tokenizer = M2M100Tokenizer.from_pretrained("facebook/m2m100_418M", src_lang="zh")
>>> model = M2M100ForConditionalGeneration.from_pretrained("facebook/m2m100_418M")
```

Tokenize the text:

```py
>>> encoded_zh = tokenizer(chinese_text, return_tensors="pt")
```

M2M100 forces the target language id as the first generated token to translate to the target language. Set the `forced_bos_token_id` to `en` in the `generate` method to translate to English:

```py
>>> generated_tokens = model.generate(**encoded_zh, forced_bos_token_id=tokenizer.get_lang_id("en"))
>>> tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
'Do not interfere with the matters of the witches, because they are delicate and will soon be angry.'


key: _node_content
string_value:
{"id_": "29ab9778-f91f-47eb-8b07-0ad3113d6f9b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/multilingual/", "title": "Multilingual models for inference", "tokens": 2131, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5a015970-bcf8-4baf-8cf0-f1ff3c80cf9c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/multilingual/", "title": "Multilingual models for inference", "tokens": 2131, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4923899e6781666bba557b0a4f2ca8e1c475029a627a1da6de67d84bae3b2a98", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8508ec70-07d9-4bf9-a4b2-a38c79ff77c0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/multilingual/", "title": "Multilingual models for inference", "tokens": 2131, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "14113e496b9325d2369e45a86c2e83fb94422e74fdf65fe2b4f994483346446c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a8b0fe70-7446-4b24-a985-9cb2f6833204", "node_type": "1", "metadata": {}, "hash": "17e8accc0199a9afbabeffe6c84019a2b4b25c21e397fd28984dfeebced517cf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4293, "end_char_idx": 7079, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
It provides strong gains over previously released multilingual models like mBERT or XLM on downstream tasks like classification, sequence labeling, and question answering.

## M2M100

The following M2M100 models can be used for multilingual translation:

- `facebook/m2m100_418M` (Translation)
- `facebook/m2m100_1.2B` (Translation)

In this example, load the `facebook/m2m100_418M` checkpoint to translate from Chinese to English. You can set the source language in the tokenizer:

```py
>>> from transformers import M2M100ForConditionalGeneration, M2M100Tokenizer

>>> en_text = "Do not meddle in the affairs of wizards, for they are subtle and quick to anger."
>>> chinese_text = "不要插手巫師的事務, 因為他們是微妙的, 很快就會發怒."

>>> tokenizer = M2M100Tokenizer.from_pretrained("facebook/m2m100_418M", src_lang="zh")
>>> model = M2M100ForConditionalGeneration.from_pretrained("facebook/m2m100_418M")
```

Tokenize the text:

```py
>>> encoded_zh = tokenizer(chinese_text, return_tensors="pt")
```

M2M100 forces the target language id as the first generated token to translate to the target language. Set the `forced_bos_token_id` to `en` in the `generate` method to translate to English:

```py
>>> generated_tokens = model.generate(**encoded_zh, forced_bos_token_id=tokenizer.get_lang_id("en"))
>>> tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
'Do not interfere with the matters of the witches, because they are delicate and will soon be angry.'
```

## MBart

The following MBart models can be used for multilingual translation:

- `facebook/mbart-large-50-one-to-many-mmt` (One-to-many multilingual machine translation, 50 languages)
- `facebook/mbart-large-50-many-to-many-mmt` (Many-to-many multilingual machine translation, 50 languages)
- `facebook/mbart-large-50-many-to-one-mmt` (Many-to-one multilingual machine translation, 50 languages)
- `facebook/mbart-large-50` (Multilingual translation, 50 languages)
- `facebook/mbart-large-cc25`

In this example, load the `facebook/mbart-large-50-many-to-many-mmt` checkpoint to translate Finnish to English. You can set the source language in the tokenizer:

```py
>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

>>> en_text = "Do not meddle in the affairs of wizards, for they are subtle and quick to anger."
>>> fi_text = "Älä sekaannu velhojen asioihin, sillä ne ovat hienovaraisia ja nopeasti vihaisia."

>>> tokenizer = AutoTokenizer.from_pretrained("facebook/mbart-large-50-many-to-many-mmt", src_lang="fi_FI")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("facebook/mbart-large-50-many-to-many-mmt")
```

Tokenize the text:

```py
>>> encoded_en = tokenizer(en_text, return_tensors="pt")
```

MBart forces the target language id as the first generated token to translate to the target language.


key: _node_content
string_value:
{"id_": "a8b0fe70-7446-4b24-a985-9cb2f6833204", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/multilingual/", "title": "Multilingual models for inference", "tokens": 2131, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5a015970-bcf8-4baf-8cf0-f1ff3c80cf9c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/multilingual/", "title": "Multilingual models for inference", "tokens": 2131, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4923899e6781666bba557b0a4f2ca8e1c475029a627a1da6de67d84bae3b2a98", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "29ab9778-f91f-47eb-8b07-0ad3113d6f9b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/multilingual/", "title": "Multilingual models for inference", "tokens": 2131, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "9ac56cfe7192cd76680ae1de02da4fa58925a5a63e52894369cf70abdbe201d7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5748, "end_char_idx": 7633, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```

## MBart

The following MBart models can be used for multilingual translation:

- `facebook/mbart-large-50-one-to-many-mmt` (One-to-many multilingual machine translation, 50 languages)
- `facebook/mbart-large-50-many-to-many-mmt` (Many-to-many multilingual machine translation, 50 languages)
- `facebook/mbart-large-50-many-to-one-mmt` (Many-to-one multilingual machine translation, 50 languages)
- `facebook/mbart-large-50` (Multilingual translation, 50 languages)
- `facebook/mbart-large-cc25`

In this example, load the `facebook/mbart-large-50-many-to-many-mmt` checkpoint to translate Finnish to English. You can set the source language in the tokenizer:

```py
>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

>>> en_text = "Do not meddle in the affairs of wizards, for they are subtle and quick to anger."
>>> fi_text = "Älä sekaannu velhojen asioihin, sillä ne ovat hienovaraisia ja nopeasti vihaisia."

>>> tokenizer = AutoTokenizer.from_pretrained("facebook/mbart-large-50-many-to-many-mmt", src_lang="fi_FI")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("facebook/mbart-large-50-many-to-many-mmt")
```

Tokenize the text:

```py
>>> encoded_en = tokenizer(en_text, return_tensors="pt")
```

MBart forces the target language id as the first generated token to translate to the target language. Set the `forced_bos_token_id` to `en` in the `generate` method to translate to English:

```py
>>> generated_tokens = model.generate(**encoded_en, forced_bos_token_id=tokenizer.lang_code_to_id["en_XX"])
>>> tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
"Don't interfere with the wizard's affairs, because they are subtle, will soon get angry."
```

If you are using the `facebook/mbart-large-50-many-to-one-mmt` checkpoint, you don't need to force the target language id as the first generated token otherwise the usage is the same.


key: _node_content
string_value:
{"id_": "f04e3316-eed5-4fd6-a343-2216446d0f3f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "76509118-4217-4e77-808d-b28da782333e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "ce4ac9fc34d09d2761a0d7cab7dd26c34ec62514a20ba7c2e604bed2dd024a79", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ce82d3b7-5fa3-4ca2-a1f8-0fefe4e0b84a", "node_type": "1", "metadata": {}, "hash": "1635fd4e22cf59f4b430f695011bf44b5fffba2c7c4672a4914ecc2f1e15e203", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3468, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Debugging

Training on multiple GPUs can be a tricky endeavor whether you're running into installation issues or communication problems between your GPUs. This debugging guide covers some issues you may run into and how to resolve them.

## DeepSpeed CUDA installation

If you're using DeepSpeed, you've probably already installed it with the following command.

```bash
pip install deepspeed
```

DeepSpeed compiles CUDA C++ code and it can be a potential source of errors when building PyTorch extensions that require CUDA. These errors depend on how CUDA is installed on your system, and this section focuses on PyTorch built with *CUDA 10.2*.

<Tip>

For any other installation issues, please [open an issue](https://github.com/microsoft/DeepSpeed/issues) with the DeepSpeed team.

</Tip>

### Non-identical CUDA toolkits

PyTorch comes with its own CUDA toolkit, but to use DeepSpeed with PyTorch, you need to have an identical version of CUDA installed system-wide. For example, if you installed PyTorch with `cudatoolkit==10.2` in your Python environment, then you'll also need to have CUDA 10.2 installed system-wide. If you don't have CUDA installed system-wide, you should install it first.

The exact location may vary from system to system, but `usr/local/cuda-10.2` is the most common location on many Unix systems. When CUDA is correctly setup and added to your `PATH` environment variable, you can find the installation location with the following command:

```bash
which nvcc
```

### Multiple CUDA toolkits

You may also have more than one CUDA toolkit installed system-wide.

```bash
/usr/local/cuda-10.2
/usr/local/cuda-11.0
```

Typically, package installers set the paths to whatever the last version was installed. If the package build fails because it can't find the right CUDA version (despite it being installed system-wide already), then you need to configure the `PATH` and `LD_LIBRARY_PATH` environment variables to point to the correct path.

Take a look at the contents of these environment variables first:

```bash
echo $PATH
echo $LD_LIBRARY_PATH
```

`PATH` lists the locations of the executables and `LD_LIBRARY_PATH` lists where to look for shared libraries. Earlier entries are prioritized over later ones, and `:` is used to separate multiple entries. To tell the build program where to find the specific CUDA toolkit you want, insert the correct path to list first. This command prepends rather than overwrites the existing values.

```bash
# adjust the version and full path if needed
export PATH=/usr/local/cuda-10.2/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-10.2/lib64:$LD_LIBRARY_PATH
```

In addition, you should also check the directories you assign actually exist. The `lib64` sub-directory contains various CUDA `.so` objects (like `libcudart.so`) and while it is unlikely your system names them differently, you should check the actual names and change them accordingly.

### Older CUDA versions

Sometimes, older CUDA versions may refuse to build with newer compilers. For example, if you have `gcc-9` but CUDA wants `gcc-7`. Usually, installing the latest CUDA toolkit enables support for the newer compiler.

You could also install an older version of the compiler in addition to the one you're currently using (or it may already be installed but it's not used by default and the build system can't see it). To resolve this, you can create a symlink to give the build system visibility to the older compiler.


key: _node_content
string_value:
{"id_": "ce82d3b7-5fa3-4ca2-a1f8-0fefe4e0b84a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "76509118-4217-4e77-808d-b28da782333e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "ce4ac9fc34d09d2761a0d7cab7dd26c34ec62514a20ba7c2e604bed2dd024a79", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f04e3316-eed5-4fd6-a343-2216446d0f3f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5c0c25542737bdd028d9f9c5ebd5a64cad5c10ae7a2dd6aa609e0366806b124d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fc6e60c9-8c5f-41c5-b293-c30d6e82170b", "node_type": "1", "metadata": {}, "hash": "b17cdd84ce2e3fef5fce5abab17b3b17fe97571a06420e6851585ff069949e56", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1739, "end_char_idx": 4520, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If the package build fails because it can't find the right CUDA version (despite it being installed system-wide already), then you need to configure the `PATH` and `LD_LIBRARY_PATH` environment variables to point to the correct path.

Take a look at the contents of these environment variables first:

```bash
echo $PATH
echo $LD_LIBRARY_PATH
```

`PATH` lists the locations of the executables and `LD_LIBRARY_PATH` lists where to look for shared libraries. Earlier entries are prioritized over later ones, and `:` is used to separate multiple entries. To tell the build program where to find the specific CUDA toolkit you want, insert the correct path to list first. This command prepends rather than overwrites the existing values.

```bash
# adjust the version and full path if needed
export PATH=/usr/local/cuda-10.2/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-10.2/lib64:$LD_LIBRARY_PATH
```

In addition, you should also check the directories you assign actually exist. The `lib64` sub-directory contains various CUDA `.so` objects (like `libcudart.so`) and while it is unlikely your system names them differently, you should check the actual names and change them accordingly.

### Older CUDA versions

Sometimes, older CUDA versions may refuse to build with newer compilers. For example, if you have `gcc-9` but CUDA wants `gcc-7`. Usually, installing the latest CUDA toolkit enables support for the newer compiler.

You could also install an older version of the compiler in addition to the one you're currently using (or it may already be installed but it's not used by default and the build system can't see it). To resolve this, you can create a symlink to give the build system visibility to the older compiler.

```bash
# adapt the path to your system
sudo ln -s /usr/bin/gcc-7  /usr/local/cuda-10.2/bin/gcc
sudo ln -s /usr/bin/g++-7  /usr/local/cuda-10.2/bin/g++
```

### Prebuild

If you're still having issues with installing DeepSpeed or if you're building DeepSpeed at run time, you can try to prebuild the DeepSpeed modules before installing them. To make a local build for DeepSpeed:

```bash
git clone https://github.com/microsoft/DeepSpeed/
cd DeepSpeed
rm -rf build
TORCH_CUDA_ARCH_LIST="8.6" DS_BUILD_CPU_ADAM=1 DS_BUILD_UTILS=1 pip install . \
--global-option="build_ext" --global-option="-j8" --no-cache -v \
--disable-pip-version-check 2>&1 | tee build.log
```

<Tip>

To use NVMe offload, add the `DS_BUILD_AIO=1` parameter to the build command and make sure you install the libaio-dev package system-wide.

</Tip>

Next, you'll have to specify your GPU's architecture by editing the `TORCH_CUDA_ARCH_LIST` variable (find a complete list of NVIDIA GPUs and their corresponding architectures on this [page](https://developer.nvidia.com/cuda-gpus)).


key: _node_content
string_value:
{"id_": "fc6e60c9-8c5f-41c5-b293-c30d6e82170b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "76509118-4217-4e77-808d-b28da782333e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "ce4ac9fc34d09d2761a0d7cab7dd26c34ec62514a20ba7c2e604bed2dd024a79", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ce82d3b7-5fa3-4ca2-a1f8-0fefe4e0b84a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d81a7f795e172aa086253a47d2301920b65bf5e5ab821dfbfa3f5568e3ab668b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5aab5f46-53af-45f8-8808-0700c9e293a4", "node_type": "1", "metadata": {}, "hash": "8a12dec6c11fbf78aa24e990ad2d9ad9c584ecff3a4e98d1d9f0b156676110d3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3027, "end_char_idx": 5739, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For example, if you have `gcc-9` but CUDA wants `gcc-7`. Usually, installing the latest CUDA toolkit enables support for the newer compiler.

You could also install an older version of the compiler in addition to the one you're currently using (or it may already be installed but it's not used by default and the build system can't see it). To resolve this, you can create a symlink to give the build system visibility to the older compiler.

```bash
# adapt the path to your system
sudo ln -s /usr/bin/gcc-7  /usr/local/cuda-10.2/bin/gcc
sudo ln -s /usr/bin/g++-7  /usr/local/cuda-10.2/bin/g++
```

### Prebuild

If you're still having issues with installing DeepSpeed or if you're building DeepSpeed at run time, you can try to prebuild the DeepSpeed modules before installing them. To make a local build for DeepSpeed:

```bash
git clone https://github.com/microsoft/DeepSpeed/
cd DeepSpeed
rm -rf build
TORCH_CUDA_ARCH_LIST="8.6" DS_BUILD_CPU_ADAM=1 DS_BUILD_UTILS=1 pip install . \
--global-option="build_ext" --global-option="-j8" --no-cache -v \
--disable-pip-version-check 2>&1 | tee build.log
```

<Tip>

To use NVMe offload, add the `DS_BUILD_AIO=1` parameter to the build command and make sure you install the libaio-dev package system-wide.

</Tip>

Next, you'll have to specify your GPU's architecture by editing the `TORCH_CUDA_ARCH_LIST` variable (find a complete list of NVIDIA GPUs and their corresponding architectures on this [page](https://developer.nvidia.com/cuda-gpus)). To check the PyTorch version that corresponds to your architecture, run the following command:

```bash
python -c "import torch; print(torch.cuda.get_arch_list())"
```

Find the architecture for a GPU with the following command:

<hfoptions id="arch">
<hfoption id="same GPUs">

```bash
CUDA_VISIBLE_DEVICES=0 python -c "import torch; print(torch.cuda.get_device_capability())"
```

</hfoption>
<hfoption id="specific GPU">

To find the architecture for GPU `0`:

```bash
CUDA_VISIBLE_DEVICES=0 python -c "import torch; \
print(torch.cuda.get_device_properties(torch.device('cuda')))
"_CudaDeviceProperties(name='GeForce RTX 3090', major=8, minor=6, total_memory=24268MB, multi_processor_count=82)"
```

This means your GPU architecture is `8.6`.

</hfoption>
</hfoptions>

If you get `8, 6`, then you can set `TORCH_CUDA_ARCH_LIST="8.6"`. For multiple GPUs with different architectures, list them like `TORCH_CUDA_ARCH_LIST="6.1;8.6"`.

It is also possible to not specify `TORCH_CUDA_ARCH_LIST` and the build program automatically queries the GPU architecture of the build. However, it may or may not match the actual GPU on the target machine which is why it is better to explicitly specify the correct architecture.


key: _node_content
string_value:
{"id_": "5aab5f46-53af-45f8-8808-0700c9e293a4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "76509118-4217-4e77-808d-b28da782333e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "ce4ac9fc34d09d2761a0d7cab7dd26c34ec62514a20ba7c2e604bed2dd024a79", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fc6e60c9-8c5f-41c5-b293-c30d6e82170b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8013f2e229d56bcbf220dc06abecd2b15fe5304e7ce62ea3d241c1997595a8e4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "36eebe6a-ac8f-4333-ba15-c157abe21360", "node_type": "1", "metadata": {}, "hash": "d41a0b1ff0448b3bca8072a63edd39838056f5c3adb7a07e32a1afb634b19516", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4281, "end_char_idx": 7107, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

Next, you'll have to specify your GPU's architecture by editing the `TORCH_CUDA_ARCH_LIST` variable (find a complete list of NVIDIA GPUs and their corresponding architectures on this [page](https://developer.nvidia.com/cuda-gpus)). To check the PyTorch version that corresponds to your architecture, run the following command:

```bash
python -c "import torch; print(torch.cuda.get_arch_list())"
```

Find the architecture for a GPU with the following command:

<hfoptions id="arch">
<hfoption id="same GPUs">

```bash
CUDA_VISIBLE_DEVICES=0 python -c "import torch; print(torch.cuda.get_device_capability())"
```

</hfoption>
<hfoption id="specific GPU">

To find the architecture for GPU `0`:

```bash
CUDA_VISIBLE_DEVICES=0 python -c "import torch; \
print(torch.cuda.get_device_properties(torch.device('cuda')))
"_CudaDeviceProperties(name='GeForce RTX 3090', major=8, minor=6, total_memory=24268MB, multi_processor_count=82)"
```

This means your GPU architecture is `8.6`.

</hfoption>
</hfoptions>

If you get `8, 6`, then you can set `TORCH_CUDA_ARCH_LIST="8.6"`. For multiple GPUs with different architectures, list them like `TORCH_CUDA_ARCH_LIST="6.1;8.6"`.

It is also possible to not specify `TORCH_CUDA_ARCH_LIST` and the build program automatically queries the GPU architecture of the build. However, it may or may not match the actual GPU on the target machine which is why it is better to explicitly specify the correct architecture.

For training on multiple machines with the same setup, you'll need to make a binary wheel:

```bash
git clone https://github.com/microsoft/DeepSpeed/
cd DeepSpeed
rm -rf build
TORCH_CUDA_ARCH_LIST="8.6" DS_BUILD_CPU_ADAM=1 DS_BUILD_UTILS=1 \
python setup.py build_ext -j8 bdist_wheel
```

This command generates a binary wheel that'll look something like `dist/deepspeed-0.3.13+8cd046f-cp38-cp38-linux_x86_64.whl`. Now you can install this wheel locally or on another machine.

```bash
pip install deepspeed-0.3.13+8cd046f-cp38-cp38-linux_x86_64.whl
```

## Multi-GPU Network Issues Debug

When training or inferencing with `DistributedDataParallel` and multiple GPU, if you run into issue of inter-communication between processes and/or nodes, you can use the following script to diagnose network issues.

```bash
wget https://raw.githubusercontent.com/huggingface/transformers/main/scripts/distributed/torch-distributed-gpu-test.py
```

For example to test how 2 GPUs interact do:

```bash
python -m torch.distributed.run --nproc_per_node 2 --nnodes 1 torch-distributed-gpu-test.py
```
If both processes can talk to each and allocate GPU memory each will print an OK status.

For more GPUs or nodes adjust the arguments in the script.

You will find a lot more details inside the diagnostics script and even a recipe to how you could run it in a SLURM environment.


key: _node_content
string_value:
{"id_": "36eebe6a-ac8f-4333-ba15-c157abe21360", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "76509118-4217-4e77-808d-b28da782333e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "ce4ac9fc34d09d2761a0d7cab7dd26c34ec62514a20ba7c2e604bed2dd024a79", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5aab5f46-53af-45f8-8808-0700c9e293a4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "74f5f25b41e143f7e00830e12689f6b72f63dc6c6f2da382783291f289dab88d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5a28d167-79fe-44f4-9825-862d07abde56", "node_type": "1", "metadata": {}, "hash": "576d36afd6b8fa648b546efa9eacd29548d4bf6136e8638374b50434909888bf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5596, "end_char_idx": 8770, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, it may or may not match the actual GPU on the target machine which is why it is better to explicitly specify the correct architecture.

For training on multiple machines with the same setup, you'll need to make a binary wheel:

```bash
git clone https://github.com/microsoft/DeepSpeed/
cd DeepSpeed
rm -rf build
TORCH_CUDA_ARCH_LIST="8.6" DS_BUILD_CPU_ADAM=1 DS_BUILD_UTILS=1 \
python setup.py build_ext -j8 bdist_wheel
```

This command generates a binary wheel that'll look something like `dist/deepspeed-0.3.13+8cd046f-cp38-cp38-linux_x86_64.whl`. Now you can install this wheel locally or on another machine.

```bash
pip install deepspeed-0.3.13+8cd046f-cp38-cp38-linux_x86_64.whl
```

## Multi-GPU Network Issues Debug

When training or inferencing with `DistributedDataParallel` and multiple GPU, if you run into issue of inter-communication between processes and/or nodes, you can use the following script to diagnose network issues.

```bash
wget https://raw.githubusercontent.com/huggingface/transformers/main/scripts/distributed/torch-distributed-gpu-test.py
```

For example to test how 2 GPUs interact do:

```bash
python -m torch.distributed.run --nproc_per_node 2 --nnodes 1 torch-distributed-gpu-test.py
```
If both processes can talk to each and allocate GPU memory each will print an OK status.

For more GPUs or nodes adjust the arguments in the script.

You will find a lot more details inside the diagnostics script and even a recipe to how you could run it in a SLURM environment.

An additional level of debug is to add `NCCL_DEBUG=INFO` environment variable as follows:

```bash
NCCL_DEBUG=INFO python -m torch.distributed.run --nproc_per_node 2 --nnodes 1 torch-distributed-gpu-test.py
```

This will dump a lot of NCCL-related debug information, which you can then search online if you find that some problems are reported. Or if you're not sure how to interpret the output you can share the log file in an Issue.



## Underflow and Overflow Detection

<Tip>

This feature is currently available for PyTorch-only.

</Tip>

<Tip>

For multi-GPU training it requires DDP (`torch.distributed.launch`).

</Tip>

<Tip>

This feature can be used with any `nn.Module`-based model.

</Tip>

If you start getting `loss=NaN` or the model inhibits some other abnormal behavior due to `inf` or `nan` in
activations or weights one needs to discover where the first underflow or overflow happens and what led to it. Luckily
you can accomplish that easily by activating a special module that will do the detection automatically.

If you're using [`Trainer`], you just need to add:

```bash
--debug underflow_overflow
```

to the normal command line arguments, or pass `debug="underflow_overflow"` when creating the
[`TrainingArguments`] object.

If you're using your own training loop or another Trainer you can accomplish the same with:

```python
from transformers.debug_utils import DebugUnderflowOverflow

debug_overflow = DebugUnderflowOverflow(model)
```

[`~debug_utils.DebugUnderflowOverflow`] inserts hooks into the model that immediately after each
forward call will test input and output variables and also the corresponding module's weights.


key: _node_content
string_value:
{"id_": "5a28d167-79fe-44f4-9825-862d07abde56", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "76509118-4217-4e77-808d-b28da782333e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "ce4ac9fc34d09d2761a0d7cab7dd26c34ec62514a20ba7c2e604bed2dd024a79", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "36eebe6a-ac8f-4333-ba15-c157abe21360", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "25a5ec4e517684b6d1223d66d2248626af372a555523430a3532da4831ad326a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6f49497d-0c8d-4aa9-b921-9ea0b4a9dffe", "node_type": "1", "metadata": {}, "hash": "3b519da78cade1bab1818163fb8e6cd49b88da400252a093db6444e3b11b5a7c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7109, "end_char_idx": 10398, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
An additional level of debug is to add `NCCL_DEBUG=INFO` environment variable as follows:

```bash
NCCL_DEBUG=INFO python -m torch.distributed.run --nproc_per_node 2 --nnodes 1 torch-distributed-gpu-test.py
```

This will dump a lot of NCCL-related debug information, which you can then search online if you find that some problems are reported. Or if you're not sure how to interpret the output you can share the log file in an Issue.



## Underflow and Overflow Detection

<Tip>

This feature is currently available for PyTorch-only.

</Tip>

<Tip>

For multi-GPU training it requires DDP (`torch.distributed.launch`).

</Tip>

<Tip>

This feature can be used with any `nn.Module`-based model.

</Tip>

If you start getting `loss=NaN` or the model inhibits some other abnormal behavior due to `inf` or `nan` in
activations or weights one needs to discover where the first underflow or overflow happens and what led to it. Luckily
you can accomplish that easily by activating a special module that will do the detection automatically.

If you're using [`Trainer`], you just need to add:

```bash
--debug underflow_overflow
```

to the normal command line arguments, or pass `debug="underflow_overflow"` when creating the
[`TrainingArguments`] object.

If you're using your own training loop or another Trainer you can accomplish the same with:

```python
from transformers.debug_utils import DebugUnderflowOverflow

debug_overflow = DebugUnderflowOverflow(model)
```

[`~debug_utils.DebugUnderflowOverflow`] inserts hooks into the model that immediately after each
forward call will test input and output variables and also the corresponding module's weights. As soon as `inf` or
`nan` is detected in at least one element of the activations or weights, the program will assert and print a report
like this (this was caught with `google/mt5-small` under fp16 mixed precision):

```
Detected inf/nan during batch_number=0
Last 21 forward frames:
abs min  abs max  metadata
                  encoder.block.1.layer.1.DenseReluDense.dropout Dropout
0.00e+00 2.57e+02 input[0]
0.00e+00 2.85e+02 output
[...]
                  encoder.block.2.layer.0 T5LayerSelfAttention
6.78e-04 3.15e+03 input[0]
2.65e-04 3.42e+03 output[0]
             None output[1]
2.25e-01 1.00e+04 output[2]
                  encoder.block.2.layer.1.layer_norm T5LayerNorm
8.69e-02 4.18e-01 weight
2.65e-04 3.42e+03 input[0]
1.79e-06 4.65e+00 output
                  encoder.block.2.layer.1.DenseReluDense.wi_0 Linear
2.17e-07 4.50e+00 weight
1.79e-06 4.65e+00 input[0]
2.68e-06 3.70e+01 output
                  encoder.block.2.layer.1.DenseReluDense.wi_1 Linear
8.08e-07 2.66e+01 weight
1.79e-06 4.65e+00 input[0]
1.27e-04 2.37e+02 output
                  encoder.block.2.layer.1.DenseReluDense.dropout Dropout
0.00e+00 8.76e+03 input[0]
0.00e+00 9.74e+03 output
                  encoder.block.2.layer.1.DenseReluDense.wo Linear
1.01e-06 6.44e+00 weight
0.00e+00 9.74e+03 input[0]
3.18e-04 6.27e+04 output
                  encoder.block.2.layer.1.DenseReluDense T5DenseGatedGeluDense
1.79e-06 4.65e+00 input[0]
3.18e-04 6.27e+04 output
                  encoder.block.2.layer.1.dropout Dropout
3.18e-04 6.27e+04 input[0]
0.00e+00      inf output
```

The example output has been trimmed in the middle for brevity.


key: _node_content
string_value:
{"id_": "6f49497d-0c8d-4aa9-b921-9ea0b4a9dffe", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "76509118-4217-4e77-808d-b28da782333e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "ce4ac9fc34d09d2761a0d7cab7dd26c34ec62514a20ba7c2e604bed2dd024a79", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5a28d167-79fe-44f4-9825-862d07abde56", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "644618890cbe21d9db128311e54cc03e1890a9ff1a60b629c1cf73afedbdcc1d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9a008609-345d-4b00-af41-a618465d4a43", "node_type": "1", "metadata": {}, "hash": "e8bca47fc6f6d00970b20c72a3aebb19b78a20df0ae9f0d10da2fd63e2693fcd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10400, "end_char_idx": 12865, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The second column shows the value of the absolute largest element, so if you have a closer look at the last few frames,
the inputs and outputs were in the range of `1e4`. So when this training was done under fp16 mixed precision the very
last step overflowed (since under `fp16` the largest number before `inf` is `64e3`). To avoid overflows under
`fp16` the activations must remain way below `1e4`, because `1e4 * 1e4 = 1e8` so any matrix multiplication with
large activations is going to lead to a numerical overflow condition.

At the very start of the trace you can discover at which batch number the problem occurred (here `Detected inf/nan during batch_number=0` means the problem occurred on the first batch).

Each reported frame starts by declaring the fully qualified entry for the corresponding module this frame is reporting
for. If we look just at this frame:

```
                  encoder.block.2.layer.1.layer_norm T5LayerNorm
8.69e-02 4.18e-01 weight
2.65e-04 3.42e+03 input[0]
1.79e-06 4.65e+00 output
```

Here, `encoder.block.2.layer.1.layer_norm` indicates that it was a layer norm for the first layer, of the second
block of the encoder. And the specific calls of the `forward` is `T5LayerNorm`.

Let's look at the last few frames of that report:

```
Detected inf/nan during batch_number=0
Last 21 forward frames:
abs min  abs max  metadata
[...]
                  encoder.block.2.layer.1.DenseReluDense.wi_0 Linear
2.17e-07 4.50e+00 weight
1.79e-06 4.65e+00 input[0]
2.68e-06 3.70e+01 output
                  encoder.block.2.layer.1.DenseReluDense.wi_1 Linear
8.08e-07 2.66e+01 weight
1.79e-06 4.65e+00 input[0]
1.27e-04 2.37e+02 output
                  encoder.block.2.layer.1.DenseReluDense.wo Linear
1.01e-06 6.44e+00 weight
0.00e+00 9.74e+03 input[0]
3.18e-04 6.27e+04 output
                  encoder.block.2.layer.1.DenseReluDense T5DenseGatedGeluDense
1.79e-06 4.65e+00 input[0]
3.18e-04 6.27e+04 output
                  encoder.block.2.layer.1.dropout Dropout
3.18e-04 6.27e+04 input[0]
0.00e+00      inf output
```

The last frame reports for `Dropout.forward` function with the first entry for the only input and the second for the
only output. You can see that it was called from an attribute `dropout` inside `DenseReluDense` class. We can see
that it happened during the first layer, of the 2nd block, during the very first batch. Finally, the absolute largest
input elements was `6.27e+04` and same for the output was `inf`.


key: _node_content
string_value:
{"id_": "9a008609-345d-4b00-af41-a618465d4a43", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "76509118-4217-4e77-808d-b28da782333e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "ce4ac9fc34d09d2761a0d7cab7dd26c34ec62514a20ba7c2e604bed2dd024a79", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6f49497d-0c8d-4aa9-b921-9ea0b4a9dffe", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "1c3909a4f73b6c85dbe3868ccc969964caf44f8d9c331135855e0cd1ddded35a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4f93a467-7b4f-46f4-876a-0521ae36378e", "node_type": "1", "metadata": {}, "hash": "2f2e14cd8e43031c6afd03293286b1960d0b617145d293f7e238aee961e1a13d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12582, "end_char_idx": 15743, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can see that it was called from an attribute `dropout` inside `DenseReluDense` class. We can see
that it happened during the first layer, of the 2nd block, during the very first batch. Finally, the absolute largest
input elements was `6.27e+04` and same for the output was `inf`.

You can see here, that `T5DenseGatedGeluDense.forward` resulted in output activations, whose absolute max value was
around 62.7K, which is very close to fp16's top limit of 64K. In the next frame we have `Dropout` which renormalizes
the weights, after it zeroed some of the elements, which pushes the absolute max value to more than 64K, and we get an
overflow (`inf`).

As you can see it's the previous frames that we need to look into when the numbers start going into very large for fp16
numbers.

Let's match the report to the code from `models/t5/modeling_t5.py`:

```python
class T5DenseGatedGeluDense(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.wi_0 = nn.Linear(config.d_model, config.d_ff, bias=False)
        self.wi_1 = nn.Linear(config.d_model, config.d_ff, bias=False)
        self.wo = nn.Linear(config.d_ff, config.d_model, bias=False)
        self.dropout = nn.Dropout(config.dropout_rate)
        self.gelu_act = ACT2FN["gelu_new"]

    def forward(self, hidden_states):
        hidden_gelu = self.gelu_act(self.wi_0(hidden_states))
        hidden_linear = self.wi_1(hidden_states)
        hidden_states = hidden_gelu * hidden_linear
        hidden_states = self.dropout(hidden_states)
        hidden_states = self.wo(hidden_states)
        return hidden_states
```

Now it's easy to see the `dropout` call, and all the previous calls as well.

Since the detection is happening in a forward hook, these reports are printed immediately after each `forward`
returns.

Going back to the full report, to act on it and to fix the problem, we need to go a few frames up where the numbers
started to go up and most likely switch to the `fp32` mode here, so that the numbers don't overflow when multiplied
or summed up. Of course, there might be other solutions. For example, we could turn off `amp` temporarily if it's
enabled, after moving the original `forward` into a helper wrapper, like so:

```python
def _forward(self, hidden_states):
    hidden_gelu = self.gelu_act(self.wi_0(hidden_states))
    hidden_linear = self.wi_1(hidden_states)
    hidden_states = hidden_gelu * hidden_linear
    hidden_states = self.dropout(hidden_states)
    hidden_states = self.wo(hidden_states)
    return hidden_states


import torch


def forward(self, hidden_states):
    if torch.is_autocast_enabled():
        with torch.cuda.amp.autocast(enabled=False):
            return self._forward(hidden_states)
    else:
        return self._forward(hidden_states)
```

Since the automatic detector only reports on inputs and outputs of full frames, once you know where to look, you may
want to analyse the intermediary stages of any specific `forward` function as well. In such a case you can use the
`detect_overflow` helper function to inject the detector where you want it, for example:

```python
from debug_utils import detect_overflow


key: _node_content
string_value:
{"id_": "4f93a467-7b4f-46f4-876a-0521ae36378e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "76509118-4217-4e77-808d-b28da782333e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "ce4ac9fc34d09d2761a0d7cab7dd26c34ec62514a20ba7c2e604bed2dd024a79", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9a008609-345d-4b00-af41-a618465d4a43", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "49f3b3f40b5a394255811314d48bd7b4b6b01c1b62673c103ed757f40c82121a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d1286500-d27d-4d7b-857c-ec9931811385", "node_type": "1", "metadata": {}, "hash": "22e9aef06fa3af10f84e5e9b19dfdb4165cbfd8a0a16ea156b8dce3c357c6e5e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14268, "end_char_idx": 17480, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Since the detection is happening in a forward hook, these reports are printed immediately after each `forward`
returns.

Going back to the full report, to act on it and to fix the problem, we need to go a few frames up where the numbers
started to go up and most likely switch to the `fp32` mode here, so that the numbers don't overflow when multiplied
or summed up. Of course, there might be other solutions. For example, we could turn off `amp` temporarily if it's
enabled, after moving the original `forward` into a helper wrapper, like so:

```python
def _forward(self, hidden_states):
    hidden_gelu = self.gelu_act(self.wi_0(hidden_states))
    hidden_linear = self.wi_1(hidden_states)
    hidden_states = hidden_gelu * hidden_linear
    hidden_states = self.dropout(hidden_states)
    hidden_states = self.wo(hidden_states)
    return hidden_states


import torch


def forward(self, hidden_states):
    if torch.is_autocast_enabled():
        with torch.cuda.amp.autocast(enabled=False):
            return self._forward(hidden_states)
    else:
        return self._forward(hidden_states)
```

Since the automatic detector only reports on inputs and outputs of full frames, once you know where to look, you may
want to analyse the intermediary stages of any specific `forward` function as well. In such a case you can use the
`detect_overflow` helper function to inject the detector where you want it, for example:

```python
from debug_utils import detect_overflow


class T5LayerFF(nn.Module):
    [...]

    def forward(self, hidden_states):
        forwarded_states = self.layer_norm(hidden_states)
        detect_overflow(forwarded_states, "after layer_norm")
        forwarded_states = self.DenseReluDense(forwarded_states)
        detect_overflow(forwarded_states, "after DenseReluDense")
        return hidden_states + self.dropout(forwarded_states)
```

You can see that we added 2 of these and now we track if `inf` or `nan` for `forwarded_states` was detected
somewhere in between.

Actually, the detector already reports these because each of the calls in the example above is a `nn.Module`, but
let's say if you had some local direct calculations this is how you'd do that.

Additionally, if you're instantiating the debugger in your own code, you can adjust the number of frames printed from
its default, e.g.:

```python
from transformers.debug_utils import DebugUnderflowOverflow

debug_overflow = DebugUnderflowOverflow(model, max_frames_to_save=100)
```

### Specific batch absolute min and max value tracing

The same debugging class can be used for per-batch tracing with the underflow/overflow detection feature turned off.

Let's say you want to watch the absolute min and max values for all the ingredients of each `forward` call of a given
batch, and only do that for batches 1 and 3. Then you instantiate this class as:

```python
debug_overflow = DebugUnderflowOverflow(model, trace_batch_nums=[1, 3])
```

And now full batches 1 and 3 will be traced using the same format as the underflow/overflow detector does.

Batches are 0-indexed.

This is helpful if you know that the program starts misbehaving after a certain batch number, so you can fast-forward
right to that area.


key: _node_content
string_value:
{"id_": "d1286500-d27d-4d7b-857c-ec9931811385", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "76509118-4217-4e77-808d-b28da782333e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "ce4ac9fc34d09d2761a0d7cab7dd26c34ec62514a20ba7c2e604bed2dd024a79", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4f93a467-7b4f-46f4-876a-0521ae36378e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "28351dc705bb9f0c731a3175c1d8f52e096ea4e7abb67cc2d55bba506ae5a708", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "56858bed-8eb2-4f07-b51a-39834c0dfc01", "node_type": "1", "metadata": {}, "hash": "2f48625c654a5991b3ced2a4abe32384204c45601fad667a829f56a555f6c2f5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15746, "end_char_idx": 18653, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
class T5LayerFF(nn.Module):
    [...]

    def forward(self, hidden_states):
        forwarded_states = self.layer_norm(hidden_states)
        detect_overflow(forwarded_states, "after layer_norm")
        forwarded_states = self.DenseReluDense(forwarded_states)
        detect_overflow(forwarded_states, "after DenseReluDense")
        return hidden_states + self.dropout(forwarded_states)
```

You can see that we added 2 of these and now we track if `inf` or `nan` for `forwarded_states` was detected
somewhere in between.

Actually, the detector already reports these because each of the calls in the example above is a `nn.Module`, but
let's say if you had some local direct calculations this is how you'd do that.

Additionally, if you're instantiating the debugger in your own code, you can adjust the number of frames printed from
its default, e.g.:

```python
from transformers.debug_utils import DebugUnderflowOverflow

debug_overflow = DebugUnderflowOverflow(model, max_frames_to_save=100)
```

### Specific batch absolute min and max value tracing

The same debugging class can be used for per-batch tracing with the underflow/overflow detection feature turned off.

Let's say you want to watch the absolute min and max values for all the ingredients of each `forward` call of a given
batch, and only do that for batches 1 and 3. Then you instantiate this class as:

```python
debug_overflow = DebugUnderflowOverflow(model, trace_batch_nums=[1, 3])
```

And now full batches 1 and 3 will be traced using the same format as the underflow/overflow detector does.

Batches are 0-indexed.

This is helpful if you know that the program starts misbehaving after a certain batch number, so you can fast-forward
right to that area. Here is a sample truncated output for such configuration:

```
                  *** Starting batch number=1 ***
abs min  abs max  metadata
                  shared Embedding
1.01e-06 7.92e+02 weight
0.00e+00 2.47e+04 input[0]
5.36e-05 7.92e+02 output
[...]
                  decoder.dropout Dropout
1.60e-07 2.27e+01 input[0]
0.00e+00 2.52e+01 output
                  decoder T5Stack
     not a tensor output
                  lm_head Linear
1.01e-06 7.92e+02 weight
0.00e+00 1.11e+00 input[0]
6.06e-02 8.39e+01 output
                   T5ForConditionalGeneration
     not a tensor output

                  *** Starting batch number=3 ***
abs min  abs max  metadata
                  shared Embedding
1.01e-06 7.92e+02 weight
0.00e+00 2.78e+04 input[0]
5.36e-05 7.92e+02 output
[...]
```

Here you will get a huge number of frames dumped - as many as there were forward calls in your model, so it may or may
not what you want, but sometimes it can be easier to use for debugging purposes than a normal debugger. For example, if
a problem starts happening at batch number 150. So you can dump traces for batches 149 and 150 and compare where
numbers started to diverge.


key: _node_content
string_value:
{"id_": "56858bed-8eb2-4f07-b51a-39834c0dfc01", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "76509118-4217-4e77-808d-b28da782333e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "ce4ac9fc34d09d2761a0d7cab7dd26c34ec62514a20ba7c2e604bed2dd024a79", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d1286500-d27d-4d7b-857c-ec9931811385", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/debugging/", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "cbd1b3af045dd6a9c37e984c9be358befbcca5ade479d7a3a8bdbd8885443047", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17319, "end_char_idx": 18844, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Batches are 0-indexed.

This is helpful if you know that the program starts misbehaving after a certain batch number, so you can fast-forward
right to that area. Here is a sample truncated output for such configuration:

```
                  *** Starting batch number=1 ***
abs min  abs max  metadata
                  shared Embedding
1.01e-06 7.92e+02 weight
0.00e+00 2.47e+04 input[0]
5.36e-05 7.92e+02 output
[...]
                  decoder.dropout Dropout
1.60e-07 2.27e+01 input[0]
0.00e+00 2.52e+01 output
                  decoder T5Stack
     not a tensor output
                  lm_head Linear
1.01e-06 7.92e+02 weight
0.00e+00 1.11e+00 input[0]
6.06e-02 8.39e+01 output
                   T5ForConditionalGeneration
     not a tensor output

                  *** Starting batch number=3 ***
abs min  abs max  metadata
                  shared Embedding
1.01e-06 7.92e+02 weight
0.00e+00 2.78e+04 input[0]
5.36e-05 7.92e+02 output
[...]
```

Here you will get a huge number of frames dumped - as many as there were forward calls in your model, so it may or may
not what you want, but sometimes it can be easier to use for debugging purposes than a normal debugger. For example, if
a problem starts happening at batch number 150. So you can dump traces for batches 149 and 150 and compare where
numbers started to diverge.

You can also specify the batch number after which to stop the training, with:

```python
debug_overflow = DebugUnderflowOverflow(model, trace_batch_nums=[1, 3], abort_after_batch_num=3)
```


key: _node_content
string_value:
{"id_": "71eaef83-1b33-47f7-b9b2-032d78fce488", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8c9fb3ec-42ec-4923-83e4-199288c3bc95", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c580f6c975aa9151af462a3224645ef200b0db69bd9186349d1e84fa758e1ab0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e29bb787-e3aa-43f5-8eda-2c9ca2e42f20", "node_type": "1", "metadata": {}, "hash": "55da9ed7b6c4c530cd7dc6f6ee7194d2350b0b6d00419bd06423fc197fcdbddc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2285, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Create a custom architecture

An [`AutoClass`](model_doc/auto) automatically infers the model architecture and downloads pretrained configuration and weights. Generally, we recommend using an `AutoClass` to produce checkpoint-agnostic code. But users who want more control over specific model parameters can create a custom 🤗 Transformers model from just a few base classes. This could be particularly useful for anyone who is interested in studying, training or experimenting with a 🤗 Transformers model. In this guide, dive deeper into creating a custom model without an `AutoClass`. Learn how to:

- Load and customize a model configuration.
- Create a model architecture.
- Create a slow and fast tokenizer for text.
- Create an image processor for vision tasks.
- Create a feature extractor for audio tasks.
- Create a processor for multimodal tasks.

## Configuration

A [configuration](main_classes/configuration) refers to a model's specific attributes. Each model configuration has different attributes; for instance, all NLP models have the `hidden_size`, `num_attention_heads`, `num_hidden_layers` and `vocab_size` attributes in common. These attributes specify the number of attention heads or hidden layers to construct a model with.

Get a closer look at [DistilBERT](model_doc/distilbert) by accessing [`DistilBertConfig`] to inspect it's attributes:

```py
>>> from transformers import DistilBertConfig

>>> config = DistilBertConfig()
>>> print(config)
DistilBertConfig {
  "activation": "gelu",
  "attention_dropout": 0.1,
  "dim": 768,
  "dropout": 0.1,
  "hidden_dim": 3072,
  "initializer_range": 0.02,
  "max_position_embeddings": 512,
  "model_type": "distilbert",
  "n_heads": 12,
  "n_layers": 6,
  "pad_token_id": 0,
  "qa_dropout": 0.1,
  "seq_classif_dropout": 0.2,
  "sinusoidal_pos_embds": false,
  "transformers_version": "4.16.2",
  "vocab_size": 30522
}
```

[`DistilBertConfig`] displays all the default attributes used to build a base [`DistilBertModel`]. All attributes are customizable, creating space for experimentation. For example, you can customize a default model to:

- Try a different activation function with the `activation` parameter.
- Use a higher dropout ratio for the attention probabilities with the `attention_dropout` parameter.


key: _node_content
string_value:
{"id_": "e29bb787-e3aa-43f5-8eda-2c9ca2e42f20", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8c9fb3ec-42ec-4923-83e4-199288c3bc95", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c580f6c975aa9151af462a3224645ef200b0db69bd9186349d1e84fa758e1ab0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "71eaef83-1b33-47f7-b9b2-032d78fce488", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e17a6d530c69436f0a0bae77602158d255ad2906e7425ec9ca1a159e39affb4b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d0d9db6f-b079-4808-a767-25e60a42c1ba", "node_type": "1", "metadata": {}, "hash": "65a6652ee58185cb887dbe54220094d198f815b7a5908b56f44800dac1340e38", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 769, "end_char_idx": 3693, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- Create a feature extractor for audio tasks.
- Create a processor for multimodal tasks.

## Configuration

A [configuration](main_classes/configuration) refers to a model's specific attributes. Each model configuration has different attributes; for instance, all NLP models have the `hidden_size`, `num_attention_heads`, `num_hidden_layers` and `vocab_size` attributes in common. These attributes specify the number of attention heads or hidden layers to construct a model with.

Get a closer look at [DistilBERT](model_doc/distilbert) by accessing [`DistilBertConfig`] to inspect it's attributes:

```py
>>> from transformers import DistilBertConfig

>>> config = DistilBertConfig()
>>> print(config)
DistilBertConfig {
  "activation": "gelu",
  "attention_dropout": 0.1,
  "dim": 768,
  "dropout": 0.1,
  "hidden_dim": 3072,
  "initializer_range": 0.02,
  "max_position_embeddings": 512,
  "model_type": "distilbert",
  "n_heads": 12,
  "n_layers": 6,
  "pad_token_id": 0,
  "qa_dropout": 0.1,
  "seq_classif_dropout": 0.2,
  "sinusoidal_pos_embds": false,
  "transformers_version": "4.16.2",
  "vocab_size": 30522
}
```

[`DistilBertConfig`] displays all the default attributes used to build a base [`DistilBertModel`]. All attributes are customizable, creating space for experimentation. For example, you can customize a default model to:

- Try a different activation function with the `activation` parameter.
- Use a higher dropout ratio for the attention probabilities with the `attention_dropout` parameter.

```py
>>> my_config = DistilBertConfig(activation="relu", attention_dropout=0.4)
>>> print(my_config)
DistilBertConfig {
  "activation": "relu",
  "attention_dropout": 0.4,
  "dim": 768,
  "dropout": 0.1,
  "hidden_dim": 3072,
  "initializer_range": 0.02,
  "max_position_embeddings": 512,
  "model_type": "distilbert",
  "n_heads": 12,
  "n_layers": 6,
  "pad_token_id": 0,
  "qa_dropout": 0.1,
  "seq_classif_dropout": 0.2,
  "sinusoidal_pos_embds": false,
  "transformers_version": "4.16.2",
  "vocab_size": 30522
}
```

Pretrained model attributes can be modified in the [`~PretrainedConfig.from_pretrained`] function:

```py
>>> my_config = DistilBertConfig.from_pretrained("distilbert/distilbert-base-uncased", activation="relu", attention_dropout=0.4)
```

Once you are satisfied with your model configuration, you can save it with [`~PretrainedConfig.save_pretrained`]. Your configuration file is stored as a JSON file in the specified save directory:

```py
>>> my_config.save_pretrained(save_directory="./your_model_save_path")
```

To reuse the configuration file, load it with [`~PretrainedConfig.from_pretrained`]:

```py
>>> my_config = DistilBertConfig.from_pretrained("./your_model_save_path/config.json")
```

<Tip>

You can also save your configuration file as a dictionary or even just the difference between your custom configuration attributes and the default configuration attributes!


key: _node_content
string_value:
{"id_": "d0d9db6f-b079-4808-a767-25e60a42c1ba", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8c9fb3ec-42ec-4923-83e4-199288c3bc95", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c580f6c975aa9151af462a3224645ef200b0db69bd9186349d1e84fa758e1ab0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e29bb787-e3aa-43f5-8eda-2c9ca2e42f20", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "849ae132d14cc2fcf08c817c51cc49d697d366968071bd546e08cdc7148267bd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "64186f8a-d86a-4d2a-b608-e2cae8b22153", "node_type": "1", "metadata": {}, "hash": "9c38af66dfc1c87f16960c9505dcc96df55315b6add0afdbb93600e3c8c778fa", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2185, "end_char_idx": 5237, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- Use a higher dropout ratio for the attention probabilities with the `attention_dropout` parameter.

```py
>>> my_config = DistilBertConfig(activation="relu", attention_dropout=0.4)
>>> print(my_config)
DistilBertConfig {
  "activation": "relu",
  "attention_dropout": 0.4,
  "dim": 768,
  "dropout": 0.1,
  "hidden_dim": 3072,
  "initializer_range": 0.02,
  "max_position_embeddings": 512,
  "model_type": "distilbert",
  "n_heads": 12,
  "n_layers": 6,
  "pad_token_id": 0,
  "qa_dropout": 0.1,
  "seq_classif_dropout": 0.2,
  "sinusoidal_pos_embds": false,
  "transformers_version": "4.16.2",
  "vocab_size": 30522
}
```

Pretrained model attributes can be modified in the [`~PretrainedConfig.from_pretrained`] function:

```py
>>> my_config = DistilBertConfig.from_pretrained("distilbert/distilbert-base-uncased", activation="relu", attention_dropout=0.4)
```

Once you are satisfied with your model configuration, you can save it with [`~PretrainedConfig.save_pretrained`]. Your configuration file is stored as a JSON file in the specified save directory:

```py
>>> my_config.save_pretrained(save_directory="./your_model_save_path")
```

To reuse the configuration file, load it with [`~PretrainedConfig.from_pretrained`]:

```py
>>> my_config = DistilBertConfig.from_pretrained("./your_model_save_path/config.json")
```

<Tip>

You can also save your configuration file as a dictionary or even just the difference between your custom configuration attributes and the default configuration attributes! See the [configuration](main_classes/configuration) documentation for more details.

</Tip>

## Model

The next step is to create a [model](main_classes/models). The model - also loosely referred to as the architecture - defines what each layer is doing and what operations are happening. Attributes like `num_hidden_layers` from the configuration are used to define the architecture. Every model shares the base class [`PreTrainedModel`] and a few common methods like resizing input embeddings and pruning self-attention heads. In addition, all models are also either a [`torch.nn.Module`](https://pytorch.org/docs/stable/generated/torch.nn.Module.html), [`tf.keras.Model`](https://www.tensorflow.org/api_docs/python/tf/keras/Model) or [`flax.linen.Module`](https://flax.readthedocs.io/en/latest/api_reference/flax.linen/module.html) subclass. This means models are compatible with each of their respective framework's usage.

<frameworkcontent>
<pt>
Load your custom configuration attributes into the model:

```py
>>> from transformers import DistilBertModel

>>> my_config = DistilBertConfig.from_pretrained("./your_model_save_path/config.json")
>>> model = DistilBertModel(my_config)
```

This creates a model with random values instead of pretrained weights. You won't be able to use this model for anything useful yet until you train it. Training is a costly and time-consuming process. It is generally better to use a pretrained model to obtain better results faster, while using only a fraction of the resources required for training.


key: _node_content
string_value:
{"id_": "64186f8a-d86a-4d2a-b608-e2cae8b22153", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8c9fb3ec-42ec-4923-83e4-199288c3bc95", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c580f6c975aa9151af462a3224645ef200b0db69bd9186349d1e84fa758e1ab0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d0d9db6f-b079-4808-a767-25e60a42c1ba", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "9e4cb5ddec62bf3df377ed782bd351f1d03d1641a7bd44e41a5749a789d99668", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "926a9176-2cfc-4c76-ae0c-49b5a952c7ec", "node_type": "1", "metadata": {}, "hash": "a3462cecb106da4d9bfa1683f5c57d35282982009306f5e6cc00e7ffb2763289", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3694, "end_char_idx": 7135, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
See the [configuration](main_classes/configuration) documentation for more details.

</Tip>

## Model

The next step is to create a [model](main_classes/models). The model - also loosely referred to as the architecture - defines what each layer is doing and what operations are happening. Attributes like `num_hidden_layers` from the configuration are used to define the architecture. Every model shares the base class [`PreTrainedModel`] and a few common methods like resizing input embeddings and pruning self-attention heads. In addition, all models are also either a [`torch.nn.Module`](https://pytorch.org/docs/stable/generated/torch.nn.Module.html), [`tf.keras.Model`](https://www.tensorflow.org/api_docs/python/tf/keras/Model) or [`flax.linen.Module`](https://flax.readthedocs.io/en/latest/api_reference/flax.linen/module.html) subclass. This means models are compatible with each of their respective framework's usage.

<frameworkcontent>
<pt>
Load your custom configuration attributes into the model:

```py
>>> from transformers import DistilBertModel

>>> my_config = DistilBertConfig.from_pretrained("./your_model_save_path/config.json")
>>> model = DistilBertModel(my_config)
```

This creates a model with random values instead of pretrained weights. You won't be able to use this model for anything useful yet until you train it. Training is a costly and time-consuming process. It is generally better to use a pretrained model to obtain better results faster, while using only a fraction of the resources required for training.

Create a pretrained model with [`~PreTrainedModel.from_pretrained`]:

```py
>>> model = DistilBertModel.from_pretrained("distilbert/distilbert-base-uncased")
```

When you load pretrained weights, the default model configuration is automatically loaded if the model is provided by 🤗 Transformers. However, you can still replace - some or all of - the default model configuration attributes with your own if you'd like:

```py
>>> model = DistilBertModel.from_pretrained("distilbert/distilbert-base-uncased", config=my_config)
```
</pt>
<tf>
Load your custom configuration attributes into the model:

```py
>>> from transformers import TFDistilBertModel

>>> my_config = DistilBertConfig.from_pretrained("./your_model_save_path/my_config.json")
>>> tf_model = TFDistilBertModel(my_config)
```

This creates a model with random values instead of pretrained weights. You won't be able to use this model for anything useful yet until you train it. Training is a costly and time-consuming process. It is generally better to use a pretrained model to obtain better results faster, while using only a fraction of the resources required for training.

Create a pretrained model with [`~TFPreTrainedModel.from_pretrained`]:

```py
>>> tf_model = TFDistilBertModel.from_pretrained("distilbert/distilbert-base-uncased")
```

When you load pretrained weights, the default model configuration is automatically loaded if the model is provided by 🤗 Transformers. However, you can still replace - some or all of - the default model configuration attributes with your own if you'd like:

```py
>>> tf_model = TFDistilBertModel.from_pretrained("distilbert/distilbert-base-uncased", config=my_config)
```
</tf>
</frameworkcontent>

### Model heads

At this point, you have a base DistilBERT model which outputs the *hidden states*. The hidden states are passed as inputs to a model head to produce the final output.


key: _node_content
string_value:
{"id_": "926a9176-2cfc-4c76-ae0c-49b5a952c7ec", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8c9fb3ec-42ec-4923-83e4-199288c3bc95", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c580f6c975aa9151af462a3224645ef200b0db69bd9186349d1e84fa758e1ab0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "64186f8a-d86a-4d2a-b608-e2cae8b22153", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "cd5c2c1dfc0d8ed43504911fa9fcbed1d36f98489070e750458ae0bfcab37923", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4ad181d4-8a7e-449e-aed6-51523203da80", "node_type": "1", "metadata": {}, "hash": "51e542240f3b7a26af8efd5d7567d61fd9b441bfdf9bed810352ed3448a0f288", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5536, "end_char_idx": 8930, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, you can still replace - some or all of - the default model configuration attributes with your own if you'd like:

```py
>>> model = DistilBertModel.from_pretrained("distilbert/distilbert-base-uncased", config=my_config)
```
</pt>
<tf>
Load your custom configuration attributes into the model:

```py
>>> from transformers import TFDistilBertModel

>>> my_config = DistilBertConfig.from_pretrained("./your_model_save_path/my_config.json")
>>> tf_model = TFDistilBertModel(my_config)
```

This creates a model with random values instead of pretrained weights. You won't be able to use this model for anything useful yet until you train it. Training is a costly and time-consuming process. It is generally better to use a pretrained model to obtain better results faster, while using only a fraction of the resources required for training.

Create a pretrained model with [`~TFPreTrainedModel.from_pretrained`]:

```py
>>> tf_model = TFDistilBertModel.from_pretrained("distilbert/distilbert-base-uncased")
```

When you load pretrained weights, the default model configuration is automatically loaded if the model is provided by 🤗 Transformers. However, you can still replace - some or all of - the default model configuration attributes with your own if you'd like:

```py
>>> tf_model = TFDistilBertModel.from_pretrained("distilbert/distilbert-base-uncased", config=my_config)
```
</tf>
</frameworkcontent>

### Model heads

At this point, you have a base DistilBERT model which outputs the *hidden states*. The hidden states are passed as inputs to a model head to produce the final output. 🤗 Transformers provides a different model head for each task as long as a model supports the task (i.e., you can't use DistilBERT for a sequence-to-sequence task like translation).

<frameworkcontent>
<pt>
For example, [`DistilBertForSequenceClassification`] is a base DistilBERT model with a sequence classification head. The sequence classification head is a linear layer on top of the pooled outputs.

```py
>>> from transformers import DistilBertForSequenceClassification

>>> model = DistilBertForSequenceClassification.from_pretrained("distilbert/distilbert-base-uncased")
```

Easily reuse this checkpoint for another task by switching to a different model head. For a question answering task, you would use the [`DistilBertForQuestionAnswering`] model head. The question answering head is similar to the sequence classification head except it is a linear layer on top of the hidden states output.

```py
>>> from transformers import DistilBertForQuestionAnswering

>>> model = DistilBertForQuestionAnswering.from_pretrained("distilbert/distilbert-base-uncased")
```
</pt>
<tf>
For example, [`TFDistilBertForSequenceClassification`] is a base DistilBERT model with a sequence classification head. The sequence classification head is a linear layer on top of the pooled outputs.

```py
>>> from transformers import TFDistilBertForSequenceClassification

>>> tf_model = TFDistilBertForSequenceClassification.from_pretrained("distilbert/distilbert-base-uncased")
```

Easily reuse this checkpoint for another task by switching to a different model head. For a question answering task, you would use the [`TFDistilBertForQuestionAnswering`] model head. The question answering head is similar to the sequence classification head except it is a linear layer on top of the hidden states output.


key: _node_content
string_value:
{"id_": "4ad181d4-8a7e-449e-aed6-51523203da80", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8c9fb3ec-42ec-4923-83e4-199288c3bc95", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c580f6c975aa9151af462a3224645ef200b0db69bd9186349d1e84fa758e1ab0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "926a9176-2cfc-4c76-ae0c-49b5a952c7ec", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "66ba263d1a1bd00a14a6c63fafc1375d60eba9db3e708f23e367e4daedda0201", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3f57254b-efac-4a67-b50c-92250b56801c", "node_type": "1", "metadata": {}, "hash": "b8c4e69f0cea427df77fb8f616ef9eb3b49e0d8d0834dca5ed2167c5a4668ee6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7318, "end_char_idx": 10639, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<frameworkcontent>
<pt>
For example, [`DistilBertForSequenceClassification`] is a base DistilBERT model with a sequence classification head. The sequence classification head is a linear layer on top of the pooled outputs.

```py
>>> from transformers import DistilBertForSequenceClassification

>>> model = DistilBertForSequenceClassification.from_pretrained("distilbert/distilbert-base-uncased")
```

Easily reuse this checkpoint for another task by switching to a different model head. For a question answering task, you would use the [`DistilBertForQuestionAnswering`] model head. The question answering head is similar to the sequence classification head except it is a linear layer on top of the hidden states output.

```py
>>> from transformers import DistilBertForQuestionAnswering

>>> model = DistilBertForQuestionAnswering.from_pretrained("distilbert/distilbert-base-uncased")
```
</pt>
<tf>
For example, [`TFDistilBertForSequenceClassification`] is a base DistilBERT model with a sequence classification head. The sequence classification head is a linear layer on top of the pooled outputs.

```py
>>> from transformers import TFDistilBertForSequenceClassification

>>> tf_model = TFDistilBertForSequenceClassification.from_pretrained("distilbert/distilbert-base-uncased")
```

Easily reuse this checkpoint for another task by switching to a different model head. For a question answering task, you would use the [`TFDistilBertForQuestionAnswering`] model head. The question answering head is similar to the sequence classification head except it is a linear layer on top of the hidden states output.

```py
>>> from transformers import TFDistilBertForQuestionAnswering

>>> tf_model = TFDistilBertForQuestionAnswering.from_pretrained("distilbert/distilbert-base-uncased")
```
</tf>
</frameworkcontent>

## Tokenizer

The last base class you need before using a model for textual data is a [tokenizer](main_classes/tokenizer) to convert raw text to tensors. There are two types of tokenizers you can use with 🤗 Transformers:

- [`PreTrainedTokenizer`]: a Python implementation of a tokenizer.
- [`PreTrainedTokenizerFast`]: a tokenizer from our Rust-based [🤗 Tokenizer](https://huggingface.co/docs/tokenizers/python/latest/) library. This tokenizer type is significantly faster - especially during batch tokenization - due to its Rust implementation. The fast tokenizer also offers additional methods like *offset mapping* which maps tokens to their original words or characters.

Both tokenizers support common methods such as encoding and decoding, adding new tokens, and managing special tokens.

<Tip warning={true}>

Not every model supports a fast tokenizer. Take a look at this [table](index#supported-frameworks) to check if a model has fast tokenizer support.

</Tip>

If you trained your own tokenizer, you can create one from your *vocabulary* file:

```py
>>> from transformers import DistilBertTokenizer

>>> my_tokenizer = DistilBertTokenizer(vocab_file="my_vocab_file.txt", do_lower_case=False, padding_side="left")
```

It is important to remember the vocabulary from a custom tokenizer will be different from the vocabulary generated by a pretrained model's tokenizer. You need to use a pretrained model's vocabulary if you are using a pretrained model, otherwise the inputs won't make sense.


key: _node_content
string_value:
{"id_": "3f57254b-efac-4a67-b50c-92250b56801c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8c9fb3ec-42ec-4923-83e4-199288c3bc95", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c580f6c975aa9151af462a3224645ef200b0db69bd9186349d1e84fa758e1ab0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4ad181d4-8a7e-449e-aed6-51523203da80", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "05a8e29a89fabe96577a49cdc9f8b43bf60d4c29c23ec287afcc542eb53f311c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9968ccac-82db-44d4-9b57-2b930b23c457", "node_type": "1", "metadata": {}, "hash": "386792257a6e23b1b96aa9b61f0d8aa3a29a42ce3292925605633a905181b9c9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8932, "end_char_idx": 12134, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> from transformers import TFDistilBertForQuestionAnswering

>>> tf_model = TFDistilBertForQuestionAnswering.from_pretrained("distilbert/distilbert-base-uncased")
```
</tf>
</frameworkcontent>

## Tokenizer

The last base class you need before using a model for textual data is a [tokenizer](main_classes/tokenizer) to convert raw text to tensors. There are two types of tokenizers you can use with 🤗 Transformers:

- [`PreTrainedTokenizer`]: a Python implementation of a tokenizer.
- [`PreTrainedTokenizerFast`]: a tokenizer from our Rust-based [🤗 Tokenizer](https://huggingface.co/docs/tokenizers/python/latest/) library. This tokenizer type is significantly faster - especially during batch tokenization - due to its Rust implementation. The fast tokenizer also offers additional methods like *offset mapping* which maps tokens to their original words or characters.

Both tokenizers support common methods such as encoding and decoding, adding new tokens, and managing special tokens.

<Tip warning={true}>

Not every model supports a fast tokenizer. Take a look at this [table](index#supported-frameworks) to check if a model has fast tokenizer support.

</Tip>

If you trained your own tokenizer, you can create one from your *vocabulary* file:

```py
>>> from transformers import DistilBertTokenizer

>>> my_tokenizer = DistilBertTokenizer(vocab_file="my_vocab_file.txt", do_lower_case=False, padding_side="left")
```

It is important to remember the vocabulary from a custom tokenizer will be different from the vocabulary generated by a pretrained model's tokenizer. You need to use a pretrained model's vocabulary if you are using a pretrained model, otherwise the inputs won't make sense. Create a tokenizer with a pretrained model's vocabulary with the [`DistilBertTokenizer`] class:

```py
>>> from transformers import DistilBertTokenizer

>>> slow_tokenizer = DistilBertTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
```

Create a fast tokenizer with the [`DistilBertTokenizerFast`] class:

```py
>>> from transformers import DistilBertTokenizerFast

>>> fast_tokenizer = DistilBertTokenizerFast.from_pretrained("distilbert/distilbert-base-uncased")
```

<Tip>

By default, [`AutoTokenizer`] will try to load a fast tokenizer. You can disable this behavior by setting `use_fast=False` in `from_pretrained`.

</Tip>

## Image processor

An image processor processes vision inputs. It inherits from the base [`~image_processing_utils.ImageProcessingMixin`] class.

To use, create an image processor associated with the model you're using. For example, create a default [`ViTImageProcessor`] if you are using [ViT](model_doc/vit) for image classification:

```py
>>> from transformers import ViTImageProcessor

>>> vit_extractor = ViTImageProcessor()
>>> print(vit_extractor)
ViTImageProcessor {
  "do_normalize": true,
  "do_resize": true,
  "image_processor_type": "ViTImageProcessor",
  "image_mean": [
    0.5,
    0.5,
    0.5
  ],
  "image_std": [
    0.5,
    0.5,
    0.5
  ],
  "resample": 2,
  "size": 224
}
```

<Tip>

If you aren't looking for any customization, just use the `from_pretrained` method to load a model's default image processor parameters.


key: _node_content
string_value:
{"id_": "9968ccac-82db-44d4-9b57-2b930b23c457", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8c9fb3ec-42ec-4923-83e4-199288c3bc95", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c580f6c975aa9151af462a3224645ef200b0db69bd9186349d1e84fa758e1ab0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3f57254b-efac-4a67-b50c-92250b56801c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8e2471b0086043c9da48a5c7ffe1aa32ff942525422deea15cb87744792bca28", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e529d669-6517-469e-8287-6054a55abe41", "node_type": "1", "metadata": {}, "hash": "71c014e8650b38c38e188df63fc00f7d332c8586144f797ba4af2a795a633d41", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10640, "end_char_idx": 13600, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Create a tokenizer with a pretrained model's vocabulary with the [`DistilBertTokenizer`] class:

```py
>>> from transformers import DistilBertTokenizer

>>> slow_tokenizer = DistilBertTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
```

Create a fast tokenizer with the [`DistilBertTokenizerFast`] class:

```py
>>> from transformers import DistilBertTokenizerFast

>>> fast_tokenizer = DistilBertTokenizerFast.from_pretrained("distilbert/distilbert-base-uncased")
```

<Tip>

By default, [`AutoTokenizer`] will try to load a fast tokenizer. You can disable this behavior by setting `use_fast=False` in `from_pretrained`.

</Tip>

## Image processor

An image processor processes vision inputs. It inherits from the base [`~image_processing_utils.ImageProcessingMixin`] class.

To use, create an image processor associated with the model you're using. For example, create a default [`ViTImageProcessor`] if you are using [ViT](model_doc/vit) for image classification:

```py
>>> from transformers import ViTImageProcessor

>>> vit_extractor = ViTImageProcessor()
>>> print(vit_extractor)
ViTImageProcessor {
  "do_normalize": true,
  "do_resize": true,
  "image_processor_type": "ViTImageProcessor",
  "image_mean": [
    0.5,
    0.5,
    0.5
  ],
  "image_std": [
    0.5,
    0.5,
    0.5
  ],
  "resample": 2,
  "size": 224
}
```

<Tip>

If you aren't looking for any customization, just use the `from_pretrained` method to load a model's default image processor parameters.

</Tip>

Modify any of the [`ViTImageProcessor`] parameters to create your custom image processor:

```py
>>> from transformers import ViTImageProcessor

>>> my_vit_extractor = ViTImageProcessor(resample="PIL.Image.BOX", do_normalize=False, image_mean=[0.3, 0.3, 0.3])
>>> print(my_vit_extractor)
ViTImageProcessor {
  "do_normalize": false,
  "do_resize": true,
  "image_processor_type": "ViTImageProcessor",
  "image_mean": [
    0.3,
    0.3,
    0.3
  ],
  "image_std": [
    0.5,
    0.5,
    0.5
  ],
  "resample": "PIL.Image.BOX",
  "size": 224
}
```

## Backbone

<div style="text-align: center">
  <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/Backbone.png">
</div>

Computer vision models consist of a backbone, neck, and head. The backbone extracts features from an input image, the neck combines and enhances the extracted features, and the head is used for the main task (e.g., object detection). Start by initializing a backbone in the model config and specify whether you want to load pretrained weights or load randomly initialized weights. Then you can pass the model config to the model head.

For example, to load a [ResNet](../model_doc/resnet) backbone into a [MaskFormer](../model_doc/maskformer) model with an instance segmentation head:

<hfoptions id="backbone">
<hfoption id="pretrained weights">

Set `use_pretrained_backbone=True` to load pretrained ResNet weights for the backbone.


key: _node_content
string_value:
{"id_": "e529d669-6517-469e-8287-6054a55abe41", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8c9fb3ec-42ec-4923-83e4-199288c3bc95", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c580f6c975aa9151af462a3224645ef200b0db69bd9186349d1e84fa758e1ab0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9968ccac-82db-44d4-9b57-2b930b23c457", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8734c6c9875154d6077b518ab0a92f19bcc6fb89824d51ade2a91fcecf949f33", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ecde499a-beed-4f38-bcf4-dbde646674f2", "node_type": "1", "metadata": {}, "hash": "885dfaaafbdcd8392b25cb48456c8e879b47959425aca733eb95436e4bd4d3c2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12136, "end_char_idx": 14896, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

Modify any of the [`ViTImageProcessor`] parameters to create your custom image processor:

```py
>>> from transformers import ViTImageProcessor

>>> my_vit_extractor = ViTImageProcessor(resample="PIL.Image.BOX", do_normalize=False, image_mean=[0.3, 0.3, 0.3])
>>> print(my_vit_extractor)
ViTImageProcessor {
  "do_normalize": false,
  "do_resize": true,
  "image_processor_type": "ViTImageProcessor",
  "image_mean": [
    0.3,
    0.3,
    0.3
  ],
  "image_std": [
    0.5,
    0.5,
    0.5
  ],
  "resample": "PIL.Image.BOX",
  "size": 224
}
```

## Backbone

<div style="text-align: center">
  <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/Backbone.png">
</div>

Computer vision models consist of a backbone, neck, and head. The backbone extracts features from an input image, the neck combines and enhances the extracted features, and the head is used for the main task (e.g., object detection). Start by initializing a backbone in the model config and specify whether you want to load pretrained weights or load randomly initialized weights. Then you can pass the model config to the model head.

For example, to load a [ResNet](../model_doc/resnet) backbone into a [MaskFormer](../model_doc/maskformer) model with an instance segmentation head:

<hfoptions id="backbone">
<hfoption id="pretrained weights">

Set `use_pretrained_backbone=True` to load pretrained ResNet weights for the backbone.

```py
from transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation

config = MaskFormerConfig(backbone="microsoft/resnet-50", use_pretrained_backbone=True) # backbone and neck config
model = MaskFormerForInstanceSegmentation(config) # head
```

</hfoption>
<hfoption id="random weights">

Set `use_pretrained_backbone=False` to randomly initialize a ResNet backbone.

```py
from transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation

config = MaskFormerConfig(backbone="microsoft/resnet-50", use_pretrained_backbone=False) # backbone and neck config
model = MaskFormerForInstanceSegmentation(config) # head
```

You could also load the backbone config separately and then pass it to the model config.

```py
from transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation, ResNetConfig

backbone_config = ResNetConfig()
config = MaskFormerConfig(backbone_config=backbone_config)
model = MaskFormerForInstanceSegmentation(config)
```

</hfoption>
</hfoptions id="timm backbone">

[timm](https://hf.co/docs/timm/index) models are loaded within a model with `use_timm_backbone=True` or with [`TimmBackbone`] and [`TimmBackboneConfig`].

Use `use_timm_backbone=True` and `use_pretrained_backbone=True` to load pretrained timm weights for the backbone.


key: _node_content
string_value:
{"id_": "ecde499a-beed-4f38-bcf4-dbde646674f2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8c9fb3ec-42ec-4923-83e4-199288c3bc95", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c580f6c975aa9151af462a3224645ef200b0db69bd9186349d1e84fa758e1ab0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e529d669-6517-469e-8287-6054a55abe41", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "1773250c53746994e3883d86a75403805a9224b8040178f185ec4ae905372711", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bd0c5a1d-4061-4908-b976-81eb57e4aad4", "node_type": "1", "metadata": {}, "hash": "a5a0ec48555ad6214e2ba002241f55c0926715c1cf9fb41be65af115e72cd50e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13247, "end_char_idx": 16584, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Then you can pass the model config to the model head.

For example, to load a [ResNet](../model_doc/resnet) backbone into a [MaskFormer](../model_doc/maskformer) model with an instance segmentation head:

<hfoptions id="backbone">
<hfoption id="pretrained weights">

Set `use_pretrained_backbone=True` to load pretrained ResNet weights for the backbone.

```py
from transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation

config = MaskFormerConfig(backbone="microsoft/resnet-50", use_pretrained_backbone=True) # backbone and neck config
model = MaskFormerForInstanceSegmentation(config) # head
```

</hfoption>
<hfoption id="random weights">

Set `use_pretrained_backbone=False` to randomly initialize a ResNet backbone.

```py
from transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation

config = MaskFormerConfig(backbone="microsoft/resnet-50", use_pretrained_backbone=False) # backbone and neck config
model = MaskFormerForInstanceSegmentation(config) # head
```

You could also load the backbone config separately and then pass it to the model config.

```py
from transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation, ResNetConfig

backbone_config = ResNetConfig()
config = MaskFormerConfig(backbone_config=backbone_config)
model = MaskFormerForInstanceSegmentation(config)
```

</hfoption>
</hfoptions id="timm backbone">

[timm](https://hf.co/docs/timm/index) models are loaded within a model with `use_timm_backbone=True` or with [`TimmBackbone`] and [`TimmBackboneConfig`].

Use `use_timm_backbone=True` and `use_pretrained_backbone=True` to load pretrained timm weights for the backbone.

```python
from transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation

config = MaskFormerConfig(backbone="resnet50", use_pretrained_backbone=True, use_timm_backbone=True) # backbone and neck config
model = MaskFormerForInstanceSegmentation(config) # head
```

Set `use_timm_backbone=True` and `use_pretrained_backbone=False` to load a randomly initialized timm backbone.

```python
from transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation

config = MaskFormerConfig(backbone="resnet50", use_pretrained_backbone=False, use_timm_backbone=True) # backbone and neck config
model = MaskFormerForInstanceSegmentation(config) # head
```

You could also load the backbone config and use it to create a `TimmBackbone` or pass it to the model config. Timm backbones will load pretrained weights by default. Set `use_pretrained_backbone=False` to load randomly initialized weights.

```python
from transformers import TimmBackboneConfig, TimmBackbone

backbone_config = TimmBackboneConfig("resnet50", use_pretrained_backbone=False)

# Create a backbone class
backbone = TimmBackbone(config=backbone_config)

# Create a model with a timm backbone
from transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation

config = MaskFormerConfig(backbone_config=backbone_config)
model = MaskFormerForInstanceSegmentation(config)
```

## Feature extractor

A feature extractor processes audio inputs. It inherits from the base [`~feature_extraction_utils.FeatureExtractionMixin`] class, and may also inherit from the [`SequenceFeatureExtractor`] class for processing audio inputs.

To use, create a feature extractor associated with the model you're using.


key: _node_content
string_value:
{"id_": "bd0c5a1d-4061-4908-b976-81eb57e4aad4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8c9fb3ec-42ec-4923-83e4-199288c3bc95", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c580f6c975aa9151af462a3224645ef200b0db69bd9186349d1e84fa758e1ab0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ecde499a-beed-4f38-bcf4-dbde646674f2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "ac538f38953c69a808c0823e1e383f84cbb3eb12d229250d724f8d61376fcfb8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "80ab1b1d-9877-4552-841c-c68e144db86c", "node_type": "1", "metadata": {}, "hash": "f60fa4c524bd51c61f93430adbef83bfb17dbe28ebd3c4be56967f996eff408f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14898, "end_char_idx": 18073, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation

config = MaskFormerConfig(backbone="resnet50", use_pretrained_backbone=True, use_timm_backbone=True) # backbone and neck config
model = MaskFormerForInstanceSegmentation(config) # head
```

Set `use_timm_backbone=True` and `use_pretrained_backbone=False` to load a randomly initialized timm backbone.

```python
from transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation

config = MaskFormerConfig(backbone="resnet50", use_pretrained_backbone=False, use_timm_backbone=True) # backbone and neck config
model = MaskFormerForInstanceSegmentation(config) # head
```

You could also load the backbone config and use it to create a `TimmBackbone` or pass it to the model config. Timm backbones will load pretrained weights by default. Set `use_pretrained_backbone=False` to load randomly initialized weights.

```python
from transformers import TimmBackboneConfig, TimmBackbone

backbone_config = TimmBackboneConfig("resnet50", use_pretrained_backbone=False)

# Create a backbone class
backbone = TimmBackbone(config=backbone_config)

# Create a model with a timm backbone
from transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation

config = MaskFormerConfig(backbone_config=backbone_config)
model = MaskFormerForInstanceSegmentation(config)
```

## Feature extractor

A feature extractor processes audio inputs. It inherits from the base [`~feature_extraction_utils.FeatureExtractionMixin`] class, and may also inherit from the [`SequenceFeatureExtractor`] class for processing audio inputs.

To use, create a feature extractor associated with the model you're using. For example, create a default [`Wav2Vec2FeatureExtractor`] if you are using [Wav2Vec2](model_doc/wav2vec2) for audio classification:

```py
>>> from transformers import Wav2Vec2FeatureExtractor

>>> w2v2_extractor = Wav2Vec2FeatureExtractor()
>>> print(w2v2_extractor)
Wav2Vec2FeatureExtractor {
  "do_normalize": true,
  "feature_extractor_type": "Wav2Vec2FeatureExtractor",
  "feature_size": 1,
  "padding_side": "right",
  "padding_value": 0.0,
  "return_attention_mask": false,
  "sampling_rate": 16000
}
```

<Tip>

If you aren't looking for any customization, just use the `from_pretrained` method to load a model's default feature extractor parameters.

</Tip>

Modify any of the [`Wav2Vec2FeatureExtractor`] parameters to create your custom feature extractor:

```py
>>> from transformers import Wav2Vec2FeatureExtractor

>>> w2v2_extractor = Wav2Vec2FeatureExtractor(sampling_rate=8000, do_normalize=False)
>>> print(w2v2_extractor)
Wav2Vec2FeatureExtractor {
  "do_normalize": false,
  "feature_extractor_type": "Wav2Vec2FeatureExtractor",
  "feature_size": 1,
  "padding_side": "right",
  "padding_value": 0.0,
  "return_attention_mask": false,
  "sampling_rate": 8000
}
```

## Processor

For models that support multimodal tasks, 🤗 Transformers offers a processor class that conveniently wraps processing classes such as a feature extractor and a tokenizer into a single object. For example, let's use the [`Wav2Vec2Processor`] for an automatic speech recognition task (ASR).


key: _node_content
string_value:
{"id_": "80ab1b1d-9877-4552-841c-c68e144db86c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8c9fb3ec-42ec-4923-83e4-199288c3bc95", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c580f6c975aa9151af462a3224645ef200b0db69bd9186349d1e84fa758e1ab0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bd0c5a1d-4061-4908-b976-81eb57e4aad4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model/", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "1fca72f6987f2d3cefd52beadd60725391ca205e16b90fdb083facb2a641ea33", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16585, "end_char_idx": 19176, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For example, create a default [`Wav2Vec2FeatureExtractor`] if you are using [Wav2Vec2](model_doc/wav2vec2) for audio classification:

```py
>>> from transformers import Wav2Vec2FeatureExtractor

>>> w2v2_extractor = Wav2Vec2FeatureExtractor()
>>> print(w2v2_extractor)
Wav2Vec2FeatureExtractor {
  "do_normalize": true,
  "feature_extractor_type": "Wav2Vec2FeatureExtractor",
  "feature_size": 1,
  "padding_side": "right",
  "padding_value": 0.0,
  "return_attention_mask": false,
  "sampling_rate": 16000
}
```

<Tip>

If you aren't looking for any customization, just use the `from_pretrained` method to load a model's default feature extractor parameters.

</Tip>

Modify any of the [`Wav2Vec2FeatureExtractor`] parameters to create your custom feature extractor:

```py
>>> from transformers import Wav2Vec2FeatureExtractor

>>> w2v2_extractor = Wav2Vec2FeatureExtractor(sampling_rate=8000, do_normalize=False)
>>> print(w2v2_extractor)
Wav2Vec2FeatureExtractor {
  "do_normalize": false,
  "feature_extractor_type": "Wav2Vec2FeatureExtractor",
  "feature_size": 1,
  "padding_side": "right",
  "padding_value": 0.0,
  "return_attention_mask": false,
  "sampling_rate": 8000
}
```

## Processor

For models that support multimodal tasks, 🤗 Transformers offers a processor class that conveniently wraps processing classes such as a feature extractor and a tokenizer into a single object. For example, let's use the [`Wav2Vec2Processor`] for an automatic speech recognition task (ASR). ASR transcribes audio to text, so you will need a feature extractor and a tokenizer.

Create a feature extractor to handle the audio inputs:

```py
>>> from transformers import Wav2Vec2FeatureExtractor

>>> feature_extractor = Wav2Vec2FeatureExtractor(padding_value=1.0, do_normalize=True)
```

Create a tokenizer to handle the text inputs:

```py
>>> from transformers import Wav2Vec2CTCTokenizer

>>> tokenizer = Wav2Vec2CTCTokenizer(vocab_file="my_vocab_file.txt")
```

Combine the feature extractor and tokenizer in [`Wav2Vec2Processor`]:

```py
>>> from transformers import Wav2Vec2Processor

>>> processor = Wav2Vec2Processor(feature_extractor=feature_extractor, tokenizer=tokenizer)
```

With two basic classes - configuration and model - and an additional preprocessing class (tokenizer, image processor, feature extractor, or processor), you can create any of the models supported by 🤗 Transformers. Each of these base classes are configurable, allowing you to use the specific attributes you want. You can easily setup a model for training or modify an existing pretrained model to fine-tune.


key: _node_content
string_value:
{"id_": "5b1db86d-361e-4ba1-8e96-cffa24a77860", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0cf5ca4d-95cf-4dd2-b5df-388357294619", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a3fdd2959a6b3a168b4c1d5e1bef5668fde197ae3e72bffd417165d445ce9f83", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e9969cc3-2bd5-471c-82f3-bbeeb214c738", "node_type": "1", "metadata": {}, "hash": "1d294bb065ef66ccb5a16ce1cead7cfbf88423f3d17eca5286d30ac185b77512", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 4663, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Methods and tools for efficient training on a single GPU

This guide demonstrates practical techniques that you can use to increase the efficiency of your model's training by 
optimizing memory utilization, speeding up the training, or both. If you'd like to understand how GPU is utilized during 
training, please refer to the [Model training anatomy](model_memory_anatomy) conceptual guide first. This guide 
focuses on practical techniques.  

<Tip>

If you have access to a machine with multiple GPUs, these approaches are still valid, plus you can leverage additional methods outlined in the [multi-GPU section](perf_train_gpu_many).

</Tip>

When training large models, there are two aspects that should be considered at the same time: 

* Data throughput/training time
* Model performance

Maximizing the throughput (samples/second) leads to lower training cost. This is generally achieved by utilizing the GPU 
as much as possible and thus filling GPU memory to its limit. If the desired batch size exceeds the limits of the GPU memory, 
the memory optimization techniques, such as gradient accumulation, can help.

However, if the preferred batch size fits into memory, there's no reason to apply memory-optimizing techniques because they can 
slow down the training. Just because one can use a large batch size, does not necessarily mean they should. As part of 
hyperparameter tuning, you should determine which batch size yields the best results and then optimize resources accordingly.

The methods and tools covered in this guide can be classified based on the effect they have on the training process:

| Method/tool                                                                                                                                             | Improves training speed | Optimizes memory utilization |
|:--------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------|:-----------------------------|
| [Batch size choice](#batch-size-choice)                                                                                                                 | Yes                     | Yes                          |
| [Gradient accumulation](#gradient-accumulation)                                                                                                         | No                      | Yes                          |
| [Gradient checkpointing](#gradient-checkpointing)                                                                                                       | No                      | Yes                          |
| [Mixed precision training](#mixed-precision-training)                                                                                                   | Yes                     | Maybe*                       |
| [torch_empty_cache_steps](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.TrainingArguments.torch_empty_cache_steps) | No                      | Yes                          |
| [Optimizer choice](#optimizer-choice)                                                                                                                   | Yes                     | Yes                          |
| [Data preloading](#data-preloading)                                                                                                                     | Yes                     | No                           |
| [DeepSpeed Zero](#deepspeed-zero)                                                                                                                       | No                      | Yes                          |
| [torch.compile](#using-torchcompile)                                                                                                                    | Yes                     | No                           |
| [Parameter-Efficient Fine Tuning (PEFT)](#using--peft)                                                                                                  | No                      | Yes                          |
 
<Tip>

*Note: when using mixed precision with a small model and a large batch size, there will be some memory savings but with a 
large model and a small batch size, the memory use will be larger.

</Tip>

You can combine the above methods to get a cumulative effect. These techniques are available to you whether you are 
training your model with [`Trainer`] or writing a pure PyTorch loop, in which case you can [configure these optimizations 
with 🤗 Accelerate](#using--accelerate).


key: _node_content
string_value:
{"id_": "e9969cc3-2bd5-471c-82f3-bbeeb214c738", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0cf5ca4d-95cf-4dd2-b5df-388357294619", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a3fdd2959a6b3a168b4c1d5e1bef5668fde197ae3e72bffd417165d445ce9f83", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5b1db86d-361e-4ba1-8e96-cffa24a77860", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "ea074fccf2f5d5470efaf8f0a67843e061192ddcc3d185f4ed9ff4e1bf0a784d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "94027b9d-897c-4ed9-b9e7-e5887621a784", "node_type": "1", "metadata": {}, "hash": "eb294357966085c238f3a027bb664ba22bb21e2e71a2a0c452ee62cfd9986e40", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1502, "end_char_idx": 6318, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The methods and tools covered in this guide can be classified based on the effect they have on the training process:

| Method/tool                                                                                                                                             | Improves training speed | Optimizes memory utilization |
|:--------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------|:-----------------------------|
| [Batch size choice](#batch-size-choice)                                                                                                                 | Yes                     | Yes                          |
| [Gradient accumulation](#gradient-accumulation)                                                                                                         | No                      | Yes                          |
| [Gradient checkpointing](#gradient-checkpointing)                                                                                                       | No                      | Yes                          |
| [Mixed precision training](#mixed-precision-training)                                                                                                   | Yes                     | Maybe*                       |
| [torch_empty_cache_steps](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.TrainingArguments.torch_empty_cache_steps) | No                      | Yes                          |
| [Optimizer choice](#optimizer-choice)                                                                                                                   | Yes                     | Yes                          |
| [Data preloading](#data-preloading)                                                                                                                     | Yes                     | No                           |
| [DeepSpeed Zero](#deepspeed-zero)                                                                                                                       | No                      | Yes                          |
| [torch.compile](#using-torchcompile)                                                                                                                    | Yes                     | No                           |
| [Parameter-Efficient Fine Tuning (PEFT)](#using--peft)                                                                                                  | No                      | Yes                          |
 
<Tip>

*Note: when using mixed precision with a small model and a large batch size, there will be some memory savings but with a 
large model and a small batch size, the memory use will be larger.

</Tip>

You can combine the above methods to get a cumulative effect. These techniques are available to you whether you are 
training your model with [`Trainer`] or writing a pure PyTorch loop, in which case you can [configure these optimizations 
with 🤗 Accelerate](#using--accelerate).

If these methods do not result in sufficient gains, you can explore the following options: 
* [Look into building your own custom Docker container with efficient software prebuilds](#efficient-software-prebuilds)
* [Consider a model that uses Mixture of Experts (MoE)](#mixture-of-experts)
* [Convert your model to BetterTransformer to leverage PyTorch native attention](#using-pytorch-native-attention-and-flash-attention)

Finally, if all of the above is still not enough, even after switching to a server-grade GPU like A100, consider moving 
to a multi-GPU setup. All these approaches are still valid in a multi-GPU setup, plus you can leverage additional parallelism 
techniques outlined in the [multi-GPU section](perf_train_gpu_many). 

## Batch size choice

To achieve optimal performance, start by identifying the appropriate batch size. It is recommended to use batch sizes and 
input/output neuron counts that are of size 2^N. Often it's a multiple of 8, but it can be 
higher depending on the hardware being used and the model's dtype.

For reference, check out NVIDIA's recommendation for [input/output neuron counts](
https://docs.nvidia.com/deeplearning/performance/dl-performance-fully-connected/index.html#input-features) and 
[batch size](https://docs.nvidia.com/deeplearning/performance/dl-performance-fully-connected/index.html#batch-size) for 
fully connected layers (which are involved in GEMMs (General Matrix Multiplications)).

[Tensor Core Requirements](https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc) 
define the multiplier based on the dtype and the hardware.


key: _node_content
string_value:
{"id_": "94027b9d-897c-4ed9-b9e7-e5887621a784", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0cf5ca4d-95cf-4dd2-b5df-388357294619", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a3fdd2959a6b3a168b4c1d5e1bef5668fde197ae3e72bffd417165d445ce9f83", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e9969cc3-2bd5-471c-82f3-bbeeb214c738", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "56bcf9444cb6c2fffd06666287c5b32f42a9e079b3001f9747be7d3d99bd72f5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9134fd4e-fd89-4625-b75f-d9c4ceccdcf3", "node_type": "1", "metadata": {}, "hash": "892334beae99862449fa14745627e2e6fb3411fb67ddeac0469caa24e1c160ba", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4665, "end_char_idx": 8258, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If these methods do not result in sufficient gains, you can explore the following options: 
* [Look into building your own custom Docker container with efficient software prebuilds](#efficient-software-prebuilds)
* [Consider a model that uses Mixture of Experts (MoE)](#mixture-of-experts)
* [Convert your model to BetterTransformer to leverage PyTorch native attention](#using-pytorch-native-attention-and-flash-attention)

Finally, if all of the above is still not enough, even after switching to a server-grade GPU like A100, consider moving 
to a multi-GPU setup. All these approaches are still valid in a multi-GPU setup, plus you can leverage additional parallelism 
techniques outlined in the [multi-GPU section](perf_train_gpu_many). 

## Batch size choice

To achieve optimal performance, start by identifying the appropriate batch size. It is recommended to use batch sizes and 
input/output neuron counts that are of size 2^N. Often it's a multiple of 8, but it can be 
higher depending on the hardware being used and the model's dtype.

For reference, check out NVIDIA's recommendation for [input/output neuron counts](
https://docs.nvidia.com/deeplearning/performance/dl-performance-fully-connected/index.html#input-features) and 
[batch size](https://docs.nvidia.com/deeplearning/performance/dl-performance-fully-connected/index.html#batch-size) for 
fully connected layers (which are involved in GEMMs (General Matrix Multiplications)).

[Tensor Core Requirements](https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc) 
define the multiplier based on the dtype and the hardware. For instance, for fp16 data type a multiple of 8 is recommended, unless 
it's an A100 GPU, in which case use multiples of 64.

For parameters that are small, consider also [Dimension Quantization Effects](https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#dim-quantization). 
This is where tiling happens and the right multiplier can have a significant speedup.

## Gradient Accumulation

The **gradient accumulation** method aims to calculate gradients in smaller increments instead of computing them for the 
entire batch at once. This approach involves iteratively calculating gradients in smaller batches by performing forward 
and backward passes through the model and accumulating the gradients during the process. Once a sufficient number of 
gradients have been accumulated, the model's optimization step is executed. By employing gradient accumulation, it 
becomes possible to increase the **effective batch size** beyond the limitations imposed by the GPU's memory capacity. 
However, it is important to note that the additional forward and backward passes introduced by gradient accumulation can 
slow down the training process.

You can enable gradient accumulation by adding the `gradient_accumulation_steps` argument to  [`TrainingArguments`]: 

```py
training_args = TrainingArguments(per_device_train_batch_size=1, gradient_accumulation_steps=4, **default_args)
```

In the above example, your effective batch size becomes 4. 

Alternatively, use 🤗 Accelerate to gain full control over the training loop. Find the 🤗 Accelerate example 
[further down in this guide](#using--accelerate).

While it is advised to max out GPU usage as much as possible, a high number of gradient accumulation steps can 
result in a more pronounced training slowdown. Consider the following example. Let's say, the `per_device_train_batch_size=4` 
without gradient accumulation hits the GPU's limit.


key: _node_content
string_value:
{"id_": "9134fd4e-fd89-4625-b75f-d9c4ceccdcf3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0cf5ca4d-95cf-4dd2-b5df-388357294619", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a3fdd2959a6b3a168b4c1d5e1bef5668fde197ae3e72bffd417165d445ce9f83", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "94027b9d-897c-4ed9-b9e7-e5887621a784", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6a0b195b9fa1fc81e29236e4f3856100303415ed7e5c69a8fd5a583cca1f862b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4b794f11-1313-497f-9d28-ce2f29ebe9f6", "node_type": "1", "metadata": {}, "hash": "da9605a755e7e7b3993b8b98eae6d58f29cf8b39ebba6fab809a05668083f7a5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6446, "end_char_idx": 9864, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For parameters that are small, consider also [Dimension Quantization Effects](https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#dim-quantization). 
This is where tiling happens and the right multiplier can have a significant speedup.

## Gradient Accumulation

The **gradient accumulation** method aims to calculate gradients in smaller increments instead of computing them for the 
entire batch at once. This approach involves iteratively calculating gradients in smaller batches by performing forward 
and backward passes through the model and accumulating the gradients during the process. Once a sufficient number of 
gradients have been accumulated, the model's optimization step is executed. By employing gradient accumulation, it 
becomes possible to increase the **effective batch size** beyond the limitations imposed by the GPU's memory capacity. 
However, it is important to note that the additional forward and backward passes introduced by gradient accumulation can 
slow down the training process.

You can enable gradient accumulation by adding the `gradient_accumulation_steps` argument to  [`TrainingArguments`]: 

```py
training_args = TrainingArguments(per_device_train_batch_size=1, gradient_accumulation_steps=4, **default_args)
```

In the above example, your effective batch size becomes 4. 

Alternatively, use 🤗 Accelerate to gain full control over the training loop. Find the 🤗 Accelerate example 
[further down in this guide](#using--accelerate).

While it is advised to max out GPU usage as much as possible, a high number of gradient accumulation steps can 
result in a more pronounced training slowdown. Consider the following example. Let's say, the `per_device_train_batch_size=4` 
without gradient accumulation hits the GPU's limit. If you would like to train with batches of size 64, do not set the 
`per_device_train_batch_size` to 1 and `gradient_accumulation_steps` to 64. Instead, keep `per_device_train_batch_size=4` 
and set `gradient_accumulation_steps=16`. This results in the same effective batch size while making better use of 
the available GPU resources.

For additional information, please refer to batch size and gradient accumulation benchmarks for [RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004392537)
and [A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1005033957).

## Gradient Checkpointing

Some large models may still face memory issues even when the batch size is set to 1 and gradient accumulation is used. 
This is because there are other components that also require memory storage.

Saving all activations from the forward pass in order to compute the gradients during the backward pass can result in 
significant memory overhead. The alternative approach of discarding the activations and recalculating them when needed 
during the backward pass, would introduce a considerable computational overhead and slow down the training process.

**Gradient checkpointing** offers a compromise between these two approaches and saves strategically selected activations 
throughout the computational graph so only a fraction of the activations need to be re-computed for the gradients. For 
an in-depth explanation of gradient checkpointing, refer to [this great article](https://medium.com/tensorflow/fitting-larger-networks-into-memory-583e3c758ff9).


key: _node_content
string_value:
{"id_": "4b794f11-1313-497f-9d28-ce2f29ebe9f6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0cf5ca4d-95cf-4dd2-b5df-388357294619", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a3fdd2959a6b3a168b4c1d5e1bef5668fde197ae3e72bffd417165d445ce9f83", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9134fd4e-fd89-4625-b75f-d9c4ceccdcf3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3b22ab96a9806231069bee39da7d50c783338604fd674e97154f938843736ada", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "60676925-e26b-4fc4-ba00-854487a68bd2", "node_type": "1", "metadata": {}, "hash": "44d8cb7ee24f612184b3c99d5457da56a579a26848c689859c000e11e87d8ad7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8127, "end_char_idx": 11768, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Consider the following example. Let's say, the `per_device_train_batch_size=4` 
without gradient accumulation hits the GPU's limit. If you would like to train with batches of size 64, do not set the 
`per_device_train_batch_size` to 1 and `gradient_accumulation_steps` to 64. Instead, keep `per_device_train_batch_size=4` 
and set `gradient_accumulation_steps=16`. This results in the same effective batch size while making better use of 
the available GPU resources.

For additional information, please refer to batch size and gradient accumulation benchmarks for [RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004392537)
and [A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1005033957).

## Gradient Checkpointing

Some large models may still face memory issues even when the batch size is set to 1 and gradient accumulation is used. 
This is because there are other components that also require memory storage.

Saving all activations from the forward pass in order to compute the gradients during the backward pass can result in 
significant memory overhead. The alternative approach of discarding the activations and recalculating them when needed 
during the backward pass, would introduce a considerable computational overhead and slow down the training process.

**Gradient checkpointing** offers a compromise between these two approaches and saves strategically selected activations 
throughout the computational graph so only a fraction of the activations need to be re-computed for the gradients. For 
an in-depth explanation of gradient checkpointing, refer to [this great article](https://medium.com/tensorflow/fitting-larger-networks-into-memory-583e3c758ff9).

To enable gradient checkpointing in the [`Trainer`], pass the corresponding a flag to [`TrainingArguments`]:

```py
training_args = TrainingArguments(
    per_device_train_batch_size=1, gradient_accumulation_steps=4, gradient_checkpointing=True, **default_args
)
```

Alternatively, use 🤗 Accelerate - find the 🤗 Accelerate example [further in this guide](#using--accelerate). 

<Tip>

While gradient checkpointing may improve memory efficiency, it slows training by approximately 20%.

</Tip>

## Mixed precision training

**Mixed precision training** is a technique that aims to optimize the computational efficiency of training models by 
utilizing lower-precision numerical formats for certain variables. Traditionally, most models use 32-bit floating point 
precision (fp32 or float32) to represent and process variables. However, not all variables require this high precision 
level to achieve accurate results. By reducing the precision of certain variables to lower numerical formats like 16-bit 
floating point (fp16 or float16), we can speed up the computations. Because in this approach some computations are performed 
in half-precision, while some are still in full precision, the approach is called mixed precision training.

Most commonly mixed precision training is achieved by using fp16 (float16) data types, however, some GPU architectures 
(such as the Ampere architecture) offer bf16 and tf32 (CUDA internal data type) data types. Check 
out the [NVIDIA Blog](https://developer.nvidia.com/blog/accelerating-ai-training-with-tf32-tensor-cores/) to learn more about 
the differences between these data types.

### fp16

The main advantage of mixed precision training comes from saving the activations in half precision (fp16). 
Although the gradients are also computed in half precision they are converted back to full precision for the optimization 
step so no memory is saved here.


key: _node_content
string_value:
{"id_": "60676925-e26b-4fc4-ba00-854487a68bd2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0cf5ca4d-95cf-4dd2-b5df-388357294619", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a3fdd2959a6b3a168b4c1d5e1bef5668fde197ae3e72bffd417165d445ce9f83", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4b794f11-1313-497f-9d28-ce2f29ebe9f6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d08711c4ce986a7bc7f1747080651cb1d079200b68ed00b7881534079419cfaf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "624e03ff-c8cd-439f-8fef-ccc95306fd5c", "node_type": "1", "metadata": {}, "hash": "4b3721e222332d2bf8efbb5456adc6108af4f418d419433dac15209598569963", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10245, "end_char_idx": 13584, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<Tip>

While gradient checkpointing may improve memory efficiency, it slows training by approximately 20%.

</Tip>

## Mixed precision training

**Mixed precision training** is a technique that aims to optimize the computational efficiency of training models by 
utilizing lower-precision numerical formats for certain variables. Traditionally, most models use 32-bit floating point 
precision (fp32 or float32) to represent and process variables. However, not all variables require this high precision 
level to achieve accurate results. By reducing the precision of certain variables to lower numerical formats like 16-bit 
floating point (fp16 or float16), we can speed up the computations. Because in this approach some computations are performed 
in half-precision, while some are still in full precision, the approach is called mixed precision training.

Most commonly mixed precision training is achieved by using fp16 (float16) data types, however, some GPU architectures 
(such as the Ampere architecture) offer bf16 and tf32 (CUDA internal data type) data types. Check 
out the [NVIDIA Blog](https://developer.nvidia.com/blog/accelerating-ai-training-with-tf32-tensor-cores/) to learn more about 
the differences between these data types.

### fp16

The main advantage of mixed precision training comes from saving the activations in half precision (fp16). 
Although the gradients are also computed in half precision they are converted back to full precision for the optimization 
step so no memory is saved here. 
While mixed precision training results in faster computations, it can also lead to more GPU memory being utilized, especially for small batch sizes.
This is because the model is now present on the GPU in both 16-bit and 32-bit precision (1.5x the original model on the GPU).

To enable mixed precision training, set the `fp16` flag to `True`:

```py
training_args = TrainingArguments(per_device_train_batch_size=4, fp16=True, **default_args)
```

If you prefer to use 🤗 Accelerate, find the 🤗 Accelerate example [further in this guide](#using--accelerate). 

### BF16

If you have access to an Ampere or newer hardware you can use bf16 for mixed precision training and evaluation. While 
bf16 has a worse precision than fp16, it has a much bigger dynamic range. In fp16 the biggest number you can have 
is `65535` and any number above that will result in an overflow. A bf16 number can be as large as `3.39e+38` (!) which 
is about the same as fp32 - because both have 8-bits used for the numerical range.

You can enable BF16 in the 🤗 Trainer with:

```python
training_args = TrainingArguments(bf16=True, **default_args)
```

### TF32

The Ampere hardware uses a magical data type called tf32. It has the same numerical range as fp32 (8-bits), but instead 
of 23 bits precision it has only 10 bits (same as fp16) and uses only 19 bits in total. It's "magical" in the sense that 
you can use the normal fp32 training and/or inference code and by enabling tf32 support you can get up to 3x throughput 
improvement. All you need to do is to add the following to your code:

```python
import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.allow_tf32 = True
```

CUDA will automatically switch to using tf32 instead of fp32 where possible, assuming that the used GPU is from the Ampere series.


key: _node_content
string_value:
{"id_": "624e03ff-c8cd-439f-8fef-ccc95306fd5c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0cf5ca4d-95cf-4dd2-b5df-388357294619", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a3fdd2959a6b3a168b4c1d5e1bef5668fde197ae3e72bffd417165d445ce9f83", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "60676925-e26b-4fc4-ba00-854487a68bd2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "2a755723fe14ecc4b319b6a10cef4b54047d9bf38f99c15d2a8d95b0c9018504", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cd0dbc23-7849-42ce-ad0b-8b5a7263ee15", "node_type": "1", "metadata": {}, "hash": "81275c55185587f37ff37d2e5261d369d5a345a48d0ba8b2a4a92abb5cee2638", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12329, "end_char_idx": 15435, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### BF16

If you have access to an Ampere or newer hardware you can use bf16 for mixed precision training and evaluation. While 
bf16 has a worse precision than fp16, it has a much bigger dynamic range. In fp16 the biggest number you can have 
is `65535` and any number above that will result in an overflow. A bf16 number can be as large as `3.39e+38` (!) which 
is about the same as fp32 - because both have 8-bits used for the numerical range.

You can enable BF16 in the 🤗 Trainer with:

```python
training_args = TrainingArguments(bf16=True, **default_args)
```

### TF32

The Ampere hardware uses a magical data type called tf32. It has the same numerical range as fp32 (8-bits), but instead 
of 23 bits precision it has only 10 bits (same as fp16) and uses only 19 bits in total. It's "magical" in the sense that 
you can use the normal fp32 training and/or inference code and by enabling tf32 support you can get up to 3x throughput 
improvement. All you need to do is to add the following to your code:

```python
import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.allow_tf32 = True
```

CUDA will automatically switch to using tf32 instead of fp32 where possible, assuming that the used GPU is from the Ampere series.

According to [NVIDIA research](https://developer.nvidia.com/blog/accelerating-ai-training-with-tf32-tensor-cores/), the 
majority of machine learning training workloads show the same perplexity and convergence with tf32 training as with fp32. 
If you're already using fp16 or bf16 mixed precision it may help with the throughput as well.

You can enable this mode in the 🤗 Trainer:

```python
TrainingArguments(tf32=True, **default_args)
```

<Tip>

tf32 can't be accessed directly via `tensor.to(dtype=torch.tf32)` because it is an internal CUDA data type. You need `torch>=1.7` to use tf32 data types.

</Tip>

For additional information on tf32 vs other precisions, please refer to the following benchmarks: 
[RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004390803) and
[A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1004543189).

## Flash Attention 2

You can speedup the training throughput by using Flash Attention 2 integration in transformers. Check out the appropriate section in the [single GPU section](./perf_infer_gpu_one#Flash-Attention-2) to learn more about how to load a model with Flash Attention 2 modules. 

## Optimizer choice

The most common optimizer used to train transformer models is Adam or AdamW (Adam with weight decay). Adam achieves 
good convergence by storing the rolling average of the previous gradients; however, it adds an additional memory 
footprint of the order of the number of model parameters. To remedy this, you can use an alternative optimizer. 
For example if you have [NVIDIA/apex](https://github.com/NVIDIA/apex) installed for NVIDIA GPUs, or [ROCmSoftwarePlatform/apex](https://github.com/ROCmSoftwarePlatform/apex) for AMD GPUs, `adamw_apex_fused` will give you the
fastest training experience among all supported AdamW optimizers.


key: _node_content
string_value:
{"id_": "cd0dbc23-7849-42ce-ad0b-8b5a7263ee15", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0cf5ca4d-95cf-4dd2-b5df-388357294619", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a3fdd2959a6b3a168b4c1d5e1bef5668fde197ae3e72bffd417165d445ce9f83", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "624e03ff-c8cd-439f-8fef-ccc95306fd5c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "bf718f7c47f692951f57daecdc3a8d896f70118afe24318f391005b9ee8d8cb6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "57a71860-8cf6-48cd-b338-6ed47b461ec0", "node_type": "1", "metadata": {}, "hash": "7ca15e68e49fa56cdac15503edeb2bc9ada2eba13b8c85e1385faa9a67fab7cd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13830, "end_char_idx": 16652, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If you're already using fp16 or bf16 mixed precision it may help with the throughput as well.

You can enable this mode in the 🤗 Trainer:

```python
TrainingArguments(tf32=True, **default_args)
```

<Tip>

tf32 can't be accessed directly via `tensor.to(dtype=torch.tf32)` because it is an internal CUDA data type. You need `torch>=1.7` to use tf32 data types.

</Tip>

For additional information on tf32 vs other precisions, please refer to the following benchmarks: 
[RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004390803) and
[A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1004543189).

## Flash Attention 2

You can speedup the training throughput by using Flash Attention 2 integration in transformers. Check out the appropriate section in the [single GPU section](./perf_infer_gpu_one#Flash-Attention-2) to learn more about how to load a model with Flash Attention 2 modules. 

## Optimizer choice

The most common optimizer used to train transformer models is Adam or AdamW (Adam with weight decay). Adam achieves 
good convergence by storing the rolling average of the previous gradients; however, it adds an additional memory 
footprint of the order of the number of model parameters. To remedy this, you can use an alternative optimizer. 
For example if you have [NVIDIA/apex](https://github.com/NVIDIA/apex) installed for NVIDIA GPUs, or [ROCmSoftwarePlatform/apex](https://github.com/ROCmSoftwarePlatform/apex) for AMD GPUs, `adamw_apex_fused` will give you the
fastest training experience among all supported AdamW optimizers.

[`Trainer`] integrates a variety of optimizers that can be used out of box: `adamw_hf`, `adamw_torch`, `adamw_torch_fused`, 
`adamw_apex_fused`, `adamw_anyprecision`, `adafactor`, or `adamw_bnb_8bit`. More optimizers can be plugged in via a third-party implementation.

Let's take a closer look at two alternatives to AdamW optimizer:
1. `adafactor` which is available in [`Trainer`]
2. `adamw_bnb_8bit` is also available in Trainer, but a third-party integration is provided below for demonstration.

For comparison, for a 3B-parameter model, like “google-t5/t5-3b”: 
* A standard AdamW optimizer will need 24GB of GPU memory because it uses 8 bytes for each parameter (8*3 => 24GB)
* Adafactor optimizer will need more than 12GB. It uses slightly more than 4 bytes for each parameter, so 4*3 and then some extra.
* 8bit BNB quantized optimizer will use only (2*3) 6GB if all optimizer states are quantized.

### Adafactor

Adafactor doesn't store rolling averages for each element in weight matrices. Instead, it keeps aggregated information 
(sums of rolling averages row- and column-wise), significantly reducing its footprint. However, compared to Adam, 
Adafactor may have slower convergence in certain cases.


key: _node_content
string_value:
{"id_": "57a71860-8cf6-48cd-b338-6ed47b461ec0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0cf5ca4d-95cf-4dd2-b5df-388357294619", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a3fdd2959a6b3a168b4c1d5e1bef5668fde197ae3e72bffd417165d445ce9f83", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cd0dbc23-7849-42ce-ad0b-8b5a7263ee15", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "417f49d4d6782d3f111a0c2b88ef411e685ce8c8ed2eaf83be1e93f819af3d81", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "642ed6df-133f-4506-8b7b-c0126f68a2a1", "node_type": "1", "metadata": {}, "hash": "2b16bfcf163efe9d449daea34e45248b6aec8a573e35473da48374c8fdff8925", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15145, "end_char_idx": 18226, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For example if you have [NVIDIA/apex](https://github.com/NVIDIA/apex) installed for NVIDIA GPUs, or [ROCmSoftwarePlatform/apex](https://github.com/ROCmSoftwarePlatform/apex) for AMD GPUs, `adamw_apex_fused` will give you the
fastest training experience among all supported AdamW optimizers.

[`Trainer`] integrates a variety of optimizers that can be used out of box: `adamw_hf`, `adamw_torch`, `adamw_torch_fused`, 
`adamw_apex_fused`, `adamw_anyprecision`, `adafactor`, or `adamw_bnb_8bit`. More optimizers can be plugged in via a third-party implementation.

Let's take a closer look at two alternatives to AdamW optimizer:
1. `adafactor` which is available in [`Trainer`]
2. `adamw_bnb_8bit` is also available in Trainer, but a third-party integration is provided below for demonstration.

For comparison, for a 3B-parameter model, like “google-t5/t5-3b”: 
* A standard AdamW optimizer will need 24GB of GPU memory because it uses 8 bytes for each parameter (8*3 => 24GB)
* Adafactor optimizer will need more than 12GB. It uses slightly more than 4 bytes for each parameter, so 4*3 and then some extra.
* 8bit BNB quantized optimizer will use only (2*3) 6GB if all optimizer states are quantized.

### Adafactor

Adafactor doesn't store rolling averages for each element in weight matrices. Instead, it keeps aggregated information 
(sums of rolling averages row- and column-wise), significantly reducing its footprint. However, compared to Adam, 
Adafactor may have slower convergence in certain cases.

You can switch to Adafactor by setting `optim="adafactor"` in [`TrainingArguments`]:

```py
training_args = TrainingArguments(per_device_train_batch_size=4, optim="adafactor", **default_args)
```

Combined with other approaches (gradient accumulation, gradient checkpointing, and mixed precision training) 
you can notice up to 3x improvement while maintaining the throughput! However, as mentioned before, the convergence of 
Adafactor can be worse than Adam. 

### 8-bit Adam

Instead of aggregating optimizer states like Adafactor, 8-bit Adam keeps the full state and quantizes it. Quantization 
means that it stores the state with lower precision and dequantizes it only for the optimization. This is similar to the 
idea behind mixed precision training.

To use `adamw_bnb_8bit`, you simply need to set `optim="adamw_bnb_8bit"` in [`TrainingArguments`]:

```py
training_args = TrainingArguments(per_device_train_batch_size=4, optim="adamw_bnb_8bit", **default_args)
```

However, we can also use a third-party implementation of the 8-bit optimizer for demonstration purposes to see how that can be integrated.

First, follow the installation guide in the GitHub [repo](https://github.com/TimDettmers/bitsandbytes) to install the `bitsandbytes` library 
that implements the 8-bit Adam optimizer.

Next you need to initialize the optimizer. This involves two steps: 
* First, group the model's parameters into two groups - one where weight decay should be applied, and the other one where it should not. Usually, biases and layer norm parameters are not weight decayed.


key: _node_content
string_value:
{"id_": "642ed6df-133f-4506-8b7b-c0126f68a2a1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0cf5ca4d-95cf-4dd2-b5df-388357294619", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a3fdd2959a6b3a168b4c1d5e1bef5668fde197ae3e72bffd417165d445ce9f83", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "57a71860-8cf6-48cd-b338-6ed47b461ec0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3d6d6b2d440ba45a6e96e9cc24838725946fd363cfc9c60ae1f51acb647b78d0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d5cab6f3-a5e3-4065-9482-7199d2e8e85f", "node_type": "1", "metadata": {}, "hash": "b6819d4428a3491e37fd7e89bffdfcedb3ceda16e2c099ab3eb2d29af1aa8062", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16569, "end_char_idx": 19786, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, compared to Adam, 
Adafactor may have slower convergence in certain cases.

You can switch to Adafactor by setting `optim="adafactor"` in [`TrainingArguments`]:

```py
training_args = TrainingArguments(per_device_train_batch_size=4, optim="adafactor", **default_args)
```

Combined with other approaches (gradient accumulation, gradient checkpointing, and mixed precision training) 
you can notice up to 3x improvement while maintaining the throughput! However, as mentioned before, the convergence of 
Adafactor can be worse than Adam. 

### 8-bit Adam

Instead of aggregating optimizer states like Adafactor, 8-bit Adam keeps the full state and quantizes it. Quantization 
means that it stores the state with lower precision and dequantizes it only for the optimization. This is similar to the 
idea behind mixed precision training.

To use `adamw_bnb_8bit`, you simply need to set `optim="adamw_bnb_8bit"` in [`TrainingArguments`]:

```py
training_args = TrainingArguments(per_device_train_batch_size=4, optim="adamw_bnb_8bit", **default_args)
```

However, we can also use a third-party implementation of the 8-bit optimizer for demonstration purposes to see how that can be integrated.

First, follow the installation guide in the GitHub [repo](https://github.com/TimDettmers/bitsandbytes) to install the `bitsandbytes` library 
that implements the 8-bit Adam optimizer.

Next you need to initialize the optimizer. This involves two steps: 
* First, group the model's parameters into two groups - one where weight decay should be applied, and the other one where it should not. Usually, biases and layer norm parameters are not weight decayed. 
* Then do some argument housekeeping to use the same parameters as the previously used AdamW optimizer.

```py
import bitsandbytes as bnb
from torch import nn
from transformers.trainer_pt_utils import get_parameter_names

training_args = TrainingArguments(per_device_train_batch_size=4, **default_args)

decay_parameters = get_parameter_names(model, [nn.LayerNorm])
decay_parameters = [name for name in decay_parameters if "bias" not in name]
optimizer_grouped_parameters = [
    {
        "params": [p for n, p in model.named_parameters() if n in decay_parameters],
        "weight_decay": training_args.weight_decay,
    },
    {
        "params": [p for n, p in model.named_parameters() if n not in decay_parameters],
        "weight_decay": 0.0,
    },
]

optimizer_kwargs = {
    "betas": (training_args.adam_beta1, training_args.adam_beta2),
    "eps": training_args.adam_epsilon,
}
optimizer_kwargs["lr"] = training_args.learning_rate
adam_bnb_optim = bnb.optim.Adam8bit(
    optimizer_grouped_parameters,
    betas=(training_args.adam_beta1, training_args.adam_beta2),
    eps=training_args.adam_epsilon,
    lr=training_args.learning_rate,
)
```

Finally, pass the custom optimizer as an argument to the `Trainer`:

```py
trainer = Trainer(model=model, args=training_args, train_dataset=ds, optimizers=(adam_bnb_optim, None))
```

Combined with other approaches (gradient accumulation, gradient checkpointing, and mixed precision training), 
you can expect to get about a 3x memory improvement and even slightly higher throughput as using Adafactor.


key: _node_content
string_value:
{"id_": "d5cab6f3-a5e3-4065-9482-7199d2e8e85f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0cf5ca4d-95cf-4dd2-b5df-388357294619", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a3fdd2959a6b3a168b4c1d5e1bef5668fde197ae3e72bffd417165d445ce9f83", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "642ed6df-133f-4506-8b7b-c0126f68a2a1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a4f285b981b598c4c92116b76025d77b4ae6437eeaf9353bdd5e23ee6c9db198", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2ef088a1-a7a5-42be-a62b-e6352f9965bd", "node_type": "1", "metadata": {}, "hash": "d2c34d9f7fc4235098958035c9c54c0fd03d75043ca2f9e6a6c3eccbd3d34ad4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18161, "end_char_idx": 21625, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Usually, biases and layer norm parameters are not weight decayed. 
* Then do some argument housekeeping to use the same parameters as the previously used AdamW optimizer.

```py
import bitsandbytes as bnb
from torch import nn
from transformers.trainer_pt_utils import get_parameter_names

training_args = TrainingArguments(per_device_train_batch_size=4, **default_args)

decay_parameters = get_parameter_names(model, [nn.LayerNorm])
decay_parameters = [name for name in decay_parameters if "bias" not in name]
optimizer_grouped_parameters = [
    {
        "params": [p for n, p in model.named_parameters() if n in decay_parameters],
        "weight_decay": training_args.weight_decay,
    },
    {
        "params": [p for n, p in model.named_parameters() if n not in decay_parameters],
        "weight_decay": 0.0,
    },
]

optimizer_kwargs = {
    "betas": (training_args.adam_beta1, training_args.adam_beta2),
    "eps": training_args.adam_epsilon,
}
optimizer_kwargs["lr"] = training_args.learning_rate
adam_bnb_optim = bnb.optim.Adam8bit(
    optimizer_grouped_parameters,
    betas=(training_args.adam_beta1, training_args.adam_beta2),
    eps=training_args.adam_epsilon,
    lr=training_args.learning_rate,
)
```

Finally, pass the custom optimizer as an argument to the `Trainer`:

```py
trainer = Trainer(model=model, args=training_args, train_dataset=ds, optimizers=(adam_bnb_optim, None))
```

Combined with other approaches (gradient accumulation, gradient checkpointing, and mixed precision training), 
you can expect to get about a 3x memory improvement and even slightly higher throughput as using Adafactor. 

### multi_tensor

pytorch-nightly introduced `torch.optim._multi_tensor` which should significantly speed up the optimizers for situations 
with lots of small feature tensors. It should eventually become the default, but if you want to experiment with it sooner, take a look at this GitHub [issue](https://github.com/huggingface/transformers/issues/9965).

## Data preloading

One of the important requirements to reach great training speed is the ability to feed the GPU at the maximum speed it 
can handle. By default, everything happens in the main process, and it might not be able to read the data from disk fast 
enough, and thus create a bottleneck, leading to GPU under-utilization. Configure the following arguments to reduce the bottleneck:

- `DataLoader(pin_memory=True, ...)` - ensures the data gets preloaded into the pinned memory on CPU and typically leads to much faster transfers from CPU to GPU memory.
- `DataLoader(num_workers=4, ...)` - spawn several workers to preload data faster. During training, watch the GPU utilization stats; if it's far from 100%, experiment with increasing the number of workers. Of course, the problem could be elsewhere, so many workers won't necessarily lead to better performance.

When using [`Trainer`], the corresponding [`TrainingArguments`] are: `dataloader_pin_memory` (`True` by default), and `dataloader_num_workers` (defaults to `0`).

## DeepSpeed ZeRO

DeepSpeed is an open-source deep learning optimization library that is integrated with 🤗 Transformers and 🤗 Accelerate.
It provides a wide range of features and optimizations designed to improve the efficiency and scalability of large-scale 
deep learning training.

If your model fits onto a single GPU and you have enough space to fit a small batch size, you don't need to use DeepSpeed
as it'll only slow things down.


key: _node_content
string_value:
{"id_": "2ef088a1-a7a5-42be-a62b-e6352f9965bd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0cf5ca4d-95cf-4dd2-b5df-388357294619", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a3fdd2959a6b3a168b4c1d5e1bef5668fde197ae3e72bffd417165d445ce9f83", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d5cab6f3-a5e3-4065-9482-7199d2e8e85f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e5fa188a95d44c3b988e7b6212be582ce6ab385b32a289bc51956c595cafe32c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2e37a132-2b45-414c-a2cb-08b39b1c8891", "node_type": "1", "metadata": {}, "hash": "ea54760135da62d0f528915ec57b120139cd3e480751c7e3ad3f1c4d67ce63ee", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19789, "end_char_idx": 23266, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### multi_tensor

pytorch-nightly introduced `torch.optim._multi_tensor` which should significantly speed up the optimizers for situations 
with lots of small feature tensors. It should eventually become the default, but if you want to experiment with it sooner, take a look at this GitHub [issue](https://github.com/huggingface/transformers/issues/9965).

## Data preloading

One of the important requirements to reach great training speed is the ability to feed the GPU at the maximum speed it 
can handle. By default, everything happens in the main process, and it might not be able to read the data from disk fast 
enough, and thus create a bottleneck, leading to GPU under-utilization. Configure the following arguments to reduce the bottleneck:

- `DataLoader(pin_memory=True, ...)` - ensures the data gets preloaded into the pinned memory on CPU and typically leads to much faster transfers from CPU to GPU memory.
- `DataLoader(num_workers=4, ...)` - spawn several workers to preload data faster. During training, watch the GPU utilization stats; if it's far from 100%, experiment with increasing the number of workers. Of course, the problem could be elsewhere, so many workers won't necessarily lead to better performance.

When using [`Trainer`], the corresponding [`TrainingArguments`] are: `dataloader_pin_memory` (`True` by default), and `dataloader_num_workers` (defaults to `0`).

## DeepSpeed ZeRO

DeepSpeed is an open-source deep learning optimization library that is integrated with 🤗 Transformers and 🤗 Accelerate.
It provides a wide range of features and optimizations designed to improve the efficiency and scalability of large-scale 
deep learning training.

If your model fits onto a single GPU and you have enough space to fit a small batch size, you don't need to use DeepSpeed
as it'll only slow things down. However, if the model doesn't fit onto a single GPU or you can't fit a small batch, you can 
leverage DeepSpeed ZeRO + CPU Offload, or NVMe Offload for much larger models. In this case, you need to separately
[install the library](main_classes/deepspeed#installation), then follow one of the guides to create a configuration file 
and launch DeepSpeed: 
 
* For an in-depth guide on DeepSpeed integration with [`Trainer`], review [the corresponding documentation](main_classes/deepspeed), specifically the 
[section for a single GPU](main_classes/deepspeed#deployment-with-one-gpu). Some adjustments are required to use DeepSpeed in a notebook; please take a look at the [corresponding guide](main_classes/deepspeed#deployment-in-notebooks).
* If you prefer to use 🤗 Accelerate, refer to [🤗 Accelerate DeepSpeed guide](https://huggingface.co/docs/accelerate/en/usage_guides/deepspeed).

## Using torch.compile

PyTorch 2.0 introduced a new compile function that doesn't require any modification to existing PyTorch code but can 
optimize your code by adding a single line of code: `model = torch.compile(model)`.

If using [`Trainer`], you only need `to` pass the `torch_compile` option in the [`TrainingArguments`]: 

```python
training_args = TrainingArguments(torch_compile=True, **default_args)
```

`torch.compile` uses Python's frame evaluation API to automatically create a graph from existing PyTorch programs. After 
capturing the graph, different backends can be deployed to lower the graph to an optimized engine. 
You can find more details and benchmarks in [PyTorch documentation](https://pytorch.org/get-started/pytorch-2.0/).


key: _node_content
string_value:
{"id_": "2e37a132-2b45-414c-a2cb-08b39b1c8891", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0cf5ca4d-95cf-4dd2-b5df-388357294619", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a3fdd2959a6b3a168b4c1d5e1bef5668fde197ae3e72bffd417165d445ce9f83", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2ef088a1-a7a5-42be-a62b-e6352f9965bd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8ca48de467e235b66ab1b83e1053cc45d17468b6c81373aea9c574c9095cb56c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f265d4c2-9aa2-4379-81b4-5420cd92f098", "node_type": "1", "metadata": {}, "hash": "60e50f119917ab4c312c0b4a8936ac35b4b376f68ea1811fd22322a4238cfe77", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21626, "end_char_idx": 24671, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, if the model doesn't fit onto a single GPU or you can't fit a small batch, you can 
leverage DeepSpeed ZeRO + CPU Offload, or NVMe Offload for much larger models. In this case, you need to separately
[install the library](main_classes/deepspeed#installation), then follow one of the guides to create a configuration file 
and launch DeepSpeed: 
 
* For an in-depth guide on DeepSpeed integration with [`Trainer`], review [the corresponding documentation](main_classes/deepspeed), specifically the 
[section for a single GPU](main_classes/deepspeed#deployment-with-one-gpu). Some adjustments are required to use DeepSpeed in a notebook; please take a look at the [corresponding guide](main_classes/deepspeed#deployment-in-notebooks).
* If you prefer to use 🤗 Accelerate, refer to [🤗 Accelerate DeepSpeed guide](https://huggingface.co/docs/accelerate/en/usage_guides/deepspeed).

## Using torch.compile

PyTorch 2.0 introduced a new compile function that doesn't require any modification to existing PyTorch code but can 
optimize your code by adding a single line of code: `model = torch.compile(model)`.

If using [`Trainer`], you only need `to` pass the `torch_compile` option in the [`TrainingArguments`]: 

```python
training_args = TrainingArguments(torch_compile=True, **default_args)
```

`torch.compile` uses Python's frame evaluation API to automatically create a graph from existing PyTorch programs. After 
capturing the graph, different backends can be deployed to lower the graph to an optimized engine. 
You can find more details and benchmarks in [PyTorch documentation](https://pytorch.org/get-started/pytorch-2.0/).

`torch.compile` has a growing list of backends, which can be found in by calling `torchdynamo.list_backends()`, each of which with its optional dependencies.

Choose which backend to use by specifying it via `torch_compile_backend` in the [`TrainingArguments`].  Some of the most commonly used backends are:

**Debugging backends**:
* `dynamo.optimize("eager")` - Uses PyTorch to run the extracted GraphModule. This is quite useful in debugging TorchDynamo issues.
* `dynamo.optimize("aot_eager")` - Uses AotAutograd with no compiler, i.e, just using PyTorch eager for the AotAutograd's extracted forward and backward graphs. This is useful for debugging, and unlikely to give speedups.

**Training & inference backends**:
* `dynamo.optimize("inductor")` - Uses TorchInductor backend with AotAutograd and cudagraphs by leveraging codegened Triton kernels  [Read more](https://dev-discuss.pytorch.org/t/torchinductor-a-pytorch-native-compiler-with-define-by-run-ir-and-symbolic-shapes/747)
* `dynamo.optimize("nvfuser")` -  nvFuser with TorchScript. [Read more](https://dev-discuss.pytorch.org/t/tracing-with-primitives-update-1-nvfuser-and-its-primitives/593)
* `dynamo.optimize("aot_nvfuser")` -  nvFuser with AotAutograd. [Read more](https://dev-discuss.pytorch.org/t/tracing-with-primitives-update-1-nvfuser-and-its-primitives/593)
* `dynamo.optimize("aot_cudagraphs")` - cudagraphs with AotAutograd.


key: _node_content
string_value:
{"id_": "f265d4c2-9aa2-4379-81b4-5420cd92f098", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0cf5ca4d-95cf-4dd2-b5df-388357294619", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a3fdd2959a6b3a168b4c1d5e1bef5668fde197ae3e72bffd417165d445ce9f83", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2e37a132-2b45-414c-a2cb-08b39b1c8891", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e49fad3edfc2aa5cb46154e4a672cce08aba46e73f85876c4d3c52cb216f998b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9388bfe8-e9e2-4748-bec2-5ee724a8da60", "node_type": "1", "metadata": {}, "hash": "29337587d7fc986bff5cb8345aaca667658278785ffe84c1e075a328b736fd5d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23268, "end_char_idx": 26002, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
`torch.compile` has a growing list of backends, which can be found in by calling `torchdynamo.list_backends()`, each of which with its optional dependencies.

Choose which backend to use by specifying it via `torch_compile_backend` in the [`TrainingArguments`].  Some of the most commonly used backends are:

**Debugging backends**:
* `dynamo.optimize("eager")` - Uses PyTorch to run the extracted GraphModule. This is quite useful in debugging TorchDynamo issues.
* `dynamo.optimize("aot_eager")` - Uses AotAutograd with no compiler, i.e, just using PyTorch eager for the AotAutograd's extracted forward and backward graphs. This is useful for debugging, and unlikely to give speedups.

**Training & inference backends**:
* `dynamo.optimize("inductor")` - Uses TorchInductor backend with AotAutograd and cudagraphs by leveraging codegened Triton kernels  [Read more](https://dev-discuss.pytorch.org/t/torchinductor-a-pytorch-native-compiler-with-define-by-run-ir-and-symbolic-shapes/747)
* `dynamo.optimize("nvfuser")` -  nvFuser with TorchScript. [Read more](https://dev-discuss.pytorch.org/t/tracing-with-primitives-update-1-nvfuser-and-its-primitives/593)
* `dynamo.optimize("aot_nvfuser")` -  nvFuser with AotAutograd. [Read more](https://dev-discuss.pytorch.org/t/tracing-with-primitives-update-1-nvfuser-and-its-primitives/593)
* `dynamo.optimize("aot_cudagraphs")` - cudagraphs with AotAutograd. [Read more](https://github.com/pytorch/torchdynamo/pull/757)

**Inference-only backend**s:
* `dynamo.optimize("ofi")` -  Uses Torchscript optimize_for_inference.  [Read more](https://pytorch.org/docs/stable/generated/torch.jit.optimize_for_inference.html)
* `dynamo.optimize("fx2trt")` -  Uses NVIDIA TensorRT for inference optimizations.  [Read more](https://pytorch.org/TensorRT/tutorials/getting_started_with_fx_path.html)
* `dynamo.optimize("onnxrt")` -  Uses ONNXRT for inference on CPU/GPU.  [Read more](https://onnxruntime.ai/)
* `dynamo.optimize("ipex")` -  Uses IPEX for inference on CPU.  [Read more](https://github.com/intel/intel-extension-for-pytorch)

For an example of using `torch.compile` with 🤗 Transformers, check out this [blog post on fine-tuning a BERT model for Text Classification using the newest PyTorch 2.0 features](https://www.philschmid.de/getting-started-pytorch-2-0-transformers)

## Using 🤗 PEFT

[Parameter-Efficient Fine Tuning (PEFT)](https://huggingface.co/blog/peft) methods freeze the pretrained model parameters during fine-tuning and add a small number of trainable parameters (the adapters) on top of it.

As a result the [memory associated to the optimizer states and gradients](https://huggingface.co/docs/transformers/model_memory_anatomy#anatomy-of-models-memory) are greatly reduced.


key: _node_content
string_value:
{"id_": "9388bfe8-e9e2-4748-bec2-5ee724a8da60", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0cf5ca4d-95cf-4dd2-b5df-388357294619", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a3fdd2959a6b3a168b4c1d5e1bef5668fde197ae3e72bffd417165d445ce9f83", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f265d4c2-9aa2-4379-81b4-5420cd92f098", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "429d267c07d68adf7ef348c229f32d912b563478d072df713092aad87072a898", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "036262ac-653f-4d88-906e-5f8063fcd4ec", "node_type": "1", "metadata": {}, "hash": "a870ec7023092bed4b349be6dea9e3eaa33de491146a52856b3e2a69fe6efb3f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24672, "end_char_idx": 27057, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Read more](https://github.com/pytorch/torchdynamo/pull/757)

**Inference-only backend**s:
* `dynamo.optimize("ofi")` -  Uses Torchscript optimize_for_inference.  [Read more](https://pytorch.org/docs/stable/generated/torch.jit.optimize_for_inference.html)
* `dynamo.optimize("fx2trt")` -  Uses NVIDIA TensorRT for inference optimizations.  [Read more](https://pytorch.org/TensorRT/tutorials/getting_started_with_fx_path.html)
* `dynamo.optimize("onnxrt")` -  Uses ONNXRT for inference on CPU/GPU.  [Read more](https://onnxruntime.ai/)
* `dynamo.optimize("ipex")` -  Uses IPEX for inference on CPU.  [Read more](https://github.com/intel/intel-extension-for-pytorch)

For an example of using `torch.compile` with 🤗 Transformers, check out this [blog post on fine-tuning a BERT model for Text Classification using the newest PyTorch 2.0 features](https://www.philschmid.de/getting-started-pytorch-2-0-transformers)

## Using 🤗 PEFT

[Parameter-Efficient Fine Tuning (PEFT)](https://huggingface.co/blog/peft) methods freeze the pretrained model parameters during fine-tuning and add a small number of trainable parameters (the adapters) on top of it.

As a result the [memory associated to the optimizer states and gradients](https://huggingface.co/docs/transformers/model_memory_anatomy#anatomy-of-models-memory) are greatly reduced.

For example with a vanilla AdamW, the memory requirement for the optimizer state would be:
* fp32 copy of parameters: 4 bytes/param
* Momentum: 4 bytes/param
* Variance: 4 bytes/param

Suppose a model with 7B parameters and 200 millions parameters injected with [Low Rank Adapters](https://huggingface.co/docs/peft/conceptual_guides/lora).

The memory requirement for the optimizer state of the plain model would be 12 * 7 = 84 GB (assuming 7B trainable parameters).

Adding Lora increases slightly the memory associated to the model weights and substantially decreases memory requirement for the optimizer state to 12 * 0.2 = 2.4GB.

Read more about PEFT and its detailed usage in [the PEFT documentation](https://huggingface.co/docs/peft/) or [PEFT repository](https://github.com/huggingface/peft).

## Using 🤗 Accelerate

With [🤗 Accelerate](https://huggingface.co/docs/accelerate/index) you can use the above methods while gaining full 
control over the training loop and can essentially write the loop in pure PyTorch with some minor modifications.


key: _node_content
string_value:
{"id_": "036262ac-653f-4d88-906e-5f8063fcd4ec", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0cf5ca4d-95cf-4dd2-b5df-388357294619", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a3fdd2959a6b3a168b4c1d5e1bef5668fde197ae3e72bffd417165d445ce9f83", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9388bfe8-e9e2-4748-bec2-5ee724a8da60", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d601e91e75893a5b7dc739ce99440d26f971d1d6236509a18c14cfe61ae240dc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a4f393fe-e083-4b0c-8102-d1fff564c1b6", "node_type": "1", "metadata": {}, "hash": "ee7f26c7057418433d67e084828b7100097c0bf844c6c1e2ab0e11da48999ed7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 25820, "end_char_idx": 29183, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
As a result the [memory associated to the optimizer states and gradients](https://huggingface.co/docs/transformers/model_memory_anatomy#anatomy-of-models-memory) are greatly reduced.

For example with a vanilla AdamW, the memory requirement for the optimizer state would be:
* fp32 copy of parameters: 4 bytes/param
* Momentum: 4 bytes/param
* Variance: 4 bytes/param

Suppose a model with 7B parameters and 200 millions parameters injected with [Low Rank Adapters](https://huggingface.co/docs/peft/conceptual_guides/lora).

The memory requirement for the optimizer state of the plain model would be 12 * 7 = 84 GB (assuming 7B trainable parameters).

Adding Lora increases slightly the memory associated to the model weights and substantially decreases memory requirement for the optimizer state to 12 * 0.2 = 2.4GB.

Read more about PEFT and its detailed usage in [the PEFT documentation](https://huggingface.co/docs/peft/) or [PEFT repository](https://github.com/huggingface/peft).

## Using 🤗 Accelerate

With [🤗 Accelerate](https://huggingface.co/docs/accelerate/index) you can use the above methods while gaining full 
control over the training loop and can essentially write the loop in pure PyTorch with some minor modifications. 

Suppose you have combined the methods in the [`TrainingArguments`] like so:

```py
training_args = TrainingArguments(
    per_device_train_batch_size=1,
    gradient_accumulation_steps=4,
    gradient_checkpointing=True,
    fp16=True,
    **default_args,
)
```

The full example training loop with 🤗 Accelerate is only a handful of lines of code long:

```py
from accelerate import Accelerator
from torch.utils.data.dataloader import DataLoader

dataloader = DataLoader(ds, batch_size=training_args.per_device_train_batch_size)

if training_args.gradient_checkpointing:
    model.gradient_checkpointing_enable()

accelerator = Accelerator(fp16=training_args.fp16)
model, optimizer, dataloader = accelerator.prepare(model, adam_bnb_optim, dataloader)

model.train()
for step, batch in enumerate(dataloader, start=1):
    loss = model(**batch).loss
    loss = loss / training_args.gradient_accumulation_steps
    accelerator.backward(loss)
    if step % training_args.gradient_accumulation_steps == 0:
        optimizer.step()
        optimizer.zero_grad()
```

First we wrap the dataset in a [`DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader). 
Then we can enable gradient checkpointing by calling the model's [`~PreTrainedModel.gradient_checkpointing_enable`] method. 
When we initialize the [`Accelerator`](https://huggingface.co/docs/accelerate/package_reference/accelerator#accelerate.Accelerator) 
we can specify if we want to use mixed precision training and it will take care of it for us in the [`prepare`] call. 
During the [`prepare`](https://huggingface.co/docs/accelerate/package_reference/accelerator#accelerate.Accelerator.prepare) 
call the dataloader will also be distributed across workers should we use multiple GPUs. We use the same [8-bit optimizer](#8-bit-adam) from the earlier example.

Finally, we can add the main training loop. Note that the `backward` call is handled by 🤗 Accelerate. We can also see
how gradient accumulation works: we normalize the loss, so we get the average at the end of accumulation and once we have 
enough steps we run the optimization.


key: _node_content
string_value:
{"id_": "a4f393fe-e083-4b0c-8102-d1fff564c1b6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0cf5ca4d-95cf-4dd2-b5df-388357294619", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a3fdd2959a6b3a168b4c1d5e1bef5668fde197ae3e72bffd417165d445ce9f83", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "036262ac-653f-4d88-906e-5f8063fcd4ec", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5d81e53225451016da059d15da1c29d4bd06d16b9d2fa6c37f96bf5e77c6bf0f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c9f7bfde-bc73-4b0e-abdd-caa2521c95f1", "node_type": "1", "metadata": {}, "hash": "6dd543a9bd13a10453af50206de91af489c86ac8bbb09063ef7506921b74210d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 28240, "end_char_idx": 31559, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Then we can enable gradient checkpointing by calling the model's [`~PreTrainedModel.gradient_checkpointing_enable`] method. 
When we initialize the [`Accelerator`](https://huggingface.co/docs/accelerate/package_reference/accelerator#accelerate.Accelerator) 
we can specify if we want to use mixed precision training and it will take care of it for us in the [`prepare`] call. 
During the [`prepare`](https://huggingface.co/docs/accelerate/package_reference/accelerator#accelerate.Accelerator.prepare) 
call the dataloader will also be distributed across workers should we use multiple GPUs. We use the same [8-bit optimizer](#8-bit-adam) from the earlier example.

Finally, we can add the main training loop. Note that the `backward` call is handled by 🤗 Accelerate. We can also see
how gradient accumulation works: we normalize the loss, so we get the average at the end of accumulation and once we have 
enough steps we run the optimization. 

Implementing these optimization techniques with 🤗 Accelerate only takes a handful of lines of code and comes with the 
benefit of more flexibility in the training loop. For a full documentation of all features have a look at the 
[Accelerate documentation](https://huggingface.co/docs/accelerate/index).


## Efficient Software Prebuilds

PyTorch's [pip and conda builds](https://pytorch.org/get-started/locally/#start-locally) come prebuilt with the cuda toolkit 
which is enough to run PyTorch, but it is insufficient if you need to build cuda extensions.

At times, additional efforts may be required to pre-build some components. For instance, if you're using libraries like `apex` that 
don't come pre-compiled. In other situations figuring out how to install the right cuda toolkit system-wide can be complicated. 
To address these scenarios PyTorch and NVIDIA released a new version of NGC docker container which already comes with 
everything prebuilt. You just need to install your programs on it, and it will run out of the box.

This approach is also useful if you want to tweak the pytorch source and/or make a new customized build.
To find the docker image version you want start [with PyTorch release notes](https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/), 
choose one of the latest monthly releases. Go into the release's notes for the desired release, check that the environment's 
components are matching your needs (including NVIDIA Driver requirements!) and then at the very top of that document go 
to the corresponding NGC page. If for some reason you get lost, here is [the index of all PyTorch NGC images](https://ngc.nvidia.com/catalog/containers/nvidia:pytorch).

Next follow the instructions to download and deploy the docker image.

## Mixture of Experts

Some recent papers reported a 4-5x training speedup and a faster inference by integrating
Mixture of Experts (MoE) into the Transformer models.

Since it has been discovered that more parameters lead to better performance, this technique allows to increase the 
number of parameters by an order of magnitude without increasing training costs.

In this approach every other FFN layer is replaced with a MoE Layer which consists of many experts, with a gated function 
that trains each expert in a balanced way depending on the input token's position in a sequence.

!


key: _node_content
string_value:
{"id_": "c9f7bfde-bc73-4b0e-abdd-caa2521c95f1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0cf5ca4d-95cf-4dd2-b5df-388357294619", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a3fdd2959a6b3a168b4c1d5e1bef5668fde197ae3e72bffd417165d445ce9f83", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a4f393fe-e083-4b0c-8102-d1fff564c1b6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "1aee431e08ed5b12ff3aa8f78f649bbab159e4d6e00c38c67edaf380f37fa19d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a5c7d1cb-5e47-4b7d-a269-a738b0250ced", "node_type": "1", "metadata": {}, "hash": "fad7fc434cf82aed64403587dcfba7a85f85afd4baaf54269dc232e446980511", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 29745, "end_char_idx": 32427, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
At times, additional efforts may be required to pre-build some components. For instance, if you're using libraries like `apex` that 
don't come pre-compiled. In other situations figuring out how to install the right cuda toolkit system-wide can be complicated. 
To address these scenarios PyTorch and NVIDIA released a new version of NGC docker container which already comes with 
everything prebuilt. You just need to install your programs on it, and it will run out of the box.

This approach is also useful if you want to tweak the pytorch source and/or make a new customized build.
To find the docker image version you want start [with PyTorch release notes](https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/), 
choose one of the latest monthly releases. Go into the release's notes for the desired release, check that the environment's 
components are matching your needs (including NVIDIA Driver requirements!) and then at the very top of that document go 
to the corresponding NGC page. If for some reason you get lost, here is [the index of all PyTorch NGC images](https://ngc.nvidia.com/catalog/containers/nvidia:pytorch).

Next follow the instructions to download and deploy the docker image.

## Mixture of Experts

Some recent papers reported a 4-5x training speedup and a faster inference by integrating
Mixture of Experts (MoE) into the Transformer models.

Since it has been discovered that more parameters lead to better performance, this technique allows to increase the 
number of parameters by an order of magnitude without increasing training costs.

In this approach every other FFN layer is replaced with a MoE Layer which consists of many experts, with a gated function 
that trains each expert in a balanced way depending on the input token's position in a sequence.

![MoE Transformer 2x block](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/perf-moe-transformer.png)

(source: [GLAM](https://ai.googleblog.com/2021/12/more-efficient-in-context-learning-with.html))

You can find exhaustive details and comparison tables in the papers listed at the end of this section.

The main drawback of this approach is that it requires staggering amounts of GPU memory - almost an order of magnitude 
larger than its dense equivalent. Various distillation and approaches are proposed to how to overcome the much higher memory requirements.

There is direct trade-off though, you can use just a few experts with a 2-3x smaller base model instead of dozens or 
hundreds experts leading to a 5x smaller model and thus increase the training speed moderately while increasing the 
memory requirements moderately as well.


key: _node_content
string_value:
{"id_": "a5c7d1cb-5e47-4b7d-a269-a738b0250ced", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0cf5ca4d-95cf-4dd2-b5df-388357294619", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a3fdd2959a6b3a168b4c1d5e1bef5668fde197ae3e72bffd417165d445ce9f83", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c9f7bfde-bc73-4b0e-abdd-caa2521c95f1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c918706183a095256b6795baaab5824dfcef92e33309c35ae17a8f4cdec70120", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "18662d6d-6f41-4a8a-af16-b41bc84784a7", "node_type": "1", "metadata": {}, "hash": "d1685d545f4b828a3ebabcad8413e0c9f89e6f169ba9d67a2e692d9127d8ef81", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30525, "end_char_idx": 33971, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Go into the release's notes for the desired release, check that the environment's 
components are matching your needs (including NVIDIA Driver requirements!) and then at the very top of that document go 
to the corresponding NGC page. If for some reason you get lost, here is [the index of all PyTorch NGC images](https://ngc.nvidia.com/catalog/containers/nvidia:pytorch).

Next follow the instructions to download and deploy the docker image.

## Mixture of Experts

Some recent papers reported a 4-5x training speedup and a faster inference by integrating
Mixture of Experts (MoE) into the Transformer models.

Since it has been discovered that more parameters lead to better performance, this technique allows to increase the 
number of parameters by an order of magnitude without increasing training costs.

In this approach every other FFN layer is replaced with a MoE Layer which consists of many experts, with a gated function 
that trains each expert in a balanced way depending on the input token's position in a sequence.

![MoE Transformer 2x block](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/perf-moe-transformer.png)

(source: [GLAM](https://ai.googleblog.com/2021/12/more-efficient-in-context-learning-with.html))

You can find exhaustive details and comparison tables in the papers listed at the end of this section.

The main drawback of this approach is that it requires staggering amounts of GPU memory - almost an order of magnitude 
larger than its dense equivalent. Various distillation and approaches are proposed to how to overcome the much higher memory requirements.

There is direct trade-off though, you can use just a few experts with a 2-3x smaller base model instead of dozens or 
hundreds experts leading to a 5x smaller model and thus increase the training speed moderately while increasing the 
memory requirements moderately as well.

Most related papers and implementations are built around Tensorflow/TPUs:

- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://arxiv.org/abs/2006.16668)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://arxiv.org/abs/2101.03961)
- [GLaM: Generalist Language Model (GLaM)](https://ai.googleblog.com/2021/12/more-efficient-in-context-learning-with.html)

And for Pytorch DeepSpeed has built one as well: [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://arxiv.org/abs/2201.05596), [Mixture of Experts](https://www.deepspeed.ai/tutorials/mixture-of-experts/) - blog posts:  [1](https://www.microsoft.com/en-us/research/blog/deepspeed-powers-8x-larger-moe-model-training-with-high-performance/), [2](https://www.microsoft.com/en-us/research/publication/scalable-and-efficient-moe-training-for-multitask-multilingual-models/) and specific deployment with large transformer-based natural language generation models: [blog post](https://www.deepspeed.ai/2021/12/09/deepspeed-moe-nlg.html), [Megatron-Deepspeed branch](https://github.com/microsoft/Megatron-DeepSpeed/tree/moe-training).

## Using PyTorch native attention and Flash Attention

PyTorch's [`torch.nn.functional.scaled_dot_product_attention`](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html) (SDPA) can also call FlashAttention and memory-efficient attention kernels under the hood.


key: _node_content
string_value:
{"id_": "18662d6d-6f41-4a8a-af16-b41bc84784a7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0cf5ca4d-95cf-4dd2-b5df-388357294619", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a3fdd2959a6b3a168b4c1d5e1bef5668fde197ae3e72bffd417165d445ce9f83", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a5c7d1cb-5e47-4b7d-a269-a738b0250ced", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one/", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c564ad671a9cf421bc6c17aed67ae506b4391e196dce91424483d3b93d14f5b0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 32429, "end_char_idx": 34460, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Most related papers and implementations are built around Tensorflow/TPUs:

- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://arxiv.org/abs/2006.16668)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://arxiv.org/abs/2101.03961)
- [GLaM: Generalist Language Model (GLaM)](https://ai.googleblog.com/2021/12/more-efficient-in-context-learning-with.html)

And for Pytorch DeepSpeed has built one as well: [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://arxiv.org/abs/2201.05596), [Mixture of Experts](https://www.deepspeed.ai/tutorials/mixture-of-experts/) - blog posts:  [1](https://www.microsoft.com/en-us/research/blog/deepspeed-powers-8x-larger-moe-model-training-with-high-performance/), [2](https://www.microsoft.com/en-us/research/publication/scalable-and-efficient-moe-training-for-multitask-multilingual-models/) and specific deployment with large transformer-based natural language generation models: [blog post](https://www.deepspeed.ai/2021/12/09/deepspeed-moe-nlg.html), [Megatron-Deepspeed branch](https://github.com/microsoft/Megatron-DeepSpeed/tree/moe-training).

## Using PyTorch native attention and Flash Attention

PyTorch's [`torch.nn.functional.scaled_dot_product_attention`](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html) (SDPA) can also call FlashAttention and memory-efficient attention kernels under the hood. SDPA support is currently being added natively in Transformers and is used by default for `torch>=2.1.1` when an implementation is available. Please refer to [PyTorch scaled dot product attention](https://huggingface.co/docs/transformers/perf_infer_gpu_one#pytorch-scaled-dot-product-attention) for a list of supported models and more details.

Check out this [blogpost](https://pytorch.org/blog/out-of-the-box-acceleration/) to learn more about acceleration and memory-savings with SDPA.


key: _node_content
string_value:
{"id_": "9118258a-08f7-4ebd-beba-7dcf31c75052", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/installation/", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "faff96d7-bc9e-4d68-aaba-b19506f35b12", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/installation/", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "64c602ceb99afa1c55aa97651856bb2fd73523e224c98f6e51268159f76f2def", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9d4ed9ac-1370-4309-93b8-b220f1632a6b", "node_type": "1", "metadata": {}, "hash": "eaf39928c104f616403f50a4ec03f81bd2d1ff8b54de892267a1798b2f08099e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3287, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<!---
Copyright 2022 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.

-->

# Installation

Install 🤗 Transformers for whichever deep learning library you're working with, setup your cache, and optionally configure 🤗 Transformers to run offline.

🤗 Transformers is tested on Python 3.6+, PyTorch 1.1.0+, TensorFlow 2.0+, and Flax. Follow the installation instructions below for the deep learning library you are using:

* [PyTorch](https://pytorch.org/get-started/locally/) installation instructions.
* [TensorFlow 2.0](https://www.tensorflow.org/install/pip) installation instructions.
* [Flax](https://flax.readthedocs.io/en/latest/) installation instructions.

## Install with pip

You should install 🤗 Transformers in a [virtual environment](https://docs.python.org/3/library/venv.html). If you're unfamiliar with Python virtual environments, take a look at this [guide](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/). A virtual environment makes it easier to manage different projects, and avoid compatibility issues between dependencies.

Start by creating a virtual environment in your project directory:

```bash
python -m venv .env
```

Activate the virtual environment. On Linux and MacOs:

```bash
source .env/bin/activate
```
Activate Virtual environment on Windows

```bash
.env/Scripts/activate
```

Now you're ready to install 🤗 Transformers with the following command:

```bash
pip install transformers
```

For CPU-support only, you can conveniently install 🤗 Transformers and a deep learning library in one line. For example, install 🤗 Transformers and PyTorch with:

```bash
pip install 'transformers[torch]'
```

🤗 Transformers and TensorFlow 2.0:

```bash
pip install 'transformers[tf-cpu]'
```

<Tip warning={true}>

M1 / ARM Users

You will need to install the following before installing TensorFLow 2.0
```bash
brew install cmake
brew install pkg-config
```

</Tip>

🤗 Transformers and Flax:

```bash
pip install 'transformers[flax]'
```

Finally, check if 🤗 Transformers has been properly installed by running the following command. It will download a pretrained model:

```bash
python -c "from transformers import pipeline; print(pipeline('sentiment-analysis')('we love you'))"
```

Then print out the label and score:

```bash
[{'label': 'POSITIVE', 'score': 0.9998704791069031}]
```

## Install from source

Install 🤗 Transformers from source with the following command:

```bash
pip install git+https://github.com/huggingface/transformers
```

This command installs the bleeding edge `main` version rather than the latest `stable` version.


key: _node_content
string_value:
{"id_": "9d4ed9ac-1370-4309-93b8-b220f1632a6b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/installation/", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "faff96d7-bc9e-4d68-aaba-b19506f35b12", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/installation/", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "64c602ceb99afa1c55aa97651856bb2fd73523e224c98f6e51268159f76f2def", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9118258a-08f7-4ebd-beba-7dcf31c75052", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/installation/", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e849795d0367edb37d91447b6f872878df70498ed4f12d73ce1e3cfe8c5b03ae", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cbcfdcb7-c773-439b-8545-6ee46288f9bb", "node_type": "1", "metadata": {}, "hash": "b5d0d21984db14aa1802940e6d744bd6ef0f340de8005559300114e9e2b4ccb3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1642, "end_char_idx": 4848, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
A virtual environment makes it easier to manage different projects, and avoid compatibility issues between dependencies.

Start by creating a virtual environment in your project directory:

```bash
python -m venv .env
```

Activate the virtual environment. On Linux and MacOs:

```bash
source .env/bin/activate
```
Activate Virtual environment on Windows

```bash
.env/Scripts/activate
```

Now you're ready to install 🤗 Transformers with the following command:

```bash
pip install transformers
```

For CPU-support only, you can conveniently install 🤗 Transformers and a deep learning library in one line. For example, install 🤗 Transformers and PyTorch with:

```bash
pip install 'transformers[torch]'
```

🤗 Transformers and TensorFlow 2.0:

```bash
pip install 'transformers[tf-cpu]'
```

<Tip warning={true}>

M1 / ARM Users

You will need to install the following before installing TensorFLow 2.0
```bash
brew install cmake
brew install pkg-config
```

</Tip>

🤗 Transformers and Flax:

```bash
pip install 'transformers[flax]'
```

Finally, check if 🤗 Transformers has been properly installed by running the following command. It will download a pretrained model:

```bash
python -c "from transformers import pipeline; print(pipeline('sentiment-analysis')('we love you'))"
```

Then print out the label and score:

```bash
[{'label': 'POSITIVE', 'score': 0.9998704791069031}]
```

## Install from source

Install 🤗 Transformers from source with the following command:

```bash
pip install git+https://github.com/huggingface/transformers
```

This command installs the bleeding edge `main` version rather than the latest `stable` version. The `main` version is useful for staying up-to-date with the latest developments. For instance, if a bug has been fixed since the last official release but a new release hasn't been rolled out yet. However, this means the `main` version may not always be stable. We strive to keep the `main` version operational, and most issues are usually resolved within a few hours or a day. If you run into a problem, please open an [Issue](https://github.com/huggingface/transformers/issues) so we can fix it even sooner!

Check if 🤗 Transformers has been properly installed by running the following command:

```bash
python -c "from transformers import pipeline; print(pipeline('sentiment-analysis')('I love you'))"
```

## Editable install

You will need an editable install if you'd like to:

* Use the `main` version of the source code.
* Contribute to 🤗 Transformers and need to test changes in the code.

Clone the repository and install 🤗 Transformers with the following commands:

```bash
git clone https://github.com/huggingface/transformers.git
cd transformers
pip install -e .
```

These commands will link the folder you cloned the repository to and your Python library paths. Python will now look inside the folder you cloned to in addition to the normal library paths. For example, if your Python packages are typically installed in `~/anaconda3/envs/main/lib/python3.7/site-packages/`, Python will also search the folder you cloned to: `~/transformers/`.

<Tip warning={true}>

You must keep the `transformers` folder if you want to keep using the library.


key: _node_content
string_value:
{"id_": "cbcfdcb7-c773-439b-8545-6ee46288f9bb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/installation/", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "faff96d7-bc9e-4d68-aaba-b19506f35b12", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/installation/", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "64c602ceb99afa1c55aa97651856bb2fd73523e224c98f6e51268159f76f2def", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9d4ed9ac-1370-4309-93b8-b220f1632a6b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/installation/", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "f39bcb726e26e3f20998f9f0e582dcb9d3446ceddf72c105fb1d74617c91e82b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ea2a2d01-279a-4b73-9c65-08deb494ef36", "node_type": "1", "metadata": {}, "hash": "cbb13bdfa3d6f1604e84f17fb180aa9742a49094c04025c6eae711f93f8d4164", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3288, "end_char_idx": 6442, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The `main` version is useful for staying up-to-date with the latest developments. For instance, if a bug has been fixed since the last official release but a new release hasn't been rolled out yet. However, this means the `main` version may not always be stable. We strive to keep the `main` version operational, and most issues are usually resolved within a few hours or a day. If you run into a problem, please open an [Issue](https://github.com/huggingface/transformers/issues) so we can fix it even sooner!

Check if 🤗 Transformers has been properly installed by running the following command:

```bash
python -c "from transformers import pipeline; print(pipeline('sentiment-analysis')('I love you'))"
```

## Editable install

You will need an editable install if you'd like to:

* Use the `main` version of the source code.
* Contribute to 🤗 Transformers and need to test changes in the code.

Clone the repository and install 🤗 Transformers with the following commands:

```bash
git clone https://github.com/huggingface/transformers.git
cd transformers
pip install -e .
```

These commands will link the folder you cloned the repository to and your Python library paths. Python will now look inside the folder you cloned to in addition to the normal library paths. For example, if your Python packages are typically installed in `~/anaconda3/envs/main/lib/python3.7/site-packages/`, Python will also search the folder you cloned to: `~/transformers/`.

<Tip warning={true}>

You must keep the `transformers` folder if you want to keep using the library.

</Tip>

Now you can easily update your clone to the latest version of 🤗 Transformers with the following command:

```bash
cd ~/transformers/
git pull
```

Your Python environment will find the `main` version of 🤗 Transformers on the next run.

## Install with conda

Install from the conda channel `conda-forge`:

```bash
conda install conda-forge::transformers
```

## Cache setup

Pretrained models are downloaded and locally cached at: `~/.cache/huggingface/hub`. This is the default directory given by the shell environment variable `TRANSFORMERS_CACHE`. On Windows, the default directory is given by `C:\Users\username\.cache\huggingface\hub`. You can change the shell environment variables shown below - in order of priority - to specify a different cache directory:

1. Shell environment variable (default): `HUGGINGFACE_HUB_CACHE` or `TRANSFORMERS_CACHE`.
2. Shell environment variable: `HF_HOME`.
3. Shell environment variable: `XDG_CACHE_HOME` + `/huggingface`.

<Tip>

🤗 Transformers will use the shell environment variables `PYTORCH_TRANSFORMERS_CACHE` or `PYTORCH_PRETRAINED_BERT_CACHE` if you are coming from an earlier iteration of this library and have set those environment variables, unless you specify the shell environment variable `TRANSFORMERS_CACHE`.

</Tip>

## Offline mode

Run 🤗 Transformers in a firewalled or offline environment with locally cached files by setting the environment variable `HF_HUB_OFFLINE=1`.

<Tip>

Add [🤗 Datasets](https://huggingface.co/docs/datasets/) to your offline training workflow with the environment variable `HF_DATASETS_OFFLINE=1`.


key: _node_content
string_value:
{"id_": "ea2a2d01-279a-4b73-9c65-08deb494ef36", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/installation/", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "faff96d7-bc9e-4d68-aaba-b19506f35b12", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/installation/", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "64c602ceb99afa1c55aa97651856bb2fd73523e224c98f6e51268159f76f2def", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cbcfdcb7-c773-439b-8545-6ee46288f9bb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/installation/", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "9dd3d9c10cc080a72a6267e2214d413435a8596353877dc9fe7745a9f0cf1bee", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d5af37d7-cae9-4fac-a1cf-d6e900e0615b", "node_type": "1", "metadata": {}, "hash": "3a564744caca7f9bdad5df12c335ba36f51fa3013d4c26b05c2167990d7e2dc5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4850, "end_char_idx": 7996, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

Now you can easily update your clone to the latest version of 🤗 Transformers with the following command:

```bash
cd ~/transformers/
git pull
```

Your Python environment will find the `main` version of 🤗 Transformers on the next run.

## Install with conda

Install from the conda channel `conda-forge`:

```bash
conda install conda-forge::transformers
```

## Cache setup

Pretrained models are downloaded and locally cached at: `~/.cache/huggingface/hub`. This is the default directory given by the shell environment variable `TRANSFORMERS_CACHE`. On Windows, the default directory is given by `C:\Users\username\.cache\huggingface\hub`. You can change the shell environment variables shown below - in order of priority - to specify a different cache directory:

1. Shell environment variable (default): `HUGGINGFACE_HUB_CACHE` or `TRANSFORMERS_CACHE`.
2. Shell environment variable: `HF_HOME`.
3. Shell environment variable: `XDG_CACHE_HOME` + `/huggingface`.

<Tip>

🤗 Transformers will use the shell environment variables `PYTORCH_TRANSFORMERS_CACHE` or `PYTORCH_PRETRAINED_BERT_CACHE` if you are coming from an earlier iteration of this library and have set those environment variables, unless you specify the shell environment variable `TRANSFORMERS_CACHE`.

</Tip>

## Offline mode

Run 🤗 Transformers in a firewalled or offline environment with locally cached files by setting the environment variable `HF_HUB_OFFLINE=1`.

<Tip>

Add [🤗 Datasets](https://huggingface.co/docs/datasets/) to your offline training workflow with the environment variable `HF_DATASETS_OFFLINE=1`.

</Tip>

```bash
HF_DATASETS_OFFLINE=1 HF_HUB_OFFLINE=1 \
python examples/pytorch/translation/run_translation.py --model_name_or_path google-t5/t5-small --dataset_name wmt16 --dataset_config ro-en ...
```

This script should run without hanging or waiting to timeout because it won't attempt to download the model from the Hub.

You can also bypass loading a model from the Hub from each [`~PreTrainedModel.from_pretrained`] call with the [`local_files_only`] parameter. When set to `True`, only local files are loaded:

```py
from transformers import T5Model

model = T5Model.from_pretrained("./path/to/local/directory", local_files_only=True)
```

### Fetch models and tokenizers to use offline

Another option for using 🤗 Transformers offline is to download the files ahead of time, and then point to their local path when you need to use them offline. There are three ways to do this:

* Download a file through the user interface on the [Model Hub](https://huggingface.co/models) by clicking on the ↓ icon.

    ![download-icon](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/download-icon.png)

* Use the [`PreTrainedModel.from_pretrained`] and [`PreTrainedModel.save_pretrained`] workflow:

    1. Download your files ahead of time with [`PreTrainedModel.from_pretrained`]:

    ```py
    >>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

    >>> tokenizer = AutoTokenizer.from_pretrained("bigscience/T0_3B")
    >>> model = AutoModelForSeq2SeqLM.from_pretrained("bigscience/T0_3B")
    ```

    2.


key: _node_content
string_value:
{"id_": "d5af37d7-cae9-4fac-a1cf-d6e900e0615b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/installation/", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "faff96d7-bc9e-4d68-aaba-b19506f35b12", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/installation/", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "64c602ceb99afa1c55aa97651856bb2fd73523e224c98f6e51268159f76f2def", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ea2a2d01-279a-4b73-9c65-08deb494ef36", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/installation/", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c607cbaac757d145a4302d6ee77897668a76dce74080f778efae41bc6ebcb3da", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8b532dd2-a6f2-44ff-978a-2f4450b40857", "node_type": "1", "metadata": {}, "hash": "6777ec7304a04bb36cc1a727105977ae48e32e9dc0a90fa456c3835e1f28a445", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6444, "end_char_idx": 8960, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

```bash
HF_DATASETS_OFFLINE=1 HF_HUB_OFFLINE=1 \
python examples/pytorch/translation/run_translation.py --model_name_or_path google-t5/t5-small --dataset_name wmt16 --dataset_config ro-en ...
```

This script should run without hanging or waiting to timeout because it won't attempt to download the model from the Hub.

You can also bypass loading a model from the Hub from each [`~PreTrainedModel.from_pretrained`] call with the [`local_files_only`] parameter. When set to `True`, only local files are loaded:

```py
from transformers import T5Model

model = T5Model.from_pretrained("./path/to/local/directory", local_files_only=True)
```

### Fetch models and tokenizers to use offline

Another option for using 🤗 Transformers offline is to download the files ahead of time, and then point to their local path when you need to use them offline. There are three ways to do this:

* Download a file through the user interface on the [Model Hub](https://huggingface.co/models) by clicking on the ↓ icon.

    ![download-icon](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/download-icon.png)

* Use the [`PreTrainedModel.from_pretrained`] and [`PreTrainedModel.save_pretrained`] workflow:

    1. Download your files ahead of time with [`PreTrainedModel.from_pretrained`]:

    ```py
    >>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

    >>> tokenizer = AutoTokenizer.from_pretrained("bigscience/T0_3B")
    >>> model = AutoModelForSeq2SeqLM.from_pretrained("bigscience/T0_3B")
    ```

    2. Save your files to a specified directory with [`PreTrainedModel.save_pretrained`]:

    ```py
    >>> tokenizer.save_pretrained("./your/path/bigscience_t0")
    >>> model.save_pretrained("./your/path/bigscience_t0")
    ```

    3. Now when you're offline, reload your files with [`PreTrainedModel.from_pretrained`] from the specified directory:

    ```py
    >>> tokenizer = AutoTokenizer.from_pretrained("./your/path/bigscience_t0")
    >>> model = AutoModel.from_pretrained("./your/path/bigscience_t0")
    ```

* Programmatically download files with the [huggingface_hub](https://github.com/huggingface/huggingface_hub/tree/main/src/huggingface_hub) library:

    1. Install the `huggingface_hub` library in your virtual environment:

    ```bash
    python -m pip install huggingface_hub
    ```

    2. Use the [`hf_hub_download`](https://huggingface.co/docs/hub/adding-a-library#download-files-from-the-hub) function to download a file to a specific path.


key: _node_content
string_value:
{"id_": "8b532dd2-a6f2-44ff-978a-2f4450b40857", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/installation/", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "faff96d7-bc9e-4d68-aaba-b19506f35b12", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/installation/", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "64c602ceb99afa1c55aa97651856bb2fd73523e224c98f6e51268159f76f2def", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d5af37d7-cae9-4fac-a1cf-d6e900e0615b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/installation/", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "9f5d9e65faef1032ba95580da39221052a98f96b91c5f1faf950e77f9a8fc794", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7460, "end_char_idx": 9702, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
![download-icon](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/download-icon.png)

* Use the [`PreTrainedModel.from_pretrained`] and [`PreTrainedModel.save_pretrained`] workflow:

    1. Download your files ahead of time with [`PreTrainedModel.from_pretrained`]:

    ```py
    >>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

    >>> tokenizer = AutoTokenizer.from_pretrained("bigscience/T0_3B")
    >>> model = AutoModelForSeq2SeqLM.from_pretrained("bigscience/T0_3B")
    ```

    2. Save your files to a specified directory with [`PreTrainedModel.save_pretrained`]:

    ```py
    >>> tokenizer.save_pretrained("./your/path/bigscience_t0")
    >>> model.save_pretrained("./your/path/bigscience_t0")
    ```

    3. Now when you're offline, reload your files with [`PreTrainedModel.from_pretrained`] from the specified directory:

    ```py
    >>> tokenizer = AutoTokenizer.from_pretrained("./your/path/bigscience_t0")
    >>> model = AutoModel.from_pretrained("./your/path/bigscience_t0")
    ```

* Programmatically download files with the [huggingface_hub](https://github.com/huggingface/huggingface_hub/tree/main/src/huggingface_hub) library:

    1. Install the `huggingface_hub` library in your virtual environment:

    ```bash
    python -m pip install huggingface_hub
    ```

    2. Use the [`hf_hub_download`](https://huggingface.co/docs/hub/adding-a-library#download-files-from-the-hub) function to download a file to a specific path. For example, the following command downloads the `config.json` file from the [T0](https://huggingface.co/bigscience/T0_3B) model to your desired path:

    ```py
    >>> from huggingface_hub import hf_hub_download

    >>> hf_hub_download(repo_id="bigscience/T0_3B", filename="config.json", cache_dir="./your/path/bigscience_t0")
    ```

Once your file is downloaded and locally cached, specify it's local path to load and use it:

```py
>>> from transformers import AutoConfig

>>> config = AutoConfig.from_pretrained("./your/path/bigscience_t0/config.json")
```

<Tip>

See the [How to download files from the Hub](https://huggingface.co/docs/hub/how-to-downstream) section for more details on downloading files stored on the Hub.

</Tip>


key: _node_content
string_value:
{"id_": "06a94731-eab9-4092-8f75-c790ff984d44", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks/", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "71ccd574-526c-4952-a66d-468f27a84aa9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks/", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4a844f363b7c467e60ce1e93bf4a95d6c56faa42cd044ecb249b3d1fc0e4a1ba", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "be345724-1632-4354-be09-955d97d39869", "node_type": "1", "metadata": {}, "hash": "cae151ad16374d59543be6e232b84e8c332430214e522f972eef428170d42e81", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3417, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<!---
Copyright 2020 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.

-->

# Checks on a Pull Request

When you open a pull request on 🤗 Transformers, a fair number of checks will be run to make sure the patch you are adding is not breaking anything existing. Those checks are of four types:
- regular tests
- documentation build
- code and documentation style
- general repository consistency

In this document, we will take a stab at explaining what those various checks are and the reason behind them, as well as how to debug them locally if one of them fails on your PR.

Note that, ideally, they require you to have a dev install:

```bash
pip install transformers[dev]
```

or for an editable install:

```bash
pip install -e .[dev]
```

inside the Transformers repo. Since the number of optional dependencies of Transformers has grown a lot, it's possible you don't manage to get all of them. If the dev install fails, make sure to install the Deep Learning framework you are working with (PyTorch, TensorFlow and/or Flax) then do

```bash
pip install transformers[quality]
```

or for an editable install:

```bash
pip install -e .[quality]
```


## Tests

All the jobs that begin with `ci/circleci: run_tests_` run parts of the Transformers testing suite. Each of those jobs focuses on a part of the library in a certain environment: for instance `ci/circleci: run_tests_pipelines_tf` runs the pipelines test in an environment where TensorFlow only is installed.

Note that to avoid running tests when there is no real change in the modules they are testing, only part of the test suite is run each time: a utility is run to determine the differences in the library between before and after the PR (what GitHub shows you in the "Files changes" tab) and picks the tests impacted by that diff. That utility can be run locally with:

```bash
python utils/tests_fetcher.py
```

from the root of the Transformers repo. It will:

1. Check for each file in the diff if the changes are in the code or only in comments or docstrings. Only the files with real code changes are kept.
2. Build an internal map that gives for each file of the source code of the library all the files it recursively impacts. Module A is said to impact module B if module B imports module A. For the recursive impact, we need a chain of modules going from module A to module B in which each module imports the previous one.
3. Apply this map on the files gathered in step 1, which  gives us the list of model files impacted by the PR.
4. Map each of those files to their corresponding test file(s) and get the list of tests to run.

When executing the script locally, you should get the results of step 1, 3 and 4 printed and thus know which tests are run.


key: _node_content
string_value:
{"id_": "be345724-1632-4354-be09-955d97d39869", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks/", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "71ccd574-526c-4952-a66d-468f27a84aa9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks/", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4a844f363b7c467e60ce1e93bf4a95d6c56faa42cd044ecb249b3d1fc0e4a1ba", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "06a94731-eab9-4092-8f75-c790ff984d44", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks/", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a118c9030244ab329f664a7f8be1ae87b4a44779fb56ded1d8e4d757f566ece0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0ad25347-db00-4117-9f41-5fdb6530238c", "node_type": "1", "metadata": {}, "hash": "16a310c00d2fef1fe9a44664b28695bae9b1556ce3af39ee3954139a115f73b2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1838, "end_char_idx": 5082, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Tests

All the jobs that begin with `ci/circleci: run_tests_` run parts of the Transformers testing suite. Each of those jobs focuses on a part of the library in a certain environment: for instance `ci/circleci: run_tests_pipelines_tf` runs the pipelines test in an environment where TensorFlow only is installed.

Note that to avoid running tests when there is no real change in the modules they are testing, only part of the test suite is run each time: a utility is run to determine the differences in the library between before and after the PR (what GitHub shows you in the "Files changes" tab) and picks the tests impacted by that diff. That utility can be run locally with:

```bash
python utils/tests_fetcher.py
```

from the root of the Transformers repo. It will:

1. Check for each file in the diff if the changes are in the code or only in comments or docstrings. Only the files with real code changes are kept.
2. Build an internal map that gives for each file of the source code of the library all the files it recursively impacts. Module A is said to impact module B if module B imports module A. For the recursive impact, we need a chain of modules going from module A to module B in which each module imports the previous one.
3. Apply this map on the files gathered in step 1, which  gives us the list of model files impacted by the PR.
4. Map each of those files to their corresponding test file(s) and get the list of tests to run.

When executing the script locally, you should get the results of step 1, 3 and 4 printed and thus know which tests are run. The script will also create a file named `test_list.txt` which contains the list of tests to run, and you can run them locally with the following command:

```bash
python -m pytest -n 8 --dist=loadfile -rA -s $(cat test_list.txt)
```

Just in case anything slipped through the cracks, the full test suite is also run daily.

## Documentation build

The `build_pr_documentation` job builds and generates a preview of the documentation to make sure everything looks okay once your PR is merged. A bot will add a link to preview the documentation in your PR. Any changes you make to the PR are automatically updated in the preview. If the documentation fails to build, click on **Details** next to the failed job to see where things went wrong. Often, the error is as simple as a missing file in the `toctree`.

If you're interested in building or previewing the documentation locally, take a look at the [`README.md`](https://github.com/huggingface/transformers/tree/main/docs) in the docs folder.

## Code and documentation style

Code formatting is applied to all the source files, the examples and the tests using `black` and `ruff`. We also have a custom tool taking care of the formatting of docstrings and `rst` files (`utils/style_doc.py`), as well as the order of the lazy imports performed in the Transformers `__init__.py` files (`utils/custom_init_isort.py`). All of this can be launched by executing

```bash
make style
```

The CI checks those have been applied inside the `ci/circleci: check_code_quality` check. It also runs `ruff`, that will have a basic look at your code and will complain if it finds an undefined variable, or one that is not used.


key: _node_content
string_value:
{"id_": "0ad25347-db00-4117-9f41-5fdb6530238c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks/", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "71ccd574-526c-4952-a66d-468f27a84aa9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks/", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4a844f363b7c467e60ce1e93bf4a95d6c56faa42cd044ecb249b3d1fc0e4a1ba", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "be345724-1632-4354-be09-955d97d39869", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks/", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3328156dd485b16d834a6568c433621857394d8b7bd7f18b8ba0ce0bfe75fa71", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d0bb4369-9bd9-4420-9212-614588450816", "node_type": "1", "metadata": {}, "hash": "de20bfcf29b1ccbde76088ab3b3afc431cf2d082830fb92581db07adf31ce4ac", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3418, "end_char_idx": 5585, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The script will also create a file named `test_list.txt` which contains the list of tests to run, and you can run them locally with the following command:

```bash
python -m pytest -n 8 --dist=loadfile -rA -s $(cat test_list.txt)
```

Just in case anything slipped through the cracks, the full test suite is also run daily.

## Documentation build

The `build_pr_documentation` job builds and generates a preview of the documentation to make sure everything looks okay once your PR is merged. A bot will add a link to preview the documentation in your PR. Any changes you make to the PR are automatically updated in the preview. If the documentation fails to build, click on **Details** next to the failed job to see where things went wrong. Often, the error is as simple as a missing file in the `toctree`.

If you're interested in building or previewing the documentation locally, take a look at the [`README.md`](https://github.com/huggingface/transformers/tree/main/docs) in the docs folder.

## Code and documentation style

Code formatting is applied to all the source files, the examples and the tests using `black` and `ruff`. We also have a custom tool taking care of the formatting of docstrings and `rst` files (`utils/style_doc.py`), as well as the order of the lazy imports performed in the Transformers `__init__.py` files (`utils/custom_init_isort.py`). All of this can be launched by executing

```bash
make style
```

The CI checks those have been applied inside the `ci/circleci: check_code_quality` check. It also runs `ruff`, that will have a basic look at your code and will complain if it finds an undefined variable, or one that is not used. To run that check locally, use

```bash
make quality
```

This can take a lot of time, so to run the same thing on only the files you modified in the current branch, run

```bash
make fixup
```

This last command will also run all the additional checks for the repository consistency. Let's have a look at them.

## Repository consistency

This regroups all the tests to make sure your PR leaves the repository in a good state, and is performed by the `ci/circleci: check_repository_consistency` check.


key: _node_content
string_value:
{"id_": "d0bb4369-9bd9-4420-9212-614588450816", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks/", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "71ccd574-526c-4952-a66d-468f27a84aa9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks/", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4a844f363b7c467e60ce1e93bf4a95d6c56faa42cd044ecb249b3d1fc0e4a1ba", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0ad25347-db00-4117-9f41-5fdb6530238c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks/", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a6ac378c858638ab81a15a2ad0d1327d50e573cf711454ada8691a6a5aca2a73", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f1b79f14-a0d0-4898-997d-afff8d21eaa9", "node_type": "1", "metadata": {}, "hash": "c2305b09c0845a1cd234086ef51fd1574f2924d76b81af7b56eed2572f606667", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3911, "end_char_idx": 7747, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
A bot will add a link to preview the documentation in your PR. Any changes you make to the PR are automatically updated in the preview. If the documentation fails to build, click on **Details** next to the failed job to see where things went wrong. Often, the error is as simple as a missing file in the `toctree`.

If you're interested in building or previewing the documentation locally, take a look at the [`README.md`](https://github.com/huggingface/transformers/tree/main/docs) in the docs folder.

## Code and documentation style

Code formatting is applied to all the source files, the examples and the tests using `black` and `ruff`. We also have a custom tool taking care of the formatting of docstrings and `rst` files (`utils/style_doc.py`), as well as the order of the lazy imports performed in the Transformers `__init__.py` files (`utils/custom_init_isort.py`). All of this can be launched by executing

```bash
make style
```

The CI checks those have been applied inside the `ci/circleci: check_code_quality` check. It also runs `ruff`, that will have a basic look at your code and will complain if it finds an undefined variable, or one that is not used. To run that check locally, use

```bash
make quality
```

This can take a lot of time, so to run the same thing on only the files you modified in the current branch, run

```bash
make fixup
```

This last command will also run all the additional checks for the repository consistency. Let's have a look at them.

## Repository consistency

This regroups all the tests to make sure your PR leaves the repository in a good state, and is performed by the `ci/circleci: check_repository_consistency` check. You can locally run that check by executing the following:

```bash
make repo-consistency
```

This checks that:

- All objects added to the init are documented (performed by `utils/check_repo.py`)
- All `__init__.py` files have the same content in their two sections (performed by `utils/check_inits.py`)
- All code identified as a copy from another module is consistent with the original (performed by `utils/check_copies.py`)
- All configuration classes have at least one valid checkpoint mentioned in their docstrings (performed by `utils/check_config_docstrings.py`)
- All configuration classes only contain attributes that are used in corresponding modeling files (performed by `utils/check_config_attributes.py`)
- The translations of the READMEs and the index of the doc have the same model list as the main README (performed by `utils/check_copies.py`)
- The auto-generated tables in the documentation are up to date (performed by `utils/check_table.py`)
- The library has all objects available even if not all optional dependencies are installed (performed by `utils/check_dummies.py`)
- All docstrings properly document the arguments in the signature of the object (performed by `utils/check_docstrings.py`)

Should this check fail, the first two items require manual fixing, the last four can be fixed automatically for you by running the command

```bash
make fix-copies
```

Additional checks concern PRs that add new models, mainly that:

- All models added are in an Auto-mapping (performed by `utils/check_repo.py`)
<!-- TODO Sylvain, add a check that makes sure the common tests are implemented.-->
- All models are properly tested (performed by `utils/check_repo.py`)

<!-- TODO Sylvain, add the following
- All models are added to the main README, inside the main doc
- All checkpoints used actually exist on the Hub

-->

### Check copies

Since the Transformers library is very opinionated with respect to model code, and each model should fully be implemented in a single file without relying on other models, we have added a mechanism that checks whether a copy of the code of a layer of a given model stays consistent with the original.


