# embedding_metadata

key: _node_content
string_value:
{"id_": "9b843449-0ca2-4683-88f9-b2c07d508372", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/debugging", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3afbb714-b9f6-4e67-bda1-c03b779c2b6b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/debugging", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "transformers"}, "hash": "547ee66977115761f6778ec77a0f4395de63815dbe390968fa02dc78b9a4a944", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "755af5b4-1ba6-48f8-a798-322d3124bc34", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/debugging", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "transformers"}, "hash": "42be12e575702d8edaf2d5e5380658935b71b208c4384a279974d5ef335e969d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "be2da9de-bb08-48f3-8342-41d413700c3d", "node_type": "1", "metadata": {}, "hash": "2f48625c654a5991b3ced2a4abe32384204c45601fad667a829f56a555f6c2f5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15746, "end_char_idx": 18653, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
class T5LayerFF(nn.Module):
    [...]

    def forward(self, hidden_states):
        forwarded_states = self.layer_norm(hidden_states)
        detect_overflow(forwarded_states, "after layer_norm")
        forwarded_states = self.DenseReluDense(forwarded_states)
        detect_overflow(forwarded_states, "after DenseReluDense")
        return hidden_states + self.dropout(forwarded_states)
```

You can see that we added 2 of these and now we track if `inf` or `nan` for `forwarded_states` was detected
somewhere in between.

Actually, the detector already reports these because each of the calls in the example above is a `nn.Module`, but
let's say if you had some local direct calculations this is how you'd do that.

Additionally, if you're instantiating the debugger in your own code, you can adjust the number of frames printed from
its default, e.g.:

```python
from transformers.debug_utils import DebugUnderflowOverflow

debug_overflow = DebugUnderflowOverflow(model, max_frames_to_save=100)
```

### Specific batch absolute min and max value tracing

The same debugging class can be used for per-batch tracing with the underflow/overflow detection feature turned off.

Let's say you want to watch the absolute min and max values for all the ingredients of each `forward` call of a given
batch, and only do that for batches 1 and 3. Then you instantiate this class as:

```python
debug_overflow = DebugUnderflowOverflow(model, trace_batch_nums=[1, 3])
```

And now full batches 1 and 3 will be traced using the same format as the underflow/overflow detector does.

Batches are 0-indexed.

This is helpful if you know that the program starts misbehaving after a certain batch number, so you can fast-forward
right to that area. Here is a sample truncated output for such configuration:

```
                  *** Starting batch number=1 ***
abs min  abs max  metadata
                  shared Embedding
1.01e-06 7.92e+02 weight
0.00e+00 2.47e+04 input[0]
5.36e-05 7.92e+02 output
[...]
                  decoder.dropout Dropout
1.60e-07 2.27e+01 input[0]
0.00e+00 2.52e+01 output
                  decoder T5Stack
     not a tensor output
                  lm_head Linear
1.01e-06 7.92e+02 weight
0.00e+00 1.11e+00 input[0]
6.06e-02 8.39e+01 output
                   T5ForConditionalGeneration
     not a tensor output

                  *** Starting batch number=3 ***
abs min  abs max  metadata
                  shared Embedding
1.01e-06 7.92e+02 weight
0.00e+00 2.78e+04 input[0]
5.36e-05 7.92e+02 output
[...]
```

Here you will get a huge number of frames dumped - as many as there were forward calls in your model, so it may or may
not what you want, but sometimes it can be easier to use for debugging purposes than a normal debugger. For example, if
a problem starts happening at batch number 150. So you can dump traces for batches 149 and 150 and compare where
numbers started to diverge.


key: _node_content
string_value:
{"id_": "be2da9de-bb08-48f3-8342-41d413700c3d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/debugging", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3afbb714-b9f6-4e67-bda1-c03b779c2b6b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/debugging", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "transformers"}, "hash": "547ee66977115761f6778ec77a0f4395de63815dbe390968fa02dc78b9a4a944", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9b843449-0ca2-4683-88f9-b2c07d508372", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/debugging", "title": "Debugging", "tokens": 5079, "retrieve_doc": true, "source": "transformers"}, "hash": "f525c0e0e088c2ddbbfe6b70d4c18b512c74ed37f2e5be1ce05fe5be9d69e04b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17319, "end_char_idx": 18844, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Batches are 0-indexed.

This is helpful if you know that the program starts misbehaving after a certain batch number, so you can fast-forward
right to that area. Here is a sample truncated output for such configuration:

```
                  *** Starting batch number=1 ***
abs min  abs max  metadata
                  shared Embedding
1.01e-06 7.92e+02 weight
0.00e+00 2.47e+04 input[0]
5.36e-05 7.92e+02 output
[...]
                  decoder.dropout Dropout
1.60e-07 2.27e+01 input[0]
0.00e+00 2.52e+01 output
                  decoder T5Stack
     not a tensor output
                  lm_head Linear
1.01e-06 7.92e+02 weight
0.00e+00 1.11e+00 input[0]
6.06e-02 8.39e+01 output
                   T5ForConditionalGeneration
     not a tensor output

                  *** Starting batch number=3 ***
abs min  abs max  metadata
                  shared Embedding
1.01e-06 7.92e+02 weight
0.00e+00 2.78e+04 input[0]
5.36e-05 7.92e+02 output
[...]
```

Here you will get a huge number of frames dumped - as many as there were forward calls in your model, so it may or may
not what you want, but sometimes it can be easier to use for debugging purposes than a normal debugger. For example, if
a problem starts happening at batch number 150. So you can dump traces for batches 149 and 150 and compare where
numbers started to diverge.

You can also specify the batch number after which to stop the training, with:

```python
debug_overflow = DebugUnderflowOverflow(model, trace_batch_nums=[1, 3], abort_after_batch_num=3)
```


key: _node_content
string_value:
{"id_": "bf38e92a-7e04-42ce-8d71-af5d2f8c8c40", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e9ad52a7-418f-4f38-80d8-afeca381c91e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "hash": "8ba46afe581db3eec89c4dc44fce458850ae1d6e4a1482c90791c118a7846287", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6c088b50-0765-44ef-8ffe-bd591d218b75", "node_type": "1", "metadata": {}, "hash": "55da9ed7b6c4c530cd7dc6f6ee7194d2350b0b6d00419bd06423fc197fcdbddc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2285, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Create a custom architecture

An [`AutoClass`](model_doc/auto) automatically infers the model architecture and downloads pretrained configuration and weights. Generally, we recommend using an `AutoClass` to produce checkpoint-agnostic code. But users who want more control over specific model parameters can create a custom 🤗 Transformers model from just a few base classes. This could be particularly useful for anyone who is interested in studying, training or experimenting with a 🤗 Transformers model. In this guide, dive deeper into creating a custom model without an `AutoClass`. Learn how to:

- Load and customize a model configuration.
- Create a model architecture.
- Create a slow and fast tokenizer for text.
- Create an image processor for vision tasks.
- Create a feature extractor for audio tasks.
- Create a processor for multimodal tasks.

## Configuration

A [configuration](main_classes/configuration) refers to a model's specific attributes. Each model configuration has different attributes; for instance, all NLP models have the `hidden_size`, `num_attention_heads`, `num_hidden_layers` and `vocab_size` attributes in common. These attributes specify the number of attention heads or hidden layers to construct a model with.

Get a closer look at [DistilBERT](model_doc/distilbert) by accessing [`DistilBertConfig`] to inspect it's attributes:

```py
>>> from transformers import DistilBertConfig

>>> config = DistilBertConfig()
>>> print(config)
DistilBertConfig {
  "activation": "gelu",
  "attention_dropout": 0.1,
  "dim": 768,
  "dropout": 0.1,
  "hidden_dim": 3072,
  "initializer_range": 0.02,
  "max_position_embeddings": 512,
  "model_type": "distilbert",
  "n_heads": 12,
  "n_layers": 6,
  "pad_token_id": 0,
  "qa_dropout": 0.1,
  "seq_classif_dropout": 0.2,
  "sinusoidal_pos_embds": false,
  "transformers_version": "4.16.2",
  "vocab_size": 30522
}
```

[`DistilBertConfig`] displays all the default attributes used to build a base [`DistilBertModel`]. All attributes are customizable, creating space for experimentation. For example, you can customize a default model to:

- Try a different activation function with the `activation` parameter.
- Use a higher dropout ratio for the attention probabilities with the `attention_dropout` parameter.


key: _node_content
string_value:
{"id_": "6c088b50-0765-44ef-8ffe-bd591d218b75", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e9ad52a7-418f-4f38-80d8-afeca381c91e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "hash": "8ba46afe581db3eec89c4dc44fce458850ae1d6e4a1482c90791c118a7846287", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bf38e92a-7e04-42ce-8d71-af5d2f8c8c40", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "hash": "2d53c9c9228d58d35d3e11cb756241329e8f205ac060cdbf5fc0bde50632bbd8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "39c64eb3-7b8c-4799-9764-b09e830cce67", "node_type": "1", "metadata": {}, "hash": "65a6652ee58185cb887dbe54220094d198f815b7a5908b56f44800dac1340e38", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 769, "end_char_idx": 3693, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- Create a feature extractor for audio tasks.
- Create a processor for multimodal tasks.

## Configuration

A [configuration](main_classes/configuration) refers to a model's specific attributes. Each model configuration has different attributes; for instance, all NLP models have the `hidden_size`, `num_attention_heads`, `num_hidden_layers` and `vocab_size` attributes in common. These attributes specify the number of attention heads or hidden layers to construct a model with.

Get a closer look at [DistilBERT](model_doc/distilbert) by accessing [`DistilBertConfig`] to inspect it's attributes:

```py
>>> from transformers import DistilBertConfig

>>> config = DistilBertConfig()
>>> print(config)
DistilBertConfig {
  "activation": "gelu",
  "attention_dropout": 0.1,
  "dim": 768,
  "dropout": 0.1,
  "hidden_dim": 3072,
  "initializer_range": 0.02,
  "max_position_embeddings": 512,
  "model_type": "distilbert",
  "n_heads": 12,
  "n_layers": 6,
  "pad_token_id": 0,
  "qa_dropout": 0.1,
  "seq_classif_dropout": 0.2,
  "sinusoidal_pos_embds": false,
  "transformers_version": "4.16.2",
  "vocab_size": 30522
}
```

[`DistilBertConfig`] displays all the default attributes used to build a base [`DistilBertModel`]. All attributes are customizable, creating space for experimentation. For example, you can customize a default model to:

- Try a different activation function with the `activation` parameter.
- Use a higher dropout ratio for the attention probabilities with the `attention_dropout` parameter.

```py
>>> my_config = DistilBertConfig(activation="relu", attention_dropout=0.4)
>>> print(my_config)
DistilBertConfig {
  "activation": "relu",
  "attention_dropout": 0.4,
  "dim": 768,
  "dropout": 0.1,
  "hidden_dim": 3072,
  "initializer_range": 0.02,
  "max_position_embeddings": 512,
  "model_type": "distilbert",
  "n_heads": 12,
  "n_layers": 6,
  "pad_token_id": 0,
  "qa_dropout": 0.1,
  "seq_classif_dropout": 0.2,
  "sinusoidal_pos_embds": false,
  "transformers_version": "4.16.2",
  "vocab_size": 30522
}
```

Pretrained model attributes can be modified in the [`~PretrainedConfig.from_pretrained`] function:

```py
>>> my_config = DistilBertConfig.from_pretrained("distilbert/distilbert-base-uncased", activation="relu", attention_dropout=0.4)
```

Once you are satisfied with your model configuration, you can save it with [`~PretrainedConfig.save_pretrained`]. Your configuration file is stored as a JSON file in the specified save directory:

```py
>>> my_config.save_pretrained(save_directory="./your_model_save_path")
```

To reuse the configuration file, load it with [`~PretrainedConfig.from_pretrained`]:

```py
>>> my_config = DistilBertConfig.from_pretrained("./your_model_save_path/config.json")
```

<Tip>

You can also save your configuration file as a dictionary or even just the difference between your custom configuration attributes and the default configuration attributes!


key: _node_content
string_value:
{"id_": "39c64eb3-7b8c-4799-9764-b09e830cce67", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e9ad52a7-418f-4f38-80d8-afeca381c91e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "hash": "8ba46afe581db3eec89c4dc44fce458850ae1d6e4a1482c90791c118a7846287", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6c088b50-0765-44ef-8ffe-bd591d218b75", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "hash": "30aa4d18f6c54944f006e65568b15bdcdd1786cbf4c081423c11e780afcb1e9d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "968749c8-1181-483f-99c8-2f91f5054c7e", "node_type": "1", "metadata": {}, "hash": "9c38af66dfc1c87f16960c9505dcc96df55315b6add0afdbb93600e3c8c778fa", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2185, "end_char_idx": 5237, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- Use a higher dropout ratio for the attention probabilities with the `attention_dropout` parameter.

```py
>>> my_config = DistilBertConfig(activation="relu", attention_dropout=0.4)
>>> print(my_config)
DistilBertConfig {
  "activation": "relu",
  "attention_dropout": 0.4,
  "dim": 768,
  "dropout": 0.1,
  "hidden_dim": 3072,
  "initializer_range": 0.02,
  "max_position_embeddings": 512,
  "model_type": "distilbert",
  "n_heads": 12,
  "n_layers": 6,
  "pad_token_id": 0,
  "qa_dropout": 0.1,
  "seq_classif_dropout": 0.2,
  "sinusoidal_pos_embds": false,
  "transformers_version": "4.16.2",
  "vocab_size": 30522
}
```

Pretrained model attributes can be modified in the [`~PretrainedConfig.from_pretrained`] function:

```py
>>> my_config = DistilBertConfig.from_pretrained("distilbert/distilbert-base-uncased", activation="relu", attention_dropout=0.4)
```

Once you are satisfied with your model configuration, you can save it with [`~PretrainedConfig.save_pretrained`]. Your configuration file is stored as a JSON file in the specified save directory:

```py
>>> my_config.save_pretrained(save_directory="./your_model_save_path")
```

To reuse the configuration file, load it with [`~PretrainedConfig.from_pretrained`]:

```py
>>> my_config = DistilBertConfig.from_pretrained("./your_model_save_path/config.json")
```

<Tip>

You can also save your configuration file as a dictionary or even just the difference between your custom configuration attributes and the default configuration attributes! See the [configuration](main_classes/configuration) documentation for more details.

</Tip>

## Model

The next step is to create a [model](main_classes/models). The model - also loosely referred to as the architecture - defines what each layer is doing and what operations are happening. Attributes like `num_hidden_layers` from the configuration are used to define the architecture. Every model shares the base class [`PreTrainedModel`] and a few common methods like resizing input embeddings and pruning self-attention heads. In addition, all models are also either a [`torch.nn.Module`](https://pytorch.org/docs/stable/generated/torch.nn.Module.html), [`tf.keras.Model`](https://www.tensorflow.org/api_docs/python/tf/keras/Model) or [`flax.linen.Module`](https://flax.readthedocs.io/en/latest/api_reference/flax.linen/module.html) subclass. This means models are compatible with each of their respective framework's usage.

<frameworkcontent>
<pt>
Load your custom configuration attributes into the model:

```py
>>> from transformers import DistilBertModel

>>> my_config = DistilBertConfig.from_pretrained("./your_model_save_path/config.json")
>>> model = DistilBertModel(my_config)
```

This creates a model with random values instead of pretrained weights. You won't be able to use this model for anything useful yet until you train it. Training is a costly and time-consuming process. It is generally better to use a pretrained model to obtain better results faster, while using only a fraction of the resources required for training.


key: _node_content
string_value:
{"id_": "968749c8-1181-483f-99c8-2f91f5054c7e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e9ad52a7-418f-4f38-80d8-afeca381c91e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "hash": "8ba46afe581db3eec89c4dc44fce458850ae1d6e4a1482c90791c118a7846287", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "39c64eb3-7b8c-4799-9764-b09e830cce67", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "hash": "b830837d89a7bb447ae983ea0bd2c6dcd8449bae82f0d816a9a5e9d7ee84671d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0e3dee86-3378-4b1d-88b5-c3ccff3fd452", "node_type": "1", "metadata": {}, "hash": "a3462cecb106da4d9bfa1683f5c57d35282982009306f5e6cc00e7ffb2763289", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3694, "end_char_idx": 7135, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
See the [configuration](main_classes/configuration) documentation for more details.

</Tip>

## Model

The next step is to create a [model](main_classes/models). The model - also loosely referred to as the architecture - defines what each layer is doing and what operations are happening. Attributes like `num_hidden_layers` from the configuration are used to define the architecture. Every model shares the base class [`PreTrainedModel`] and a few common methods like resizing input embeddings and pruning self-attention heads. In addition, all models are also either a [`torch.nn.Module`](https://pytorch.org/docs/stable/generated/torch.nn.Module.html), [`tf.keras.Model`](https://www.tensorflow.org/api_docs/python/tf/keras/Model) or [`flax.linen.Module`](https://flax.readthedocs.io/en/latest/api_reference/flax.linen/module.html) subclass. This means models are compatible with each of their respective framework's usage.

<frameworkcontent>
<pt>
Load your custom configuration attributes into the model:

```py
>>> from transformers import DistilBertModel

>>> my_config = DistilBertConfig.from_pretrained("./your_model_save_path/config.json")
>>> model = DistilBertModel(my_config)
```

This creates a model with random values instead of pretrained weights. You won't be able to use this model for anything useful yet until you train it. Training is a costly and time-consuming process. It is generally better to use a pretrained model to obtain better results faster, while using only a fraction of the resources required for training.

Create a pretrained model with [`~PreTrainedModel.from_pretrained`]:

```py
>>> model = DistilBertModel.from_pretrained("distilbert/distilbert-base-uncased")
```

When you load pretrained weights, the default model configuration is automatically loaded if the model is provided by 🤗 Transformers. However, you can still replace - some or all of - the default model configuration attributes with your own if you'd like:

```py
>>> model = DistilBertModel.from_pretrained("distilbert/distilbert-base-uncased", config=my_config)
```
</pt>
<tf>
Load your custom configuration attributes into the model:

```py
>>> from transformers import TFDistilBertModel

>>> my_config = DistilBertConfig.from_pretrained("./your_model_save_path/my_config.json")
>>> tf_model = TFDistilBertModel(my_config)
```

This creates a model with random values instead of pretrained weights. You won't be able to use this model for anything useful yet until you train it. Training is a costly and time-consuming process. It is generally better to use a pretrained model to obtain better results faster, while using only a fraction of the resources required for training.

Create a pretrained model with [`~TFPreTrainedModel.from_pretrained`]:

```py
>>> tf_model = TFDistilBertModel.from_pretrained("distilbert/distilbert-base-uncased")
```

When you load pretrained weights, the default model configuration is automatically loaded if the model is provided by 🤗 Transformers. However, you can still replace - some or all of - the default model configuration attributes with your own if you'd like:

```py
>>> tf_model = TFDistilBertModel.from_pretrained("distilbert/distilbert-base-uncased", config=my_config)
```
</tf>
</frameworkcontent>

### Model heads

At this point, you have a base DistilBERT model which outputs the *hidden states*. The hidden states are passed as inputs to a model head to produce the final output.


key: _node_content
string_value:
{"id_": "0e3dee86-3378-4b1d-88b5-c3ccff3fd452", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e9ad52a7-418f-4f38-80d8-afeca381c91e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "hash": "8ba46afe581db3eec89c4dc44fce458850ae1d6e4a1482c90791c118a7846287", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "968749c8-1181-483f-99c8-2f91f5054c7e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "hash": "3caabefb0490c3cb81e628b8028965bf2f283f3a66b098b079024619c6c4debf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ef3e53a7-ab5f-42c2-8557-ef37d1f111da", "node_type": "1", "metadata": {}, "hash": "51e542240f3b7a26af8efd5d7567d61fd9b441bfdf9bed810352ed3448a0f288", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5536, "end_char_idx": 8930, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, you can still replace - some or all of - the default model configuration attributes with your own if you'd like:

```py
>>> model = DistilBertModel.from_pretrained("distilbert/distilbert-base-uncased", config=my_config)
```
</pt>
<tf>
Load your custom configuration attributes into the model:

```py
>>> from transformers import TFDistilBertModel

>>> my_config = DistilBertConfig.from_pretrained("./your_model_save_path/my_config.json")
>>> tf_model = TFDistilBertModel(my_config)
```

This creates a model with random values instead of pretrained weights. You won't be able to use this model for anything useful yet until you train it. Training is a costly and time-consuming process. It is generally better to use a pretrained model to obtain better results faster, while using only a fraction of the resources required for training.

Create a pretrained model with [`~TFPreTrainedModel.from_pretrained`]:

```py
>>> tf_model = TFDistilBertModel.from_pretrained("distilbert/distilbert-base-uncased")
```

When you load pretrained weights, the default model configuration is automatically loaded if the model is provided by 🤗 Transformers. However, you can still replace - some or all of - the default model configuration attributes with your own if you'd like:

```py
>>> tf_model = TFDistilBertModel.from_pretrained("distilbert/distilbert-base-uncased", config=my_config)
```
</tf>
</frameworkcontent>

### Model heads

At this point, you have a base DistilBERT model which outputs the *hidden states*. The hidden states are passed as inputs to a model head to produce the final output. 🤗 Transformers provides a different model head for each task as long as a model supports the task (i.e., you can't use DistilBERT for a sequence-to-sequence task like translation).

<frameworkcontent>
<pt>
For example, [`DistilBertForSequenceClassification`] is a base DistilBERT model with a sequence classification head. The sequence classification head is a linear layer on top of the pooled outputs.

```py
>>> from transformers import DistilBertForSequenceClassification

>>> model = DistilBertForSequenceClassification.from_pretrained("distilbert/distilbert-base-uncased")
```

Easily reuse this checkpoint for another task by switching to a different model head. For a question answering task, you would use the [`DistilBertForQuestionAnswering`] model head. The question answering head is similar to the sequence classification head except it is a linear layer on top of the hidden states output.

```py
>>> from transformers import DistilBertForQuestionAnswering

>>> model = DistilBertForQuestionAnswering.from_pretrained("distilbert/distilbert-base-uncased")
```
</pt>
<tf>
For example, [`TFDistilBertForSequenceClassification`] is a base DistilBERT model with a sequence classification head. The sequence classification head is a linear layer on top of the pooled outputs.

```py
>>> from transformers import TFDistilBertForSequenceClassification

>>> tf_model = TFDistilBertForSequenceClassification.from_pretrained("distilbert/distilbert-base-uncased")
```

Easily reuse this checkpoint for another task by switching to a different model head. For a question answering task, you would use the [`TFDistilBertForQuestionAnswering`] model head. The question answering head is similar to the sequence classification head except it is a linear layer on top of the hidden states output.


key: _node_content
string_value:
{"id_": "ef3e53a7-ab5f-42c2-8557-ef37d1f111da", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e9ad52a7-418f-4f38-80d8-afeca381c91e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "hash": "8ba46afe581db3eec89c4dc44fce458850ae1d6e4a1482c90791c118a7846287", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0e3dee86-3378-4b1d-88b5-c3ccff3fd452", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "hash": "b5646942feb2cdbfd989b3464da2ae561cfffd6195dbb2cc816e86762e4145fc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5ac806bd-c2c7-414e-9eb5-b4cd84abf805", "node_type": "1", "metadata": {}, "hash": "b8c4e69f0cea427df77fb8f616ef9eb3b49e0d8d0834dca5ed2167c5a4668ee6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7318, "end_char_idx": 10639, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<frameworkcontent>
<pt>
For example, [`DistilBertForSequenceClassification`] is a base DistilBERT model with a sequence classification head. The sequence classification head is a linear layer on top of the pooled outputs.

```py
>>> from transformers import DistilBertForSequenceClassification

>>> model = DistilBertForSequenceClassification.from_pretrained("distilbert/distilbert-base-uncased")
```

Easily reuse this checkpoint for another task by switching to a different model head. For a question answering task, you would use the [`DistilBertForQuestionAnswering`] model head. The question answering head is similar to the sequence classification head except it is a linear layer on top of the hidden states output.

```py
>>> from transformers import DistilBertForQuestionAnswering

>>> model = DistilBertForQuestionAnswering.from_pretrained("distilbert/distilbert-base-uncased")
```
</pt>
<tf>
For example, [`TFDistilBertForSequenceClassification`] is a base DistilBERT model with a sequence classification head. The sequence classification head is a linear layer on top of the pooled outputs.

```py
>>> from transformers import TFDistilBertForSequenceClassification

>>> tf_model = TFDistilBertForSequenceClassification.from_pretrained("distilbert/distilbert-base-uncased")
```

Easily reuse this checkpoint for another task by switching to a different model head. For a question answering task, you would use the [`TFDistilBertForQuestionAnswering`] model head. The question answering head is similar to the sequence classification head except it is a linear layer on top of the hidden states output.

```py
>>> from transformers import TFDistilBertForQuestionAnswering

>>> tf_model = TFDistilBertForQuestionAnswering.from_pretrained("distilbert/distilbert-base-uncased")
```
</tf>
</frameworkcontent>

## Tokenizer

The last base class you need before using a model for textual data is a [tokenizer](main_classes/tokenizer) to convert raw text to tensors. There are two types of tokenizers you can use with 🤗 Transformers:

- [`PreTrainedTokenizer`]: a Python implementation of a tokenizer.
- [`PreTrainedTokenizerFast`]: a tokenizer from our Rust-based [🤗 Tokenizer](https://huggingface.co/docs/tokenizers/python/latest/) library. This tokenizer type is significantly faster - especially during batch tokenization - due to its Rust implementation. The fast tokenizer also offers additional methods like *offset mapping* which maps tokens to their original words or characters.

Both tokenizers support common methods such as encoding and decoding, adding new tokens, and managing special tokens.

<Tip warning={true}>

Not every model supports a fast tokenizer. Take a look at this [table](index#supported-frameworks) to check if a model has fast tokenizer support.

</Tip>

If you trained your own tokenizer, you can create one from your *vocabulary* file:

```py
>>> from transformers import DistilBertTokenizer

>>> my_tokenizer = DistilBertTokenizer(vocab_file="my_vocab_file.txt", do_lower_case=False, padding_side="left")
```

It is important to remember the vocabulary from a custom tokenizer will be different from the vocabulary generated by a pretrained model's tokenizer. You need to use a pretrained model's vocabulary if you are using a pretrained model, otherwise the inputs won't make sense.


key: _node_content
string_value:
{"id_": "5ac806bd-c2c7-414e-9eb5-b4cd84abf805", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e9ad52a7-418f-4f38-80d8-afeca381c91e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "hash": "8ba46afe581db3eec89c4dc44fce458850ae1d6e4a1482c90791c118a7846287", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ef3e53a7-ab5f-42c2-8557-ef37d1f111da", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "hash": "128e5084bde51bbc8dbfd725ffff2f868e26ede5c6070ad3405a1d496b90fa47", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ad17cc37-17ef-499a-ab3f-10f9432b7c07", "node_type": "1", "metadata": {}, "hash": "386792257a6e23b1b96aa9b61f0d8aa3a29a42ce3292925605633a905181b9c9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8932, "end_char_idx": 12134, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> from transformers import TFDistilBertForQuestionAnswering

>>> tf_model = TFDistilBertForQuestionAnswering.from_pretrained("distilbert/distilbert-base-uncased")
```
</tf>
</frameworkcontent>

## Tokenizer

The last base class you need before using a model for textual data is a [tokenizer](main_classes/tokenizer) to convert raw text to tensors. There are two types of tokenizers you can use with 🤗 Transformers:

- [`PreTrainedTokenizer`]: a Python implementation of a tokenizer.
- [`PreTrainedTokenizerFast`]: a tokenizer from our Rust-based [🤗 Tokenizer](https://huggingface.co/docs/tokenizers/python/latest/) library. This tokenizer type is significantly faster - especially during batch tokenization - due to its Rust implementation. The fast tokenizer also offers additional methods like *offset mapping* which maps tokens to their original words or characters.

Both tokenizers support common methods such as encoding and decoding, adding new tokens, and managing special tokens.

<Tip warning={true}>

Not every model supports a fast tokenizer. Take a look at this [table](index#supported-frameworks) to check if a model has fast tokenizer support.

</Tip>

If you trained your own tokenizer, you can create one from your *vocabulary* file:

```py
>>> from transformers import DistilBertTokenizer

>>> my_tokenizer = DistilBertTokenizer(vocab_file="my_vocab_file.txt", do_lower_case=False, padding_side="left")
```

It is important to remember the vocabulary from a custom tokenizer will be different from the vocabulary generated by a pretrained model's tokenizer. You need to use a pretrained model's vocabulary if you are using a pretrained model, otherwise the inputs won't make sense. Create a tokenizer with a pretrained model's vocabulary with the [`DistilBertTokenizer`] class:

```py
>>> from transformers import DistilBertTokenizer

>>> slow_tokenizer = DistilBertTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
```

Create a fast tokenizer with the [`DistilBertTokenizerFast`] class:

```py
>>> from transformers import DistilBertTokenizerFast

>>> fast_tokenizer = DistilBertTokenizerFast.from_pretrained("distilbert/distilbert-base-uncased")
```

<Tip>

By default, [`AutoTokenizer`] will try to load a fast tokenizer. You can disable this behavior by setting `use_fast=False` in `from_pretrained`.

</Tip>

## Image processor

An image processor processes vision inputs. It inherits from the base [`~image_processing_utils.ImageProcessingMixin`] class.

To use, create an image processor associated with the model you're using. For example, create a default [`ViTImageProcessor`] if you are using [ViT](model_doc/vit) for image classification:

```py
>>> from transformers import ViTImageProcessor

>>> vit_extractor = ViTImageProcessor()
>>> print(vit_extractor)
ViTImageProcessor {
  "do_normalize": true,
  "do_resize": true,
  "image_processor_type": "ViTImageProcessor",
  "image_mean": [
    0.5,
    0.5,
    0.5
  ],
  "image_std": [
    0.5,
    0.5,
    0.5
  ],
  "resample": 2,
  "size": 224
}
```

<Tip>

If you aren't looking for any customization, just use the `from_pretrained` method to load a model's default image processor parameters.


key: _node_content
string_value:
{"id_": "ad17cc37-17ef-499a-ab3f-10f9432b7c07", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e9ad52a7-418f-4f38-80d8-afeca381c91e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "hash": "8ba46afe581db3eec89c4dc44fce458850ae1d6e4a1482c90791c118a7846287", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5ac806bd-c2c7-414e-9eb5-b4cd84abf805", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "hash": "96d9dfc5ab4f55c7f7ed6fc982e079881fa69f1ad70ab5ef5128f14d41ff8b5d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dbc1477e-d7a4-4d8b-bb01-283a85a29e57", "node_type": "1", "metadata": {}, "hash": "9d4eb4740ad3e484a8e1400034106a3158e0fc738a15de1cc6c4ef78965d2e41", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10640, "end_char_idx": 13600, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Create a tokenizer with a pretrained model's vocabulary with the [`DistilBertTokenizer`] class:

```py
>>> from transformers import DistilBertTokenizer

>>> slow_tokenizer = DistilBertTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
```

Create a fast tokenizer with the [`DistilBertTokenizerFast`] class:

```py
>>> from transformers import DistilBertTokenizerFast

>>> fast_tokenizer = DistilBertTokenizerFast.from_pretrained("distilbert/distilbert-base-uncased")
```

<Tip>

By default, [`AutoTokenizer`] will try to load a fast tokenizer. You can disable this behavior by setting `use_fast=False` in `from_pretrained`.

</Tip>

## Image processor

An image processor processes vision inputs. It inherits from the base [`~image_processing_utils.ImageProcessingMixin`] class.

To use, create an image processor associated with the model you're using. For example, create a default [`ViTImageProcessor`] if you are using [ViT](model_doc/vit) for image classification:

```py
>>> from transformers import ViTImageProcessor

>>> vit_extractor = ViTImageProcessor()
>>> print(vit_extractor)
ViTImageProcessor {
  "do_normalize": true,
  "do_resize": true,
  "image_processor_type": "ViTImageProcessor",
  "image_mean": [
    0.5,
    0.5,
    0.5
  ],
  "image_std": [
    0.5,
    0.5,
    0.5
  ],
  "resample": 2,
  "size": 224
}
```

<Tip>

If you aren't looking for any customization, just use the `from_pretrained` method to load a model's default image processor parameters.

</Tip>

Modify any of the [`ViTImageProcessor`] parameters to create your custom image processor:

```py
>>> from transformers import ViTImageProcessor

>>> my_vit_extractor = ViTImageProcessor(resample="PIL.Image.BOX", do_normalize=False, image_mean=[0.3, 0.3, 0.3])
>>> print(my_vit_extractor)
ViTImageProcessor {
  "do_normalize": false,
  "do_resize": true,
  "image_processor_type": "ViTImageProcessor",
  "image_mean": [
    0.3,
    0.3,
    0.3
  ],
  "image_std": [
    0.5,
    0.5,
    0.5
  ],
  "resample": "PIL.Image.BOX",
  "size": 224
}
```

## Backbone

<div style="text-align: center">
  <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/Backbone.png">
</div>

Computer vision models consist of a backbone, neck, and head. The backbone extracts features from an input image, the neck combines and enhances the extracted features, and the head is used for the main task (e.g., object detection). Start by initializing a backbone in the model config and specify whether you want to load pretrained weights or load randomly initialized weights. Then you can pass the model config to the model head.

For example, to load a [ResNet](../model_doc/resnet) backbone into a [MaskFormer](../model_doc/maskformer) model with an instance segmentation head:

<hfoptions id="backbone">
<hfoption id="pretrained weights">

Set `use_pretrained_backbone=True` to load pretrained ResNet weights for the backbone.


key: _node_content
string_value:
{"id_": "dbc1477e-d7a4-4d8b-bb01-283a85a29e57", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e9ad52a7-418f-4f38-80d8-afeca381c91e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "hash": "8ba46afe581db3eec89c4dc44fce458850ae1d6e4a1482c90791c118a7846287", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ad17cc37-17ef-499a-ab3f-10f9432b7c07", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "hash": "8608cc8d1d78d9328aa85859c1a7c6441178d689be7fae981c8e62ec12c52e0c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4cf01dee-cd3d-4d97-add9-7e6aab05da6b", "node_type": "1", "metadata": {}, "hash": "2eba7ff5fec9533153c7fe7fec51544cfc938d8a05de662c05779927fbded44b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12136, "end_char_idx": 15286, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

Modify any of the [`ViTImageProcessor`] parameters to create your custom image processor:

```py
>>> from transformers import ViTImageProcessor

>>> my_vit_extractor = ViTImageProcessor(resample="PIL.Image.BOX", do_normalize=False, image_mean=[0.3, 0.3, 0.3])
>>> print(my_vit_extractor)
ViTImageProcessor {
  "do_normalize": false,
  "do_resize": true,
  "image_processor_type": "ViTImageProcessor",
  "image_mean": [
    0.3,
    0.3,
    0.3
  ],
  "image_std": [
    0.5,
    0.5,
    0.5
  ],
  "resample": "PIL.Image.BOX",
  "size": 224
}
```

## Backbone

<div style="text-align: center">
  <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/Backbone.png">
</div>

Computer vision models consist of a backbone, neck, and head. The backbone extracts features from an input image, the neck combines and enhances the extracted features, and the head is used for the main task (e.g., object detection). Start by initializing a backbone in the model config and specify whether you want to load pretrained weights or load randomly initialized weights. Then you can pass the model config to the model head.

For example, to load a [ResNet](../model_doc/resnet) backbone into a [MaskFormer](../model_doc/maskformer) model with an instance segmentation head:

<hfoptions id="backbone">
<hfoption id="pretrained weights">

Set `use_pretrained_backbone=True` to load pretrained ResNet weights for the backbone.

```py
from transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation

config = MaskFormerConfig(backbone="microsoft/resnet-50", use_pretrained_backbone=True) # backbone and neck config
model = MaskFormerForInstanceSegmentation(config) # head
```

</hfoption>
<hfoption id="random weights">

Set `use_pretrained_backbone=False` to randomly initialize a ResNet backbone.

```py
from transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation

config = MaskFormerConfig(backbone="microsoft/resnet-50", use_pretrained_backbone=False) # backbone and neck config
model = MaskFormerForInstanceSegmentation(config) # head
```

You could also load the backbone config separately and then pass it to the model config.

```py
from transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation, ResNetConfig

backbone_config = ResNetConfig()
config = MaskFormerConfig(backbone_config=backbone_config)
model = MaskFormerForInstanceSegmentation(config)
```

</hfoption>
</hfoptions id="timm backbone">

[timm](https://hf.co/docs/timm/index) models are loaded within a model with `use_timm_backbone=True` or with [`TimmBackbone`] and [`TimmBackboneConfig`].

Use `use_timm_backbone=True` and `use_pretrained_backbone=True` to load pretrained timm weights for the backbone.

```python
from transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation

config = MaskFormerConfig(backbone="resnet50", use_pretrained_backbone=True, use_timm_backbone=True) # backbone and neck config
model = MaskFormerForInstanceSegmentation(config) # head
```

Set `use_timm_backbone=True` and `use_pretrained_backbone=False` to load a randomly initialized timm backbone.


key: _node_content
string_value:
{"id_": "4cf01dee-cd3d-4d97-add9-7e6aab05da6b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e9ad52a7-418f-4f38-80d8-afeca381c91e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "hash": "8ba46afe581db3eec89c4dc44fce458850ae1d6e4a1482c90791c118a7846287", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dbc1477e-d7a4-4d8b-bb01-283a85a29e57", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "hash": "837b35f10edb1b92ba996718e48600077ea887d04cd93ef25249fee54691612f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "083305f2-b579-4fab-af4e-5cdbfb33d2ba", "node_type": "1", "metadata": {}, "hash": "a5a0ec48555ad6214e2ba002241f55c0926715c1cf9fb41be65af115e72cd50e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13602, "end_char_idx": 16584, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
from transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation

config = MaskFormerConfig(backbone="microsoft/resnet-50", use_pretrained_backbone=True) # backbone and neck config
model = MaskFormerForInstanceSegmentation(config) # head
```

</hfoption>
<hfoption id="random weights">

Set `use_pretrained_backbone=False` to randomly initialize a ResNet backbone.

```py
from transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation

config = MaskFormerConfig(backbone="microsoft/resnet-50", use_pretrained_backbone=False) # backbone and neck config
model = MaskFormerForInstanceSegmentation(config) # head
```

You could also load the backbone config separately and then pass it to the model config.

```py
from transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation, ResNetConfig

backbone_config = ResNetConfig()
config = MaskFormerConfig(backbone_config=backbone_config)
model = MaskFormerForInstanceSegmentation(config)
```

</hfoption>
</hfoptions id="timm backbone">

[timm](https://hf.co/docs/timm/index) models are loaded within a model with `use_timm_backbone=True` or with [`TimmBackbone`] and [`TimmBackboneConfig`].

Use `use_timm_backbone=True` and `use_pretrained_backbone=True` to load pretrained timm weights for the backbone.

```python
from transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation

config = MaskFormerConfig(backbone="resnet50", use_pretrained_backbone=True, use_timm_backbone=True) # backbone and neck config
model = MaskFormerForInstanceSegmentation(config) # head
```

Set `use_timm_backbone=True` and `use_pretrained_backbone=False` to load a randomly initialized timm backbone.

```python
from transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation

config = MaskFormerConfig(backbone="resnet50", use_pretrained_backbone=False, use_timm_backbone=True) # backbone and neck config
model = MaskFormerForInstanceSegmentation(config) # head
```

You could also load the backbone config and use it to create a `TimmBackbone` or pass it to the model config. Timm backbones will load pretrained weights by default. Set `use_pretrained_backbone=False` to load randomly initialized weights.

```python
from transformers import TimmBackboneConfig, TimmBackbone

backbone_config = TimmBackboneConfig("resnet50", use_pretrained_backbone=False)

# Create a backbone class
backbone = TimmBackbone(config=backbone_config)

# Create a model with a timm backbone
from transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation

config = MaskFormerConfig(backbone_config=backbone_config)
model = MaskFormerForInstanceSegmentation(config)
```

## Feature extractor

A feature extractor processes audio inputs. It inherits from the base [`~feature_extraction_utils.FeatureExtractionMixin`] class, and may also inherit from the [`SequenceFeatureExtractor`] class for processing audio inputs.

To use, create a feature extractor associated with the model you're using.


key: _node_content
string_value:
{"id_": "083305f2-b579-4fab-af4e-5cdbfb33d2ba", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e9ad52a7-418f-4f38-80d8-afeca381c91e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "hash": "8ba46afe581db3eec89c4dc44fce458850ae1d6e4a1482c90791c118a7846287", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4cf01dee-cd3d-4d97-add9-7e6aab05da6b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "hash": "a46ed4f592de4ff7b1e7712b0f999506a52a400975ee09c0dd86b7183fc0cb8b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "59898cd1-57a3-42d0-9547-5d1f6cf7f86f", "node_type": "1", "metadata": {}, "hash": "f60fa4c524bd51c61f93430adbef83bfb17dbe28ebd3c4be56967f996eff408f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14898, "end_char_idx": 18073, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation

config = MaskFormerConfig(backbone="resnet50", use_pretrained_backbone=True, use_timm_backbone=True) # backbone and neck config
model = MaskFormerForInstanceSegmentation(config) # head
```

Set `use_timm_backbone=True` and `use_pretrained_backbone=False` to load a randomly initialized timm backbone.

```python
from transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation

config = MaskFormerConfig(backbone="resnet50", use_pretrained_backbone=False, use_timm_backbone=True) # backbone and neck config
model = MaskFormerForInstanceSegmentation(config) # head
```

You could also load the backbone config and use it to create a `TimmBackbone` or pass it to the model config. Timm backbones will load pretrained weights by default. Set `use_pretrained_backbone=False` to load randomly initialized weights.

```python
from transformers import TimmBackboneConfig, TimmBackbone

backbone_config = TimmBackboneConfig("resnet50", use_pretrained_backbone=False)

# Create a backbone class
backbone = TimmBackbone(config=backbone_config)

# Create a model with a timm backbone
from transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation

config = MaskFormerConfig(backbone_config=backbone_config)
model = MaskFormerForInstanceSegmentation(config)
```

## Feature extractor

A feature extractor processes audio inputs. It inherits from the base [`~feature_extraction_utils.FeatureExtractionMixin`] class, and may also inherit from the [`SequenceFeatureExtractor`] class for processing audio inputs.

To use, create a feature extractor associated with the model you're using. For example, create a default [`Wav2Vec2FeatureExtractor`] if you are using [Wav2Vec2](model_doc/wav2vec2) for audio classification:

```py
>>> from transformers import Wav2Vec2FeatureExtractor

>>> w2v2_extractor = Wav2Vec2FeatureExtractor()
>>> print(w2v2_extractor)
Wav2Vec2FeatureExtractor {
  "do_normalize": true,
  "feature_extractor_type": "Wav2Vec2FeatureExtractor",
  "feature_size": 1,
  "padding_side": "right",
  "padding_value": 0.0,
  "return_attention_mask": false,
  "sampling_rate": 16000
}
```

<Tip>

If you aren't looking for any customization, just use the `from_pretrained` method to load a model's default feature extractor parameters.

</Tip>

Modify any of the [`Wav2Vec2FeatureExtractor`] parameters to create your custom feature extractor:

```py
>>> from transformers import Wav2Vec2FeatureExtractor

>>> w2v2_extractor = Wav2Vec2FeatureExtractor(sampling_rate=8000, do_normalize=False)
>>> print(w2v2_extractor)
Wav2Vec2FeatureExtractor {
  "do_normalize": false,
  "feature_extractor_type": "Wav2Vec2FeatureExtractor",
  "feature_size": 1,
  "padding_side": "right",
  "padding_value": 0.0,
  "return_attention_mask": false,
  "sampling_rate": 8000
}
```

## Processor

For models that support multimodal tasks, 🤗 Transformers offers a processor class that conveniently wraps processing classes such as a feature extractor and a tokenizer into a single object. For example, let's use the [`Wav2Vec2Processor`] for an automatic speech recognition task (ASR).


key: _node_content
string_value:
{"id_": "59898cd1-57a3-42d0-9547-5d1f6cf7f86f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e9ad52a7-418f-4f38-80d8-afeca381c91e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "hash": "8ba46afe581db3eec89c4dc44fce458850ae1d6e4a1482c90791c118a7846287", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "083305f2-b579-4fab-af4e-5cdbfb33d2ba", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/create_a_model", "title": "Create a custom architecture", "tokens": 4699, "retrieve_doc": true, "source": "transformers"}, "hash": "70377aacd483deb02318033f778c383c6603aec1d077483ab963cec3fdd37223", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16585, "end_char_idx": 19176, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For example, create a default [`Wav2Vec2FeatureExtractor`] if you are using [Wav2Vec2](model_doc/wav2vec2) for audio classification:

```py
>>> from transformers import Wav2Vec2FeatureExtractor

>>> w2v2_extractor = Wav2Vec2FeatureExtractor()
>>> print(w2v2_extractor)
Wav2Vec2FeatureExtractor {
  "do_normalize": true,
  "feature_extractor_type": "Wav2Vec2FeatureExtractor",
  "feature_size": 1,
  "padding_side": "right",
  "padding_value": 0.0,
  "return_attention_mask": false,
  "sampling_rate": 16000
}
```

<Tip>

If you aren't looking for any customization, just use the `from_pretrained` method to load a model's default feature extractor parameters.

</Tip>

Modify any of the [`Wav2Vec2FeatureExtractor`] parameters to create your custom feature extractor:

```py
>>> from transformers import Wav2Vec2FeatureExtractor

>>> w2v2_extractor = Wav2Vec2FeatureExtractor(sampling_rate=8000, do_normalize=False)
>>> print(w2v2_extractor)
Wav2Vec2FeatureExtractor {
  "do_normalize": false,
  "feature_extractor_type": "Wav2Vec2FeatureExtractor",
  "feature_size": 1,
  "padding_side": "right",
  "padding_value": 0.0,
  "return_attention_mask": false,
  "sampling_rate": 8000
}
```

## Processor

For models that support multimodal tasks, 🤗 Transformers offers a processor class that conveniently wraps processing classes such as a feature extractor and a tokenizer into a single object. For example, let's use the [`Wav2Vec2Processor`] for an automatic speech recognition task (ASR). ASR transcribes audio to text, so you will need a feature extractor and a tokenizer.

Create a feature extractor to handle the audio inputs:

```py
>>> from transformers import Wav2Vec2FeatureExtractor

>>> feature_extractor = Wav2Vec2FeatureExtractor(padding_value=1.0, do_normalize=True)
```

Create a tokenizer to handle the text inputs:

```py
>>> from transformers import Wav2Vec2CTCTokenizer

>>> tokenizer = Wav2Vec2CTCTokenizer(vocab_file="my_vocab_file.txt")
```

Combine the feature extractor and tokenizer in [`Wav2Vec2Processor`]:

```py
>>> from transformers import Wav2Vec2Processor

>>> processor = Wav2Vec2Processor(feature_extractor=feature_extractor, tokenizer=tokenizer)
```

With two basic classes - configuration and model - and an additional preprocessing class (tokenizer, image processor, feature extractor, or processor), you can create any of the models supported by 🤗 Transformers. Each of these base classes are configurable, allowing you to use the specific attributes you want. You can easily setup a model for training or modify an existing pretrained model to fine-tune.


key: _node_content
string_value:
{"id_": "cc12e4c1-adcc-4978-b258-f5ce7216660d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "32235eb0-3add-4442-a981-82a445254d5b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2e853d92-2c11-40d0-81fd-ac5e16e85110", "node_type": "1", "metadata": {}, "hash": "1d294bb065ef66ccb5a16ce1cead7cfbf88423f3d17eca5286d30ac185b77512", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 4663, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Methods and tools for efficient training on a single GPU

This guide demonstrates practical techniques that you can use to increase the efficiency of your model's training by 
optimizing memory utilization, speeding up the training, or both. If you'd like to understand how GPU is utilized during 
training, please refer to the [Model training anatomy](model_memory_anatomy) conceptual guide first. This guide 
focuses on practical techniques.  

<Tip>

If you have access to a machine with multiple GPUs, these approaches are still valid, plus you can leverage additional methods outlined in the [multi-GPU section](perf_train_gpu_many).

</Tip>

When training large models, there are two aspects that should be considered at the same time: 

* Data throughput/training time
* Model performance

Maximizing the throughput (samples/second) leads to lower training cost. This is generally achieved by utilizing the GPU 
as much as possible and thus filling GPU memory to its limit. If the desired batch size exceeds the limits of the GPU memory, 
the memory optimization techniques, such as gradient accumulation, can help.

However, if the preferred batch size fits into memory, there's no reason to apply memory-optimizing techniques because they can 
slow down the training. Just because one can use a large batch size, does not necessarily mean they should. As part of 
hyperparameter tuning, you should determine which batch size yields the best results and then optimize resources accordingly.

The methods and tools covered in this guide can be classified based on the effect they have on the training process:

| Method/tool                                                                                                                                             | Improves training speed | Optimizes memory utilization |
|:--------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------|:-----------------------------|
| [Batch size choice](#batch-size-choice)                                                                                                                 | Yes                     | Yes                          |
| [Gradient accumulation](#gradient-accumulation)                                                                                                         | No                      | Yes                          |
| [Gradient checkpointing](#gradient-checkpointing)                                                                                                       | No                      | Yes                          |
| [Mixed precision training](#mixed-precision-training)                                                                                                   | Yes                     | Maybe*                       |
| [torch_empty_cache_steps](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.TrainingArguments.torch_empty_cache_steps) | No                      | Yes                          |
| [Optimizer choice](#optimizer-choice)                                                                                                                   | Yes                     | Yes                          |
| [Data preloading](#data-preloading)                                                                                                                     | Yes                     | No                           |
| [DeepSpeed Zero](#deepspeed-zero)                                                                                                                       | No                      | Yes                          |
| [torch.compile](#using-torchcompile)                                                                                                                    | Yes                     | No                           |
| [Parameter-Efficient Fine Tuning (PEFT)](#using--peft)                                                                                                  | No                      | Yes                          |
 
<Tip>

*Note: when using mixed precision with a small model and a large batch size, there will be some memory savings but with a 
large model and a small batch size, the memory use will be larger.

</Tip>

You can combine the above methods to get a cumulative effect. These techniques are available to you whether you are 
training your model with [`Trainer`] or writing a pure PyTorch loop, in which case you can [configure these optimizations 
with 🤗 Accelerate](#using--accelerate).


key: _node_content
string_value:
{"id_": "2e853d92-2c11-40d0-81fd-ac5e16e85110", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "32235eb0-3add-4442-a981-82a445254d5b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cc12e4c1-adcc-4978-b258-f5ce7216660d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "33662ed06fd11420e69357661dacf9342b61d4091938221c6ddf331891986828", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bdec2308-f0f9-4982-8a59-4ac147b14ec7", "node_type": "1", "metadata": {}, "hash": "eb294357966085c238f3a027bb664ba22bb21e2e71a2a0c452ee62cfd9986e40", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1502, "end_char_idx": 6318, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The methods and tools covered in this guide can be classified based on the effect they have on the training process:

| Method/tool                                                                                                                                             | Improves training speed | Optimizes memory utilization |
|:--------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------|:-----------------------------|
| [Batch size choice](#batch-size-choice)                                                                                                                 | Yes                     | Yes                          |
| [Gradient accumulation](#gradient-accumulation)                                                                                                         | No                      | Yes                          |
| [Gradient checkpointing](#gradient-checkpointing)                                                                                                       | No                      | Yes                          |
| [Mixed precision training](#mixed-precision-training)                                                                                                   | Yes                     | Maybe*                       |
| [torch_empty_cache_steps](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.TrainingArguments.torch_empty_cache_steps) | No                      | Yes                          |
| [Optimizer choice](#optimizer-choice)                                                                                                                   | Yes                     | Yes                          |
| [Data preloading](#data-preloading)                                                                                                                     | Yes                     | No                           |
| [DeepSpeed Zero](#deepspeed-zero)                                                                                                                       | No                      | Yes                          |
| [torch.compile](#using-torchcompile)                                                                                                                    | Yes                     | No                           |
| [Parameter-Efficient Fine Tuning (PEFT)](#using--peft)                                                                                                  | No                      | Yes                          |
 
<Tip>

*Note: when using mixed precision with a small model and a large batch size, there will be some memory savings but with a 
large model and a small batch size, the memory use will be larger.

</Tip>

You can combine the above methods to get a cumulative effect. These techniques are available to you whether you are 
training your model with [`Trainer`] or writing a pure PyTorch loop, in which case you can [configure these optimizations 
with 🤗 Accelerate](#using--accelerate).

If these methods do not result in sufficient gains, you can explore the following options: 
* [Look into building your own custom Docker container with efficient software prebuilds](#efficient-software-prebuilds)
* [Consider a model that uses Mixture of Experts (MoE)](#mixture-of-experts)
* [Convert your model to BetterTransformer to leverage PyTorch native attention](#using-pytorch-native-attention-and-flash-attention)

Finally, if all of the above is still not enough, even after switching to a server-grade GPU like A100, consider moving 
to a multi-GPU setup. All these approaches are still valid in a multi-GPU setup, plus you can leverage additional parallelism 
techniques outlined in the [multi-GPU section](perf_train_gpu_many). 

## Batch size choice

To achieve optimal performance, start by identifying the appropriate batch size. It is recommended to use batch sizes and 
input/output neuron counts that are of size 2^N. Often it's a multiple of 8, but it can be 
higher depending on the hardware being used and the model's dtype.

For reference, check out NVIDIA's recommendation for [input/output neuron counts](
https://docs.nvidia.com/deeplearning/performance/dl-performance-fully-connected/index.html#input-features) and 
[batch size](https://docs.nvidia.com/deeplearning/performance/dl-performance-fully-connected/index.html#batch-size) for 
fully connected layers (which are involved in GEMMs (General Matrix Multiplications)).

[Tensor Core Requirements](https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc) 
define the multiplier based on the dtype and the hardware.


key: _node_content
string_value:
{"id_": "bdec2308-f0f9-4982-8a59-4ac147b14ec7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "32235eb0-3add-4442-a981-82a445254d5b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2e853d92-2c11-40d0-81fd-ac5e16e85110", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "63165444d87b03c9ae27ec3bbf51f4a7f2b7c64fb01ac7d6626a2905335cb9c7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6c75271d-758b-4c3e-bdfd-8fadee45d571", "node_type": "1", "metadata": {}, "hash": "892334beae99862449fa14745627e2e6fb3411fb67ddeac0469caa24e1c160ba", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4665, "end_char_idx": 8258, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If these methods do not result in sufficient gains, you can explore the following options: 
* [Look into building your own custom Docker container with efficient software prebuilds](#efficient-software-prebuilds)
* [Consider a model that uses Mixture of Experts (MoE)](#mixture-of-experts)
* [Convert your model to BetterTransformer to leverage PyTorch native attention](#using-pytorch-native-attention-and-flash-attention)

Finally, if all of the above is still not enough, even after switching to a server-grade GPU like A100, consider moving 
to a multi-GPU setup. All these approaches are still valid in a multi-GPU setup, plus you can leverage additional parallelism 
techniques outlined in the [multi-GPU section](perf_train_gpu_many). 

## Batch size choice

To achieve optimal performance, start by identifying the appropriate batch size. It is recommended to use batch sizes and 
input/output neuron counts that are of size 2^N. Often it's a multiple of 8, but it can be 
higher depending on the hardware being used and the model's dtype.

For reference, check out NVIDIA's recommendation for [input/output neuron counts](
https://docs.nvidia.com/deeplearning/performance/dl-performance-fully-connected/index.html#input-features) and 
[batch size](https://docs.nvidia.com/deeplearning/performance/dl-performance-fully-connected/index.html#batch-size) for 
fully connected layers (which are involved in GEMMs (General Matrix Multiplications)).

[Tensor Core Requirements](https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc) 
define the multiplier based on the dtype and the hardware. For instance, for fp16 data type a multiple of 8 is recommended, unless 
it's an A100 GPU, in which case use multiples of 64.

For parameters that are small, consider also [Dimension Quantization Effects](https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#dim-quantization). 
This is where tiling happens and the right multiplier can have a significant speedup.

## Gradient Accumulation

The **gradient accumulation** method aims to calculate gradients in smaller increments instead of computing them for the 
entire batch at once. This approach involves iteratively calculating gradients in smaller batches by performing forward 
and backward passes through the model and accumulating the gradients during the process. Once a sufficient number of 
gradients have been accumulated, the model's optimization step is executed. By employing gradient accumulation, it 
becomes possible to increase the **effective batch size** beyond the limitations imposed by the GPU's memory capacity. 
However, it is important to note that the additional forward and backward passes introduced by gradient accumulation can 
slow down the training process.

You can enable gradient accumulation by adding the `gradient_accumulation_steps` argument to  [`TrainingArguments`]: 

```py
training_args = TrainingArguments(per_device_train_batch_size=1, gradient_accumulation_steps=4, **default_args)
```

In the above example, your effective batch size becomes 4. 

Alternatively, use 🤗 Accelerate to gain full control over the training loop. Find the 🤗 Accelerate example 
[further down in this guide](#using--accelerate).

While it is advised to max out GPU usage as much as possible, a high number of gradient accumulation steps can 
result in a more pronounced training slowdown. Consider the following example. Let's say, the `per_device_train_batch_size=4` 
without gradient accumulation hits the GPU's limit.


key: _node_content
string_value:
{"id_": "6c75271d-758b-4c3e-bdfd-8fadee45d571", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "32235eb0-3add-4442-a981-82a445254d5b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bdec2308-f0f9-4982-8a59-4ac147b14ec7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "03bca210cb066a00cee89f818cdc36c6076434d8778cc4552becf9f29b940755", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3868e035-743f-4094-b2d8-be72f032853a", "node_type": "1", "metadata": {}, "hash": "da9605a755e7e7b3993b8b98eae6d58f29cf8b39ebba6fab809a05668083f7a5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6446, "end_char_idx": 9864, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For parameters that are small, consider also [Dimension Quantization Effects](https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#dim-quantization). 
This is where tiling happens and the right multiplier can have a significant speedup.

## Gradient Accumulation

The **gradient accumulation** method aims to calculate gradients in smaller increments instead of computing them for the 
entire batch at once. This approach involves iteratively calculating gradients in smaller batches by performing forward 
and backward passes through the model and accumulating the gradients during the process. Once a sufficient number of 
gradients have been accumulated, the model's optimization step is executed. By employing gradient accumulation, it 
becomes possible to increase the **effective batch size** beyond the limitations imposed by the GPU's memory capacity. 
However, it is important to note that the additional forward and backward passes introduced by gradient accumulation can 
slow down the training process.

You can enable gradient accumulation by adding the `gradient_accumulation_steps` argument to  [`TrainingArguments`]: 

```py
training_args = TrainingArguments(per_device_train_batch_size=1, gradient_accumulation_steps=4, **default_args)
```

In the above example, your effective batch size becomes 4. 

Alternatively, use 🤗 Accelerate to gain full control over the training loop. Find the 🤗 Accelerate example 
[further down in this guide](#using--accelerate).

While it is advised to max out GPU usage as much as possible, a high number of gradient accumulation steps can 
result in a more pronounced training slowdown. Consider the following example. Let's say, the `per_device_train_batch_size=4` 
without gradient accumulation hits the GPU's limit. If you would like to train with batches of size 64, do not set the 
`per_device_train_batch_size` to 1 and `gradient_accumulation_steps` to 64. Instead, keep `per_device_train_batch_size=4` 
and set `gradient_accumulation_steps=16`. This results in the same effective batch size while making better use of 
the available GPU resources.

For additional information, please refer to batch size and gradient accumulation benchmarks for [RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004392537)
and [A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1005033957).

## Gradient Checkpointing

Some large models may still face memory issues even when the batch size is set to 1 and gradient accumulation is used. 
This is because there are other components that also require memory storage.

Saving all activations from the forward pass in order to compute the gradients during the backward pass can result in 
significant memory overhead. The alternative approach of discarding the activations and recalculating them when needed 
during the backward pass, would introduce a considerable computational overhead and slow down the training process.

**Gradient checkpointing** offers a compromise between these two approaches and saves strategically selected activations 
throughout the computational graph so only a fraction of the activations need to be re-computed for the gradients. For 
an in-depth explanation of gradient checkpointing, refer to [this great article](https://medium.com/tensorflow/fitting-larger-networks-into-memory-583e3c758ff9).


key: _node_content
string_value:
{"id_": "3868e035-743f-4094-b2d8-be72f032853a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "32235eb0-3add-4442-a981-82a445254d5b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6c75271d-758b-4c3e-bdfd-8fadee45d571", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "14ec32b5c955abd6993caa094cb597e650b576c879fb0afb34b1e001426ce89a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2aa3d8ea-2a0e-4930-bed7-54f3cf3c24ef", "node_type": "1", "metadata": {}, "hash": "44d8cb7ee24f612184b3c99d5457da56a579a26848c689859c000e11e87d8ad7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8127, "end_char_idx": 11768, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Consider the following example. Let's say, the `per_device_train_batch_size=4` 
without gradient accumulation hits the GPU's limit. If you would like to train with batches of size 64, do not set the 
`per_device_train_batch_size` to 1 and `gradient_accumulation_steps` to 64. Instead, keep `per_device_train_batch_size=4` 
and set `gradient_accumulation_steps=16`. This results in the same effective batch size while making better use of 
the available GPU resources.

For additional information, please refer to batch size and gradient accumulation benchmarks for [RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004392537)
and [A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1005033957).

## Gradient Checkpointing

Some large models may still face memory issues even when the batch size is set to 1 and gradient accumulation is used. 
This is because there are other components that also require memory storage.

Saving all activations from the forward pass in order to compute the gradients during the backward pass can result in 
significant memory overhead. The alternative approach of discarding the activations and recalculating them when needed 
during the backward pass, would introduce a considerable computational overhead and slow down the training process.

**Gradient checkpointing** offers a compromise between these two approaches and saves strategically selected activations 
throughout the computational graph so only a fraction of the activations need to be re-computed for the gradients. For 
an in-depth explanation of gradient checkpointing, refer to [this great article](https://medium.com/tensorflow/fitting-larger-networks-into-memory-583e3c758ff9).

To enable gradient checkpointing in the [`Trainer`], pass the corresponding a flag to [`TrainingArguments`]:

```py
training_args = TrainingArguments(
    per_device_train_batch_size=1, gradient_accumulation_steps=4, gradient_checkpointing=True, **default_args
)
```

Alternatively, use 🤗 Accelerate - find the 🤗 Accelerate example [further in this guide](#using--accelerate). 

<Tip>

While gradient checkpointing may improve memory efficiency, it slows training by approximately 20%.

</Tip>

## Mixed precision training

**Mixed precision training** is a technique that aims to optimize the computational efficiency of training models by 
utilizing lower-precision numerical formats for certain variables. Traditionally, most models use 32-bit floating point 
precision (fp32 or float32) to represent and process variables. However, not all variables require this high precision 
level to achieve accurate results. By reducing the precision of certain variables to lower numerical formats like 16-bit 
floating point (fp16 or float16), we can speed up the computations. Because in this approach some computations are performed 
in half-precision, while some are still in full precision, the approach is called mixed precision training.

Most commonly mixed precision training is achieved by using fp16 (float16) data types, however, some GPU architectures 
(such as the Ampere architecture) offer bf16 and tf32 (CUDA internal data type) data types. Check 
out the [NVIDIA Blog](https://developer.nvidia.com/blog/accelerating-ai-training-with-tf32-tensor-cores/) to learn more about 
the differences between these data types.

### fp16

The main advantage of mixed precision training comes from saving the activations in half precision (fp16). 
Although the gradients are also computed in half precision they are converted back to full precision for the optimization 
step so no memory is saved here.


key: _node_content
string_value:
{"id_": "2aa3d8ea-2a0e-4930-bed7-54f3cf3c24ef", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "32235eb0-3add-4442-a981-82a445254d5b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3868e035-743f-4094-b2d8-be72f032853a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "76ae7d5f524559e6aa10ed38fe45e29e575a30a7a3346896a5587c9cf0c5b447", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9ea3b385-9014-40a4-b246-00794939066e", "node_type": "1", "metadata": {}, "hash": "4b3721e222332d2bf8efbb5456adc6108af4f418d419433dac15209598569963", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10245, "end_char_idx": 13584, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<Tip>

While gradient checkpointing may improve memory efficiency, it slows training by approximately 20%.

</Tip>

## Mixed precision training

**Mixed precision training** is a technique that aims to optimize the computational efficiency of training models by 
utilizing lower-precision numerical formats for certain variables. Traditionally, most models use 32-bit floating point 
precision (fp32 or float32) to represent and process variables. However, not all variables require this high precision 
level to achieve accurate results. By reducing the precision of certain variables to lower numerical formats like 16-bit 
floating point (fp16 or float16), we can speed up the computations. Because in this approach some computations are performed 
in half-precision, while some are still in full precision, the approach is called mixed precision training.

Most commonly mixed precision training is achieved by using fp16 (float16) data types, however, some GPU architectures 
(such as the Ampere architecture) offer bf16 and tf32 (CUDA internal data type) data types. Check 
out the [NVIDIA Blog](https://developer.nvidia.com/blog/accelerating-ai-training-with-tf32-tensor-cores/) to learn more about 
the differences between these data types.

### fp16

The main advantage of mixed precision training comes from saving the activations in half precision (fp16). 
Although the gradients are also computed in half precision they are converted back to full precision for the optimization 
step so no memory is saved here. 
While mixed precision training results in faster computations, it can also lead to more GPU memory being utilized, especially for small batch sizes.
This is because the model is now present on the GPU in both 16-bit and 32-bit precision (1.5x the original model on the GPU).

To enable mixed precision training, set the `fp16` flag to `True`:

```py
training_args = TrainingArguments(per_device_train_batch_size=4, fp16=True, **default_args)
```

If you prefer to use 🤗 Accelerate, find the 🤗 Accelerate example [further in this guide](#using--accelerate). 

### BF16

If you have access to an Ampere or newer hardware you can use bf16 for mixed precision training and evaluation. While 
bf16 has a worse precision than fp16, it has a much bigger dynamic range. In fp16 the biggest number you can have 
is `65535` and any number above that will result in an overflow. A bf16 number can be as large as `3.39e+38` (!) which 
is about the same as fp32 - because both have 8-bits used for the numerical range.

You can enable BF16 in the 🤗 Trainer with:

```python
training_args = TrainingArguments(bf16=True, **default_args)
```

### TF32

The Ampere hardware uses a magical data type called tf32. It has the same numerical range as fp32 (8-bits), but instead 
of 23 bits precision it has only 10 bits (same as fp16) and uses only 19 bits in total. It's "magical" in the sense that 
you can use the normal fp32 training and/or inference code and by enabling tf32 support you can get up to 3x throughput 
improvement. All you need to do is to add the following to your code:

```python
import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.allow_tf32 = True
```

CUDA will automatically switch to using tf32 instead of fp32 where possible, assuming that the used GPU is from the Ampere series.


key: _node_content
string_value:
{"id_": "9ea3b385-9014-40a4-b246-00794939066e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "32235eb0-3add-4442-a981-82a445254d5b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2aa3d8ea-2a0e-4930-bed7-54f3cf3c24ef", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "0d91525d66274f7f2595a19fb84ecb59add85380e23867b615bb5cbebb756536", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "861b2071-664d-4720-a6c5-95230e2b0f2a", "node_type": "1", "metadata": {}, "hash": "81275c55185587f37ff37d2e5261d369d5a345a48d0ba8b2a4a92abb5cee2638", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12329, "end_char_idx": 15435, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### BF16

If you have access to an Ampere or newer hardware you can use bf16 for mixed precision training and evaluation. While 
bf16 has a worse precision than fp16, it has a much bigger dynamic range. In fp16 the biggest number you can have 
is `65535` and any number above that will result in an overflow. A bf16 number can be as large as `3.39e+38` (!) which 
is about the same as fp32 - because both have 8-bits used for the numerical range.

You can enable BF16 in the 🤗 Trainer with:

```python
training_args = TrainingArguments(bf16=True, **default_args)
```

### TF32

The Ampere hardware uses a magical data type called tf32. It has the same numerical range as fp32 (8-bits), but instead 
of 23 bits precision it has only 10 bits (same as fp16) and uses only 19 bits in total. It's "magical" in the sense that 
you can use the normal fp32 training and/or inference code and by enabling tf32 support you can get up to 3x throughput 
improvement. All you need to do is to add the following to your code:

```python
import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.allow_tf32 = True
```

CUDA will automatically switch to using tf32 instead of fp32 where possible, assuming that the used GPU is from the Ampere series.

According to [NVIDIA research](https://developer.nvidia.com/blog/accelerating-ai-training-with-tf32-tensor-cores/), the 
majority of machine learning training workloads show the same perplexity and convergence with tf32 training as with fp32. 
If you're already using fp16 or bf16 mixed precision it may help with the throughput as well.

You can enable this mode in the 🤗 Trainer:

```python
TrainingArguments(tf32=True, **default_args)
```

<Tip>

tf32 can't be accessed directly via `tensor.to(dtype=torch.tf32)` because it is an internal CUDA data type. You need `torch>=1.7` to use tf32 data types.

</Tip>

For additional information on tf32 vs other precisions, please refer to the following benchmarks: 
[RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004390803) and
[A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1004543189).

## Flash Attention 2

You can speedup the training throughput by using Flash Attention 2 integration in transformers. Check out the appropriate section in the [single GPU section](./perf_infer_gpu_one#Flash-Attention-2) to learn more about how to load a model with Flash Attention 2 modules. 

## Optimizer choice

The most common optimizer used to train transformer models is Adam or AdamW (Adam with weight decay). Adam achieves 
good convergence by storing the rolling average of the previous gradients; however, it adds an additional memory 
footprint of the order of the number of model parameters. To remedy this, you can use an alternative optimizer. 
For example if you have [NVIDIA/apex](https://github.com/NVIDIA/apex) installed for NVIDIA GPUs, or [ROCmSoftwarePlatform/apex](https://github.com/ROCmSoftwarePlatform/apex) for AMD GPUs, `adamw_apex_fused` will give you the
fastest training experience among all supported AdamW optimizers.


key: _node_content
string_value:
{"id_": "861b2071-664d-4720-a6c5-95230e2b0f2a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "32235eb0-3add-4442-a981-82a445254d5b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9ea3b385-9014-40a4-b246-00794939066e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "5e641d90c68823367efcc89bd3e2fe44aba65a5b010c6a7f34ade38706c2d5d9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ab1b2358-a588-4cba-baaa-461262318cfe", "node_type": "1", "metadata": {}, "hash": "7ca15e68e49fa56cdac15503edeb2bc9ada2eba13b8c85e1385faa9a67fab7cd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13830, "end_char_idx": 16652, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If you're already using fp16 or bf16 mixed precision it may help with the throughput as well.

You can enable this mode in the 🤗 Trainer:

```python
TrainingArguments(tf32=True, **default_args)
```

<Tip>

tf32 can't be accessed directly via `tensor.to(dtype=torch.tf32)` because it is an internal CUDA data type. You need `torch>=1.7` to use tf32 data types.

</Tip>

For additional information on tf32 vs other precisions, please refer to the following benchmarks: 
[RTX-3090](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004390803) and
[A100](https://github.com/huggingface/transformers/issues/15026#issuecomment-1004543189).

## Flash Attention 2

You can speedup the training throughput by using Flash Attention 2 integration in transformers. Check out the appropriate section in the [single GPU section](./perf_infer_gpu_one#Flash-Attention-2) to learn more about how to load a model with Flash Attention 2 modules. 

## Optimizer choice

The most common optimizer used to train transformer models is Adam or AdamW (Adam with weight decay). Adam achieves 
good convergence by storing the rolling average of the previous gradients; however, it adds an additional memory 
footprint of the order of the number of model parameters. To remedy this, you can use an alternative optimizer. 
For example if you have [NVIDIA/apex](https://github.com/NVIDIA/apex) installed for NVIDIA GPUs, or [ROCmSoftwarePlatform/apex](https://github.com/ROCmSoftwarePlatform/apex) for AMD GPUs, `adamw_apex_fused` will give you the
fastest training experience among all supported AdamW optimizers.

[`Trainer`] integrates a variety of optimizers that can be used out of box: `adamw_hf`, `adamw_torch`, `adamw_torch_fused`, 
`adamw_apex_fused`, `adamw_anyprecision`, `adafactor`, or `adamw_bnb_8bit`. More optimizers can be plugged in via a third-party implementation.

Let's take a closer look at two alternatives to AdamW optimizer:
1. `adafactor` which is available in [`Trainer`]
2. `adamw_bnb_8bit` is also available in Trainer, but a third-party integration is provided below for demonstration.

For comparison, for a 3B-parameter model, like “google-t5/t5-3b”: 
* A standard AdamW optimizer will need 24GB of GPU memory because it uses 8 bytes for each parameter (8*3 => 24GB)
* Adafactor optimizer will need more than 12GB. It uses slightly more than 4 bytes for each parameter, so 4*3 and then some extra.
* 8bit BNB quantized optimizer will use only (2*3) 6GB if all optimizer states are quantized.

### Adafactor

Adafactor doesn't store rolling averages for each element in weight matrices. Instead, it keeps aggregated information 
(sums of rolling averages row- and column-wise), significantly reducing its footprint. However, compared to Adam, 
Adafactor may have slower convergence in certain cases.


key: _node_content
string_value:
{"id_": "ab1b2358-a588-4cba-baaa-461262318cfe", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "32235eb0-3add-4442-a981-82a445254d5b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "861b2071-664d-4720-a6c5-95230e2b0f2a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "c2f23574e66f817044add77084db00decac9de5c09583df22bd1f5cc5dd6c485", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "245247c0-8b05-4aa7-8617-22eeafa0ab2e", "node_type": "1", "metadata": {}, "hash": "2b16bfcf163efe9d449daea34e45248b6aec8a573e35473da48374c8fdff8925", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15145, "end_char_idx": 18226, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For example if you have [NVIDIA/apex](https://github.com/NVIDIA/apex) installed for NVIDIA GPUs, or [ROCmSoftwarePlatform/apex](https://github.com/ROCmSoftwarePlatform/apex) for AMD GPUs, `adamw_apex_fused` will give you the
fastest training experience among all supported AdamW optimizers.

[`Trainer`] integrates a variety of optimizers that can be used out of box: `adamw_hf`, `adamw_torch`, `adamw_torch_fused`, 
`adamw_apex_fused`, `adamw_anyprecision`, `adafactor`, or `adamw_bnb_8bit`. More optimizers can be plugged in via a third-party implementation.

Let's take a closer look at two alternatives to AdamW optimizer:
1. `adafactor` which is available in [`Trainer`]
2. `adamw_bnb_8bit` is also available in Trainer, but a third-party integration is provided below for demonstration.

For comparison, for a 3B-parameter model, like “google-t5/t5-3b”: 
* A standard AdamW optimizer will need 24GB of GPU memory because it uses 8 bytes for each parameter (8*3 => 24GB)
* Adafactor optimizer will need more than 12GB. It uses slightly more than 4 bytes for each parameter, so 4*3 and then some extra.
* 8bit BNB quantized optimizer will use only (2*3) 6GB if all optimizer states are quantized.

### Adafactor

Adafactor doesn't store rolling averages for each element in weight matrices. Instead, it keeps aggregated information 
(sums of rolling averages row- and column-wise), significantly reducing its footprint. However, compared to Adam, 
Adafactor may have slower convergence in certain cases.

You can switch to Adafactor by setting `optim="adafactor"` in [`TrainingArguments`]:

```py
training_args = TrainingArguments(per_device_train_batch_size=4, optim="adafactor", **default_args)
```

Combined with other approaches (gradient accumulation, gradient checkpointing, and mixed precision training) 
you can notice up to 3x improvement while maintaining the throughput! However, as mentioned before, the convergence of 
Adafactor can be worse than Adam. 

### 8-bit Adam

Instead of aggregating optimizer states like Adafactor, 8-bit Adam keeps the full state and quantizes it. Quantization 
means that it stores the state with lower precision and dequantizes it only for the optimization. This is similar to the 
idea behind mixed precision training.

To use `adamw_bnb_8bit`, you simply need to set `optim="adamw_bnb_8bit"` in [`TrainingArguments`]:

```py
training_args = TrainingArguments(per_device_train_batch_size=4, optim="adamw_bnb_8bit", **default_args)
```

However, we can also use a third-party implementation of the 8-bit optimizer for demonstration purposes to see how that can be integrated.

First, follow the installation guide in the GitHub [repo](https://github.com/TimDettmers/bitsandbytes) to install the `bitsandbytes` library 
that implements the 8-bit Adam optimizer.

Next you need to initialize the optimizer. This involves two steps: 
* First, group the model's parameters into two groups - one where weight decay should be applied, and the other one where it should not. Usually, biases and layer norm parameters are not weight decayed.


key: _node_content
string_value:
{"id_": "245247c0-8b05-4aa7-8617-22eeafa0ab2e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "32235eb0-3add-4442-a981-82a445254d5b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ab1b2358-a588-4cba-baaa-461262318cfe", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "0de4b966e5ea07a4c6f7b2c193a2e4f31c73ee961f8f264212076c31c2d88f66", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8e1a536f-145e-4d06-b222-091472956e31", "node_type": "1", "metadata": {}, "hash": "b6819d4428a3491e37fd7e89bffdfcedb3ceda16e2c099ab3eb2d29af1aa8062", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16569, "end_char_idx": 19786, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, compared to Adam, 
Adafactor may have slower convergence in certain cases.

You can switch to Adafactor by setting `optim="adafactor"` in [`TrainingArguments`]:

```py
training_args = TrainingArguments(per_device_train_batch_size=4, optim="adafactor", **default_args)
```

Combined with other approaches (gradient accumulation, gradient checkpointing, and mixed precision training) 
you can notice up to 3x improvement while maintaining the throughput! However, as mentioned before, the convergence of 
Adafactor can be worse than Adam. 

### 8-bit Adam

Instead of aggregating optimizer states like Adafactor, 8-bit Adam keeps the full state and quantizes it. Quantization 
means that it stores the state with lower precision and dequantizes it only for the optimization. This is similar to the 
idea behind mixed precision training.

To use `adamw_bnb_8bit`, you simply need to set `optim="adamw_bnb_8bit"` in [`TrainingArguments`]:

```py
training_args = TrainingArguments(per_device_train_batch_size=4, optim="adamw_bnb_8bit", **default_args)
```

However, we can also use a third-party implementation of the 8-bit optimizer for demonstration purposes to see how that can be integrated.

First, follow the installation guide in the GitHub [repo](https://github.com/TimDettmers/bitsandbytes) to install the `bitsandbytes` library 
that implements the 8-bit Adam optimizer.

Next you need to initialize the optimizer. This involves two steps: 
* First, group the model's parameters into two groups - one where weight decay should be applied, and the other one where it should not. Usually, biases and layer norm parameters are not weight decayed. 
* Then do some argument housekeeping to use the same parameters as the previously used AdamW optimizer.

```py
import bitsandbytes as bnb
from torch import nn
from transformers.trainer_pt_utils import get_parameter_names

training_args = TrainingArguments(per_device_train_batch_size=4, **default_args)

decay_parameters = get_parameter_names(model, [nn.LayerNorm])
decay_parameters = [name for name in decay_parameters if "bias" not in name]
optimizer_grouped_parameters = [
    {
        "params": [p for n, p in model.named_parameters() if n in decay_parameters],
        "weight_decay": training_args.weight_decay,
    },
    {
        "params": [p for n, p in model.named_parameters() if n not in decay_parameters],
        "weight_decay": 0.0,
    },
]

optimizer_kwargs = {
    "betas": (training_args.adam_beta1, training_args.adam_beta2),
    "eps": training_args.adam_epsilon,
}
optimizer_kwargs["lr"] = training_args.learning_rate
adam_bnb_optim = bnb.optim.Adam8bit(
    optimizer_grouped_parameters,
    betas=(training_args.adam_beta1, training_args.adam_beta2),
    eps=training_args.adam_epsilon,
    lr=training_args.learning_rate,
)
```

Finally, pass the custom optimizer as an argument to the `Trainer`:

```py
trainer = Trainer(model=model, args=training_args, train_dataset=ds, optimizers=(adam_bnb_optim, None))
```

Combined with other approaches (gradient accumulation, gradient checkpointing, and mixed precision training), 
you can expect to get about a 3x memory improvement and even slightly higher throughput as using Adafactor.


key: _node_content
string_value:
{"id_": "8e1a536f-145e-4d06-b222-091472956e31", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "32235eb0-3add-4442-a981-82a445254d5b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "245247c0-8b05-4aa7-8617-22eeafa0ab2e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "867185769a723146e497c74818a853319704e6b0b6fb8de9956f8efebd9330f4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f20e9e76-94a5-49d1-b4db-b07e65a62ea3", "node_type": "1", "metadata": {}, "hash": "d2c34d9f7fc4235098958035c9c54c0fd03d75043ca2f9e6a6c3eccbd3d34ad4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18161, "end_char_idx": 21625, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Usually, biases and layer norm parameters are not weight decayed. 
* Then do some argument housekeeping to use the same parameters as the previously used AdamW optimizer.

```py
import bitsandbytes as bnb
from torch import nn
from transformers.trainer_pt_utils import get_parameter_names

training_args = TrainingArguments(per_device_train_batch_size=4, **default_args)

decay_parameters = get_parameter_names(model, [nn.LayerNorm])
decay_parameters = [name for name in decay_parameters if "bias" not in name]
optimizer_grouped_parameters = [
    {
        "params": [p for n, p in model.named_parameters() if n in decay_parameters],
        "weight_decay": training_args.weight_decay,
    },
    {
        "params": [p for n, p in model.named_parameters() if n not in decay_parameters],
        "weight_decay": 0.0,
    },
]

optimizer_kwargs = {
    "betas": (training_args.adam_beta1, training_args.adam_beta2),
    "eps": training_args.adam_epsilon,
}
optimizer_kwargs["lr"] = training_args.learning_rate
adam_bnb_optim = bnb.optim.Adam8bit(
    optimizer_grouped_parameters,
    betas=(training_args.adam_beta1, training_args.adam_beta2),
    eps=training_args.adam_epsilon,
    lr=training_args.learning_rate,
)
```

Finally, pass the custom optimizer as an argument to the `Trainer`:

```py
trainer = Trainer(model=model, args=training_args, train_dataset=ds, optimizers=(adam_bnb_optim, None))
```

Combined with other approaches (gradient accumulation, gradient checkpointing, and mixed precision training), 
you can expect to get about a 3x memory improvement and even slightly higher throughput as using Adafactor. 

### multi_tensor

pytorch-nightly introduced `torch.optim._multi_tensor` which should significantly speed up the optimizers for situations 
with lots of small feature tensors. It should eventually become the default, but if you want to experiment with it sooner, take a look at this GitHub [issue](https://github.com/huggingface/transformers/issues/9965).

## Data preloading

One of the important requirements to reach great training speed is the ability to feed the GPU at the maximum speed it 
can handle. By default, everything happens in the main process, and it might not be able to read the data from disk fast 
enough, and thus create a bottleneck, leading to GPU under-utilization. Configure the following arguments to reduce the bottleneck:

- `DataLoader(pin_memory=True, ...)` - ensures the data gets preloaded into the pinned memory on CPU and typically leads to much faster transfers from CPU to GPU memory.
- `DataLoader(num_workers=4, ...)` - spawn several workers to preload data faster. During training, watch the GPU utilization stats; if it's far from 100%, experiment with increasing the number of workers. Of course, the problem could be elsewhere, so many workers won't necessarily lead to better performance.

When using [`Trainer`], the corresponding [`TrainingArguments`] are: `dataloader_pin_memory` (`True` by default), and `dataloader_num_workers` (defaults to `0`).

## DeepSpeed ZeRO

DeepSpeed is an open-source deep learning optimization library that is integrated with 🤗 Transformers and 🤗 Accelerate.
It provides a wide range of features and optimizations designed to improve the efficiency and scalability of large-scale 
deep learning training.

If your model fits onto a single GPU and you have enough space to fit a small batch size, you don't need to use DeepSpeed
as it'll only slow things down.


key: _node_content
string_value:
{"id_": "f20e9e76-94a5-49d1-b4db-b07e65a62ea3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "32235eb0-3add-4442-a981-82a445254d5b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8e1a536f-145e-4d06-b222-091472956e31", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "0795a2b7a461aef9efd6aa0f189ad4b1b7210a613bcd7cc881f5f75cd427391e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "57c275b6-18bb-4500-8c27-3b2745fd7799", "node_type": "1", "metadata": {}, "hash": "ea54760135da62d0f528915ec57b120139cd3e480751c7e3ad3f1c4d67ce63ee", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19789, "end_char_idx": 23266, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### multi_tensor

pytorch-nightly introduced `torch.optim._multi_tensor` which should significantly speed up the optimizers for situations 
with lots of small feature tensors. It should eventually become the default, but if you want to experiment with it sooner, take a look at this GitHub [issue](https://github.com/huggingface/transformers/issues/9965).

## Data preloading

One of the important requirements to reach great training speed is the ability to feed the GPU at the maximum speed it 
can handle. By default, everything happens in the main process, and it might not be able to read the data from disk fast 
enough, and thus create a bottleneck, leading to GPU under-utilization. Configure the following arguments to reduce the bottleneck:

- `DataLoader(pin_memory=True, ...)` - ensures the data gets preloaded into the pinned memory on CPU and typically leads to much faster transfers from CPU to GPU memory.
- `DataLoader(num_workers=4, ...)` - spawn several workers to preload data faster. During training, watch the GPU utilization stats; if it's far from 100%, experiment with increasing the number of workers. Of course, the problem could be elsewhere, so many workers won't necessarily lead to better performance.

When using [`Trainer`], the corresponding [`TrainingArguments`] are: `dataloader_pin_memory` (`True` by default), and `dataloader_num_workers` (defaults to `0`).

## DeepSpeed ZeRO

DeepSpeed is an open-source deep learning optimization library that is integrated with 🤗 Transformers and 🤗 Accelerate.
It provides a wide range of features and optimizations designed to improve the efficiency and scalability of large-scale 
deep learning training.

If your model fits onto a single GPU and you have enough space to fit a small batch size, you don't need to use DeepSpeed
as it'll only slow things down. However, if the model doesn't fit onto a single GPU or you can't fit a small batch, you can 
leverage DeepSpeed ZeRO + CPU Offload, or NVMe Offload for much larger models. In this case, you need to separately
[install the library](main_classes/deepspeed#installation), then follow one of the guides to create a configuration file 
and launch DeepSpeed: 
 
* For an in-depth guide on DeepSpeed integration with [`Trainer`], review [the corresponding documentation](main_classes/deepspeed), specifically the 
[section for a single GPU](main_classes/deepspeed#deployment-with-one-gpu). Some adjustments are required to use DeepSpeed in a notebook; please take a look at the [corresponding guide](main_classes/deepspeed#deployment-in-notebooks).
* If you prefer to use 🤗 Accelerate, refer to [🤗 Accelerate DeepSpeed guide](https://huggingface.co/docs/accelerate/en/usage_guides/deepspeed).

## Using torch.compile

PyTorch 2.0 introduced a new compile function that doesn't require any modification to existing PyTorch code but can 
optimize your code by adding a single line of code: `model = torch.compile(model)`.

If using [`Trainer`], you only need `to` pass the `torch_compile` option in the [`TrainingArguments`]: 

```python
training_args = TrainingArguments(torch_compile=True, **default_args)
```

`torch.compile` uses Python's frame evaluation API to automatically create a graph from existing PyTorch programs. After 
capturing the graph, different backends can be deployed to lower the graph to an optimized engine. 
You can find more details and benchmarks in [PyTorch documentation](https://pytorch.org/get-started/pytorch-2.0/).


key: _node_content
string_value:
{"id_": "57c275b6-18bb-4500-8c27-3b2745fd7799", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "32235eb0-3add-4442-a981-82a445254d5b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f20e9e76-94a5-49d1-b4db-b07e65a62ea3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "8bef8b7c7f10c387163f5db918068b2175903a1d23eee374c1a50e218808e586", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0ec42b3e-cbca-4e76-8753-a7bff3211124", "node_type": "1", "metadata": {}, "hash": "60e50f119917ab4c312c0b4a8936ac35b4b376f68ea1811fd22322a4238cfe77", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21626, "end_char_idx": 24671, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, if the model doesn't fit onto a single GPU or you can't fit a small batch, you can 
leverage DeepSpeed ZeRO + CPU Offload, or NVMe Offload for much larger models. In this case, you need to separately
[install the library](main_classes/deepspeed#installation), then follow one of the guides to create a configuration file 
and launch DeepSpeed: 
 
* For an in-depth guide on DeepSpeed integration with [`Trainer`], review [the corresponding documentation](main_classes/deepspeed), specifically the 
[section for a single GPU](main_classes/deepspeed#deployment-with-one-gpu). Some adjustments are required to use DeepSpeed in a notebook; please take a look at the [corresponding guide](main_classes/deepspeed#deployment-in-notebooks).
* If you prefer to use 🤗 Accelerate, refer to [🤗 Accelerate DeepSpeed guide](https://huggingface.co/docs/accelerate/en/usage_guides/deepspeed).

## Using torch.compile

PyTorch 2.0 introduced a new compile function that doesn't require any modification to existing PyTorch code but can 
optimize your code by adding a single line of code: `model = torch.compile(model)`.

If using [`Trainer`], you only need `to` pass the `torch_compile` option in the [`TrainingArguments`]: 

```python
training_args = TrainingArguments(torch_compile=True, **default_args)
```

`torch.compile` uses Python's frame evaluation API to automatically create a graph from existing PyTorch programs. After 
capturing the graph, different backends can be deployed to lower the graph to an optimized engine. 
You can find more details and benchmarks in [PyTorch documentation](https://pytorch.org/get-started/pytorch-2.0/).

`torch.compile` has a growing list of backends, which can be found in by calling `torchdynamo.list_backends()`, each of which with its optional dependencies.

Choose which backend to use by specifying it via `torch_compile_backend` in the [`TrainingArguments`].  Some of the most commonly used backends are:

**Debugging backends**:
* `dynamo.optimize("eager")` - Uses PyTorch to run the extracted GraphModule. This is quite useful in debugging TorchDynamo issues.
* `dynamo.optimize("aot_eager")` - Uses AotAutograd with no compiler, i.e, just using PyTorch eager for the AotAutograd's extracted forward and backward graphs. This is useful for debugging, and unlikely to give speedups.

**Training & inference backends**:
* `dynamo.optimize("inductor")` - Uses TorchInductor backend with AotAutograd and cudagraphs by leveraging codegened Triton kernels  [Read more](https://dev-discuss.pytorch.org/t/torchinductor-a-pytorch-native-compiler-with-define-by-run-ir-and-symbolic-shapes/747)
* `dynamo.optimize("nvfuser")` -  nvFuser with TorchScript. [Read more](https://dev-discuss.pytorch.org/t/tracing-with-primitives-update-1-nvfuser-and-its-primitives/593)
* `dynamo.optimize("aot_nvfuser")` -  nvFuser with AotAutograd. [Read more](https://dev-discuss.pytorch.org/t/tracing-with-primitives-update-1-nvfuser-and-its-primitives/593)
* `dynamo.optimize("aot_cudagraphs")` - cudagraphs with AotAutograd.


key: _node_content
string_value:
{"id_": "0ec42b3e-cbca-4e76-8753-a7bff3211124", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "32235eb0-3add-4442-a981-82a445254d5b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "57c275b6-18bb-4500-8c27-3b2745fd7799", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "20f4a91b07fc6961f2f362b506ea632fed596aa20940a048a35ac6ddc44e84f2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "20560969-0b9c-41c5-bc40-6ada9ed8d18e", "node_type": "1", "metadata": {}, "hash": "29337587d7fc986bff5cb8345aaca667658278785ffe84c1e075a328b736fd5d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23268, "end_char_idx": 26002, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
`torch.compile` has a growing list of backends, which can be found in by calling `torchdynamo.list_backends()`, each of which with its optional dependencies.

Choose which backend to use by specifying it via `torch_compile_backend` in the [`TrainingArguments`].  Some of the most commonly used backends are:

**Debugging backends**:
* `dynamo.optimize("eager")` - Uses PyTorch to run the extracted GraphModule. This is quite useful in debugging TorchDynamo issues.
* `dynamo.optimize("aot_eager")` - Uses AotAutograd with no compiler, i.e, just using PyTorch eager for the AotAutograd's extracted forward and backward graphs. This is useful for debugging, and unlikely to give speedups.

**Training & inference backends**:
* `dynamo.optimize("inductor")` - Uses TorchInductor backend with AotAutograd and cudagraphs by leveraging codegened Triton kernels  [Read more](https://dev-discuss.pytorch.org/t/torchinductor-a-pytorch-native-compiler-with-define-by-run-ir-and-symbolic-shapes/747)
* `dynamo.optimize("nvfuser")` -  nvFuser with TorchScript. [Read more](https://dev-discuss.pytorch.org/t/tracing-with-primitives-update-1-nvfuser-and-its-primitives/593)
* `dynamo.optimize("aot_nvfuser")` -  nvFuser with AotAutograd. [Read more](https://dev-discuss.pytorch.org/t/tracing-with-primitives-update-1-nvfuser-and-its-primitives/593)
* `dynamo.optimize("aot_cudagraphs")` - cudagraphs with AotAutograd. [Read more](https://github.com/pytorch/torchdynamo/pull/757)

**Inference-only backend**s:
* `dynamo.optimize("ofi")` -  Uses Torchscript optimize_for_inference.  [Read more](https://pytorch.org/docs/stable/generated/torch.jit.optimize_for_inference.html)
* `dynamo.optimize("fx2trt")` -  Uses NVIDIA TensorRT for inference optimizations.  [Read more](https://pytorch.org/TensorRT/tutorials/getting_started_with_fx_path.html)
* `dynamo.optimize("onnxrt")` -  Uses ONNXRT for inference on CPU/GPU.  [Read more](https://onnxruntime.ai/)
* `dynamo.optimize("ipex")` -  Uses IPEX for inference on CPU.  [Read more](https://github.com/intel/intel-extension-for-pytorch)

For an example of using `torch.compile` with 🤗 Transformers, check out this [blog post on fine-tuning a BERT model for Text Classification using the newest PyTorch 2.0 features](https://www.philschmid.de/getting-started-pytorch-2-0-transformers)

## Using 🤗 PEFT

[Parameter-Efficient Fine Tuning (PEFT)](https://huggingface.co/blog/peft) methods freeze the pretrained model parameters during fine-tuning and add a small number of trainable parameters (the adapters) on top of it.

As a result the [memory associated to the optimizer states and gradients](https://huggingface.co/docs/transformers/model_memory_anatomy#anatomy-of-models-memory) are greatly reduced.


key: _node_content
string_value:
{"id_": "20560969-0b9c-41c5-bc40-6ada9ed8d18e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "32235eb0-3add-4442-a981-82a445254d5b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0ec42b3e-cbca-4e76-8753-a7bff3211124", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "c339ce2c6460ee94f2972582b759f9187a8ae72ff7713dd1740ed08a51d5cafe", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "71b90abc-8637-4c83-9ede-7e087e19476c", "node_type": "1", "metadata": {}, "hash": "a870ec7023092bed4b349be6dea9e3eaa33de491146a52856b3e2a69fe6efb3f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24672, "end_char_idx": 27057, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Read more](https://github.com/pytorch/torchdynamo/pull/757)

**Inference-only backend**s:
* `dynamo.optimize("ofi")` -  Uses Torchscript optimize_for_inference.  [Read more](https://pytorch.org/docs/stable/generated/torch.jit.optimize_for_inference.html)
* `dynamo.optimize("fx2trt")` -  Uses NVIDIA TensorRT for inference optimizations.  [Read more](https://pytorch.org/TensorRT/tutorials/getting_started_with_fx_path.html)
* `dynamo.optimize("onnxrt")` -  Uses ONNXRT for inference on CPU/GPU.  [Read more](https://onnxruntime.ai/)
* `dynamo.optimize("ipex")` -  Uses IPEX for inference on CPU.  [Read more](https://github.com/intel/intel-extension-for-pytorch)

For an example of using `torch.compile` with 🤗 Transformers, check out this [blog post on fine-tuning a BERT model for Text Classification using the newest PyTorch 2.0 features](https://www.philschmid.de/getting-started-pytorch-2-0-transformers)

## Using 🤗 PEFT

[Parameter-Efficient Fine Tuning (PEFT)](https://huggingface.co/blog/peft) methods freeze the pretrained model parameters during fine-tuning and add a small number of trainable parameters (the adapters) on top of it.

As a result the [memory associated to the optimizer states and gradients](https://huggingface.co/docs/transformers/model_memory_anatomy#anatomy-of-models-memory) are greatly reduced.

For example with a vanilla AdamW, the memory requirement for the optimizer state would be:
* fp32 copy of parameters: 4 bytes/param
* Momentum: 4 bytes/param
* Variance: 4 bytes/param

Suppose a model with 7B parameters and 200 millions parameters injected with [Low Rank Adapters](https://huggingface.co/docs/peft/conceptual_guides/lora).

The memory requirement for the optimizer state of the plain model would be 12 * 7 = 84 GB (assuming 7B trainable parameters).

Adding Lora increases slightly the memory associated to the model weights and substantially decreases memory requirement for the optimizer state to 12 * 0.2 = 2.4GB.

Read more about PEFT and its detailed usage in [the PEFT documentation](https://huggingface.co/docs/peft/) or [PEFT repository](https://github.com/huggingface/peft).

## Using 🤗 Accelerate

With [🤗 Accelerate](https://huggingface.co/docs/accelerate/index) you can use the above methods while gaining full 
control over the training loop and can essentially write the loop in pure PyTorch with some minor modifications.


key: _node_content
string_value:
{"id_": "71b90abc-8637-4c83-9ede-7e087e19476c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "32235eb0-3add-4442-a981-82a445254d5b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "20560969-0b9c-41c5-bc40-6ada9ed8d18e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "d1a10e175b4b787eebec746b077edd7abc855e8d46ec8ac446c0820ec020c3fe", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3ba2c6e0-c19a-4c3a-9098-453fd20451e1", "node_type": "1", "metadata": {}, "hash": "ee7f26c7057418433d67e084828b7100097c0bf844c6c1e2ab0e11da48999ed7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 25820, "end_char_idx": 29183, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
As a result the [memory associated to the optimizer states and gradients](https://huggingface.co/docs/transformers/model_memory_anatomy#anatomy-of-models-memory) are greatly reduced.

For example with a vanilla AdamW, the memory requirement for the optimizer state would be:
* fp32 copy of parameters: 4 bytes/param
* Momentum: 4 bytes/param
* Variance: 4 bytes/param

Suppose a model with 7B parameters and 200 millions parameters injected with [Low Rank Adapters](https://huggingface.co/docs/peft/conceptual_guides/lora).

The memory requirement for the optimizer state of the plain model would be 12 * 7 = 84 GB (assuming 7B trainable parameters).

Adding Lora increases slightly the memory associated to the model weights and substantially decreases memory requirement for the optimizer state to 12 * 0.2 = 2.4GB.

Read more about PEFT and its detailed usage in [the PEFT documentation](https://huggingface.co/docs/peft/) or [PEFT repository](https://github.com/huggingface/peft).

## Using 🤗 Accelerate

With [🤗 Accelerate](https://huggingface.co/docs/accelerate/index) you can use the above methods while gaining full 
control over the training loop and can essentially write the loop in pure PyTorch with some minor modifications. 

Suppose you have combined the methods in the [`TrainingArguments`] like so:

```py
training_args = TrainingArguments(
    per_device_train_batch_size=1,
    gradient_accumulation_steps=4,
    gradient_checkpointing=True,
    fp16=True,
    **default_args,
)
```

The full example training loop with 🤗 Accelerate is only a handful of lines of code long:

```py
from accelerate import Accelerator
from torch.utils.data.dataloader import DataLoader

dataloader = DataLoader(ds, batch_size=training_args.per_device_train_batch_size)

if training_args.gradient_checkpointing:
    model.gradient_checkpointing_enable()

accelerator = Accelerator(fp16=training_args.fp16)
model, optimizer, dataloader = accelerator.prepare(model, adam_bnb_optim, dataloader)

model.train()
for step, batch in enumerate(dataloader, start=1):
    loss = model(**batch).loss
    loss = loss / training_args.gradient_accumulation_steps
    accelerator.backward(loss)
    if step % training_args.gradient_accumulation_steps == 0:
        optimizer.step()
        optimizer.zero_grad()
```

First we wrap the dataset in a [`DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader). 
Then we can enable gradient checkpointing by calling the model's [`~PreTrainedModel.gradient_checkpointing_enable`] method. 
When we initialize the [`Accelerator`](https://huggingface.co/docs/accelerate/package_reference/accelerator#accelerate.Accelerator) 
we can specify if we want to use mixed precision training and it will take care of it for us in the [`prepare`] call. 
During the [`prepare`](https://huggingface.co/docs/accelerate/package_reference/accelerator#accelerate.Accelerator.prepare) 
call the dataloader will also be distributed across workers should we use multiple GPUs. We use the same [8-bit optimizer](#8-bit-adam) from the earlier example.

Finally, we can add the main training loop. Note that the `backward` call is handled by 🤗 Accelerate. We can also see
how gradient accumulation works: we normalize the loss, so we get the average at the end of accumulation and once we have 
enough steps we run the optimization.


key: _node_content
string_value:
{"id_": "3ba2c6e0-c19a-4c3a-9098-453fd20451e1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "32235eb0-3add-4442-a981-82a445254d5b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "71b90abc-8637-4c83-9ede-7e087e19476c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "ed6ad165a90d4c0e6f7c4c0e96a43ccb6c61e25dfd239cf83095f87cdc1d21de", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2050127c-296e-4bd6-8594-50cc2cef38fc", "node_type": "1", "metadata": {}, "hash": "6dd543a9bd13a10453af50206de91af489c86ac8bbb09063ef7506921b74210d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 28240, "end_char_idx": 31559, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Then we can enable gradient checkpointing by calling the model's [`~PreTrainedModel.gradient_checkpointing_enable`] method. 
When we initialize the [`Accelerator`](https://huggingface.co/docs/accelerate/package_reference/accelerator#accelerate.Accelerator) 
we can specify if we want to use mixed precision training and it will take care of it for us in the [`prepare`] call. 
During the [`prepare`](https://huggingface.co/docs/accelerate/package_reference/accelerator#accelerate.Accelerator.prepare) 
call the dataloader will also be distributed across workers should we use multiple GPUs. We use the same [8-bit optimizer](#8-bit-adam) from the earlier example.

Finally, we can add the main training loop. Note that the `backward` call is handled by 🤗 Accelerate. We can also see
how gradient accumulation works: we normalize the loss, so we get the average at the end of accumulation and once we have 
enough steps we run the optimization. 

Implementing these optimization techniques with 🤗 Accelerate only takes a handful of lines of code and comes with the 
benefit of more flexibility in the training loop. For a full documentation of all features have a look at the 
[Accelerate documentation](https://huggingface.co/docs/accelerate/index).


## Efficient Software Prebuilds

PyTorch's [pip and conda builds](https://pytorch.org/get-started/locally/#start-locally) come prebuilt with the cuda toolkit 
which is enough to run PyTorch, but it is insufficient if you need to build cuda extensions.

At times, additional efforts may be required to pre-build some components. For instance, if you're using libraries like `apex` that 
don't come pre-compiled. In other situations figuring out how to install the right cuda toolkit system-wide can be complicated. 
To address these scenarios PyTorch and NVIDIA released a new version of NGC docker container which already comes with 
everything prebuilt. You just need to install your programs on it, and it will run out of the box.

This approach is also useful if you want to tweak the pytorch source and/or make a new customized build.
To find the docker image version you want start [with PyTorch release notes](https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/), 
choose one of the latest monthly releases. Go into the release's notes for the desired release, check that the environment's 
components are matching your needs (including NVIDIA Driver requirements!) and then at the very top of that document go 
to the corresponding NGC page. If for some reason you get lost, here is [the index of all PyTorch NGC images](https://ngc.nvidia.com/catalog/containers/nvidia:pytorch).

Next follow the instructions to download and deploy the docker image.

## Mixture of Experts

Some recent papers reported a 4-5x training speedup and a faster inference by integrating
Mixture of Experts (MoE) into the Transformer models.

Since it has been discovered that more parameters lead to better performance, this technique allows to increase the 
number of parameters by an order of magnitude without increasing training costs.

In this approach every other FFN layer is replaced with a MoE Layer which consists of many experts, with a gated function 
that trains each expert in a balanced way depending on the input token's position in a sequence.

!


key: _node_content
string_value:
{"id_": "2050127c-296e-4bd6-8594-50cc2cef38fc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "32235eb0-3add-4442-a981-82a445254d5b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3ba2c6e0-c19a-4c3a-9098-453fd20451e1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "708fa0d836cfa698919509d6eca0915033d63a9ec393c839f584d067cb263011", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7835e437-a55d-4091-8884-070bc205f005", "node_type": "1", "metadata": {}, "hash": "fad7fc434cf82aed64403587dcfba7a85f85afd4baaf54269dc232e446980511", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 29745, "end_char_idx": 32427, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
At times, additional efforts may be required to pre-build some components. For instance, if you're using libraries like `apex` that 
don't come pre-compiled. In other situations figuring out how to install the right cuda toolkit system-wide can be complicated. 
To address these scenarios PyTorch and NVIDIA released a new version of NGC docker container which already comes with 
everything prebuilt. You just need to install your programs on it, and it will run out of the box.

This approach is also useful if you want to tweak the pytorch source and/or make a new customized build.
To find the docker image version you want start [with PyTorch release notes](https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/), 
choose one of the latest monthly releases. Go into the release's notes for the desired release, check that the environment's 
components are matching your needs (including NVIDIA Driver requirements!) and then at the very top of that document go 
to the corresponding NGC page. If for some reason you get lost, here is [the index of all PyTorch NGC images](https://ngc.nvidia.com/catalog/containers/nvidia:pytorch).

Next follow the instructions to download and deploy the docker image.

## Mixture of Experts

Some recent papers reported a 4-5x training speedup and a faster inference by integrating
Mixture of Experts (MoE) into the Transformer models.

Since it has been discovered that more parameters lead to better performance, this technique allows to increase the 
number of parameters by an order of magnitude without increasing training costs.

In this approach every other FFN layer is replaced with a MoE Layer which consists of many experts, with a gated function 
that trains each expert in a balanced way depending on the input token's position in a sequence.

![MoE Transformer 2x block](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/perf-moe-transformer.png)

(source: [GLAM](https://ai.googleblog.com/2021/12/more-efficient-in-context-learning-with.html))

You can find exhaustive details and comparison tables in the papers listed at the end of this section.

The main drawback of this approach is that it requires staggering amounts of GPU memory - almost an order of magnitude 
larger than its dense equivalent. Various distillation and approaches are proposed to how to overcome the much higher memory requirements.

There is direct trade-off though, you can use just a few experts with a 2-3x smaller base model instead of dozens or 
hundreds experts leading to a 5x smaller model and thus increase the training speed moderately while increasing the 
memory requirements moderately as well.


key: _node_content
string_value:
{"id_": "7835e437-a55d-4091-8884-070bc205f005", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "32235eb0-3add-4442-a981-82a445254d5b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2050127c-296e-4bd6-8594-50cc2cef38fc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "65f9d35fca1713302e6bd6ddb84dba498398f6b6189b88891f1b507275c884bb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fbd9d825-c7b5-422c-b347-f43042c6f25b", "node_type": "1", "metadata": {}, "hash": "d1685d545f4b828a3ebabcad8413e0c9f89e6f169ba9d67a2e692d9127d8ef81", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30525, "end_char_idx": 33971, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Go into the release's notes for the desired release, check that the environment's 
components are matching your needs (including NVIDIA Driver requirements!) and then at the very top of that document go 
to the corresponding NGC page. If for some reason you get lost, here is [the index of all PyTorch NGC images](https://ngc.nvidia.com/catalog/containers/nvidia:pytorch).

Next follow the instructions to download and deploy the docker image.

## Mixture of Experts

Some recent papers reported a 4-5x training speedup and a faster inference by integrating
Mixture of Experts (MoE) into the Transformer models.

Since it has been discovered that more parameters lead to better performance, this technique allows to increase the 
number of parameters by an order of magnitude without increasing training costs.

In this approach every other FFN layer is replaced with a MoE Layer which consists of many experts, with a gated function 
that trains each expert in a balanced way depending on the input token's position in a sequence.

![MoE Transformer 2x block](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/perf-moe-transformer.png)

(source: [GLAM](https://ai.googleblog.com/2021/12/more-efficient-in-context-learning-with.html))

You can find exhaustive details and comparison tables in the papers listed at the end of this section.

The main drawback of this approach is that it requires staggering amounts of GPU memory - almost an order of magnitude 
larger than its dense equivalent. Various distillation and approaches are proposed to how to overcome the much higher memory requirements.

There is direct trade-off though, you can use just a few experts with a 2-3x smaller base model instead of dozens or 
hundreds experts leading to a 5x smaller model and thus increase the training speed moderately while increasing the 
memory requirements moderately as well.

Most related papers and implementations are built around Tensorflow/TPUs:

- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://arxiv.org/abs/2006.16668)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://arxiv.org/abs/2101.03961)
- [GLaM: Generalist Language Model (GLaM)](https://ai.googleblog.com/2021/12/more-efficient-in-context-learning-with.html)

And for Pytorch DeepSpeed has built one as well: [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://arxiv.org/abs/2201.05596), [Mixture of Experts](https://www.deepspeed.ai/tutorials/mixture-of-experts/) - blog posts:  [1](https://www.microsoft.com/en-us/research/blog/deepspeed-powers-8x-larger-moe-model-training-with-high-performance/), [2](https://www.microsoft.com/en-us/research/publication/scalable-and-efficient-moe-training-for-multitask-multilingual-models/) and specific deployment with large transformer-based natural language generation models: [blog post](https://www.deepspeed.ai/2021/12/09/deepspeed-moe-nlg.html), [Megatron-Deepspeed branch](https://github.com/microsoft/Megatron-DeepSpeed/tree/moe-training).

## Using PyTorch native attention and Flash Attention

PyTorch's [`torch.nn.functional.scaled_dot_product_attention`](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html) (SDPA) can also call FlashAttention and memory-efficient attention kernels under the hood.


key: _node_content
string_value:
{"id_": "fbd9d825-c7b5-422c-b347-f43042c6f25b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "32235eb0-3add-4442-a981-82a445254d5b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "f4a1199f6734143a23fa61a74b0292b4dc919543135f61bdce1e193e695c4196", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7835e437-a55d-4091-8884-070bc205f005", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_gpu_one", "title": "Methods and tools for efficient training on a single GPU", "tokens": 7716, "retrieve_doc": true, "source": "transformers"}, "hash": "e23b78d849f4c72b46dde6c31648b5a4d0dcba81f6cd4a57a715c238a255f785", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 32429, "end_char_idx": 34460, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Most related papers and implementations are built around Tensorflow/TPUs:

- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://arxiv.org/abs/2006.16668)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://arxiv.org/abs/2101.03961)
- [GLaM: Generalist Language Model (GLaM)](https://ai.googleblog.com/2021/12/more-efficient-in-context-learning-with.html)

And for Pytorch DeepSpeed has built one as well: [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://arxiv.org/abs/2201.05596), [Mixture of Experts](https://www.deepspeed.ai/tutorials/mixture-of-experts/) - blog posts:  [1](https://www.microsoft.com/en-us/research/blog/deepspeed-powers-8x-larger-moe-model-training-with-high-performance/), [2](https://www.microsoft.com/en-us/research/publication/scalable-and-efficient-moe-training-for-multitask-multilingual-models/) and specific deployment with large transformer-based natural language generation models: [blog post](https://www.deepspeed.ai/2021/12/09/deepspeed-moe-nlg.html), [Megatron-Deepspeed branch](https://github.com/microsoft/Megatron-DeepSpeed/tree/moe-training).

## Using PyTorch native attention and Flash Attention

PyTorch's [`torch.nn.functional.scaled_dot_product_attention`](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html) (SDPA) can also call FlashAttention and memory-efficient attention kernels under the hood. SDPA support is currently being added natively in Transformers and is used by default for `torch>=2.1.1` when an implementation is available. Please refer to [PyTorch scaled dot product attention](https://huggingface.co/docs/transformers/perf_infer_gpu_one#pytorch-scaled-dot-product-attention) for a list of supported models and more details.

Check out this [blogpost](https://pytorch.org/blog/out-of-the-box-acceleration/) to learn more about acceleration and memory-savings with SDPA.


key: _node_content
string_value:
{"id_": "f2e7d165-487a-424e-9478-35d8d44fbe31", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/installation", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/installation", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "transformers"}, "hash": "7cfef8db584ba695f4cebba362b20f6cab428ef5e064641c2aa9d0695f768953", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e30ed985-91a2-42e3-84b0-d1de8c9f2a3d", "node_type": "1", "metadata": {}, "hash": "eaf39928c104f616403f50a4ec03f81bd2d1ff8b54de892267a1798b2f08099e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3287, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<!---
Copyright 2022 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.

-->

# Installation

Install 🤗 Transformers for whichever deep learning library you're working with, setup your cache, and optionally configure 🤗 Transformers to run offline.

🤗 Transformers is tested on Python 3.6+, PyTorch 1.1.0+, TensorFlow 2.0+, and Flax. Follow the installation instructions below for the deep learning library you are using:

* [PyTorch](https://pytorch.org/get-started/locally/) installation instructions.
* [TensorFlow 2.0](https://www.tensorflow.org/install/pip) installation instructions.
* [Flax](https://flax.readthedocs.io/en/latest/) installation instructions.

## Install with pip

You should install 🤗 Transformers in a [virtual environment](https://docs.python.org/3/library/venv.html). If you're unfamiliar with Python virtual environments, take a look at this [guide](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/). A virtual environment makes it easier to manage different projects, and avoid compatibility issues between dependencies.

Start by creating a virtual environment in your project directory:

```bash
python -m venv .env
```

Activate the virtual environment. On Linux and MacOs:

```bash
source .env/bin/activate
```
Activate Virtual environment on Windows

```bash
.env/Scripts/activate
```

Now you're ready to install 🤗 Transformers with the following command:

```bash
pip install transformers
```

For CPU-support only, you can conveniently install 🤗 Transformers and a deep learning library in one line. For example, install 🤗 Transformers and PyTorch with:

```bash
pip install 'transformers[torch]'
```

🤗 Transformers and TensorFlow 2.0:

```bash
pip install 'transformers[tf-cpu]'
```

<Tip warning={true}>

M1 / ARM Users

You will need to install the following before installing TensorFLow 2.0
```bash
brew install cmake
brew install pkg-config
```

</Tip>

🤗 Transformers and Flax:

```bash
pip install 'transformers[flax]'
```

Finally, check if 🤗 Transformers has been properly installed by running the following command. It will download a pretrained model:

```bash
python -c "from transformers import pipeline; print(pipeline('sentiment-analysis')('we love you'))"
```

Then print out the label and score:

```bash
[{'label': 'POSITIVE', 'score': 0.9998704791069031}]
```

## Install from source

Install 🤗 Transformers from source with the following command:

```bash
pip install git+https://github.com/huggingface/transformers
```

This command installs the bleeding edge `main` version rather than the latest `stable` version.


key: _node_content
string_value:
{"id_": "e30ed985-91a2-42e3-84b0-d1de8c9f2a3d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/installation", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/installation", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "transformers"}, "hash": "7cfef8db584ba695f4cebba362b20f6cab428ef5e064641c2aa9d0695f768953", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f2e7d165-487a-424e-9478-35d8d44fbe31", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/installation", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "transformers"}, "hash": "edbeed3a858acf3816698e4bcbd4bbc950fcef1a4e31cc378113d8a2b0516739", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "72a54998-78b9-41f6-8073-647ce1c72a9e", "node_type": "1", "metadata": {}, "hash": "b5d0d21984db14aa1802940e6d744bd6ef0f340de8005559300114e9e2b4ccb3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1642, "end_char_idx": 4848, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
A virtual environment makes it easier to manage different projects, and avoid compatibility issues between dependencies.

Start by creating a virtual environment in your project directory:

```bash
python -m venv .env
```

Activate the virtual environment. On Linux and MacOs:

```bash
source .env/bin/activate
```
Activate Virtual environment on Windows

```bash
.env/Scripts/activate
```

Now you're ready to install 🤗 Transformers with the following command:

```bash
pip install transformers
```

For CPU-support only, you can conveniently install 🤗 Transformers and a deep learning library in one line. For example, install 🤗 Transformers and PyTorch with:

```bash
pip install 'transformers[torch]'
```

🤗 Transformers and TensorFlow 2.0:

```bash
pip install 'transformers[tf-cpu]'
```

<Tip warning={true}>

M1 / ARM Users

You will need to install the following before installing TensorFLow 2.0
```bash
brew install cmake
brew install pkg-config
```

</Tip>

🤗 Transformers and Flax:

```bash
pip install 'transformers[flax]'
```

Finally, check if 🤗 Transformers has been properly installed by running the following command. It will download a pretrained model:

```bash
python -c "from transformers import pipeline; print(pipeline('sentiment-analysis')('we love you'))"
```

Then print out the label and score:

```bash
[{'label': 'POSITIVE', 'score': 0.9998704791069031}]
```

## Install from source

Install 🤗 Transformers from source with the following command:

```bash
pip install git+https://github.com/huggingface/transformers
```

This command installs the bleeding edge `main` version rather than the latest `stable` version. The `main` version is useful for staying up-to-date with the latest developments. For instance, if a bug has been fixed since the last official release but a new release hasn't been rolled out yet. However, this means the `main` version may not always be stable. We strive to keep the `main` version operational, and most issues are usually resolved within a few hours or a day. If you run into a problem, please open an [Issue](https://github.com/huggingface/transformers/issues) so we can fix it even sooner!

Check if 🤗 Transformers has been properly installed by running the following command:

```bash
python -c "from transformers import pipeline; print(pipeline('sentiment-analysis')('I love you'))"
```

## Editable install

You will need an editable install if you'd like to:

* Use the `main` version of the source code.
* Contribute to 🤗 Transformers and need to test changes in the code.

Clone the repository and install 🤗 Transformers with the following commands:

```bash
git clone https://github.com/huggingface/transformers.git
cd transformers
pip install -e .
```

These commands will link the folder you cloned the repository to and your Python library paths. Python will now look inside the folder you cloned to in addition to the normal library paths. For example, if your Python packages are typically installed in `~/anaconda3/envs/main/lib/python3.7/site-packages/`, Python will also search the folder you cloned to: `~/transformers/`.

<Tip warning={true}>

You must keep the `transformers` folder if you want to keep using the library.


key: _node_content
string_value:
{"id_": "72a54998-78b9-41f6-8073-647ce1c72a9e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/installation", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/installation", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "transformers"}, "hash": "7cfef8db584ba695f4cebba362b20f6cab428ef5e064641c2aa9d0695f768953", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e30ed985-91a2-42e3-84b0-d1de8c9f2a3d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/installation", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "transformers"}, "hash": "49cd58cce254bbf930fc90689541a1fdc14bafdb9586051469ab4a2e7bf81bd5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c0d2791b-ba1a-4c78-8178-a77ff315bb87", "node_type": "1", "metadata": {}, "hash": "cbb13bdfa3d6f1604e84f17fb180aa9742a49094c04025c6eae711f93f8d4164", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3288, "end_char_idx": 6442, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The `main` version is useful for staying up-to-date with the latest developments. For instance, if a bug has been fixed since the last official release but a new release hasn't been rolled out yet. However, this means the `main` version may not always be stable. We strive to keep the `main` version operational, and most issues are usually resolved within a few hours or a day. If you run into a problem, please open an [Issue](https://github.com/huggingface/transformers/issues) so we can fix it even sooner!

Check if 🤗 Transformers has been properly installed by running the following command:

```bash
python -c "from transformers import pipeline; print(pipeline('sentiment-analysis')('I love you'))"
```

## Editable install

You will need an editable install if you'd like to:

* Use the `main` version of the source code.
* Contribute to 🤗 Transformers and need to test changes in the code.

Clone the repository and install 🤗 Transformers with the following commands:

```bash
git clone https://github.com/huggingface/transformers.git
cd transformers
pip install -e .
```

These commands will link the folder you cloned the repository to and your Python library paths. Python will now look inside the folder you cloned to in addition to the normal library paths. For example, if your Python packages are typically installed in `~/anaconda3/envs/main/lib/python3.7/site-packages/`, Python will also search the folder you cloned to: `~/transformers/`.

<Tip warning={true}>

You must keep the `transformers` folder if you want to keep using the library.

</Tip>

Now you can easily update your clone to the latest version of 🤗 Transformers with the following command:

```bash
cd ~/transformers/
git pull
```

Your Python environment will find the `main` version of 🤗 Transformers on the next run.

## Install with conda

Install from the conda channel `conda-forge`:

```bash
conda install conda-forge::transformers
```

## Cache setup

Pretrained models are downloaded and locally cached at: `~/.cache/huggingface/hub`. This is the default directory given by the shell environment variable `TRANSFORMERS_CACHE`. On Windows, the default directory is given by `C:\Users\username\.cache\huggingface\hub`. You can change the shell environment variables shown below - in order of priority - to specify a different cache directory:

1. Shell environment variable (default): `HUGGINGFACE_HUB_CACHE` or `TRANSFORMERS_CACHE`.
2. Shell environment variable: `HF_HOME`.
3. Shell environment variable: `XDG_CACHE_HOME` + `/huggingface`.

<Tip>

🤗 Transformers will use the shell environment variables `PYTORCH_TRANSFORMERS_CACHE` or `PYTORCH_PRETRAINED_BERT_CACHE` if you are coming from an earlier iteration of this library and have set those environment variables, unless you specify the shell environment variable `TRANSFORMERS_CACHE`.

</Tip>

## Offline mode

Run 🤗 Transformers in a firewalled or offline environment with locally cached files by setting the environment variable `HF_HUB_OFFLINE=1`.

<Tip>

Add [🤗 Datasets](https://huggingface.co/docs/datasets/) to your offline training workflow with the environment variable `HF_DATASETS_OFFLINE=1`.


key: _node_content
string_value:
{"id_": "c0d2791b-ba1a-4c78-8178-a77ff315bb87", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/installation", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/installation", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "transformers"}, "hash": "7cfef8db584ba695f4cebba362b20f6cab428ef5e064641c2aa9d0695f768953", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "72a54998-78b9-41f6-8073-647ce1c72a9e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/installation", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "transformers"}, "hash": "e86ca2a92d70d424ce72416abdcee943a039b0f26dc74b4f8593a7c184fe53b9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a8274ade-cf07-446f-8121-2d31bea3d0d4", "node_type": "1", "metadata": {}, "hash": "3a564744caca7f9bdad5df12c335ba36f51fa3013d4c26b05c2167990d7e2dc5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4850, "end_char_idx": 7996, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

Now you can easily update your clone to the latest version of 🤗 Transformers with the following command:

```bash
cd ~/transformers/
git pull
```

Your Python environment will find the `main` version of 🤗 Transformers on the next run.

## Install with conda

Install from the conda channel `conda-forge`:

```bash
conda install conda-forge::transformers
```

## Cache setup

Pretrained models are downloaded and locally cached at: `~/.cache/huggingface/hub`. This is the default directory given by the shell environment variable `TRANSFORMERS_CACHE`. On Windows, the default directory is given by `C:\Users\username\.cache\huggingface\hub`. You can change the shell environment variables shown below - in order of priority - to specify a different cache directory:

1. Shell environment variable (default): `HUGGINGFACE_HUB_CACHE` or `TRANSFORMERS_CACHE`.
2. Shell environment variable: `HF_HOME`.
3. Shell environment variable: `XDG_CACHE_HOME` + `/huggingface`.

<Tip>

🤗 Transformers will use the shell environment variables `PYTORCH_TRANSFORMERS_CACHE` or `PYTORCH_PRETRAINED_BERT_CACHE` if you are coming from an earlier iteration of this library and have set those environment variables, unless you specify the shell environment variable `TRANSFORMERS_CACHE`.

</Tip>

## Offline mode

Run 🤗 Transformers in a firewalled or offline environment with locally cached files by setting the environment variable `HF_HUB_OFFLINE=1`.

<Tip>

Add [🤗 Datasets](https://huggingface.co/docs/datasets/) to your offline training workflow with the environment variable `HF_DATASETS_OFFLINE=1`.

</Tip>

```bash
HF_DATASETS_OFFLINE=1 HF_HUB_OFFLINE=1 \
python examples/pytorch/translation/run_translation.py --model_name_or_path google-t5/t5-small --dataset_name wmt16 --dataset_config ro-en ...
```

This script should run without hanging or waiting to timeout because it won't attempt to download the model from the Hub.

You can also bypass loading a model from the Hub from each [`~PreTrainedModel.from_pretrained`] call with the [`local_files_only`] parameter. When set to `True`, only local files are loaded:

```py
from transformers import T5Model

model = T5Model.from_pretrained("./path/to/local/directory", local_files_only=True)
```

### Fetch models and tokenizers to use offline

Another option for using 🤗 Transformers offline is to download the files ahead of time, and then point to their local path when you need to use them offline. There are three ways to do this:

* Download a file through the user interface on the [Model Hub](https://huggingface.co/models) by clicking on the ↓ icon.

    ![download-icon](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/download-icon.png)

* Use the [`PreTrainedModel.from_pretrained`] and [`PreTrainedModel.save_pretrained`] workflow:

    1. Download your files ahead of time with [`PreTrainedModel.from_pretrained`]:

    ```py
    >>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

    >>> tokenizer = AutoTokenizer.from_pretrained("bigscience/T0_3B")
    >>> model = AutoModelForSeq2SeqLM.from_pretrained("bigscience/T0_3B")
    ```

    2.


key: _node_content
string_value:
{"id_": "a8274ade-cf07-446f-8121-2d31bea3d0d4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/installation", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/installation", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "transformers"}, "hash": "7cfef8db584ba695f4cebba362b20f6cab428ef5e064641c2aa9d0695f768953", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c0d2791b-ba1a-4c78-8178-a77ff315bb87", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/installation", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "transformers"}, "hash": "2c921bc83867404130a6f4a1d72c6144c5310c9e81ba762ae24fc13b42c79202", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ac9067c8-3e9a-44a8-91b0-71b1a30659c8", "node_type": "1", "metadata": {}, "hash": "6777ec7304a04bb36cc1a727105977ae48e32e9dc0a90fa456c3835e1f28a445", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6444, "end_char_idx": 8960, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

```bash
HF_DATASETS_OFFLINE=1 HF_HUB_OFFLINE=1 \
python examples/pytorch/translation/run_translation.py --model_name_or_path google-t5/t5-small --dataset_name wmt16 --dataset_config ro-en ...
```

This script should run without hanging or waiting to timeout because it won't attempt to download the model from the Hub.

You can also bypass loading a model from the Hub from each [`~PreTrainedModel.from_pretrained`] call with the [`local_files_only`] parameter. When set to `True`, only local files are loaded:

```py
from transformers import T5Model

model = T5Model.from_pretrained("./path/to/local/directory", local_files_only=True)
```

### Fetch models and tokenizers to use offline

Another option for using 🤗 Transformers offline is to download the files ahead of time, and then point to their local path when you need to use them offline. There are three ways to do this:

* Download a file through the user interface on the [Model Hub](https://huggingface.co/models) by clicking on the ↓ icon.

    ![download-icon](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/download-icon.png)

* Use the [`PreTrainedModel.from_pretrained`] and [`PreTrainedModel.save_pretrained`] workflow:

    1. Download your files ahead of time with [`PreTrainedModel.from_pretrained`]:

    ```py
    >>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

    >>> tokenizer = AutoTokenizer.from_pretrained("bigscience/T0_3B")
    >>> model = AutoModelForSeq2SeqLM.from_pretrained("bigscience/T0_3B")
    ```

    2. Save your files to a specified directory with [`PreTrainedModel.save_pretrained`]:

    ```py
    >>> tokenizer.save_pretrained("./your/path/bigscience_t0")
    >>> model.save_pretrained("./your/path/bigscience_t0")
    ```

    3. Now when you're offline, reload your files with [`PreTrainedModel.from_pretrained`] from the specified directory:

    ```py
    >>> tokenizer = AutoTokenizer.from_pretrained("./your/path/bigscience_t0")
    >>> model = AutoModel.from_pretrained("./your/path/bigscience_t0")
    ```

* Programmatically download files with the [huggingface_hub](https://github.com/huggingface/huggingface_hub/tree/main/src/huggingface_hub) library:

    1. Install the `huggingface_hub` library in your virtual environment:

    ```bash
    python -m pip install huggingface_hub
    ```

    2. Use the [`hf_hub_download`](https://huggingface.co/docs/hub/adding-a-library#download-files-from-the-hub) function to download a file to a specific path.


key: _node_content
string_value:
{"id_": "ac9067c8-3e9a-44a8-91b0-71b1a30659c8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/installation", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d2e0bed7-f86a-42ca-9f1e-fcd98450f6a0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/installation", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "transformers"}, "hash": "7cfef8db584ba695f4cebba362b20f6cab428ef5e064641c2aa9d0695f768953", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a8274ade-cf07-446f-8121-2d31bea3d0d4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/installation", "title": "Installation", "tokens": 2277, "retrieve_doc": true, "source": "transformers"}, "hash": "a56826f2469bf7b74763375b6a766647a543c36607c0c7ba7b353a508e039c43", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7460, "end_char_idx": 9702, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
![download-icon](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/download-icon.png)

* Use the [`PreTrainedModel.from_pretrained`] and [`PreTrainedModel.save_pretrained`] workflow:

    1. Download your files ahead of time with [`PreTrainedModel.from_pretrained`]:

    ```py
    >>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

    >>> tokenizer = AutoTokenizer.from_pretrained("bigscience/T0_3B")
    >>> model = AutoModelForSeq2SeqLM.from_pretrained("bigscience/T0_3B")
    ```

    2. Save your files to a specified directory with [`PreTrainedModel.save_pretrained`]:

    ```py
    >>> tokenizer.save_pretrained("./your/path/bigscience_t0")
    >>> model.save_pretrained("./your/path/bigscience_t0")
    ```

    3. Now when you're offline, reload your files with [`PreTrainedModel.from_pretrained`] from the specified directory:

    ```py
    >>> tokenizer = AutoTokenizer.from_pretrained("./your/path/bigscience_t0")
    >>> model = AutoModel.from_pretrained("./your/path/bigscience_t0")
    ```

* Programmatically download files with the [huggingface_hub](https://github.com/huggingface/huggingface_hub/tree/main/src/huggingface_hub) library:

    1. Install the `huggingface_hub` library in your virtual environment:

    ```bash
    python -m pip install huggingface_hub
    ```

    2. Use the [`hf_hub_download`](https://huggingface.co/docs/hub/adding-a-library#download-files-from-the-hub) function to download a file to a specific path. For example, the following command downloads the `config.json` file from the [T0](https://huggingface.co/bigscience/T0_3B) model to your desired path:

    ```py
    >>> from huggingface_hub import hf_hub_download

    >>> hf_hub_download(repo_id="bigscience/T0_3B", filename="config.json", cache_dir="./your/path/bigscience_t0")
    ```

Once your file is downloaded and locally cached, specify it's local path to load and use it:

```py
>>> from transformers import AutoConfig

>>> config = AutoConfig.from_pretrained("./your/path/bigscience_t0/config.json")
```

<Tip>

See the [How to download files from the Hub](https://huggingface.co/docs/hub/how-to-downstream) section for more details on downloading files stored on the Hub.

</Tip>


key: _node_content
string_value:
{"id_": "f639b09f-0616-42fd-884f-31bfd0cecf22", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d4f9fc61-9a38-4a78-be11-4b65d3521275", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "transformers"}, "hash": "b214e07ef3ecae09c5461993de9fccbc5447a3c7ea0d44c0ffacb76327cf9123", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c8a33ebb-980c-4e3e-9bb7-18a205d0349d", "node_type": "1", "metadata": {}, "hash": "cae151ad16374d59543be6e232b84e8c332430214e522f972eef428170d42e81", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3417, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<!---
Copyright 2020 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.

-->

# Checks on a Pull Request

When you open a pull request on 🤗 Transformers, a fair number of checks will be run to make sure the patch you are adding is not breaking anything existing. Those checks are of four types:
- regular tests
- documentation build
- code and documentation style
- general repository consistency

In this document, we will take a stab at explaining what those various checks are and the reason behind them, as well as how to debug them locally if one of them fails on your PR.

Note that, ideally, they require you to have a dev install:

```bash
pip install transformers[dev]
```

or for an editable install:

```bash
pip install -e .[dev]
```

inside the Transformers repo. Since the number of optional dependencies of Transformers has grown a lot, it's possible you don't manage to get all of them. If the dev install fails, make sure to install the Deep Learning framework you are working with (PyTorch, TensorFlow and/or Flax) then do

```bash
pip install transformers[quality]
```

or for an editable install:

```bash
pip install -e .[quality]
```


## Tests

All the jobs that begin with `ci/circleci: run_tests_` run parts of the Transformers testing suite. Each of those jobs focuses on a part of the library in a certain environment: for instance `ci/circleci: run_tests_pipelines_tf` runs the pipelines test in an environment where TensorFlow only is installed.

Note that to avoid running tests when there is no real change in the modules they are testing, only part of the test suite is run each time: a utility is run to determine the differences in the library between before and after the PR (what GitHub shows you in the "Files changes" tab) and picks the tests impacted by that diff. That utility can be run locally with:

```bash
python utils/tests_fetcher.py
```

from the root of the Transformers repo. It will:

1. Check for each file in the diff if the changes are in the code or only in comments or docstrings. Only the files with real code changes are kept.
2. Build an internal map that gives for each file of the source code of the library all the files it recursively impacts. Module A is said to impact module B if module B imports module A. For the recursive impact, we need a chain of modules going from module A to module B in which each module imports the previous one.
3. Apply this map on the files gathered in step 1, which  gives us the list of model files impacted by the PR.
4. Map each of those files to their corresponding test file(s) and get the list of tests to run.

When executing the script locally, you should get the results of step 1, 3 and 4 printed and thus know which tests are run.


key: _node_content
string_value:
{"id_": "c8a33ebb-980c-4e3e-9bb7-18a205d0349d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d4f9fc61-9a38-4a78-be11-4b65d3521275", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "transformers"}, "hash": "b214e07ef3ecae09c5461993de9fccbc5447a3c7ea0d44c0ffacb76327cf9123", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f639b09f-0616-42fd-884f-31bfd0cecf22", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "transformers"}, "hash": "f138789cd51e560264017a8f806beed1d2f3f2a36f388b03ab28636e9bfb6b68", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "027f133b-b3ac-4711-b7cc-c07485be2b42", "node_type": "1", "metadata": {}, "hash": "16a310c00d2fef1fe9a44664b28695bae9b1556ce3af39ee3954139a115f73b2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1838, "end_char_idx": 5082, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Tests

All the jobs that begin with `ci/circleci: run_tests_` run parts of the Transformers testing suite. Each of those jobs focuses on a part of the library in a certain environment: for instance `ci/circleci: run_tests_pipelines_tf` runs the pipelines test in an environment where TensorFlow only is installed.

Note that to avoid running tests when there is no real change in the modules they are testing, only part of the test suite is run each time: a utility is run to determine the differences in the library between before and after the PR (what GitHub shows you in the "Files changes" tab) and picks the tests impacted by that diff. That utility can be run locally with:

```bash
python utils/tests_fetcher.py
```

from the root of the Transformers repo. It will:

1. Check for each file in the diff if the changes are in the code or only in comments or docstrings. Only the files with real code changes are kept.
2. Build an internal map that gives for each file of the source code of the library all the files it recursively impacts. Module A is said to impact module B if module B imports module A. For the recursive impact, we need a chain of modules going from module A to module B in which each module imports the previous one.
3. Apply this map on the files gathered in step 1, which  gives us the list of model files impacted by the PR.
4. Map each of those files to their corresponding test file(s) and get the list of tests to run.

When executing the script locally, you should get the results of step 1, 3 and 4 printed and thus know which tests are run. The script will also create a file named `test_list.txt` which contains the list of tests to run, and you can run them locally with the following command:

```bash
python -m pytest -n 8 --dist=loadfile -rA -s $(cat test_list.txt)
```

Just in case anything slipped through the cracks, the full test suite is also run daily.

## Documentation build

The `build_pr_documentation` job builds and generates a preview of the documentation to make sure everything looks okay once your PR is merged. A bot will add a link to preview the documentation in your PR. Any changes you make to the PR are automatically updated in the preview. If the documentation fails to build, click on **Details** next to the failed job to see where things went wrong. Often, the error is as simple as a missing file in the `toctree`.

If you're interested in building or previewing the documentation locally, take a look at the [`README.md`](https://github.com/huggingface/transformers/tree/main/docs) in the docs folder.

## Code and documentation style

Code formatting is applied to all the source files, the examples and the tests using `black` and `ruff`. We also have a custom tool taking care of the formatting of docstrings and `rst` files (`utils/style_doc.py`), as well as the order of the lazy imports performed in the Transformers `__init__.py` files (`utils/custom_init_isort.py`). All of this can be launched by executing

```bash
make style
```

The CI checks those have been applied inside the `ci/circleci: check_code_quality` check. It also runs `ruff`, that will have a basic look at your code and will complain if it finds an undefined variable, or one that is not used.


key: _node_content
string_value:
{"id_": "027f133b-b3ac-4711-b7cc-c07485be2b42", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d4f9fc61-9a38-4a78-be11-4b65d3521275", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "transformers"}, "hash": "b214e07ef3ecae09c5461993de9fccbc5447a3c7ea0d44c0ffacb76327cf9123", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c8a33ebb-980c-4e3e-9bb7-18a205d0349d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "transformers"}, "hash": "7c4263641f20018567083d22e48b4d3e836cac8cf7dabf8917f46a3f2d228784", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f082a685-eab2-4a31-8518-def55af32ef4", "node_type": "1", "metadata": {}, "hash": "de20bfcf29b1ccbde76088ab3b3afc431cf2d082830fb92581db07adf31ce4ac", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3418, "end_char_idx": 5585, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The script will also create a file named `test_list.txt` which contains the list of tests to run, and you can run them locally with the following command:

```bash
python -m pytest -n 8 --dist=loadfile -rA -s $(cat test_list.txt)
```

Just in case anything slipped through the cracks, the full test suite is also run daily.

## Documentation build

The `build_pr_documentation` job builds and generates a preview of the documentation to make sure everything looks okay once your PR is merged. A bot will add a link to preview the documentation in your PR. Any changes you make to the PR are automatically updated in the preview. If the documentation fails to build, click on **Details** next to the failed job to see where things went wrong. Often, the error is as simple as a missing file in the `toctree`.

If you're interested in building or previewing the documentation locally, take a look at the [`README.md`](https://github.com/huggingface/transformers/tree/main/docs) in the docs folder.

## Code and documentation style

Code formatting is applied to all the source files, the examples and the tests using `black` and `ruff`. We also have a custom tool taking care of the formatting of docstrings and `rst` files (`utils/style_doc.py`), as well as the order of the lazy imports performed in the Transformers `__init__.py` files (`utils/custom_init_isort.py`). All of this can be launched by executing

```bash
make style
```

The CI checks those have been applied inside the `ci/circleci: check_code_quality` check. It also runs `ruff`, that will have a basic look at your code and will complain if it finds an undefined variable, or one that is not used. To run that check locally, use

```bash
make quality
```

This can take a lot of time, so to run the same thing on only the files you modified in the current branch, run

```bash
make fixup
```

This last command will also run all the additional checks for the repository consistency. Let's have a look at them.

## Repository consistency

This regroups all the tests to make sure your PR leaves the repository in a good state, and is performed by the `ci/circleci: check_repository_consistency` check.


key: _node_content
string_value:
{"id_": "f082a685-eab2-4a31-8518-def55af32ef4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d4f9fc61-9a38-4a78-be11-4b65d3521275", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "transformers"}, "hash": "b214e07ef3ecae09c5461993de9fccbc5447a3c7ea0d44c0ffacb76327cf9123", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "027f133b-b3ac-4711-b7cc-c07485be2b42", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "transformers"}, "hash": "b45b903e5ed4df7e2cc133e89661d3c260dfba8545e838b382fb947309dfe679", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2c6ed006-ff3c-4c41-9f91-b17dba92b3a0", "node_type": "1", "metadata": {}, "hash": "c2305b09c0845a1cd234086ef51fd1574f2924d76b81af7b56eed2572f606667", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3911, "end_char_idx": 7747, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
A bot will add a link to preview the documentation in your PR. Any changes you make to the PR are automatically updated in the preview. If the documentation fails to build, click on **Details** next to the failed job to see where things went wrong. Often, the error is as simple as a missing file in the `toctree`.

If you're interested in building or previewing the documentation locally, take a look at the [`README.md`](https://github.com/huggingface/transformers/tree/main/docs) in the docs folder.

## Code and documentation style

Code formatting is applied to all the source files, the examples and the tests using `black` and `ruff`. We also have a custom tool taking care of the formatting of docstrings and `rst` files (`utils/style_doc.py`), as well as the order of the lazy imports performed in the Transformers `__init__.py` files (`utils/custom_init_isort.py`). All of this can be launched by executing

```bash
make style
```

The CI checks those have been applied inside the `ci/circleci: check_code_quality` check. It also runs `ruff`, that will have a basic look at your code and will complain if it finds an undefined variable, or one that is not used. To run that check locally, use

```bash
make quality
```

This can take a lot of time, so to run the same thing on only the files you modified in the current branch, run

```bash
make fixup
```

This last command will also run all the additional checks for the repository consistency. Let's have a look at them.

## Repository consistency

This regroups all the tests to make sure your PR leaves the repository in a good state, and is performed by the `ci/circleci: check_repository_consistency` check. You can locally run that check by executing the following:

```bash
make repo-consistency
```

This checks that:

- All objects added to the init are documented (performed by `utils/check_repo.py`)
- All `__init__.py` files have the same content in their two sections (performed by `utils/check_inits.py`)
- All code identified as a copy from another module is consistent with the original (performed by `utils/check_copies.py`)
- All configuration classes have at least one valid checkpoint mentioned in their docstrings (performed by `utils/check_config_docstrings.py`)
- All configuration classes only contain attributes that are used in corresponding modeling files (performed by `utils/check_config_attributes.py`)
- The translations of the READMEs and the index of the doc have the same model list as the main README (performed by `utils/check_copies.py`)
- The auto-generated tables in the documentation are up to date (performed by `utils/check_table.py`)
- The library has all objects available even if not all optional dependencies are installed (performed by `utils/check_dummies.py`)
- All docstrings properly document the arguments in the signature of the object (performed by `utils/check_docstrings.py`)

Should this check fail, the first two items require manual fixing, the last four can be fixed automatically for you by running the command

```bash
make fix-copies
```

Additional checks concern PRs that add new models, mainly that:

- All models added are in an Auto-mapping (performed by `utils/check_repo.py`)
<!-- TODO Sylvain, add a check that makes sure the common tests are implemented.-->
- All models are properly tested (performed by `utils/check_repo.py`)

<!-- TODO Sylvain, add the following
- All models are added to the main README, inside the main doc
- All checkpoints used actually exist on the Hub

-->

### Check copies

Since the Transformers library is very opinionated with respect to model code, and each model should fully be implemented in a single file without relying on other models, we have added a mechanism that checks whether a copy of the code of a layer of a given model stays consistent with the original.


key: _node_content
string_value:
{"id_": "2c6ed006-ff3c-4c41-9f91-b17dba92b3a0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d4f9fc61-9a38-4a78-be11-4b65d3521275", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "transformers"}, "hash": "b214e07ef3ecae09c5461993de9fccbc5447a3c7ea0d44c0ffacb76327cf9123", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f082a685-eab2-4a31-8518-def55af32ef4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "transformers"}, "hash": "0a046faad9bc1a3e3e960d35da49e86ada96a632fcd555e1da78feb6f6563544", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f8deb596-a676-4792-982a-c4c41cd916e4", "node_type": "1", "metadata": {}, "hash": "f94a0cf9b286bf13b56756048eb29e027b83b10654d866954da7a4f982981675", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7748, "end_char_idx": 10116, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This way, when there is a bug fix, we can see all other impacted models and choose to trickle down the modification or break the copy.

<Tip>

If a file is a full copy of another file, you should register it in the constant `FULL_COPIES` of `utils/check_copies.py`.

</Tip>

This mechanism relies on comments of the form `# Copied from xxx`. The `xxx` should contain the whole path to the class of function which is being copied below. For instance, `RobertaSelfOutput` is a direct copy of the `BertSelfOutput` class, so you can see [here](https://github.com/huggingface/transformers/blob/2bd7a27a671fd1d98059124024f580f8f5c0f3b5/src/transformers/models/roberta/modeling_roberta.py#L289) it has a comment:

```py
# Copied from transformers.models.bert.modeling_bert.BertSelfOutput
```

Note that instead of applying this to a whole class, you can apply it to the relevant methods that are copied from. For instance [here](https://github.com/huggingface/transformers/blob/2bd7a27a671fd1d98059124024f580f8f5c0f3b5/src/transformers/models/roberta/modeling_roberta.py#L598) you can see how `RobertaPreTrainedModel._init_weights` is copied from the same method in `BertPreTrainedModel` with the comment:

```py
# Copied from transformers.models.bert.modeling_bert.BertPreTrainedModel._init_weights
```

Sometimes the copy is exactly the same except for names: for instance in `RobertaAttention`, we use `RobertaSelfAttention` insted of `BertSelfAttention` but other than that, the code is exactly the same. This is why `# Copied from` supports simple string replacements with the following syntax: `Copied from xxx with foo->bar`. This means the code is copied with all instances of `foo` being replaced by `bar`. You can see how it used [here](https://github.com/huggingface/transformers/blob/2bd7a27a671fd1d98059124024f580f8f5c0f3b5/src/transformers/models/roberta/modeling_roberta.py#L304C1-L304C86) in `RobertaAttention` with the comment:

```py
# Copied from transformers.models.bert.modeling_bert.BertAttention with Bert->Roberta
```

Note that there shouldn't be any spaces around the arrow (unless that space is part of the pattern to replace of course).

You can add several patterns separated by a comma. For instance here `CamemberForMaskedLM` is a direct copy of `RobertaForMaskedLM` with two replacements: `Roberta` to `Camembert` and `ROBERTA` to `CAMEMBERT`.


key: _node_content
string_value:
{"id_": "f8deb596-a676-4792-982a-c4c41cd916e4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d4f9fc61-9a38-4a78-be11-4b65d3521275", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "transformers"}, "hash": "b214e07ef3ecae09c5461993de9fccbc5447a3c7ea0d44c0ffacb76327cf9123", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2c6ed006-ff3c-4c41-9f91-b17dba92b3a0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/pr_checks", "title": "Checks on a Pull Request", "tokens": 2764, "retrieve_doc": true, "source": "transformers"}, "hash": "f9cde03352f5688a4271baa6cf177ee13193507e88d141ccd356cb4ee4c20ed6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9250, "end_char_idx": 11612, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This is why `# Copied from` supports simple string replacements with the following syntax: `Copied from xxx with foo->bar`. This means the code is copied with all instances of `foo` being replaced by `bar`. You can see how it used [here](https://github.com/huggingface/transformers/blob/2bd7a27a671fd1d98059124024f580f8f5c0f3b5/src/transformers/models/roberta/modeling_roberta.py#L304C1-L304C86) in `RobertaAttention` with the comment:

```py
# Copied from transformers.models.bert.modeling_bert.BertAttention with Bert->Roberta
```

Note that there shouldn't be any spaces around the arrow (unless that space is part of the pattern to replace of course).

You can add several patterns separated by a comma. For instance here `CamemberForMaskedLM` is a direct copy of `RobertaForMaskedLM` with two replacements: `Roberta` to `Camembert` and `ROBERTA` to `CAMEMBERT`. You can see [here](https://github.com/huggingface/transformers/blob/15082a9dc6950ecae63a0d3e5060b2fc7f15050a/src/transformers/models/camembert/modeling_camembert.py#L929) this is done with the comment:

```py
# Copied from transformers.models.roberta.modeling_roberta.RobertaForMaskedLM with Roberta->Camembert, ROBERTA->CAMEMBERT
```

If the order matters (because one of the replacements might conflict with a previous one), the replacements are executed from left to right.

<Tip>

If the replacements change the formatting (if you replace a short name by a very long name for instance), the copy is checked after applying the auto-formatter.

</Tip>

Another way when the patterns are just different casings of the same replacement (with an uppercased and a lowercased variants) is just to add the option `all-casing`. [Here](https://github.com/huggingface/transformers/blob/15082a9dc6950ecae63a0d3e5060b2fc7f15050a/src/transformers/models/mobilebert/modeling_mobilebert.py#L1237) is an example in `MobileBertForSequenceClassification` with the comment:

```py
# Copied from transformers.models.bert.modeling_bert.BertForSequenceClassification with Bert->MobileBert all-casing
```

In this case, the code is copied from `BertForSequenceClassification` by replacing:
- `Bert` by `MobileBert` (for instance when using `MobileBertModel` in the init)
- `bert` by `mobilebert` (for instance when defining `self.mobilebert`)
- `BERT` by `MOBILEBERT` (in the constant `MOBILEBERT_INPUTS_DOCSTRING`)


key: _node_content
string_value:
{"id_": "5d9ffa25-e619-417d-8c3a-1946b8436c30", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tflite", "title": "Export to TFLite", "tokens": 763, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "9129df82-e582-491b-933d-20847bfa58ec", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tflite", "title": "Export to TFLite", "tokens": 763, "retrieve_doc": true, "source": "transformers"}, "hash": "0130f2681f69d6ce6f652cbe10af6485aa43a241bb98b616df984578248e01fd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2358, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Export to TFLite

[TensorFlow Lite](https://www.tensorflow.org/lite/guide) is a lightweight framework for deploying machine learning models 
on resource-constrained devices, such as mobile phones, embedded systems, and Internet of Things (IoT) devices. 
TFLite is designed to optimize and run models efficiently on these devices with limited computational power, memory, and 
power consumption.
A TensorFlow Lite model is represented in a special efficient portable format identified by the `.tflite` file extension. 

🤗 Optimum offers functionality to export 🤗 Transformers models to TFLite through the `exporters.tflite` module. 
For the list of supported model architectures, please refer to [🤗 Optimum documentation](https://huggingface.co/docs/optimum/exporters/tflite/overview).

To export a model to TFLite, install the required dependencies:
 
```bash
pip install optimum[exporters-tf]
```

To check out all available arguments, refer to the [🤗 Optimum docs](https://huggingface.co/docs/optimum/main/en/exporters/tflite/usage_guides/export_a_model), 
or view help in command line:

```bash
optimum-cli export tflite --help
```

To export a model's checkpoint from the 🤗 Hub, for example, `google-bert/bert-base-uncased`, run the following command:

```bash
optimum-cli export tflite --model google-bert/bert-base-uncased --sequence_length 128 bert_tflite/
```

You should see the logs indicating progress and showing where the resulting `model.tflite` is saved, like this:

```bash
Validating TFLite model...
	-[✓] TFLite model output names match reference model (logits)
	- Validating TFLite Model output "logits":
		-[✓] (1, 128, 30522) matches (1, 128, 30522)
		-[x] values not close enough, max diff: 5.817413330078125e-05 (atol: 1e-05)
The TensorFlow Lite export succeeded with the warning: The maximum absolute difference between the output of the reference model and the TFLite exported model is not within the set tolerance 1e-05:
- logits: max diff = 5.817413330078125e-05.
 The exported model was saved at: bert_tflite
 ```

The example above illustrates exporting a checkpoint from 🤗 Hub. When exporting a local model, first make sure that you 
saved both the model's weights and tokenizer files in the same directory (`local_path`). When using CLI, pass the 
`local_path` to the `model` argument instead of the checkpoint name on 🤗 Hub.


key: _node_content
string_value:
{"id_": "062977ad-9651-48c3-8a44-b7f50a27c2f3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "hash": "428cb441bb5a77a89345d560e7aca430a73902daee44dd5ce2b1de5ca578c2cd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cbad4e38-e6e4-45c6-b69c-09e2e6459183", "node_type": "1", "metadata": {}, "hash": "aa61503410636d6cdc5e54d4f157ad12b345e897dc6150a6fb5978cb9be0970c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1766, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Optimize inference using torch.compile()

This guide aims to provide a benchmark on the inference speed-ups introduced with [`torch.compile()`](https://pytorch.org/tutorials/intermediate/torch_compile_tutorial.html) for [computer vision models in 🤗 Transformers](https://huggingface.co/models?pipeline_tag=image-classification&library=transformers&sort=trending).

## Benefits of torch.compile
   
Depending on the model and the GPU, `torch.compile()` yields up to 30% speed-up during inference. To use `torch.compile()`, simply install any version of `torch` above 2.0. 

Compiling a model takes time, so it's useful if you are compiling the model only once instead of every time you infer.
To compile any computer vision model of your choice, call `torch.compile()` on the model as shown below:

```diff
from transformers import AutoModelForImageClassification

model = AutoModelForImageClassification.from_pretrained(MODEL_ID).to("cuda")
+ model = torch.compile(model)
```

`compile()` comes with multiple modes for compiling, which essentially differ in compilation time and inference overhead. `max-autotune` takes longer than `reduce-overhead` but results in faster inference. Default mode is fastest for compilation but is not as efficient compared to `reduce-overhead` for inference time. In this guide, we used the default mode. You can learn more about it [here](https://pytorch.org/get-started/pytorch-2.0/#user-experience).

We benchmarked `torch.compile` with different computer vision models, tasks, types of hardware, and batch sizes on `torch` version 2.0.1.

## Benchmarking code 

Below you can find the benchmarking code for each task. We warm up the GPU before inference and take the mean time of 300 inferences, using the same image each time.


key: _node_content
string_value:
{"id_": "cbad4e38-e6e4-45c6-b69c-09e2e6459183", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "hash": "428cb441bb5a77a89345d560e7aca430a73902daee44dd5ce2b1de5ca578c2cd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "062977ad-9651-48c3-8a44-b7f50a27c2f3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "hash": "8fbf805a473a6237681974baf73e8bb26fc5bc896a04eb2676d04d0fa95569fe", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "83987ac8-1bf1-411c-8d5e-d45e4afe98c0", "node_type": "1", "metadata": {}, "hash": "64ec2e10906c55dd8900250c0c554b47201ea338ede7369b9d133c03c74e5237", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 367, "end_char_idx": 3469, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Benefits of torch.compile
   
Depending on the model and the GPU, `torch.compile()` yields up to 30% speed-up during inference. To use `torch.compile()`, simply install any version of `torch` above 2.0. 

Compiling a model takes time, so it's useful if you are compiling the model only once instead of every time you infer.
To compile any computer vision model of your choice, call `torch.compile()` on the model as shown below:

```diff
from transformers import AutoModelForImageClassification

model = AutoModelForImageClassification.from_pretrained(MODEL_ID).to("cuda")
+ model = torch.compile(model)
```

`compile()` comes with multiple modes for compiling, which essentially differ in compilation time and inference overhead. `max-autotune` takes longer than `reduce-overhead` but results in faster inference. Default mode is fastest for compilation but is not as efficient compared to `reduce-overhead` for inference time. In this guide, we used the default mode. You can learn more about it [here](https://pytorch.org/get-started/pytorch-2.0/#user-experience).

We benchmarked `torch.compile` with different computer vision models, tasks, types of hardware, and batch sizes on `torch` version 2.0.1.

## Benchmarking code 

Below you can find the benchmarking code for each task. We warm up the GPU before inference and take the mean time of 300 inferences, using the same image each time.

### Image Classification with ViT

```python 
import torch
from PIL import Image
import requests
import numpy as np
from transformers import AutoImageProcessor, AutoModelForImageClassification

url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)

processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224")
model = AutoModelForImageClassification.from_pretrained("google/vit-base-patch16-224").to("cuda")
model = torch.compile(model)

processed_input = processor(image, return_tensors='pt').to(device="cuda")

with torch.no_grad():
    _ = model(**processed_input)

```

#### Object Detection with DETR

```python 
from transformers import AutoImageProcessor, AutoModelForObjectDetection

processor = AutoImageProcessor.from_pretrained("facebook/detr-resnet-50")
model = AutoModelForObjectDetection.from_pretrained("facebook/detr-resnet-50").to("cuda")
model = torch.compile(model)

texts = ["a photo of a cat", "a photo of a dog"]
inputs = processor(text=texts, images=image, return_tensors="pt").to("cuda")

with torch.no_grad():
    _ = model(**inputs)
```

#### Image Segmentation with Segformer

```python 
from transformers import SegformerImageProcessor, SegformerForSemanticSegmentation

processor = SegformerImageProcessor.from_pretrained("nvidia/segformer-b0-finetuned-ade-512-512")
model = SegformerForSemanticSegmentation.from_pretrained("nvidia/segformer-b0-finetuned-ade-512-512").to("cuda")
model = torch.compile(model)
seg_inputs = processor(images=image, return_tensors="pt").to("cuda")

with torch.no_grad():
    _ = model(**seg_inputs)
```

Below you can find the list of the models we benchmarked.


key: _node_content
string_value:
{"id_": "83987ac8-1bf1-411c-8d5e-d45e4afe98c0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "hash": "428cb441bb5a77a89345d560e7aca430a73902daee44dd5ce2b1de5ca578c2cd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cbad4e38-e6e4-45c6-b69c-09e2e6459183", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "hash": "b839ab54ceaef4e92c2765bfa74109c09f646f7f91efc794fd279b9eee543ba2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e381c62b-0e91-4727-81a4-4c386a8adb2a", "node_type": "1", "metadata": {}, "hash": "5325ed3cddc9941ad793c36ebc7e01c0bb28b7a5a8c55c7040a129ef35e1f35f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3471, "end_char_idx": 5618, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
**Image Classification** 
- [google/vit-base-patch16-224](https://huggingface.co/google/vit-base-patch16-224)
- [microsoft/beit-base-patch16-224-pt22k-ft22k](https://huggingface.co/microsoft/beit-base-patch16-224-pt22k-ft22k)
- [facebook/convnext-large-224](https://huggingface.co/facebook/convnext-large-224)
- [microsoft/resnet-50](https://huggingface.co/microsoft/resnet-50)

**Image Segmentation** 
- [nvidia/segformer-b0-finetuned-ade-512-512](https://huggingface.co/nvidia/segformer-b0-finetuned-ade-512-512)
- [facebook/mask2former-swin-tiny-coco-panoptic](https://huggingface.co/facebook/mask2former-swin-tiny-coco-panoptic)
- [facebook/maskformer-swin-base-ade](https://huggingface.co/facebook/maskformer-swin-base-ade)
- [google/deeplabv3_mobilenet_v2_1.0_513](https://huggingface.co/google/deeplabv3_mobilenet_v2_1.0_513)

**Object Detection** 
- [google/owlvit-base-patch32](https://huggingface.co/google/owlvit-base-patch32)
- [facebook/detr-resnet-101](https://huggingface.co/facebook/detr-resnet-101)
- [microsoft/conditional-detr-resnet-50](https://huggingface.co/microsoft/conditional-detr-resnet-50)

Below you can find visualization of inference durations with and without `torch.compile()` and percentage improvements for each model in different hardware and batch sizes. 

<div class="flex">
  <div>
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/a100_batch_comp.png" />
  </div>
  <div>
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/v100_batch_comp.png" />
  </div>
   <div>
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/t4_batch_comp.png" />
  </div>
</div>

<div class="flex">
  <div>
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/A100_1_duration.png" />
  </div>
  <div>
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/A100_1_percentage.png" />
  </div>
</div>


key: _node_content
string_value:
{"id_": "e381c62b-0e91-4727-81a4-4c386a8adb2a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "hash": "428cb441bb5a77a89345d560e7aca430a73902daee44dd5ce2b1de5ca578c2cd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "83987ac8-1bf1-411c-8d5e-d45e4afe98c0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "hash": "7937efdd5c76938bf67d92f454928b1c84ccdd6f278818a271708247f52cba4f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "725f6cbf-d69f-44a4-90c3-2c61808a5424", "node_type": "1", "metadata": {}, "hash": "1ee584ad0f7e7e9e75ae09c5d5c4c2876552e2eef499c4dac771ead7c1bd8ed0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4765, "end_char_idx": 7277, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<div class="flex">
  <div>
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/a100_batch_comp.png" />
  </div>
  <div>
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/v100_batch_comp.png" />
  </div>
   <div>
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/t4_batch_comp.png" />
  </div>
</div>

<div class="flex">
  <div>
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/A100_1_duration.png" />
  </div>
  <div>
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/A100_1_percentage.png" />
  </div>
</div>


![Duration Comparison on V100 with Batch Size of 1](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/v100_1_duration.png)

![Percentage Improvement on T4 with Batch Size of 4](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/torch_compile/T4_4_percentage.png)

Below you can find inference durations in milliseconds for each model with and without `compile()`. Note that OwlViT results in OOM in larger batch sizes.

### A100 (batch size: 1)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 9.325 | 7.584 | 
| Image Segmentation/Segformer | 11.759 | 10.500 |
| Object Detection/OwlViT | 24.978 | 18.420 |
| Image Classification/BeiT | 11.282 | 8.448 | 
| Object Detection/DETR | 34.619 | 19.040 |
| Image Classification/ConvNeXT | 10.410 | 10.208 | 
| Image Classification/ResNet | 6.531 | 4.124 |
| Image Segmentation/Mask2former | 60.188 | 49.117 |
| Image Segmentation/Maskformer | 75.764 | 59.487 | 
| Image Segmentation/MobileNet | 8.583 | 3.974 |
| Object Detection/Resnet-101 | 36.276 | 18.197 |
| Object Detection/Conditional-DETR | 31.219 | 17.993 |


### A100 (batch size: 4)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 14.832 | 14.499 | 
| Image Segmentation/Segformer | 18.838 | 16.476 |
| Image Classification/BeiT | 13.205 | 13.048 | 
| Object Detection/DETR | 48.657 | 32.418|
| Image Classification/ConvNeXT | 22.940 | 21.631 | 
| Image Classification/ResNet | 6.657 | 4.


key: _node_content
string_value:
{"id_": "725f6cbf-d69f-44a4-90c3-2c61808a5424", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "hash": "428cb441bb5a77a89345d560e7aca430a73902daee44dd5ce2b1de5ca578c2cd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e381c62b-0e91-4727-81a4-4c386a8adb2a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "hash": "5875c3288d14e5f3fa87b47ec992a72e3c77ed7482a46a5ee7f3cf6b9cf49c60", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ff4761b6-fb7d-4c23-96e4-c3eaddf224ee", "node_type": "1", "metadata": {}, "hash": "d6b7857f59fc78b1b22c907248693b118eb20152ccf5e14f5bcd2463ebb449eb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6863, "end_char_idx": 8908, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### A100 (batch size: 4)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 14.832 | 14.499 | 
| Image Segmentation/Segformer | 18.838 | 16.476 |
| Image Classification/BeiT | 13.205 | 13.048 | 
| Object Detection/DETR | 48.657 | 32.418|
| Image Classification/ConvNeXT | 22.940 | 21.631 | 
| Image Classification/ResNet | 6.657 | 4.268 |
| Image Segmentation/Mask2former | 74.277 | 61.781 |
| Image Segmentation/Maskformer | 180.700 | 159.116 | 
| Image Segmentation/MobileNet | 14.174 | 8.515 |
| Object Detection/Resnet-101 | 68.101 | 44.998 |
| Object Detection/Conditional-DETR | 56.470 | 35.552 |

### A100 (batch size: 16)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 40.944 | 40.010 | 
| Image Segmentation/Segformer | 37.005 | 31.144 |
| Image Classification/BeiT | 41.854 | 41.048 | 
| Object Detection/DETR | 164.382 | 161.902 |
| Image Classification/ConvNeXT | 82.258 | 75.561 | 
| Image Classification/ResNet | 7.018 | 5.024 |
| Image Segmentation/Mask2former | 178.945 | 154.814 |
| Image Segmentation/Maskformer | 638.570 | 579.826 | 
| Image Segmentation/MobileNet | 51.693 | 30.310 |
| Object Detection/Resnet-101 | 232.887 | 155.021 |
| Object Detection/Conditional-DETR | 180.491 | 124.032 |

### V100 (batch size: 1)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 10.495 | 6.00 | 
| Image Segmentation/Segformer | 13.321 | 5.862 | 
| Object Detection/OwlViT | 25.769 | 22.395 | 
| Image Classification/BeiT | 11.347 | 7.234 | 
| Object Detection/DETR | 33.951 | 19.388 |
| Image Classification/ConvNeXT | 11.623 | 10.412 | 
| Image Classification/ResNet | 6.484 | 3.820 |
| Image Segmentation/Mask2former | 64.640 | 49.873 |
| Image Segmentation/Maskformer | 95.532 | 72.207 | 
| Image Segmentation/MobileNet | 9.217 | 4.753 |
| Object Detection/Resnet-101 | 52.818 | 28.


key: _node_content
string_value:
{"id_": "ff4761b6-fb7d-4c23-96e4-c3eaddf224ee", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "hash": "428cb441bb5a77a89345d560e7aca430a73902daee44dd5ce2b1de5ca578c2cd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "725f6cbf-d69f-44a4-90c3-2c61808a5424", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "hash": "73fe75cd007aec1a17762800401c830c06bf08d27622b26295e5372b138b1867", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a1759262-c6e5-43de-b66d-200213355225", "node_type": "1", "metadata": {}, "hash": "351268f5efc603dca17cf46b34f34824d971c976bcc080d2a098e9d731fe7789", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7855, "end_char_idx": 9930, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
382 | 161.902 |
| Image Classification/ConvNeXT | 82.258 | 75.561 | 
| Image Classification/ResNet | 7.018 | 5.024 |
| Image Segmentation/Mask2former | 178.945 | 154.814 |
| Image Segmentation/Maskformer | 638.570 | 579.826 | 
| Image Segmentation/MobileNet | 51.693 | 30.310 |
| Object Detection/Resnet-101 | 232.887 | 155.021 |
| Object Detection/Conditional-DETR | 180.491 | 124.032 |

### V100 (batch size: 1)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 10.495 | 6.00 | 
| Image Segmentation/Segformer | 13.321 | 5.862 | 
| Object Detection/OwlViT | 25.769 | 22.395 | 
| Image Classification/BeiT | 11.347 | 7.234 | 
| Object Detection/DETR | 33.951 | 19.388 |
| Image Classification/ConvNeXT | 11.623 | 10.412 | 
| Image Classification/ResNet | 6.484 | 3.820 |
| Image Segmentation/Mask2former | 64.640 | 49.873 |
| Image Segmentation/Maskformer | 95.532 | 72.207 | 
| Image Segmentation/MobileNet | 9.217 | 4.753 |
| Object Detection/Resnet-101 | 52.818 | 28.367 |
| Object Detection/Conditional-DETR | 39.512 | 20.816 |

### V100 (batch size: 4)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 15.181 | 14.501 | 
| Image Segmentation/Segformer | 16.787 | 16.188 |
| Image Classification/BeiT | 15.171 | 14.753 | 
| Object Detection/DETR | 88.529 | 64.195 |
| Image Classification/ConvNeXT | 29.574 | 27.085 | 
| Image Classification/ResNet | 6.109 | 4.731 |
| Image Segmentation/Mask2former | 90.402 | 76.926 |
| Image Segmentation/Maskformer | 234.261 | 205.456 | 
| Image Segmentation/MobileNet | 24.623 | 14.816 |
| Object Detection/Resnet-101 | 134.672 | 101.304 |
| Object Detection/Conditional-DETR | 97.464 | 69.739 |

### V100 (batch size: 16)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 52.209 | 51.633 | 
| Image Segmentation/Segformer | 61.013 | 55.499 |
| Image Classification/BeiT | 53.938 | 53.


key: _node_content
string_value:
{"id_": "a1759262-c6e5-43de-b66d-200213355225", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "hash": "428cb441bb5a77a89345d560e7aca430a73902daee44dd5ce2b1de5ca578c2cd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ff4761b6-fb7d-4c23-96e4-c3eaddf224ee", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "hash": "2a312f3bccb0de7a727d8e145e2da3771f124493e621b205262c7b31b7b445a4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "aead2690-0d35-4d6c-b7d1-0bccc94f6a28", "node_type": "1", "metadata": {}, "hash": "4f31c039172791eb9fb95cd3aeb5850d0eaeafde8a5db06dcac79c14a7b7b692", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8899, "end_char_idx": 10969, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
818 | 28.367 |
| Object Detection/Conditional-DETR | 39.512 | 20.816 |

### V100 (batch size: 4)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 15.181 | 14.501 | 
| Image Segmentation/Segformer | 16.787 | 16.188 |
| Image Classification/BeiT | 15.171 | 14.753 | 
| Object Detection/DETR | 88.529 | 64.195 |
| Image Classification/ConvNeXT | 29.574 | 27.085 | 
| Image Classification/ResNet | 6.109 | 4.731 |
| Image Segmentation/Mask2former | 90.402 | 76.926 |
| Image Segmentation/Maskformer | 234.261 | 205.456 | 
| Image Segmentation/MobileNet | 24.623 | 14.816 |
| Object Detection/Resnet-101 | 134.672 | 101.304 |
| Object Detection/Conditional-DETR | 97.464 | 69.739 |

### V100 (batch size: 16)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 52.209 | 51.633 | 
| Image Segmentation/Segformer | 61.013 | 55.499 |
| Image Classification/BeiT | 53.938 | 53.581  |
| Object Detection/DETR | OOM | OOM |
| Image Classification/ConvNeXT | 109.682 | 100.771 | 
| Image Classification/ResNet | 14.857 | 12.089 |
| Image Segmentation/Mask2former | 249.605 | 222.801 |
| Image Segmentation/Maskformer | 831.142 | 743.645 | 
| Image Segmentation/MobileNet | 93.129 | 55.365 |
| Object Detection/Resnet-101 | 482.425 | 361.843 |
| Object Detection/Conditional-DETR | 344.661 | 255.298 |

### T4 (batch size: 1)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 16.520 | 15.786 | 
| Image Segmentation/Segformer | 16.116 | 14.205 |
| Object Detection/OwlViT | 53.634 | 51.105 |
| Image Classification/BeiT | 16.464 | 15.710 | 
| Object Detection/DETR | 73.100 | 53.99 |
| Image Classification/ConvNeXT | 32.932 | 30.845 | 
| Image Classification/ResNet | 6.031 | 4.321 |
| Image Segmentation/Mask2former | 79.192 | 66.815 |
| Image Segmentation/Maskformer | 200.026 | 188.268 | 
| Image Segmentation/MobileNet | 18.908 | 11.


key: _node_content
string_value:
{"id_": "aead2690-0d35-4d6c-b7d1-0bccc94f6a28", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "hash": "428cb441bb5a77a89345d560e7aca430a73902daee44dd5ce2b1de5ca578c2cd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a1759262-c6e5-43de-b66d-200213355225", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "hash": "e1031341ae5c0cf78aa6b5f508a348b63d8617c47ed85d140ea5840abb51c686", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "096f770c-16fc-4514-a0c7-646abfc36603", "node_type": "1", "metadata": {}, "hash": "a5e248a607ec8791392e67a56ac6aa4d5ecd033ff6543db0de1bdaa177e84087", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9882, "end_char_idx": 11994, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
499 |
| Image Classification/BeiT | 53.938 | 53.581  |
| Object Detection/DETR | OOM | OOM |
| Image Classification/ConvNeXT | 109.682 | 100.771 | 
| Image Classification/ResNet | 14.857 | 12.089 |
| Image Segmentation/Mask2former | 249.605 | 222.801 |
| Image Segmentation/Maskformer | 831.142 | 743.645 | 
| Image Segmentation/MobileNet | 93.129 | 55.365 |
| Object Detection/Resnet-101 | 482.425 | 361.843 |
| Object Detection/Conditional-DETR | 344.661 | 255.298 |

### T4 (batch size: 1)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 16.520 | 15.786 | 
| Image Segmentation/Segformer | 16.116 | 14.205 |
| Object Detection/OwlViT | 53.634 | 51.105 |
| Image Classification/BeiT | 16.464 | 15.710 | 
| Object Detection/DETR | 73.100 | 53.99 |
| Image Classification/ConvNeXT | 32.932 | 30.845 | 
| Image Classification/ResNet | 6.031 | 4.321 |
| Image Segmentation/Mask2former | 79.192 | 66.815 |
| Image Segmentation/Maskformer | 200.026 | 188.268 | 
| Image Segmentation/MobileNet | 18.908 | 11.997 |
| Object Detection/Resnet-101 | 106.622 | 82.566 |
| Object Detection/Conditional-DETR | 77.594 | 56.984 |

### T4 (batch size: 4)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 43.653 | 43.626 | 
| Image Segmentation/Segformer | 45.327 | 42.445 |
| Image Classification/BeiT | 52.007 | 51.354 | 
| Object Detection/DETR | 277.850 | 268.003 |
| Image Classification/ConvNeXT | 119.259 | 105.580 | 
| Image Classification/ResNet | 13.039 | 11.388 |
| Image Segmentation/Mask2former | 201.540 | 184.670 |
| Image Segmentation/Maskformer | 764.052 | 711.280 | 
| Image Segmentation/MobileNet | 74.289 | 48.677 |
| Object Detection/Resnet-101 | 421.859 | 357.614 |
| Object Detection/Conditional-DETR | 289.002 | 226.945 |

### T4 (batch size: 16)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 163.914 | 160.907 | 
| Image Segmentation/Segformer | 192.


key: _node_content
string_value:
{"id_": "096f770c-16fc-4514-a0c7-646abfc36603", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "hash": "428cb441bb5a77a89345d560e7aca430a73902daee44dd5ce2b1de5ca578c2cd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "aead2690-0d35-4d6c-b7d1-0bccc94f6a28", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "hash": "065acf7504420687129dacbf17fae248ab7375bd4abe9b64b5eb796bf7b0a241", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e53e29ac-b016-4336-9826-647c1c2ed03c", "node_type": "1", "metadata": {}, "hash": "a9f9b97c53781c583bd40baa27483b6c8dd5a0cbb4cbb2e48db9ee7bacd832a8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10960, "end_char_idx": 12699, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
908 | 11.997 |
| Object Detection/Resnet-101 | 106.622 | 82.566 |
| Object Detection/Conditional-DETR | 77.594 | 56.984 |

### T4 (batch size: 4)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 43.653 | 43.626 | 
| Image Segmentation/Segformer | 45.327 | 42.445 |
| Image Classification/BeiT | 52.007 | 51.354 | 
| Object Detection/DETR | 277.850 | 268.003 |
| Image Classification/ConvNeXT | 119.259 | 105.580 | 
| Image Classification/ResNet | 13.039 | 11.388 |
| Image Segmentation/Mask2former | 201.540 | 184.670 |
| Image Segmentation/Maskformer | 764.052 | 711.280 | 
| Image Segmentation/MobileNet | 74.289 | 48.677 |
| Object Detection/Resnet-101 | 421.859 | 357.614 |
| Object Detection/Conditional-DETR | 289.002 | 226.945 |

### T4 (batch size: 16)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 163.914 | 160.907 | 
| Image Segmentation/Segformer | 192.412 | 163.620 |
| Image Classification/BeiT | 188.978 | 187.976 | 
| Object Detection/DETR | OOM | OOM |
| Image Classification/ConvNeXT | 422.886 | 388.078 | 
| Image Classification/ResNet | 44.114 | 37.604 |
| Image Segmentation/Mask2former | 756.337 | 695.291 |
| Image Segmentation/Maskformer | 2842.940 | 2656.88 | 
| Image Segmentation/MobileNet | 299.003 | 201.942 |
| Object Detection/Resnet-101 |  1619.505 | 1262.758 | 
| Object Detection/Conditional-DETR | 1137.513 | 897.390|

## PyTorch Nightly
We also benchmarked on PyTorch nightly (2.1.0dev, find the wheel [here](https://download.pytorch.org/whl/nightly/cu118)) and observed improvement in latency both for uncompiled and compiled models.


key: _node_content
string_value:
{"id_": "e53e29ac-b016-4336-9826-647c1c2ed03c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "hash": "428cb441bb5a77a89345d560e7aca430a73902daee44dd5ce2b1de5ca578c2cd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "096f770c-16fc-4514-a0c7-646abfc36603", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "hash": "f7da60978d7bf020d299934d26df73878bfb5f7cd8c9d3006d01ccdc9e3a45d8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4b701e71-4452-4722-b523-856897587ff6", "node_type": "1", "metadata": {}, "hash": "9192f96ee7c0c80ad4248351255391c240126ba3a4eda7f3d7e4a8853ce4b890", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11557, "end_char_idx": 14060, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
670 |
| Image Segmentation/Maskformer | 764.052 | 711.280 | 
| Image Segmentation/MobileNet | 74.289 | 48.677 |
| Object Detection/Resnet-101 | 421.859 | 357.614 |
| Object Detection/Conditional-DETR | 289.002 | 226.945 |

### T4 (batch size: 16)

| **Task/Model** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|
| Image Classification/ViT | 163.914 | 160.907 | 
| Image Segmentation/Segformer | 192.412 | 163.620 |
| Image Classification/BeiT | 188.978 | 187.976 | 
| Object Detection/DETR | OOM | OOM |
| Image Classification/ConvNeXT | 422.886 | 388.078 | 
| Image Classification/ResNet | 44.114 | 37.604 |
| Image Segmentation/Mask2former | 756.337 | 695.291 |
| Image Segmentation/Maskformer | 2842.940 | 2656.88 | 
| Image Segmentation/MobileNet | 299.003 | 201.942 |
| Object Detection/Resnet-101 |  1619.505 | 1262.758 | 
| Object Detection/Conditional-DETR | 1137.513 | 897.390|

## PyTorch Nightly
We also benchmarked on PyTorch nightly (2.1.0dev, find the wheel [here](https://download.pytorch.org/whl/nightly/cu118)) and observed improvement in latency both for uncompiled and compiled models. 

### A100

| **Task/Model** | **Batch Size** | **torch 2.0 - no compile** | **torch 2.0 -<br> compile** |
|:---:|:---:|:---:|:---:|
| Image Classification/BeiT | Unbatched | 12.462 | 6.954 | 
| Image Classification/BeiT | 4 | 14.109 | 12.851 | 
| Image Classification/BeiT | 16 | 42.179 | 42.147 | 
| Object Detection/DETR | Unbatched | 30.484 | 15.221 |
| Object Detection/DETR | 4 | 46.816 | 30.942 |
| Object Detection/DETR | 16 | 163.749 | 163.706  |

### T4

| **Task/Model** | **Batch Size** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|:---:|
| Image Classification/BeiT | Unbatched | 14.408 | 14.052 | 
| Image Classification/BeiT | 4 | 47.381 | 46.604 | 
| Image Classification/BeiT | 16 | 42.179 | 42.147  | 
| Object Detection/DETR | Unbatched | 68.382 | 53.481 |
| Object Detection/DETR | 4 | 269.615 | 204.785 |
| Object Detection/DETR | 16 | OOM | OOM   |

### V100

| **Task/Model** | **Batch Size** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|:---:|
| Image Classification/BeiT | Unbatched | 13.477 | 7.926 | 
| Image Classification/BeiT | 4 | 15.103 | 14.378 | 
| Image Classification/BeiT | 16 | 52.517 | 51.691  | 
| Object Detection/DETR | Unbatched | 28.706 | 19.077 |
| Object Detection/DETR | 4 | 88.402 | 62.949|
| Object Detection/DETR | 16 | OOM | OOM  |


key: _node_content
string_value:
{"id_": "4b701e71-4452-4722-b523-856897587ff6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3f7c5d2e-a02d-4508-8758-9c1afedcbda3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "hash": "428cb441bb5a77a89345d560e7aca430a73902daee44dd5ce2b1de5ca578c2cd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e53e29ac-b016-4336-9826-647c1c2ed03c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_torch_compile", "title": "Optimize inference using torch.compile()", "tokens": 5184, "retrieve_doc": true, "source": "transformers"}, "hash": "1d7fc368e0f3e07325b89f9a52a2bec94c5cafcfe536aab584fcf4f149f8ded0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14063, "end_char_idx": 15405, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Reduce Overhead
We benchmarked `reduce-overhead` compilation mode for A100 and T4 in Nightly.

### A100

| **Task/Model** | **Batch Size** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** |
|:---:|:---:|:---:|:---:|
| Image Classification/ConvNeXT | Unbatched | 11.758 | 7.335 | 
| Image Classification/ConvNeXT | 4 | 23.171 | 21.490 | 
| Image Classification/ResNet | Unbatched | 7.435 | 3.801 | 
| Image Classification/ResNet | 4 | 7.261 | 2.187 | 
| Object Detection/Conditional-DETR | Unbatched | 32.823 | 11.627  | 
| Object Detection/Conditional-DETR | 4 | 50.622 | 33.831  | 
| Image Segmentation/MobileNet | Unbatched | 9.869 | 4.244 |
| Image Segmentation/MobileNet | 4 | 14.385 | 7.946 |


### T4

| **Task/Model** | **Batch Size** | **torch 2.0 - <br>no compile** | **torch 2.0 - <br>compile** | 
|:---:|:---:|:---:|:---:|
| Image Classification/ConvNeXT | Unbatched | 32.137 | 31.84 | 
| Image Classification/ConvNeXT | 4 | 120.944 | 110.209 | 
| Image Classification/ResNet | Unbatched | 9.761 | 7.698 | 
| Image Classification/ResNet | 4 | 15.215 | 13.871 | 
| Object Detection/Conditional-DETR | Unbatched | 72.150 | 57.660  | 
| Object Detection/Conditional-DETR | 4 | 301.494 | 247.543  | 
| Image Segmentation/MobileNet | Unbatched | 22.266 | 19.339  |
| Image Segmentation/MobileNet | 4 | 78.311 | 50.983 |


key: _node_content
string_value:
{"id_": "e1c10abb-0f51-4120-9152-de4da0f73ca3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_special", "title": "PyTorch training on Apple silicon", "tokens": 771, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8155c6af-0dc7-424e-b5ef-cc8211a86a5e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_special", "title": "PyTorch training on Apple silicon", "tokens": 771, "retrieve_doc": true, "source": "transformers"}, "hash": "3b92385ad59a6c38344c47086105ae06547252dae5793516cdd7089ed1d393e7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2592, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# PyTorch training on Apple silicon

Previously, training models on a Mac was limited to the CPU only. With the release of PyTorch v1.12, you can take advantage of training models with Apple's silicon GPUs for significantly faster performance and training. This is powered in PyTorch by integrating Apple's Metal Performance Shaders (MPS) as a backend. The [MPS backend](https://pytorch.org/docs/stable/notes/mps.html) implements PyTorch operations as custom Metal shaders and places these modules on a `mps` device.

<Tip warning={true}>

Some PyTorch operations are not implemented in MPS yet and will throw an error. To avoid this, you should set the environment variable `PYTORCH_ENABLE_MPS_FALLBACK=1` to use the CPU kernels instead (you'll still see a `UserWarning`).

<br>

If you run into any other errors, please open an issue in the [PyTorch](https://github.com/pytorch/pytorch/issues) repository because the [`Trainer`] only integrates the MPS backend.

</Tip>

With the `mps` device set, you can:

* train larger networks or batch sizes locally
* reduce data retrieval latency because the GPU's unified memory architecture allows direct access to the full memory store
* reduce costs because you don't need to train on cloud-based GPUs or add additional local GPUs

Get started by making sure you have PyTorch installed. MPS acceleration is supported on macOS 12.3+.

```bash
pip install torch torchvision torchaudio
```

[`TrainingArguments`] uses the `mps` device by default if it's available which means you don't need to explicitly set the device. For example, you can run the [run_glue.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-classification/run_glue.py) script with the MPS backend automatically enabled without making any changes.

```diff
export TASK_NAME=mrpc

python examples/pytorch/text-classification/run_glue.py \
  --model_name_or_path google-bert/bert-base-cased \
  --task_name $TASK_NAME \
- --use_mps_device \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --output_dir /tmp/$TASK_NAME/ \
  --overwrite_output_dir
```

Backends for [distributed setups](https://pytorch.org/docs/stable/distributed.html#backends) like `gloo` and `nccl` are not supported by the `mps` device which means you can only train on a single GPU with the MPS backend.

You can learn more about the MPS backend in the [Introducing Accelerated PyTorch Training on Mac](https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/) blog post.


key: _node_content
string_value:
{"id_": "dc9ad0de-58da-4f66-9eac-90d33d1bab28", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/peft", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f08b4827-75ed-4105-b063-d8456c28e127", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/peft", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "transformers"}, "hash": "be7098b1712d5597c51cf0dd6286d1c7d52c17f66a2874c74268ab4e4502eb30", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "78272d89-897d-494e-be2f-db594a7fc9db", "node_type": "1", "metadata": {}, "hash": "e27405c2aa0699f084d144afa325ddd58ce5fd51e8c95ec25e52f539539dd320", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2778, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Load adapters with 🤗 PEFT

[[open-in-colab]]

[Parameter-Efficient Fine Tuning (PEFT)](https://huggingface.co/blog/peft) methods freeze the pretrained model parameters during fine-tuning and add a small number of trainable parameters (the adapters) on top of it. The adapters are trained to learn task-specific information. This approach has been shown to be very memory-efficient with lower compute usage while producing results comparable to a fully fine-tuned model. 

Adapters trained with PEFT are also usually an order of magnitude smaller than the full model, making it convenient to share, store, and load them.

<div class="flex flex-col justify-center">
  <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/peft/PEFT-hub-screenshot.png"/>
  <figcaption class="text-center">The adapter weights for a OPTForCausalLM model stored on the Hub are only ~6MB compared to the full size of the model weights, which can be ~700MB.</figcaption>
</div>

If you're interested in learning more about the 🤗 PEFT library, check out the [documentation](https://huggingface.co/docs/peft/index).

## Setup

Get started by installing 🤗 PEFT:

```bash
pip install peft
```

If you want to try out the brand new features, you might be interested in installing the library from source:

```bash
pip install git+https://github.com/huggingface/peft.git
```

## Supported PEFT models

🤗 Transformers natively supports some PEFT methods, meaning you can load adapter weights stored locally or on the Hub and easily run or train them with a few lines of code. The following methods are supported:

- [Low Rank Adapters](https://huggingface.co/docs/peft/conceptual_guides/lora)
- [IA3](https://huggingface.co/docs/peft/conceptual_guides/ia3)
- [AdaLoRA](https://arxiv.org/abs/2303.10512)

If you want to use other PEFT methods, such as prompt learning or prompt tuning, or about the 🤗 PEFT library in general, please refer to the [documentation](https://huggingface.co/docs/peft/index).


## Load a PEFT adapter

To load and use a PEFT adapter model from 🤗 Transformers, make sure the Hub repository or local directory contains an `adapter_config.json` file and the adapter weights, as shown in the example image above. Then you can load the PEFT adapter model using the `AutoModelFor` class. For example, to load a PEFT adapter model for causal language modeling:

1. specify the PEFT model id
2. pass it to the [`AutoModelForCausalLM`] class

```py
from transformers import AutoModelForCausalLM, AutoTokenizer

peft_model_id = "ybelkada/opt-350m-lora"
model = AutoModelForCausalLM.from_pretrained(peft_model_id)
```

<Tip>

You can load a PEFT adapter with either an `AutoModelFor` class or the base model class like `OPTForCausalLM` or `LlamaForCausalLM`.


key: _node_content
string_value:
{"id_": "78272d89-897d-494e-be2f-db594a7fc9db", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/peft", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f08b4827-75ed-4105-b063-d8456c28e127", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/peft", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "transformers"}, "hash": "be7098b1712d5597c51cf0dd6286d1c7d52c17f66a2874c74268ab4e4502eb30", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dc9ad0de-58da-4f66-9eac-90d33d1bab28", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/peft", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "transformers"}, "hash": "6d0d9305367ab16700b70ee912952630caad4fd79ce75febe6526fbd63bcb5ab", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "38b2e12b-720d-4a27-8e97-be3b10da96f3", "node_type": "1", "metadata": {}, "hash": "4439f1bbbb111b77b87e2ed770a99c44b135919652977be5dc179b7c81383eb8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2009, "end_char_idx": 4089, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Load a PEFT adapter

To load and use a PEFT adapter model from 🤗 Transformers, make sure the Hub repository or local directory contains an `adapter_config.json` file and the adapter weights, as shown in the example image above. Then you can load the PEFT adapter model using the `AutoModelFor` class. For example, to load a PEFT adapter model for causal language modeling:

1. specify the PEFT model id
2. pass it to the [`AutoModelForCausalLM`] class

```py
from transformers import AutoModelForCausalLM, AutoTokenizer

peft_model_id = "ybelkada/opt-350m-lora"
model = AutoModelForCausalLM.from_pretrained(peft_model_id)
```

<Tip>

You can load a PEFT adapter with either an `AutoModelFor` class or the base model class like `OPTForCausalLM` or `LlamaForCausalLM`.

</Tip>

You can also load a PEFT adapter by calling the `load_adapter` method:

```py
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "facebook/opt-350m"
peft_model_id = "ybelkada/opt-350m-lora"

model = AutoModelForCausalLM.from_pretrained(model_id)
model.load_adapter(peft_model_id)
```

Check out the [API documentation](#transformers.integrations.PeftAdapterMixin) section below for more details.

## Load in 8bit or 4bit

The `bitsandbytes` integration supports 8bit and 4bit precision data types, which are useful for loading large models because it saves memory (see the `bitsandbytes` integration [guide](./quantization#bitsandbytes-integration) to learn more). Add the `load_in_8bit` or `load_in_4bit` parameters to [`~PreTrainedModel.from_pretrained`] and set `device_map="auto"` to effectively distribute the model to your hardware:

```py
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

peft_model_id = "ybelkada/opt-350m-lora"
model = AutoModelForCausalLM.from_pretrained(peft_model_id, quantization_config=BitsAndBytesConfig(load_in_8bit=True))
```

## Add a new adapter

You can use [`~peft.PeftModel.add_adapter`] to add a new adapter to a model with an existing adapter as long as the new adapter is the same type as the current one.


key: _node_content
string_value:
{"id_": "38b2e12b-720d-4a27-8e97-be3b10da96f3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/peft", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f08b4827-75ed-4105-b063-d8456c28e127", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/peft", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "transformers"}, "hash": "be7098b1712d5597c51cf0dd6286d1c7d52c17f66a2874c74268ab4e4502eb30", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "78272d89-897d-494e-be2f-db594a7fc9db", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/peft", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "transformers"}, "hash": "921b7e54569fdd0057ffeb4f21bd1d970f070bd33c62bb24ac04f5c18d85d582", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b9df4a47-9d3f-4b44-b7d2-9be9a0f16df1", "node_type": "1", "metadata": {}, "hash": "e02d31294110cd2094935b3338e93851a0aa7fbf820db5e4b4ba6744047aae34", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2780, "end_char_idx": 5167, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

You can also load a PEFT adapter by calling the `load_adapter` method:

```py
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "facebook/opt-350m"
peft_model_id = "ybelkada/opt-350m-lora"

model = AutoModelForCausalLM.from_pretrained(model_id)
model.load_adapter(peft_model_id)
```

Check out the [API documentation](#transformers.integrations.PeftAdapterMixin) section below for more details.

## Load in 8bit or 4bit

The `bitsandbytes` integration supports 8bit and 4bit precision data types, which are useful for loading large models because it saves memory (see the `bitsandbytes` integration [guide](./quantization#bitsandbytes-integration) to learn more). Add the `load_in_8bit` or `load_in_4bit` parameters to [`~PreTrainedModel.from_pretrained`] and set `device_map="auto"` to effectively distribute the model to your hardware:

```py
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

peft_model_id = "ybelkada/opt-350m-lora"
model = AutoModelForCausalLM.from_pretrained(peft_model_id, quantization_config=BitsAndBytesConfig(load_in_8bit=True))
```

## Add a new adapter

You can use [`~peft.PeftModel.add_adapter`] to add a new adapter to a model with an existing adapter as long as the new adapter is the same type as the current one. For example, if you have an existing LoRA adapter attached to a model:

```py
from transformers import AutoModelForCausalLM, OPTForCausalLM, AutoTokenizer
from peft import LoraConfig

model_id = "facebook/opt-350m"
model = AutoModelForCausalLM.from_pretrained(model_id)

lora_config = LoraConfig(
    target_modules=["q_proj", "k_proj"],
    init_lora_weights=False
)

model.add_adapter(lora_config, adapter_name="adapter_1")
```

To add a new adapter:

```py
# attach new adapter with same config
model.add_adapter(lora_config, adapter_name="adapter_2")
```

Now you can use [`~peft.PeftModel.set_adapter`] to set which adapter to use:

```py
# use adapter_1
model.set_adapter("adapter_1")
output = model.generate(**inputs)
print(tokenizer.decode(output_disabled[0], skip_special_tokens=True))

# use adapter_2
model.set_adapter("adapter_2")
output_enabled = model.generate(**inputs)
print(tokenizer.decode(output_enabled[0], skip_special_tokens=True))
```

## Enable and disable adapters

Once you've added an adapter to a model, you can enable or disable the adapter module.


key: _node_content
string_value:
{"id_": "b9df4a47-9d3f-4b44-b7d2-9be9a0f16df1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/peft", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f08b4827-75ed-4105-b063-d8456c28e127", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/peft", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "transformers"}, "hash": "be7098b1712d5597c51cf0dd6286d1c7d52c17f66a2874c74268ab4e4502eb30", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "38b2e12b-720d-4a27-8e97-be3b10da96f3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/peft", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "transformers"}, "hash": "38e6e2f9e0c6e85aa43a5e6082994beadd0b09fd9b900bb2d7439075ac100da8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "be10172c-53c7-42d5-aec0-23c83d1c8bf5", "node_type": "1", "metadata": {}, "hash": "8bc89707a9d994367f824d0c23884bd9c696dc335ef2a4491d941c57e222aafa", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4090, "end_char_idx": 7112, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For example, if you have an existing LoRA adapter attached to a model:

```py
from transformers import AutoModelForCausalLM, OPTForCausalLM, AutoTokenizer
from peft import LoraConfig

model_id = "facebook/opt-350m"
model = AutoModelForCausalLM.from_pretrained(model_id)

lora_config = LoraConfig(
    target_modules=["q_proj", "k_proj"],
    init_lora_weights=False
)

model.add_adapter(lora_config, adapter_name="adapter_1")
```

To add a new adapter:

```py
# attach new adapter with same config
model.add_adapter(lora_config, adapter_name="adapter_2")
```

Now you can use [`~peft.PeftModel.set_adapter`] to set which adapter to use:

```py
# use adapter_1
model.set_adapter("adapter_1")
output = model.generate(**inputs)
print(tokenizer.decode(output_disabled[0], skip_special_tokens=True))

# use adapter_2
model.set_adapter("adapter_2")
output_enabled = model.generate(**inputs)
print(tokenizer.decode(output_enabled[0], skip_special_tokens=True))
```

## Enable and disable adapters

Once you've added an adapter to a model, you can enable or disable the adapter module. To enable the adapter module:

```py
from transformers import AutoModelForCausalLM, OPTForCausalLM, AutoTokenizer
from peft import PeftConfig

model_id = "facebook/opt-350m"
adapter_model_id = "ybelkada/opt-350m-lora"
tokenizer = AutoTokenizer.from_pretrained(model_id)
text = "Hello"
inputs = tokenizer(text, return_tensors="pt")

model = AutoModelForCausalLM.from_pretrained(model_id)
peft_config = PeftConfig.from_pretrained(adapter_model_id)

# to initiate with random weights
peft_config.init_lora_weights = False

model.add_adapter(peft_config)
model.enable_adapters()
output = model.generate(**inputs)
```

To disable the adapter module:

```py
model.disable_adapters()
output = model.generate(**inputs)
```

## Train a PEFT adapter

PEFT adapters are supported by the [`Trainer`] class so that you can train an adapter for your specific use case. It only requires adding a few more lines of code. For example, to train a LoRA adapter:

<Tip>

If you aren't familiar with fine-tuning a model with [`Trainer`], take a look at the [Fine-tune a pretrained model](training) tutorial.

</Tip>

1. Define your adapter configuration with the task type and hyperparameters (see [`~peft.LoraConfig`] for more details about what the hyperparameters do).

```py
from peft import LoraConfig

peft_config = LoraConfig(
    lora_alpha=16,
    lora_dropout=0.1,
    r=64,
    bias="none",
    task_type="CAUSAL_LM",
)
```

2. Add adapter to the model.

```py
model.add_adapter(peft_config)
```

3. Now you can pass the model to [`Trainer`]!

```py
trainer = Trainer(model=model, ...)
trainer.train()
```

To save your trained adapter and load it back:

```py
model.save_pretrained(save_dir)
model = AutoModelForCausalLM.from_pretrained(save_dir)
```

## Add additional trainable layers to a PEFT adapter

You can also fine-tune additional trainable adapters on top of a model that has adapters attached by passing `modules_to_save` in your PEFT config.


key: _node_content
string_value:
{"id_": "be10172c-53c7-42d5-aec0-23c83d1c8bf5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/peft", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f08b4827-75ed-4105-b063-d8456c28e127", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/peft", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "transformers"}, "hash": "be7098b1712d5597c51cf0dd6286d1c7d52c17f66a2874c74268ab4e4502eb30", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b9df4a47-9d3f-4b44-b7d2-9be9a0f16df1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/peft", "title": "Load adapters with \ud83e\udd17 PEFT", "tokens": 2169, "retrieve_doc": true, "source": "transformers"}, "hash": "0888577565f0dff656e787a803fb0695de17fbfd15d1f8cf7210a5f89b7c21ad", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6023, "end_char_idx": 7882, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
It only requires adding a few more lines of code. For example, to train a LoRA adapter:

<Tip>

If you aren't familiar with fine-tuning a model with [`Trainer`], take a look at the [Fine-tune a pretrained model](training) tutorial.

</Tip>

1. Define your adapter configuration with the task type and hyperparameters (see [`~peft.LoraConfig`] for more details about what the hyperparameters do).

```py
from peft import LoraConfig

peft_config = LoraConfig(
    lora_alpha=16,
    lora_dropout=0.1,
    r=64,
    bias="none",
    task_type="CAUSAL_LM",
)
```

2. Add adapter to the model.

```py
model.add_adapter(peft_config)
```

3. Now you can pass the model to [`Trainer`]!

```py
trainer = Trainer(model=model, ...)
trainer.train()
```

To save your trained adapter and load it back:

```py
model.save_pretrained(save_dir)
model = AutoModelForCausalLM.from_pretrained(save_dir)
```

## Add additional trainable layers to a PEFT adapter

You can also fine-tune additional trainable adapters on top of a model that has adapters attached by passing `modules_to_save` in your PEFT config. For example, if you want to also fine-tune the lm_head on top of a model with a LoRA adapter:

```py
from transformers import AutoModelForCausalLM, OPTForCausalLM, AutoTokenizer
from peft import LoraConfig

model_id = "facebook/opt-350m"
model = AutoModelForCausalLM.from_pretrained(model_id)

lora_config = LoraConfig(
    target_modules=["q_proj", "k_proj"],
    modules_to_save=["lm_head"],
)

model.add_adapter(lora_config)
```

## API docs

[[autodoc]] integrations.PeftAdapterMixin
    - load_adapter
    - add_adapter
    - set_adapter
    - disable_adapters
    - enable_adapters
    - active_adapters
    - get_adapter_state_dict




<!--
TODO: (@younesbelkada @stevhliu)
-   Link to PEFT docs for further details
-   Trainer  
-   8-bit / 4-bit examples ?
-->


key: _node_content
string_value:
{"id_": "b6da5fa2-0ebb-462d-b747-aaf58a3bbfcc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers"}, "hash": "80b475f773e435d776ee6a2bd14f6e95d30f3d98b5cef1c6fb431b326003cb09", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1ef853e4-b754-4cc8-93c8-545d9dbc8f17", "node_type": "1", "metadata": {}, "hash": "36c01562d2d54924e66a1d73f7069c5b734782af77a28d1c786cef6dc8296979", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3041, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Translation

[[open-in-colab]]

<Youtube id="1JvfrvZgi6c"/>

Translation converts a sequence of text from one language to another. It is one of several tasks you can formulate as a sequence-to-sequence problem, a powerful framework for returning some output from an input, like translation or summarization. Translation systems are commonly used for translation between different language texts, but it can also be used for speech or some combination in between like text-to-speech or speech-to-text.

This guide will show you how to:

1. Finetune [T5](https://huggingface.co/google-t5/t5-small) on the English-French subset of the [OPUS Books](https://huggingface.co/datasets/opus_books) dataset to translate English text to French.
2. Use your finetuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/translation).

</Tip>

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install transformers datasets evaluate sacrebleu
```

We encourage you to login to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to login:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load OPUS Books dataset

Start by loading the English-French subset of the [OPUS Books](https://huggingface.co/datasets/opus_books) dataset from the 🤗 Datasets library:

```py
>>> from datasets import load_dataset

>>> books = load_dataset("opus_books", "en-fr")
```

Split the dataset into a train and test set with the [`~datasets.Dataset.train_test_split`] method:

```py
>>> books = books["train"].train_test_split(test_size=0.2)
```

Then take a look at an example:

```py
>>> books["train"][0]
{'id': '90560',
 'translation': {'en': 'But this lofty plateau measured only a few fathoms, and soon we reentered Our Element.',
  'fr': 'Mais ce plateau élevé ne mesurait que quelques toises, et bientôt nous fûmes rentrés dans notre élément.'}}
```

`translation`: an English and French translation of the text.

## Preprocess

<Youtube id="XAR8jnZZuUs"/>

The next step is to load a T5 tokenizer to process the English-French language pairs:

```py
>>> from transformers import AutoTokenizer

>>> checkpoint = "google-t5/t5-small"
>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
```

The preprocessing function you want to create needs to:

1. Prefix the input with a prompt so T5 knows this is a translation task. Some models capable of multiple NLP tasks require prompting for specific tasks.
2. Set the target language (French) in the `text_target` parameter to ensure the tokenizer processes the target text correctly. If you don't set `text_target`, the tokenizer processes the target text as English.
3. Truncate sequences to be no longer than the maximum length set by the `max_length` parameter.

```py
>>> source_lang = "en"
>>> target_lang = "fr"
>>> prefix = "translate English to French: "


key: _node_content
string_value:
{"id_": "1ef853e4-b754-4cc8-93c8-545d9dbc8f17", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers"}, "hash": "80b475f773e435d776ee6a2bd14f6e95d30f3d98b5cef1c6fb431b326003cb09", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b6da5fa2-0ebb-462d-b747-aaf58a3bbfcc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers"}, "hash": "5873fc647971bee97da66f601c8e76e51688d79aa2364b9d241d5951ff4efabb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1e29be7c-b38a-4e26-af48-e380385bf6c8", "node_type": "1", "metadata": {}, "hash": "f424f901e7a4e1f207adbbec5fcc63901bddeae43953d7efa8b2d8c73cd3a6aa", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3044, "end_char_idx": 6300, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> def preprocess_function(examples):
...     inputs = [prefix + example[source_lang] for example in examples["translation"]]
...     targets = [example[target_lang] for example in examples["translation"]]
...     model_inputs = tokenizer(inputs, text_target=targets, max_length=128, truncation=True)
...     return model_inputs
```

To apply the preprocessing function over the entire dataset, use 🤗 Datasets [`~datasets.Dataset.map`] method. You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once:

```py
>>> tokenized_books = books.map(preprocess_function, batched=True)
```

Now create a batch of examples using [`DataCollatorForSeq2Seq`]. It's more efficient to *dynamically pad* the sentences to the longest length in a batch during collation, instead of padding the whole dataset to the maximum length.

<frameworkcontent>
<pt>
```py
>>> from transformers import DataCollatorForSeq2Seq

>>> data_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=checkpoint)
```
</pt>
<tf>

```py
>>> from transformers import DataCollatorForSeq2Seq

>>> data_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=checkpoint, return_tensors="tf")
```
</tf>
</frameworkcontent>

## Evaluate

Including a metric during training is often helpful for evaluating your model's performance. You can quickly load a evaluation method with the 🤗 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [SacreBLEU](https://huggingface.co/spaces/evaluate-metric/sacrebleu) metric (see the 🤗 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric):

```py
>>> import evaluate

>>> metric = evaluate.load("sacrebleu")
```

Then create a function that passes your predictions and labels to [`~evaluate.EvaluationModule.compute`] to calculate the SacreBLEU score:

```py
>>> import numpy as np


>>> def postprocess_text(preds, labels):
...     preds = [pred.strip() for pred in preds]
...     labels = [[label.strip()] for label in labels]

...     return preds, labels


>>> def compute_metrics(eval_preds):
...     preds, labels = eval_preds
...     if isinstance(preds, tuple):
...         preds = preds[0]
...     decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)

...     labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
...     decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

...     decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)

...     result = metric.compute(predictions=decoded_preds, references=decoded_labels)
...     result = {"bleu": result["score"]}

...     prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in preds]
...     result["gen_len"] = np.mean(prediction_lens)
...     result = {k: round(v, 4) for k, v in result.items()}
...     return result
```

Your `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.

## Train

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!


key: _node_content
string_value:
{"id_": "1e29be7c-b38a-4e26-af48-e380385bf6c8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers"}, "hash": "80b475f773e435d776ee6a2bd14f6e95d30f3d98b5cef1c6fb431b326003cb09", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1ef853e4-b754-4cc8-93c8-545d9dbc8f17", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers"}, "hash": "dc2d016e45c89c4ffcd54eda80e4525647259ee4f3c32a0b88a9e29c8627c1de", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d3f1319e-ead0-4e02-b006-22f56371f1ae", "node_type": "1", "metadata": {}, "hash": "b908af4f159c3e551b2b0af5e0c38554a20b4d7b9cf208c0c62e066f5fd9ce94", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4994, "end_char_idx": 7197, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> def postprocess_text(preds, labels):
...     preds = [pred.strip() for pred in preds]
...     labels = [[label.strip()] for label in labels]

...     return preds, labels


>>> def compute_metrics(eval_preds):
...     preds, labels = eval_preds
...     if isinstance(preds, tuple):
...         preds = preds[0]
...     decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)

...     labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
...     decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

...     decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)

...     result = metric.compute(predictions=decoded_preds, references=decoded_labels)
...     result = {"bleu": result["score"]}

...     prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in preds]
...     result["gen_len"] = np.mean(prediction_lens)
...     result = {k: round(v, 4) for k, v in result.items()}
...     return result
```

Your `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.

## Train

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load T5 with [`AutoModelForSeq2SeqLM`]:

```py
>>> from transformers import AutoModelForSeq2SeqLM, Seq2SeqTrainingArguments, Seq2SeqTrainer

>>> model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint)
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`Seq2SeqTrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the SacreBLEU metric and save the training checkpoint.
2. Pass the training arguments to [`Seq2SeqTrainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.


key: _node_content
string_value:
{"id_": "d3f1319e-ead0-4e02-b006-22f56371f1ae", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers"}, "hash": "80b475f773e435d776ee6a2bd14f6e95d30f3d98b5cef1c6fb431b326003cb09", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1e29be7c-b38a-4e26-af48-e380385bf6c8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers"}, "hash": "56f27bfc03004b4200188a4d7503e331a70f4cda132fea82ed5a127226b5f7b4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5cb86bd0-f7a2-43a1-988b-7a5c719a9a85", "node_type": "1", "metadata": {}, "hash": "dfa70486787d27016373dbaf027cd410cb9abbf2a3f7aaf98388324fb417132b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6109, "end_char_idx": 8268, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Train

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load T5 with [`AutoModelForSeq2SeqLM`]:

```py
>>> from transformers import AutoModelForSeq2SeqLM, Seq2SeqTrainingArguments, Seq2SeqTrainer

>>> model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint)
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`Seq2SeqTrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the SacreBLEU metric and save the training checkpoint.
2. Pass the training arguments to [`Seq2SeqTrainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = Seq2SeqTrainingArguments(
...     output_dir="my_awesome_opus_books_model",
...     eval_strategy="epoch",
...     learning_rate=2e-5,
...     per_device_train_batch_size=16,
...     per_device_eval_batch_size=16,
...     weight_decay=0.01,
...     save_total_limit=3,
...     num_train_epochs=2,
...     predict_with_generate=True,
...     fp16=True,
...     push_to_hub=True,
... )

>>> trainer = Seq2SeqTrainer(
...     model=model,
...     args=training_args,
...     train_dataset=tokenized_books["train"],
...     eval_dataset=tokenized_books["test"],
...     tokenizer=tokenizer,
...     data_collator=data_collator,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!


key: _node_content
string_value:
{"id_": "5cb86bd0-f7a2-43a1-988b-7a5c719a9a85", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers"}, "hash": "80b475f773e435d776ee6a2bd14f6e95d30f3d98b5cef1c6fb431b326003cb09", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d3f1319e-ead0-4e02-b006-22f56371f1ae", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers"}, "hash": "3d6e2ea9f3444122d12cd7f6d9d08fe87898c44b507c6f4afcd65408ed6fecc9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "40f566ef-9409-49f8-b39d-4d13607c907e", "node_type": "1", "metadata": {}, "hash": "60de0cecc43834dc4ca4d83458ffac3949d79a8ae69d8a6ff74334dc3e2572b7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6765, "end_char_idx": 9771, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the SacreBLEU metric and save the training checkpoint.
2. Pass the training arguments to [`Seq2SeqTrainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = Seq2SeqTrainingArguments(
...     output_dir="my_awesome_opus_books_model",
...     eval_strategy="epoch",
...     learning_rate=2e-5,
...     per_device_train_batch_size=16,
...     per_device_eval_batch_size=16,
...     weight_decay=0.01,
...     save_total_limit=3,
...     num_train_epochs=2,
...     predict_with_generate=True,
...     fp16=True,
...     push_to_hub=True,
... )

>>> trainer = Seq2SeqTrainer(
...     model=model,
...     args=training_args,
...     train_dataset=tokenized_books["train"],
...     eval_dataset=tokenized_books["test"],
...     tokenizer=tokenizer,
...     data_collator=data_collator,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!

</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import AdamWeightDecay

>>> optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)
```

Then you can load T5 with [`TFAutoModelForSeq2SeqLM`]:

```py
>>> from transformers import TFAutoModelForSeq2SeqLM

>>> model = TFAutoModelForSeq2SeqLM.from_pretrained(checkpoint)
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> tf_train_set = model.prepare_tf_dataset(
...     tokenized_books["train"],
...     shuffle=True,
...     batch_size=16,
...     collate_fn=data_collator,
... )

>>> tf_test_set = model.prepare_tf_dataset(
...     tokenized_books["test"],
...     shuffle=False,
...     batch_size=16,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method). Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> import tensorflow as tf

>>> model.compile(optimizer=optimizer)  # No loss argument!
```

The last two things to setup before you start training is to compute the SacreBLEU metric from the predictions, and provide a way to push your model to the Hub. Both are done by using [Keras callbacks](../main_classes/keras_callbacks).


key: _node_content
string_value:
{"id_": "40f566ef-9409-49f8-b39d-4d13607c907e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers"}, "hash": "80b475f773e435d776ee6a2bd14f6e95d30f3d98b5cef1c6fb431b326003cb09", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5cb86bd0-f7a2-43a1-988b-7a5c719a9a85", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers"}, "hash": "99cce7458dadef385934746b057ee85f68db20486bf52b8114cf39395d2d89e5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7f56dd5f-7640-44d8-b11f-da13aafd575f", "node_type": "1", "metadata": {}, "hash": "8fce709c167c86bdb02defdbfc2e7ccd937efd2908b41d93774123f4918b6eb1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8270, "end_char_idx": 11501, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import AdamWeightDecay

>>> optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)
```

Then you can load T5 with [`TFAutoModelForSeq2SeqLM`]:

```py
>>> from transformers import TFAutoModelForSeq2SeqLM

>>> model = TFAutoModelForSeq2SeqLM.from_pretrained(checkpoint)
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> tf_train_set = model.prepare_tf_dataset(
...     tokenized_books["train"],
...     shuffle=True,
...     batch_size=16,
...     collate_fn=data_collator,
... )

>>> tf_test_set = model.prepare_tf_dataset(
...     tokenized_books["test"],
...     shuffle=False,
...     batch_size=16,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method). Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> import tensorflow as tf

>>> model.compile(optimizer=optimizer)  # No loss argument!
```

The last two things to setup before you start training is to compute the SacreBLEU metric from the predictions, and provide a way to push your model to the Hub. Both are done by using [Keras callbacks](../main_classes/keras_callbacks).

Pass your `compute_metrics` function to [`~transformers.KerasMetricCallback`]:

```py
>>> from transformers.keras_callbacks import KerasMetricCallback

>>> metric_callback = KerasMetricCallback(metric_fn=compute_metrics, eval_dataset=tf_validation_set)
```

Specify where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:

```py
>>> from transformers.keras_callbacks import PushToHubCallback

>>> push_to_hub_callback = PushToHubCallback(
...     output_dir="my_awesome_opus_books_model",
...     tokenizer=tokenizer,
... )
```

Then bundle your callbacks together:

```py
>>> callbacks = [metric_callback, push_to_hub_callback]
```

Finally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callbacks to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_test_set, epochs=3, callbacks=callbacks)
```

Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
</tf>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for translation, take a look at the corresponding
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation-tf.ipynb).

</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Come up with some text you'd like to translate to another language. For T5, you need to prefix your input depending on the task you're working on.


key: _node_content
string_value:
{"id_": "7f56dd5f-7640-44d8-b11f-da13aafd575f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers"}, "hash": "80b475f773e435d776ee6a2bd14f6e95d30f3d98b5cef1c6fb431b326003cb09", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "40f566ef-9409-49f8-b39d-4d13607c907e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers"}, "hash": "20e01f7e2014e7568f27127b13221c9e05cea815045abe49d2cf9df579786114", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "de36cced-b418-4c00-9a4b-b80a37138c9e", "node_type": "1", "metadata": {}, "hash": "cadd133cb6e157b37e75e40f88d9a0f8eba34b1dbf7342ad37992d5fb7483578", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10486, "end_char_idx": 13498, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callbacks to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_test_set, epochs=3, callbacks=callbacks)
```

Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
</tf>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for translation, take a look at the corresponding
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/translation-tf.ipynb).

</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Come up with some text you'd like to translate to another language. For T5, you need to prefix your input depending on the task you're working on. For translation from English to French, you should prefix your input as shown below:

```py
>>> text = "translate English to French: Legumes share resources with nitrogen-fixing bacteria."
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for translation with your model, and pass your text to it:

```py
>>> from transformers import pipeline

# Change `xx` to the language of the input and `yy` to the language of the desired output.
# Examples: "en" for English, "fr" for French, "de" for German, "es" for Spanish, "zh" for Chinese, etc; translation_en_to_fr translates English to French
# You can view all the lists of languages here - https://huggingface.co/languages
>>> translator = pipeline("translation_xx_to_yy", model="my_awesome_opus_books_model")
>>> translator(text)
[{'translation_text': 'Legumes partagent des ressources avec des bactéries azotantes.'}]
```

You can also manually replicate the results of the `pipeline` if you'd like:

<frameworkcontent>
<pt>
Tokenize the text and return the `input_ids` as PyTorch tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("my_awesome_opus_books_model")
>>> inputs = tokenizer(text, return_tensors="pt").input_ids
```

Use the [`~generation.GenerationMixin.generate`] method to create the translation. For more details about the different text generation strategies and parameters for controlling generation, check out the [Text Generation](../main_classes/text_generation) API.

```py
>>> from transformers import AutoModelForSeq2SeqLM

>>> model = AutoModelForSeq2SeqLM.from_pretrained("my_awesome_opus_books_model")
>>> outputs = model.generate(inputs, max_new_tokens=40, do_sample=True, top_k=30, top_p=0.95)
```

Decode the generated token ids back into text:

```py
>>> tokenizer.decode(outputs[0], skip_special_tokens=True)
'Les lignées partagent des ressources avec des bactéries enfixant l'azote.'


key: _node_content
string_value:
{"id_": "de36cced-b418-4c00-9a4b-b80a37138c9e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7c667a00-4d05-4905-948b-7c06e021fd96", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers"}, "hash": "80b475f773e435d776ee6a2bd14f6e95d30f3d98b5cef1c6fb431b326003cb09", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7f56dd5f-7640-44d8-b11f-da13aafd575f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/translation", "title": "Translation", "tokens": 3629, "retrieve_doc": true, "source": "transformers"}, "hash": "a81898bc91ccc382b2212d260d0ee38bf1bbef8be1b429f7c9c6c94ed90b8ba0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12012, "end_char_idx": 14522, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Examples: "en" for English, "fr" for French, "de" for German, "es" for Spanish, "zh" for Chinese, etc; translation_en_to_fr translates English to French
# You can view all the lists of languages here - https://huggingface.co/languages
>>> translator = pipeline("translation_xx_to_yy", model="my_awesome_opus_books_model")
>>> translator(text)
[{'translation_text': 'Legumes partagent des ressources avec des bactéries azotantes.'}]
```

You can also manually replicate the results of the `pipeline` if you'd like:

<frameworkcontent>
<pt>
Tokenize the text and return the `input_ids` as PyTorch tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("my_awesome_opus_books_model")
>>> inputs = tokenizer(text, return_tensors="pt").input_ids
```

Use the [`~generation.GenerationMixin.generate`] method to create the translation. For more details about the different text generation strategies and parameters for controlling generation, check out the [Text Generation](../main_classes/text_generation) API.

```py
>>> from transformers import AutoModelForSeq2SeqLM

>>> model = AutoModelForSeq2SeqLM.from_pretrained("my_awesome_opus_books_model")
>>> outputs = model.generate(inputs, max_new_tokens=40, do_sample=True, top_k=30, top_p=0.95)
```

Decode the generated token ids back into text:

```py
>>> tokenizer.decode(outputs[0], skip_special_tokens=True)
'Les lignées partagent des ressources avec des bactéries enfixant l'azote.'
```
</pt>
<tf>
Tokenize the text and return the `input_ids` as TensorFlow tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("my_awesome_opus_books_model")
>>> inputs = tokenizer(text, return_tensors="tf").input_ids
```

Use the [`~transformers.generation_tf_utils.TFGenerationMixin.generate`] method to create the translation. For more details about the different text generation strategies and parameters for controlling generation, check out the [Text Generation](../main_classes/text_generation) API.

```py
>>> from transformers import TFAutoModelForSeq2SeqLM

>>> model = TFAutoModelForSeq2SeqLM.from_pretrained("my_awesome_opus_books_model")
>>> outputs = model.generate(inputs, max_new_tokens=40, do_sample=True, top_k=30, top_p=0.95)
```

Decode the generated token ids back into text:

```py
>>> tokenizer.decode(outputs[0], skip_special_tokens=True)
'Les lugumes partagent les ressources avec des bactéries fixatrices d'azote.'
```
</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "a2dafb25-55dc-4d9f-a356-76e3ff933e20", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "hash": "bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "079ba672-d6f9-484f-9dc9-608e33d4f348", "node_type": "1", "metadata": {}, "hash": "3311d53bd580533cd05c4af34f34169d9ecb33adad1a073090d19bfefc810391", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3090, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Document Question Answering

[[open-in-colab]]

Document Question Answering, also referred to as Document Visual Question Answering, is a task that involves providing
answers to questions posed about document images. The input to models supporting this task is typically a combination of an image and
a question, and the output is an answer expressed in natural language. These models utilize multiple modalities, including
text, the positions of words (bounding boxes), and the image itself.

This guide illustrates how to:

- Fine-tune [LayoutLMv2](../model_doc/layoutlmv2) on the [DocVQA dataset](https://huggingface.co/datasets/nielsr/docvqa_1200_examples_donut).
- Use your fine-tuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/image-to-text)

</Tip>

LayoutLMv2 solves the document question-answering task by adding a question-answering head on top of the final hidden
states of the tokens, to predict the positions of the start and end tokens of the
answer. In other words, the problem is treated as extractive question answering: given the context, extract which piece
of information answers the question. The context comes from the output of an OCR engine, here it is Google's Tesseract.

Before you begin, make sure you have all the necessary libraries installed. LayoutLMv2 depends on detectron2, torchvision and tesseract.

```bash
pip install -q transformers datasets
```

```bash
pip install 'git+https://github.com/facebookresearch/detectron2.git'
pip install torchvision
```

```bash
sudo apt install tesseract-ocr
pip install -q pytesseract
```

Once you have installed all of the dependencies, restart your runtime.

We encourage you to share your model with the community. Log in to your Hugging Face account to upload it to the 🤗 Hub.
When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

Let's define some global variables.

```py
>>> model_checkpoint = "microsoft/layoutlmv2-base-uncased"
>>> batch_size = 4
```

## Load the data

In this guide we use a small sample of preprocessed DocVQA that you can find on 🤗 Hub. If you'd like to use the full
DocVQA dataset, you can register and download it on [DocVQA homepage](https://rrc.cvc.uab.es/?ch=17). If you do so, to
proceed with this guide check out [how to load files into a 🤗 dataset](https://huggingface.co/docs/datasets/loading#local-and-remote-files).

```py
>>> from datasets import load_dataset

>>> dataset = load_dataset("nielsr/docvqa_1200_examples")
>>> dataset
DatasetDict({
    train: Dataset({
        features: ['id', 'image', 'query', 'answers', 'words', 'bounding_boxes', 'answer'],
        num_rows: 1000
    })
    test: Dataset({
        features: ['id', 'image', 'query', 'answers', 'words', 'bounding_boxes', 'answer'],
        num_rows: 200
    })
})
```

As you can see, the dataset is split into train and test sets already. Take a look at a random example to familiarize
yourself with the features.


key: _node_content
string_value:
{"id_": "079ba672-d6f9-484f-9dc9-608e33d4f348", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "hash": "bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a2dafb25-55dc-4d9f-a356-76e3ff933e20", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "hash": "cac7c435be5c5d274480dd3315ba006b56fcba1820decbf902461fa6bd5b4c3f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "84176b39-8f23-431f-8c65-cf04b433c4ed", "node_type": "1", "metadata": {}, "hash": "73b6abd76c0efa6563d6d8be28363ac08bfc38810478c4a81fcaa1c24f7e7f8b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1758, "end_char_idx": 4821, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We encourage you to share your model with the community. Log in to your Hugging Face account to upload it to the 🤗 Hub.
When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

Let's define some global variables.

```py
>>> model_checkpoint = "microsoft/layoutlmv2-base-uncased"
>>> batch_size = 4
```

## Load the data

In this guide we use a small sample of preprocessed DocVQA that you can find on 🤗 Hub. If you'd like to use the full
DocVQA dataset, you can register and download it on [DocVQA homepage](https://rrc.cvc.uab.es/?ch=17). If you do so, to
proceed with this guide check out [how to load files into a 🤗 dataset](https://huggingface.co/docs/datasets/loading#local-and-remote-files).

```py
>>> from datasets import load_dataset

>>> dataset = load_dataset("nielsr/docvqa_1200_examples")
>>> dataset
DatasetDict({
    train: Dataset({
        features: ['id', 'image', 'query', 'answers', 'words', 'bounding_boxes', 'answer'],
        num_rows: 1000
    })
    test: Dataset({
        features: ['id', 'image', 'query', 'answers', 'words', 'bounding_boxes', 'answer'],
        num_rows: 200
    })
})
```

As you can see, the dataset is split into train and test sets already. Take a look at a random example to familiarize
yourself with the features.

```py
>>> dataset["train"].features
```

Here's what the individual fields represent:
* `id`: the example's id
* `image`: a PIL.Image.Image object containing the document image
* `query`: the question string - natural language asked question, in several languages
* `answers`: a list of correct answers provided by human annotators
* `words` and `bounding_boxes`: the results of OCR, which we will not use here
* `answer`: an answer matched by a different model which we will not use here

Let's leave only English questions, and drop the `answer` feature which appears to contain predictions by another model.
We'll also take the first of the answers from the set provided by the annotators. Alternatively, you can randomly sample it.

```py
>>> updated_dataset = dataset.map(lambda example: {"question": example["query"]["en"]}, remove_columns=["query"])
>>> updated_dataset = updated_dataset.map(
...     lambda example: {"answer": example["answers"][0]}, remove_columns=["answer", "answers"]
... )
```

Note that the LayoutLMv2 checkpoint that we use in this guide has been trained with `max_position_embeddings = 512` (you can
find this information in the [checkpoint's `config.json` file](https://huggingface.co/microsoft/layoutlmv2-base-uncased/blob/main/config.json#L18)).
We can truncate the examples but to avoid the situation where the answer might be at the end of a large document and end up truncated,
here we'll remove the few examples where the embedding is likely to end up longer than 512.
If most of the documents in your dataset are long, you can implement a sliding window strategy - check out [this notebook](https://github.com/huggingface/notebooks/blob/main/examples/question_answering.ipynb) for details.


key: _node_content
string_value:
{"id_": "84176b39-8f23-431f-8c65-cf04b433c4ed", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "hash": "bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "079ba672-d6f9-484f-9dc9-608e33d4f348", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "hash": "cad004b9bfb0723f48bb1329c11e86bf0c9e4704ffa958fe04db04f63b4d531c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "99444eec-1809-4345-8d47-21ee72f16c07", "node_type": "1", "metadata": {}, "hash": "7c636516b9b7234d61be84065d312d4873b3300bbe82fb0cebb494b32fdf33e4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3092, "end_char_idx": 6396, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> dataset["train"].features
```

Here's what the individual fields represent:
* `id`: the example's id
* `image`: a PIL.Image.Image object containing the document image
* `query`: the question string - natural language asked question, in several languages
* `answers`: a list of correct answers provided by human annotators
* `words` and `bounding_boxes`: the results of OCR, which we will not use here
* `answer`: an answer matched by a different model which we will not use here

Let's leave only English questions, and drop the `answer` feature which appears to contain predictions by another model.
We'll also take the first of the answers from the set provided by the annotators. Alternatively, you can randomly sample it.

```py
>>> updated_dataset = dataset.map(lambda example: {"question": example["query"]["en"]}, remove_columns=["query"])
>>> updated_dataset = updated_dataset.map(
...     lambda example: {"answer": example["answers"][0]}, remove_columns=["answer", "answers"]
... )
```

Note that the LayoutLMv2 checkpoint that we use in this guide has been trained with `max_position_embeddings = 512` (you can
find this information in the [checkpoint's `config.json` file](https://huggingface.co/microsoft/layoutlmv2-base-uncased/blob/main/config.json#L18)).
We can truncate the examples but to avoid the situation where the answer might be at the end of a large document and end up truncated,
here we'll remove the few examples where the embedding is likely to end up longer than 512.
If most of the documents in your dataset are long, you can implement a sliding window strategy - check out [this notebook](https://github.com/huggingface/notebooks/blob/main/examples/question_answering.ipynb) for details.

```py
>>> updated_dataset = updated_dataset.filter(lambda x: len(x["words"]) + len(x["question"].split()) < 512)
```

At this point let's also remove the OCR features from this dataset. These are a result of OCR for fine-tuning a different
model. They would still require some processing if we wanted to use them, as they do not match the input requirements
of the model we use in this guide. Instead, we can use the [`LayoutLMv2Processor`] on the original data for both OCR and
tokenization. This way we'll get the inputs that match model's expected input. If you want to process images manually,
check out the [`LayoutLMv2` model documentation](../model_doc/layoutlmv2) to learn what input format the model expects.

```py
>>> updated_dataset = updated_dataset.remove_columns("words")
>>> updated_dataset = updated_dataset.remove_columns("bounding_boxes")
```

Finally, the data exploration won't be complete if we don't peek at an image example.

```py
>>> updated_dataset["train"][11]["image"]
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/docvqa_example.jpg" alt="DocVQA Image Example"/>
 </div>

## Preprocess the data

The Document Question Answering task is a multimodal task, and you need to make sure that the inputs from each modality
are preprocessed according to the model's expectations. Let's start by loading the [`LayoutLMv2Processor`], which internally combines an image processor that can handle image data and a tokenizer that can encode text data.


key: _node_content
string_value:
{"id_": "99444eec-1809-4345-8d47-21ee72f16c07", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "hash": "bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "84176b39-8f23-431f-8c65-cf04b433c4ed", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "hash": "3c5d624efd9c575748938da1e63a1b0a2432cb041797aa8f38eca2aa701d245c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "39618fb6-68bf-485b-936c-89bd83f4658f", "node_type": "1", "metadata": {}, "hash": "ef191df0b3d976823f38e66de4ec3b0ed801134b0ab52fe498f73f629556a331", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4823, "end_char_idx": 8148, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> updated_dataset = updated_dataset.filter(lambda x: len(x["words"]) + len(x["question"].split()) < 512)
```

At this point let's also remove the OCR features from this dataset. These are a result of OCR for fine-tuning a different
model. They would still require some processing if we wanted to use them, as they do not match the input requirements
of the model we use in this guide. Instead, we can use the [`LayoutLMv2Processor`] on the original data for both OCR and
tokenization. This way we'll get the inputs that match model's expected input. If you want to process images manually,
check out the [`LayoutLMv2` model documentation](../model_doc/layoutlmv2) to learn what input format the model expects.

```py
>>> updated_dataset = updated_dataset.remove_columns("words")
>>> updated_dataset = updated_dataset.remove_columns("bounding_boxes")
```

Finally, the data exploration won't be complete if we don't peek at an image example.

```py
>>> updated_dataset["train"][11]["image"]
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/docvqa_example.jpg" alt="DocVQA Image Example"/>
 </div>

## Preprocess the data

The Document Question Answering task is a multimodal task, and you need to make sure that the inputs from each modality
are preprocessed according to the model's expectations. Let's start by loading the [`LayoutLMv2Processor`], which internally combines an image processor that can handle image data and a tokenizer that can encode text data.

```py
>>> from transformers import AutoProcessor

>>> processor = AutoProcessor.from_pretrained(model_checkpoint)
```

### Preprocessing document images

First, let's prepare the document images for the model with the help of the `image_processor` from the processor.
By default, image processor resizes the images to 224x224, makes sure they have the correct order of color channels,
applies OCR with tesseract to get words and normalized bounding boxes. In this tutorial, all of these defaults are exactly what we need.
Write a function that applies the default image processing to a batch of images and returns the results of OCR.

```py
>>> image_processor = processor.image_processor


>>> def get_ocr_words_and_boxes(examples):
...     images = [image.convert("RGB") for image in examples["image"]]
...     encoded_inputs = image_processor(images)

...     examples["image"] = encoded_inputs.pixel_values
...     examples["words"] = encoded_inputs.words
...     examples["boxes"] = encoded_inputs.boxes

...     return examples
```

To apply this preprocessing to the entire dataset in a fast way, use [`~datasets.Dataset.map`].

```py
>>> dataset_with_ocr = updated_dataset.map(get_ocr_words_and_boxes, batched=True, batch_size=2)
```

### Preprocessing text data

Once we have applied OCR to the images, we need to encode the text part of the dataset to prepare it for the model.
This involves converting the words and boxes that we got in the previous step to token-level `input_ids`, `attention_mask`,
`token_type_ids` and `bbox`. For preprocessing text, we'll need the `tokenizer` from the processor.

```py
>>> tokenizer = processor.tokenizer
```

On top of the preprocessing mentioned above, we also need to add the labels for the model.


key: _node_content
string_value:
{"id_": "39618fb6-68bf-485b-936c-89bd83f4658f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "hash": "bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "99444eec-1809-4345-8d47-21ee72f16c07", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "hash": "c2f66e10e73ac773e3c037d0c7640a6d28664a81bef4873ce48f3efcd3c2312e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9f316b23-bd3e-42d9-a720-b635fa3c708a", "node_type": "1", "metadata": {}, "hash": "213a74c67dd3d49267d3a4713aed4d3b93947a49030f0b9a211a41b332a28ddc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6398, "end_char_idx": 9193, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> from transformers import AutoProcessor

>>> processor = AutoProcessor.from_pretrained(model_checkpoint)
```

### Preprocessing document images

First, let's prepare the document images for the model with the help of the `image_processor` from the processor.
By default, image processor resizes the images to 224x224, makes sure they have the correct order of color channels,
applies OCR with tesseract to get words and normalized bounding boxes. In this tutorial, all of these defaults are exactly what we need.
Write a function that applies the default image processing to a batch of images and returns the results of OCR.

```py
>>> image_processor = processor.image_processor


>>> def get_ocr_words_and_boxes(examples):
...     images = [image.convert("RGB") for image in examples["image"]]
...     encoded_inputs = image_processor(images)

...     examples["image"] = encoded_inputs.pixel_values
...     examples["words"] = encoded_inputs.words
...     examples["boxes"] = encoded_inputs.boxes

...     return examples
```

To apply this preprocessing to the entire dataset in a fast way, use [`~datasets.Dataset.map`].

```py
>>> dataset_with_ocr = updated_dataset.map(get_ocr_words_and_boxes, batched=True, batch_size=2)
```

### Preprocessing text data

Once we have applied OCR to the images, we need to encode the text part of the dataset to prepare it for the model.
This involves converting the words and boxes that we got in the previous step to token-level `input_ids`, `attention_mask`,
`token_type_ids` and `bbox`. For preprocessing text, we'll need the `tokenizer` from the processor.

```py
>>> tokenizer = processor.tokenizer
```

On top of the preprocessing mentioned above, we also need to add the labels for the model. For `xxxForQuestionAnswering` models
in 🤗 Transformers, the labels consist of the `start_positions` and `end_positions`, indicating which token is at the
start and which token is at the end of the answer.

Let's start with that. Define a helper function that can find a sublist (the answer split into words) in a larger list (the words list).

This function will take two lists as input, `words_list` and `answer_list`. It will then iterate over the `words_list` and check
if the current word in the `words_list` (words_list[i]) is equal to the first word of answer_list (answer_list[0]) and if
the sublist of `words_list` starting from the current word and of the same length as `answer_list` is equal `to answer_list`.
If this condition is true, it means that a match has been found, and the function will record the match, its starting index (idx),
and its ending index (idx + len(answer_list) - 1). If more than one match was found, the function will return only the first one.
If no match is found, the function returns (`None`, 0, and 0).


key: _node_content
string_value:
{"id_": "9f316b23-bd3e-42d9-a720-b635fa3c708a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "hash": "bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "39618fb6-68bf-485b-936c-89bd83f4658f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "hash": "506d558068d4251ed82cbd3df45135a6aa984d9f1c95aad9a79277de941a3f1a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "891aa365-c5fe-4bbe-ad49-574efc9171c3", "node_type": "1", "metadata": {}, "hash": "1be996007826f65ac69e35b683e3a4fd5423b560e465f82e8437dd414e3bf2e8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7534, "end_char_idx": 10500, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> dataset_with_ocr = updated_dataset.map(get_ocr_words_and_boxes, batched=True, batch_size=2)
```

### Preprocessing text data

Once we have applied OCR to the images, we need to encode the text part of the dataset to prepare it for the model.
This involves converting the words and boxes that we got in the previous step to token-level `input_ids`, `attention_mask`,
`token_type_ids` and `bbox`. For preprocessing text, we'll need the `tokenizer` from the processor.

```py
>>> tokenizer = processor.tokenizer
```

On top of the preprocessing mentioned above, we also need to add the labels for the model. For `xxxForQuestionAnswering` models
in 🤗 Transformers, the labels consist of the `start_positions` and `end_positions`, indicating which token is at the
start and which token is at the end of the answer.

Let's start with that. Define a helper function that can find a sublist (the answer split into words) in a larger list (the words list).

This function will take two lists as input, `words_list` and `answer_list`. It will then iterate over the `words_list` and check
if the current word in the `words_list` (words_list[i]) is equal to the first word of answer_list (answer_list[0]) and if
the sublist of `words_list` starting from the current word and of the same length as `answer_list` is equal `to answer_list`.
If this condition is true, it means that a match has been found, and the function will record the match, its starting index (idx),
and its ending index (idx + len(answer_list) - 1). If more than one match was found, the function will return only the first one.
If no match is found, the function returns (`None`, 0, and 0).

```py
>>> def subfinder(words_list, answer_list):
...     matches = []
...     start_indices = []
...     end_indices = []
...     for idx, i in enumerate(range(len(words_list))):
...         if words_list[i] == answer_list[0] and words_list[i : i + len(answer_list)] == answer_list:
...             matches.append(answer_list)
...             start_indices.append(idx)
...             end_indices.append(idx + len(answer_list) - 1)
...     if matches:
...         return matches[0], start_indices[0], end_indices[0]
...     else:
...         return None, 0, 0
```

To illustrate how this function finds the position of the answer, let's use it on an example:

```py
>>> example = dataset_with_ocr["train"][1]
>>> words = [word.lower() for word in example["words"]]
>>> match, word_idx_start, word_idx_end = subfinder(words, example["answer"].lower().split())
>>> print("Question: ", example["question"])
>>> print("Words:", words)
>>> print("Answer: ", example["answer"])
>>> print("start_index", word_idx_start)
>>> print("end_index", word_idx_end)
Question:  Who is in  cc in this letter?
Words: ['wie', 'baw', 'brown', '&', 'williamson', 'tobacco', 'corporation', 'research', '&', 'development', 'internal', 'correspondence', 'to:', 'r.', 'h.', 'honeycutt', 'ce:', 't.f.', 'riehl', 'from:', '.', 'c.j.


key: _node_content
string_value:
{"id_": "891aa365-c5fe-4bbe-ad49-574efc9171c3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "hash": "bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9f316b23-bd3e-42d9-a720-b635fa3c708a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "hash": "e72099d6650ec9fc97dad590d0f3864bca50700bec6225d43fe51b164161ec2a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b1045981-da24-4f84-a244-e8c8a194e0da", "node_type": "1", "metadata": {}, "hash": "ea9c2ad6fe327da3d8086b56381da8f049b2d2abf9ddd19de0e2b8824ebccb45", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9052, "end_char_idx": 11606, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If more than one match was found, the function will return only the first one.
If no match is found, the function returns (`None`, 0, and 0).

```py
>>> def subfinder(words_list, answer_list):
...     matches = []
...     start_indices = []
...     end_indices = []
...     for idx, i in enumerate(range(len(words_list))):
...         if words_list[i] == answer_list[0] and words_list[i : i + len(answer_list)] == answer_list:
...             matches.append(answer_list)
...             start_indices.append(idx)
...             end_indices.append(idx + len(answer_list) - 1)
...     if matches:
...         return matches[0], start_indices[0], end_indices[0]
...     else:
...         return None, 0, 0
```

To illustrate how this function finds the position of the answer, let's use it on an example:

```py
>>> example = dataset_with_ocr["train"][1]
>>> words = [word.lower() for word in example["words"]]
>>> match, word_idx_start, word_idx_end = subfinder(words, example["answer"].lower().split())
>>> print("Question: ", example["question"])
>>> print("Words:", words)
>>> print("Answer: ", example["answer"])
>>> print("start_index", word_idx_start)
>>> print("end_index", word_idx_end)
Question:  Who is in  cc in this letter?
Words: ['wie', 'baw', 'brown', '&', 'williamson', 'tobacco', 'corporation', 'research', '&', 'development', 'internal', 'correspondence', 'to:', 'r.', 'h.', 'honeycutt', 'ce:', 't.f.', 'riehl', 'from:', '.', 'c.j.', 'cook', 'date:', 'may', '8,', '1995', 'subject:', 'review', 'of', 'existing', 'brainstorming', 'ideas/483', 'the', 'major', 'function', 'of', 'the', 'product', 'innovation', 'graup', 'is', 'to', 'develop', 'marketable', 'nove!', 'products', 'that', 'would', 'be', 'profitable', 'to', 'manufacture', 'and', 'sell.', 'novel', 'is', 'defined', 'as:', 'of', 'a', 'new', 'kind,', 'or', 'different', 'from', 'anything', 'seen', 'or', 'known', 'before.', 'innovation', 'is', 'defined', 'as:', 'something', 'new', 'or', 'different', 'introduced;', 'act', 'of', 'innovating;', 'introduction', 'of', 'new', 'things', 'or', 'methods.', 'the', 'products', 'may', 'incorporate', 'the', 'latest', 'technologies,', 'materials', 'and', 'know-how', 'available', 'to', 'give', 'then', 'a', 'unique', 'taste', 'or', 'look.', 'the', 'first', 'task', 'of', 'the', 'product', 'innovation', 'group', 'was', 'to', 'assemble,', 'review', 'and', 'categorize', 'a', 'list', 'of', 'existing', 'brainstorming', 'ideas.', 'ideas', 'were', 'grouped', 'into', 'two', 'major', 'categories', 'labeled', 'appearance', 'and', 'taste/aroma.


key: _node_content
string_value:
{"id_": "b1045981-da24-4f84-a244-e8c8a194e0da", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "hash": "bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "891aa365-c5fe-4bbe-ad49-574efc9171c3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "hash": "8ab87b63fed3d835f7970c7b3cb08a176b735062628e597684361dd1ee6c9585", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "49a6feb8-c314-4030-a065-f063efbf2bcb", "node_type": "1", "metadata": {}, "hash": "6c23259061dd04b077ba3316885b24d1c5eb20b10475aadfe7b2a78f363e73f5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10492, "end_char_idx": 12569, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
', 'c.j.', 'cook', 'date:', 'may', '8,', '1995', 'subject:', 'review', 'of', 'existing', 'brainstorming', 'ideas/483', 'the', 'major', 'function', 'of', 'the', 'product', 'innovation', 'graup', 'is', 'to', 'develop', 'marketable', 'nove!', 'products', 'that', 'would', 'be', 'profitable', 'to', 'manufacture', 'and', 'sell.', 'novel', 'is', 'defined', 'as:', 'of', 'a', 'new', 'kind,', 'or', 'different', 'from', 'anything', 'seen', 'or', 'known', 'before.', 'innovation', 'is', 'defined', 'as:', 'something', 'new', 'or', 'different', 'introduced;', 'act', 'of', 'innovating;', 'introduction', 'of', 'new', 'things', 'or', 'methods.', 'the', 'products', 'may', 'incorporate', 'the', 'latest', 'technologies,', 'materials', 'and', 'know-how', 'available', 'to', 'give', 'then', 'a', 'unique', 'taste', 'or', 'look.', 'the', 'first', 'task', 'of', 'the', 'product', 'innovation', 'group', 'was', 'to', 'assemble,', 'review', 'and', 'categorize', 'a', 'list', 'of', 'existing', 'brainstorming', 'ideas.', 'ideas', 'were', 'grouped', 'into', 'two', 'major', 'categories', 'labeled', 'appearance', 'and', 'taste/aroma.', 'these', 'categories', 'are', 'used', 'for', 'novel', 'products', 'that', 'may', 'differ', 'from', 'a', 'visual', 'and/or', 'taste/aroma', 'point', 'of', 'view', 'compared', 'to', 'canventional', 'cigarettes.', 'other', 'categories', 'include', 'a', 'combination', 'of', 'the', 'above,', 'filters,', 'packaging', 'and', 'brand', 'extensions.', 'appearance', 'this', 'category', 'is', 'used', 'for', 'novel', 'cigarette', 'constructions', 'that', 'yield', 'visually', 'different', 'products', 'with', 'minimal', 'changes', 'in', 'smoke', 'chemistry', 'two', 'cigarettes', 'in', 'cne.', 'emulti-plug', 'te', 'build', 'yaur', 'awn', 'cigarette.', 'eswitchable', 'menthol', 'or', 'non', 'menthol', 'cigarette.', '*cigarettes', 'with', 'interspaced', 'perforations', 'to', 'enable', 'smoker', 'to', 'separate', 'unburned', 'section', 'for', 'future', 'smoking.', '«short', 'cigarette,', 'tobacco', 'section', '30', 'mm.', '«extremely', 'fast', 'buming', 'cigarette.


key: _node_content
string_value:
{"id_": "49a6feb8-c314-4030-a065-f063efbf2bcb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "hash": "bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b1045981-da24-4f84-a244-e8c8a194e0da", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "hash": "74d3177ca4c57ef59428745a8e3b053126181647678349f42aa4daccd9ce7bb5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ff83a6ac-e877-46fd-be94-214eb7fba489", "node_type": "1", "metadata": {}, "hash": "253e53b788e21308ff0d22b7e8776240d5b82a2929f38ffb5874994a70b8906d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11492, "end_char_idx": 13724, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
', 'ideas', 'were', 'grouped', 'into', 'two', 'major', 'categories', 'labeled', 'appearance', 'and', 'taste/aroma.', 'these', 'categories', 'are', 'used', 'for', 'novel', 'products', 'that', 'may', 'differ', 'from', 'a', 'visual', 'and/or', 'taste/aroma', 'point', 'of', 'view', 'compared', 'to', 'canventional', 'cigarettes.', 'other', 'categories', 'include', 'a', 'combination', 'of', 'the', 'above,', 'filters,', 'packaging', 'and', 'brand', 'extensions.', 'appearance', 'this', 'category', 'is', 'used', 'for', 'novel', 'cigarette', 'constructions', 'that', 'yield', 'visually', 'different', 'products', 'with', 'minimal', 'changes', 'in', 'smoke', 'chemistry', 'two', 'cigarettes', 'in', 'cne.', 'emulti-plug', 'te', 'build', 'yaur', 'awn', 'cigarette.', 'eswitchable', 'menthol', 'or', 'non', 'menthol', 'cigarette.', '*cigarettes', 'with', 'interspaced', 'perforations', 'to', 'enable', 'smoker', 'to', 'separate', 'unburned', 'section', 'for', 'future', 'smoking.', '«short', 'cigarette,', 'tobacco', 'section', '30', 'mm.', '«extremely', 'fast', 'buming', 'cigarette.', '«novel', 'cigarette', 'constructions', 'that', 'permit', 'a', 'significant', 'reduction', 'iretobacco', 'weight', 'while', 'maintaining', 'smoking', 'mechanics', 'and', 'visual', 'characteristics.', 'higher', 'basis', 'weight', 'paper:', 'potential', 'reduction', 'in', 'tobacco', 'weight.', '«more', 'rigid', 'tobacco', 'column;', 'stiffing', 'agent', 'for', 'tobacco;', 'e.g.', 'starch', '*colored', 'tow', 'and', 'cigarette', 'papers;', 'seasonal', 'promotions,', 'e.g.', 'pastel', 'colored', 'cigarettes', 'for', 'easter', 'or', 'in', 'an', 'ebony', 'and', 'ivory', 'brand', 'containing', 'a', 'mixture', 'of', 'all', 'black', '(black', 'paper', 'and', 'tow)', 'and', 'ail', 'white', 'cigarettes.', '499150498']
Answer:  T.F. Riehl
start_index 17
end_index 18
```

Once examples are encoded, however, they will look like this:

```py
>>> encoding = tokenizer(example["question"], example["words"], example["boxes"])
>>> tokenizer.decode(encoding["input_ids"])
[CLS] who is in cc in this letter? [SEP] wie baw brown & williamson tobacco corporation research & development ...
```

We'll need to find the position of the answer in the encoded input.


key: _node_content
string_value:
{"id_": "ff83a6ac-e877-46fd-be94-214eb7fba489", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "hash": "bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "49a6feb8-c314-4030-a065-f063efbf2bcb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "hash": "445b0700783de8e079d7d1f0f60e15ad6ec36eacaadb511009808ffd7d2f4720", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8c3e84d5-3d47-4532-be37-31fc8aa2ca33", "node_type": "1", "metadata": {}, "hash": "54858536e85b1941144a1b1dd7af4ca00dafcc251b84a08e6b6410c9b52b4338", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
', '«extremely', 'fast', 'buming', 'cigarette.', '«novel', 'cigarette', 'constructions', 'that', 'permit', 'a', 'significant', 'reduction', 'iretobacco', 'weight', 'while', 'maintaining', 'smoking', 'mechanics', 'and', 'visual', 'characteristics.', 'higher', 'basis', 'weight', 'paper:', 'potential', 'reduction', 'in', 'tobacco', 'weight.', '«more', 'rigid', 'tobacco', 'column;', 'stiffing', 'agent', 'for', 'tobacco;', 'e.g.', 'starch', '*colored', 'tow', 'and', 'cigarette', 'papers;', 'seasonal', 'promotions,', 'e.g.', 'pastel', 'colored', 'cigarettes', 'for', 'easter', 'or', 'in', 'an', 'ebony', 'and', 'ivory', 'brand', 'containing', 'a', 'mixture', 'of', 'all', 'black', '(black', 'paper', 'and', 'tow)', 'and', 'ail', 'white', 'cigarettes.', '499150498']
Answer:  T.F. Riehl
start_index 17
end_index 18
```

Once examples are encoded, however, they will look like this:

```py
>>> encoding = tokenizer(example["question"], example["words"], example["boxes"])
>>> tokenizer.decode(encoding["input_ids"])
[CLS] who is in cc in this letter? [SEP] wie baw brown & williamson tobacco corporation research & development ...
```

We'll need to find the position of the answer in the encoded input.
* `token_type_ids` tells us which tokens are part of the question, and which ones are part of the document's words.
* `tokenizer.cls_token_id` will help find the special token at the beginning of the input.
* `word_ids` will help match the answer found in the original `words` to the same answer in the full encoded input and determine
the start/end position of the answer in the encoded input.

With that in mind, let's create a function to encode a batch of examples in the dataset:

```py
>>> def encode_dataset(examples, max_length=512):
.     questions = examples["question"]
.     words = examples["words"]
.     boxes = examples["boxes"]
.     answers = examples["answer"]

.     # encode the batch of examples and initialize the start_positions and end_positions
.     encoding = tokenizer(questions, words, boxes, max_length=max_length, padding="max_length", truncation=True)
.     start_positions = []
.     end_positions = []

.     # loop through the examples in the batch
.     for i in range(len(questions)):
.         cls_index = encoding["input_ids"][i].index(tokenizer.cls_token_id)

.         # find the position of the answer in example's words
.         words_example = [word.lower() for word in words[i]]
.         answer = answers[i]
.         match, word_idx_start, word_idx_end = subfinder(words_example, answer.lower().split())

.         if match:
.             # if match is found, use `token_type_ids` to find where words start in the encoding
.             token_type_ids = encoding["token_type_ids"][i]
.             token_start_index = 0
.             while token_type_ids[token_start_index] != 1:
.                 token_start_index += 1

.


key: _node_content
string_value:
{"id_": "8c3e84d5-3d47-4532-be37-31fc8aa2ca33", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "hash": "bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ff83a6ac-e877-46fd-be94-214eb7fba489", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "hash": "49245f42e3ccf604f1e0b3d67d84b3079aed4545152c6946f0708f4ccd7d3a9a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a8b28b90-0385-4c76-81f1-8f9b5eb90d1d", "node_type": "1", "metadata": {}, "hash": "c927672daa6187c26f9de6e8cbcd69667d0de0f3c3bdca2b4b5c19d3cfa5e200", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
* `token_type_ids` tells us which tokens are part of the question, and which ones are part of the document's words.
* `tokenizer.cls_token_id` will help find the special token at the beginning of the input.
* `word_ids` will help match the answer found in the original `words` to the same answer in the full encoded input and determine
the start/end position of the answer in the encoded input.

With that in mind, let's create a function to encode a batch of examples in the dataset:

```py
>>> def encode_dataset(examples, max_length=512):
.     questions = examples["question"]
.     words = examples["words"]
.     boxes = examples["boxes"]
.     answers = examples["answer"]

.     # encode the batch of examples and initialize the start_positions and end_positions
.     encoding = tokenizer(questions, words, boxes, max_length=max_length, padding="max_length", truncation=True)
.     start_positions = []
.     end_positions = []

.     # loop through the examples in the batch
.     for i in range(len(questions)):
.         cls_index = encoding["input_ids"][i].index(tokenizer.cls_token_id)

.         # find the position of the answer in example's words
.         words_example = [word.lower() for word in words[i]]
.         answer = answers[i]
.         match, word_idx_start, word_idx_end = subfinder(words_example, answer.lower().split())

.         if match:
.             # if match is found, use `token_type_ids` to find where words start in the encoding
.             token_type_ids = encoding["token_type_ids"][i]
.             token_start_index = 0
.             while token_type_ids[token_start_index] != 1:
.                 token_start_index += 1

.             token_end_index = len(encoding["input_ids"][i]) - 1
.             while token_type_ids[token_end_index] != 1:
.                 token_end_index -= 1

.             word_ids = encoding.word_ids(i)[token_start_index : token_end_index + 1]
.             start_position = cls_index
.             end_position = cls_index

.             # loop over word_ids and increase `token_start_index` until it matches the answer position in words
.             # once it matches, save the `token_start_index` as the `start_position` of the answer in the encoding
.             for id in word_ids:
.                 if id == word_idx_start:
.                     start_position = token_start_index
.                 else:
.                     token_start_index += 1

.             # similarly loop over `word_ids` starting from the end to find the `end_position` of the answer
.             for id in word_ids[::-1]:
.                 if id == word_idx_end:
.                     end_position = token_end_index
.                 else:
.                     token_end_index -= 1

.             start_positions.append(start_position)
.             end_positions.append(end_position)

.         else:
.             start_positions.append(cls_index)
.             end_positions.append(cls_index)

.     encoding["image"] = examples["image"]
.     encoding["start_positions"] = start_positions
.     encoding["end_positions"] = end_positions

.     return encoding
```

Now that we have this preprocessing function, we can encode the entire dataset:

```py
>>> encoded_train_dataset = dataset_with_ocr["train"].map(
.     encode_dataset, batched=True, batch_size=2, remove_columns=dataset_with_ocr["train"].column_names
. )
>>> encoded_test_dataset = dataset_with_ocr["test"].map(
.     encode_dataset,


key: _node_content
string_value:
{"id_": "a8b28b90-0385-4c76-81f1-8f9b5eb90d1d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "hash": "bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8c3e84d5-3d47-4532-be37-31fc8aa2ca33", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "hash": "40ff8c00e45ca710b6202f1d4e8dd710a181554caeeafaaf720d67c14534d2ad", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "651dc2d6-b4fd-4879-a9c6-c6ec0c440db8", "node_type": "1", "metadata": {}, "hash": "5aaf0084b882778a32ed68bbac54f34aee5df2b88aa330d7e5bfd78ae2a700e8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
token_end_index = len(encoding["input_ids"][i]) - 1
.             while token_type_ids[token_end_index] != 1:
.                 token_end_index -= 1

.             word_ids = encoding.word_ids(i)[token_start_index : token_end_index + 1]
.             start_position = cls_index
.             end_position = cls_index

.             # loop over word_ids and increase `token_start_index` until it matches the answer position in words
.             # once it matches, save the `token_start_index` as the `start_position` of the answer in the encoding
.             for id in word_ids:
.                 if id == word_idx_start:
.                     start_position = token_start_index
.                 else:
.                     token_start_index += 1

.             # similarly loop over `word_ids` starting from the end to find the `end_position` of the answer
.             for id in word_ids[::-1]:
.                 if id == word_idx_end:
.                     end_position = token_end_index
.                 else:
.                     token_end_index -= 1

.             start_positions.append(start_position)
.             end_positions.append(end_position)

.         else:
.             start_positions.append(cls_index)
.             end_positions.append(cls_index)

.     encoding["image"] = examples["image"]
.     encoding["start_positions"] = start_positions
.     encoding["end_positions"] = end_positions

.     return encoding
```

Now that we have this preprocessing function, we can encode the entire dataset:

```py
>>> encoded_train_dataset = dataset_with_ocr["train"].map(
.     encode_dataset, batched=True, batch_size=2, remove_columns=dataset_with_ocr["train"].column_names
. )
>>> encoded_test_dataset = dataset_with_ocr["test"].map(
.     encode_dataset, batched=True, batch_size=2, remove_columns=dataset_with_ocr["test"].column_names
. )
```

Let's check what the features of the encoded dataset look like:

```py
>>> encoded_train_dataset.features
{'image': Sequence(feature=Sequence(feature=Sequence(feature=Value(dtype='uint8', id=None), length=-1, id=None), length=-1, id=None), length=-1, id=None),
 'input_ids': Sequence(feature=Value(dtype='int32', id=None), length=-1, id=None),
 'token_type_ids': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None),
 'attention_mask': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None),
 'bbox': Sequence(feature=Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None), length=-1, id=None),
 'start_positions': Value(dtype='int64', id=None),
 'end_positions': Value(dtype='int64', id=None)}
```

## Evaluation

Evaluation for document question answering requires a significant amount of postprocessing. To avoid taking up too much
of your time, this guide skips the evaluation step. The [`Trainer`] still calculates the evaluation loss during training so
you're not completely in the dark about your model's performance. Extractive question answering is typically evaluated using F1/exact match.
If you'd like to implement it yourself, check out the [Question Answering chapter](https://huggingface.co/course/chapter7/7?fw=pt#postprocessing)
of the Hugging Face course for inspiration.

## Train

Congratulations! You've successfully navigated the toughest part of this guide and now you are ready to train your own model.


key: _node_content
string_value:
{"id_": "651dc2d6-b4fd-4879-a9c6-c6ec0c440db8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "hash": "bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a8b28b90-0385-4c76-81f1-8f9b5eb90d1d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "hash": "22ba5c29400ec391ce9a4faeae571c4030cd38b756a78b1ff71de192c1313c52", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "59ae20c8-3b24-412a-a088-95df6846f2dd", "node_type": "1", "metadata": {}, "hash": "2e1b30a09304fb5cc22c2f9e4cdf2223944f6f2965c3355c478a475343182708", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
column_names
. )
>>> encoded_test_dataset = dataset_with_ocr["test"].map(
.     encode_dataset, batched=True, batch_size=2, remove_columns=dataset_with_ocr["test"].column_names
. )
```

Let's check what the features of the encoded dataset look like:

```py
>>> encoded_train_dataset.features
{'image': Sequence(feature=Sequence(feature=Sequence(feature=Value(dtype='uint8', id=None), length=-1, id=None), length=-1, id=None), length=-1, id=None),
 'input_ids': Sequence(feature=Value(dtype='int32', id=None), length=-1, id=None),
 'token_type_ids': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None),
 'attention_mask': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None),
 'bbox': Sequence(feature=Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None), length=-1, id=None),
 'start_positions': Value(dtype='int64', id=None),
 'end_positions': Value(dtype='int64', id=None)}
```

## Evaluation

Evaluation for document question answering requires a significant amount of postprocessing. To avoid taking up too much
of your time, this guide skips the evaluation step. The [`Trainer`] still calculates the evaluation loss during training so
you're not completely in the dark about your model's performance. Extractive question answering is typically evaluated using F1/exact match.
If you'd like to implement it yourself, check out the [Question Answering chapter](https://huggingface.co/course/chapter7/7?fw=pt#postprocessing)
of the Hugging Face course for inspiration.

## Train

Congratulations! You've successfully navigated the toughest part of this guide and now you are ready to train your own model.
Training involves the following steps:
* Load the model with [`AutoModelForDocumentQuestionAnswering`] using the same checkpoint as in the preprocessing.
* Define your training hyperparameters in [`TrainingArguments`].
* Define a function to batch examples together, here the [`DefaultDataCollator`] will do just fine
* Pass the training arguments to [`Trainer`] along with the model, dataset, and data collator.
* Call [`~Trainer.train`] to finetune your model.

```py
>>> from transformers import AutoModelForDocumentQuestionAnswering

>>> model = AutoModelForDocumentQuestionAnswering.from_pretrained(model_checkpoint)
```

In the [`TrainingArguments`] use `output_dir` to specify where to save your model, and configure hyperparameters as you see fit.
If you wish to share your model with the community, set `push_to_hub` to `True` (you must be signed in to Hugging Face to upload your model).
In this case the `output_dir` will also be the name of the repo where your model checkpoint will be pushed.

```py
>>> from transformers import TrainingArguments

>>> # REPLACE THIS WITH YOUR REPO ID
>>> repo_id = "MariaK/layoutlmv2-base-uncased_finetuned_docvqa"

>>> training_args = TrainingArguments(
...     output_dir=repo_id,
...     per_device_train_batch_size=4,
...     num_train_epochs=20,
...     save_steps=200,
...     logging_steps=50,
...     eval_strategy="steps",
...     learning_rate=5e-5,
...     save_total_limit=2,
...     remove_unused_columns=False,
...     push_to_hub=True,
... )
```

Define a simple data collator to batch examples together.


key: _node_content
string_value:
{"id_": "59ae20c8-3b24-412a-a088-95df6846f2dd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "hash": "bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "651dc2d6-b4fd-4879-a9c6-c6ec0c440db8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "hash": "55cab5a5f987f1deb3fc3c95567cce47e47407959d4e64d1915b7f478810e242", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "92f198db-5ef1-4150-b289-0c74de8b476b", "node_type": "1", "metadata": {}, "hash": "22529ad00d60bfd6d24635cffbca33838f8fe8329dcc8e26d21e8de59a4d3278", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18719, "end_char_idx": 21662, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Train

Congratulations! You've successfully navigated the toughest part of this guide and now you are ready to train your own model.
Training involves the following steps:
* Load the model with [`AutoModelForDocumentQuestionAnswering`] using the same checkpoint as in the preprocessing.
* Define your training hyperparameters in [`TrainingArguments`].
* Define a function to batch examples together, here the [`DefaultDataCollator`] will do just fine
* Pass the training arguments to [`Trainer`] along with the model, dataset, and data collator.
* Call [`~Trainer.train`] to finetune your model.

```py
>>> from transformers import AutoModelForDocumentQuestionAnswering

>>> model = AutoModelForDocumentQuestionAnswering.from_pretrained(model_checkpoint)
```

In the [`TrainingArguments`] use `output_dir` to specify where to save your model, and configure hyperparameters as you see fit.
If you wish to share your model with the community, set `push_to_hub` to `True` (you must be signed in to Hugging Face to upload your model).
In this case the `output_dir` will also be the name of the repo where your model checkpoint will be pushed.

```py
>>> from transformers import TrainingArguments

>>> # REPLACE THIS WITH YOUR REPO ID
>>> repo_id = "MariaK/layoutlmv2-base-uncased_finetuned_docvqa"

>>> training_args = TrainingArguments(
...     output_dir=repo_id,
...     per_device_train_batch_size=4,
...     num_train_epochs=20,
...     save_steps=200,
...     logging_steps=50,
...     eval_strategy="steps",
...     learning_rate=5e-5,
...     save_total_limit=2,
...     remove_unused_columns=False,
...     push_to_hub=True,
... )
```

Define a simple data collator to batch examples together.

```py
>>> from transformers import DefaultDataCollator

>>> data_collator = DefaultDataCollator()
```

Finally, bring everything together, and call [`~Trainer.train`]:

```py
>>> from transformers import Trainer

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     data_collator=data_collator,
...     train_dataset=encoded_train_dataset,
...     eval_dataset=encoded_test_dataset,
...     tokenizer=processor,
... )

>>> trainer.train()
```

To add the final model to 🤗 Hub, create a model card and call `push_to_hub`:

```py
>>> trainer.create_model_card()
>>> trainer.push_to_hub()
```

## Inference

Now that you have finetuned a LayoutLMv2 model, and uploaded it to the 🤗 Hub, you can use it for inference. The simplest
way to try out your finetuned model for inference is to use it in a [`Pipeline`].

Let's take an example:
```py
>>> example = dataset["test"][2]
>>> question = example["query"]["en"]
>>> image = example["image"]
>>> print(question)
>>> print(example["answers"])
'Who is ‘presiding’ TRRF GENERAL SESSION (PART 1)?'
['TRRF Vice President', 'lee a. waller']
```

Next, instantiate a pipeline for
document question answering with your model, and pass the image + question combination to it.


key: _node_content
string_value:
{"id_": "92f198db-5ef1-4150-b289-0c74de8b476b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0bf4a080-ad00-4d2d-b673-a7d15d31d851", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "hash": "bd3d869f99ff537c3e08ebaf3d945e0a111a2f510fea93f42e25a291804ea986", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "59ae20c8-3b24-412a-a088-95df6846f2dd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/document_question_answering", "title": "Document Question Answering", "tokens": 5943, "retrieve_doc": true, "source": "transformers"}, "hash": "ed4be3b966dfacd1bb06d0e0d2fdaa77b8b0225c42645422d1029ff3f0c37bea", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20423, "end_char_idx": 23379, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> from transformers import DefaultDataCollator

>>> data_collator = DefaultDataCollator()
```

Finally, bring everything together, and call [`~Trainer.train`]:

```py
>>> from transformers import Trainer

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     data_collator=data_collator,
...     train_dataset=encoded_train_dataset,
...     eval_dataset=encoded_test_dataset,
...     tokenizer=processor,
... )

>>> trainer.train()
```

To add the final model to 🤗 Hub, create a model card and call `push_to_hub`:

```py
>>> trainer.create_model_card()
>>> trainer.push_to_hub()
```

## Inference

Now that you have finetuned a LayoutLMv2 model, and uploaded it to the 🤗 Hub, you can use it for inference. The simplest
way to try out your finetuned model for inference is to use it in a [`Pipeline`].

Let's take an example:
```py
>>> example = dataset["test"][2]
>>> question = example["query"]["en"]
>>> image = example["image"]
>>> print(question)
>>> print(example["answers"])
'Who is ‘presiding’ TRRF GENERAL SESSION (PART 1)?'
['TRRF Vice President', 'lee a. waller']
```

Next, instantiate a pipeline for
document question answering with your model, and pass the image + question combination to it.

```py
>>> from transformers import pipeline

>>> qa_pipeline = pipeline("document-question-answering", model="MariaK/layoutlmv2-base-uncased_finetuned_docvqa")
>>> qa_pipeline(image, question)
[{'score': 0.9949808120727539,
  'answer': 'Lee A. Waller',
  'start': 55,
  'end': 57}]
```

You can also manually replicate the results of the pipeline if you'd like:
1. Take an image and a question, prepare them for the model using the processor from your model.
2. Forward the result or preprocessing through the model.
3. The model returns `start_logits` and `end_logits`, which indicate which token is at the start of the answer and
which token is at the end of the answer. Both have shape (batch_size, sequence_length).
4. Take an argmax on the last dimension of both the `start_logits` and `end_logits` to get the predicted `start_idx` and `end_idx`.
5. Decode the answer with the tokenizer.

```py
>>> import torch
>>> from transformers import AutoProcessor
>>> from transformers import AutoModelForDocumentQuestionAnswering

>>> processor = AutoProcessor.from_pretrained("MariaK/layoutlmv2-base-uncased_finetuned_docvqa")
>>> model = AutoModelForDocumentQuestionAnswering.from_pretrained("MariaK/layoutlmv2-base-uncased_finetuned_docvqa")

>>> with torch.no_grad():
...     encoding = processor(image.convert("RGB"), question, return_tensors="pt")
...     outputs = model(**encoding)
...     start_logits = outputs.start_logits
...     end_logits = outputs.end_logits
...     predicted_start_idx = start_logits.argmax(-1).item()
...     predicted_end_idx = end_logits.argmax(-1).item()

>>> processor.tokenizer.decode(encoding.input_ids.squeeze()[predicted_start_idx : predicted_end_idx + 1])
'lee a. waller'
```


key: _node_content
string_value:
{"id_": "10e6c50b-3256-4fce-a430-503e41e29c77", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "hash": "3e60ba6acb9b94b0caa16c4dcf8fd1e857d79b0cb4554071b24b5fbe5185a8c9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "59e12aa6-e3a9-4979-b7cc-7727cee8386b", "node_type": "1", "metadata": {}, "hash": "23212da2355ce05c4a828ec946ef7f3ba3f0a16cbfa6a29ddbe3703169d54c2b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3066, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Image tasks with IDEFICS

[[open-in-colab]]

While individual tasks can be tackled by fine-tuning specialized models, an alternative approach 
that has recently emerged and gained popularity is to use large models for a diverse set of tasks without fine-tuning. 
For instance, large language models can handle such NLP tasks as summarization, translation, classification, and more. 
This approach is no longer limited to a single modality, such as text, and in this guide, we will illustrate how you can 
solve image-text tasks with a large multimodal model called IDEFICS. 

[IDEFICS](../model_doc/idefics) is an open-access vision and language model based on [Flamingo](https://huggingface.co/papers/2204.14198), 
a state-of-the-art visual language model initially developed by DeepMind. The model accepts arbitrary sequences of image 
and text inputs and generates coherent text as output. It can answer questions about images, describe visual content, 
create stories grounded in multiple images, and so on. IDEFICS comes in two variants - [80 billion parameters](https://huggingface.co/HuggingFaceM4/idefics-80b) 
and [9 billion parameters](https://huggingface.co/HuggingFaceM4/idefics-9b), both of which are available on the 🤗 Hub. For each variant, you can also find fine-tuned instructed 
versions of the model adapted for conversational use cases.

This model is exceptionally versatile and can be used for a wide range of image and multimodal tasks. However, 
being a large model means it requires significant computational resources and infrastructure. It is up to you to decide whether 
this approach suits your use case better than fine-tuning specialized models for each individual task. 

In this guide, you'll learn how to: 
- [Load IDEFICS](#loading-the-model) and [load the quantized version of the model](#quantized-model)
- Use IDEFICS for: 
  - [Image captioning](#image-captioning)
  - [Prompted image captioning](#prompted-image-captioning)
  - [Few-shot prompting](#few-shot-prompting)
  - [Visual question answering](#visual-question-answering)
  - [Image classification](#image-classification)
  - [Image-guided text generation](#image-guided-text-generation)
- [Run inference in batch mode](#running-inference-in-batch-mode)
- [Run IDEFICS instruct for conversational use](#idefics-instruct-for-conversational-use)

Before you begin, make sure you have all the necessary libraries installed. 

```bash
pip install -q bitsandbytes sentencepiece accelerate transformers
```

<Tip>
To run the following examples with a non-quantized version of the model checkpoint you will need at least 20GB of GPU memory.
</Tip>

## Loading the model

Let's start by loading the model's 9 billion parameters checkpoint: 

```py
>>> checkpoint = "HuggingFaceM4/idefics-9b"
```

Just like for other Transformers models, you need to load a processor and the model itself from the checkpoint. 
The IDEFICS processor wraps a [`LlamaTokenizer`] and IDEFICS image processor into a single processor to take care of 
preparing text and image inputs for the model.


key: _node_content
string_value:
{"id_": "59e12aa6-e3a9-4979-b7cc-7727cee8386b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "hash": "3e60ba6acb9b94b0caa16c4dcf8fd1e857d79b0cb4554071b24b5fbe5185a8c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "10e6c50b-3256-4fce-a430-503e41e29c77", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "hash": "f8860b38887097e25f54e5b9ee05863af376245773043139a77bf4d4176bb075", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "242a285b-537a-46d4-80ca-2276c3dc15cc", "node_type": "1", "metadata": {}, "hash": "0e3a1787316ae602c29724caaa88031d9c8349d04c11dd73e2d51ae20a66c287", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1462, "end_char_idx": 4670, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, 
being a large model means it requires significant computational resources and infrastructure. It is up to you to decide whether 
this approach suits your use case better than fine-tuning specialized models for each individual task. 

In this guide, you'll learn how to: 
- [Load IDEFICS](#loading-the-model) and [load the quantized version of the model](#quantized-model)
- Use IDEFICS for: 
  - [Image captioning](#image-captioning)
  - [Prompted image captioning](#prompted-image-captioning)
  - [Few-shot prompting](#few-shot-prompting)
  - [Visual question answering](#visual-question-answering)
  - [Image classification](#image-classification)
  - [Image-guided text generation](#image-guided-text-generation)
- [Run inference in batch mode](#running-inference-in-batch-mode)
- [Run IDEFICS instruct for conversational use](#idefics-instruct-for-conversational-use)

Before you begin, make sure you have all the necessary libraries installed. 

```bash
pip install -q bitsandbytes sentencepiece accelerate transformers
```

<Tip>
To run the following examples with a non-quantized version of the model checkpoint you will need at least 20GB of GPU memory.
</Tip>

## Loading the model

Let's start by loading the model's 9 billion parameters checkpoint: 

```py
>>> checkpoint = "HuggingFaceM4/idefics-9b"
```

Just like for other Transformers models, you need to load a processor and the model itself from the checkpoint. 
The IDEFICS processor wraps a [`LlamaTokenizer`] and IDEFICS image processor into a single processor to take care of 
preparing text and image inputs for the model.

```py
>>> import torch

>>> from transformers import IdeficsForVisionText2Text, AutoProcessor

>>> processor = AutoProcessor.from_pretrained(checkpoint)

>>> model = IdeficsForVisionText2Text.from_pretrained(checkpoint, torch_dtype=torch.bfloat16, device_map="auto")
```

Setting `device_map` to `"auto"` will automatically determine how to load and store the model weights in the most optimized 
manner given existing devices.

### Quantized model

If high-memory GPU availability is an issue, you can load the quantized version of the model. To load the model and the 
processor in 4bit precision, pass a `BitsAndBytesConfig` to the `from_pretrained` method and the model will be compressed 
on the fly while loading.

```py
>>> import torch
>>> from transformers import IdeficsForVisionText2Text, AutoProcessor, BitsAndBytesConfig

>>> quantization_config = BitsAndBytesConfig(
...     load_in_4bit=True,
...     bnb_4bit_compute_dtype=torch.float16,
... )

>>> processor = AutoProcessor.from_pretrained(checkpoint)

>>> model = IdeficsForVisionText2Text.from_pretrained(
...     checkpoint,
...     quantization_config=quantization_config,
...     device_map="auto"
... )
```

Now that you have the model loaded in one of the suggested ways, let's move on to exploring tasks that you can use IDEFICS for.

## Image captioning
Image captioning is the task of predicting a caption for a given image. A common application is to aid visually impaired 
people navigate through different situations, for instance, explore image content online. 

To illustrate the task, get an image to be captioned, e.g.


key: _node_content
string_value:
{"id_": "242a285b-537a-46d4-80ca-2276c3dc15cc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d0be5bbc-3da9-40b5-86cd-6e745b701b3d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "hash": "3e60ba6acb9b94b0caa16c4dcf8fd1e857d79b0cb4554071b24b5fbe5185a8c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "59e12aa6-e3a9-4979-b7cc-7727cee8386b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/idefics", "title": "Image tasks with IDEFICS", "tokens": 5432, "retrieve_doc": true, "source": "transformers"}, "hash": "b8695312d2ea15464b66d0686cc53131a07384dc40bb8ed7ee2c65462f60e135", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "419af2bb-b4d9-4000-85b2-c0ae8c67d43f", "node_type": "1", "metadata": {}, "hash": "6314744aa142dec6c5699d8efa4a2c87b252929610d77a0eb3e9fa5fd71c0a4b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3068, "end_char_idx": 5366, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


