# collections

id: 5523a7e1-1ad4-449b-b059-70d441f3d0d1
name: chroma-db-transformers
database_id: 00000000-0000-0000-0000-000000000000
config_json_str:
{"hnsw_configuration": {"space": "l2", "ef_construction": 100, "ef_search": 10, "num_threads": 8, "M": 16, "resize_factor": 1.2, "batch_size": 100, "sync_threshold": 1000, "_type": "HNSWConfigurationInternal"}, "_type": "CollectionConfigurationInternal"}


# embedding_metadata

key: _node_content
string_value:
{"id_": "83800250-89f6-4012-89dc-ecda912192ca", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a701d0ef-c3ad-4dc0-92d7-7a8fd611db21", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0977e3d2-4f04-4b40-8607-875fe1f5ddb1", "node_type": "1", "metadata": {}, "hash": "7b6e926217884aa4d45d293ae4066739adfb48fddce666f2b7ac2b49e5479503", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2903, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Preprocess

[[open-in-colab]]

Before you can train a model on a dataset, it needs to be preprocessed into the expected model input format. Whether your data is text, images, or audio, they need to be converted and assembled into batches of tensors. 🤗 Transformers provides a set of preprocessing classes to help prepare your data for the model. In this tutorial, you'll learn that for:

* Text, use a [Tokenizer](./main_classes/tokenizer) to convert text into a sequence of tokens, create a numerical representation of the tokens, and assemble them into tensors.
* Speech and audio, use a [Feature extractor](./main_classes/feature_extractor) to extract sequential features from audio waveforms and convert them into tensors.
* Image inputs use a [ImageProcessor](./main_classes/image_processor) to convert images into tensors.
* Multimodal inputs, use a [Processor](./main_classes/processors) to combine a tokenizer and a feature extractor or image processor.

<Tip>

`AutoProcessor` **always** works and automatically chooses the correct class for the model you're using, whether you're using a tokenizer, image processor, feature extractor or processor.

</Tip>

Before you begin, install 🤗 Datasets so you can load some datasets to experiment with:

```bash
pip install datasets
```

## Natural Language Processing

<Youtube id="Yffk5aydLzg"/>

The main tool for preprocessing textual data is a [tokenizer](main_classes/tokenizer). A tokenizer splits text into *tokens* according to a set of rules. The tokens are converted into numbers and then tensors, which become the model inputs. Any additional inputs required by the model are added by the tokenizer.

<Tip>

If you plan on using a pretrained model, it's important to use the associated pretrained tokenizer. This ensures the text is split the same way as the pretraining corpus, and uses the same corresponding tokens-to-index (usually referred to as the *vocab*) during pretraining.

</Tip>

Get started by loading a pretrained tokenizer with the [`AutoTokenizer.from_pretrained`] method. This downloads the *vocab* a model was pretrained with:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-cased")
```

Then pass your text to the tokenizer:

```py
>>> encoded_input = tokenizer("Do not meddle in the affairs of wizards, for they are subtle and quick to anger.")
>>> print(encoded_input)
{'input_ids': [101, 2079, 2025, 19960, 10362, 1999, 1996, 3821, 1997, 16657, 1010, 2005, 2027, 2024, 11259, 1998, 4248, 2000, 4963, 1012, 102],
 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
```

The tokenizer returns a dictionary with three important items:

* [input_ids](glossary#input-ids) are the indices corresponding to each token in the sentence.


key: _node_content
string_value:
{"id_": "0977e3d2-4f04-4b40-8607-875fe1f5ddb1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a701d0ef-c3ad-4dc0-92d7-7a8fd611db21", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "83800250-89f6-4012-89dc-ecda912192ca", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "cb33f20b07ca1a9f80777824548579572d967acce85051e22799c63e763bccb9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5e97d5b5-574b-4118-b47e-4891aefd1937", "node_type": "1", "metadata": {}, "hash": "8336be552eec8e5c8f393668167fc2d4ec77acc072931d5e0e258f60c419c4c4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1773, "end_char_idx": 3814, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This ensures the text is split the same way as the pretraining corpus, and uses the same corresponding tokens-to-index (usually referred to as the *vocab*) during pretraining.

</Tip>

Get started by loading a pretrained tokenizer with the [`AutoTokenizer.from_pretrained`] method. This downloads the *vocab* a model was pretrained with:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-cased")
```

Then pass your text to the tokenizer:

```py
>>> encoded_input = tokenizer("Do not meddle in the affairs of wizards, for they are subtle and quick to anger.")
>>> print(encoded_input)
{'input_ids': [101, 2079, 2025, 19960, 10362, 1999, 1996, 3821, 1997, 16657, 1010, 2005, 2027, 2024, 11259, 1998, 4248, 2000, 4963, 1012, 102],
 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
```

The tokenizer returns a dictionary with three important items:

* [input_ids](glossary#input-ids) are the indices corresponding to each token in the sentence.
* [attention_mask](glossary#attention-mask) indicates whether a token should be attended to or not.
* [token_type_ids](glossary#token-type-ids) identifies which sequence a token belongs to when there is more than one sequence.

Return your input by decoding the `input_ids`:

```py
>>> tokenizer.decode(encoded_input["input_ids"])
'[CLS] Do not meddle in the affairs of wizards, for they are subtle and quick to anger. [SEP]'
```

As you can see, the tokenizer added two special tokens - `CLS` and `SEP` (classifier and separator) - to the sentence. Not all models need
special tokens, but if they do, the tokenizer automatically adds them for you.

If there are several sentences you want to preprocess, pass them as a list to the tokenizer:

```py
>>> batch_sentences = [
...     "But what about second breakfast?",
...     "Don't think he knows about second breakfast, Pip.",
...     "What about elevensies?


key: _node_content
string_value:
{"id_": "5e97d5b5-574b-4118-b47e-4891aefd1937", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a701d0ef-c3ad-4dc0-92d7-7a8fd611db21", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0977e3d2-4f04-4b40-8607-875fe1f5ddb1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "82dae28fc3d6d06948bcc12f0153a5e5ba9366d4de83ca10c306720f9c9ff264", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2c1719d8-a163-4e70-a5e5-5364896d7d05", "node_type": "1", "metadata": {}, "hash": "bd9f6313951e6714e51bc15e1f90fac0aac4da7a8e937e5b66bb62961712c8b5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2904, "end_char_idx": 4970, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
* [attention_mask](glossary#attention-mask) indicates whether a token should be attended to or not.
* [token_type_ids](glossary#token-type-ids) identifies which sequence a token belongs to when there is more than one sequence.

Return your input by decoding the `input_ids`:

```py
>>> tokenizer.decode(encoded_input["input_ids"])
'[CLS] Do not meddle in the affairs of wizards, for they are subtle and quick to anger. [SEP]'
```

As you can see, the tokenizer added two special tokens - `CLS` and `SEP` (classifier and separator) - to the sentence. Not all models need
special tokens, but if they do, the tokenizer automatically adds them for you.

If there are several sentences you want to preprocess, pass them as a list to the tokenizer:

```py
>>> batch_sentences = [
...     "But what about second breakfast?",
...     "Don't think he knows about second breakfast, Pip.",
...     "What about elevensies?",
... ]
>>> encoded_inputs = tokenizer(batch_sentences)
>>> print(encoded_inputs)
{'input_ids': [[101, 1252, 1184, 1164, 1248, 6462, 136, 102],
               [101, 1790, 112, 189, 1341, 1119, 3520, 1164, 1248, 6462, 117, 21902, 1643, 119, 102],
               [101, 1327, 1164, 5450, 23434, 136, 102]],
 'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0],
                    [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                    [0, 0, 0, 0, 0, 0, 0]],
 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1],
                    [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
                    [1, 1, 1, 1, 1, 1, 1]]}
```

### Pad

Sentences aren't always the same length which can be an issue because tensors, the model inputs, need to have a uniform shape. Padding is a strategy for ensuring tensors are rectangular by adding a special *padding token* to shorter sentences.

Set the `padding` parameter to `True` to pad the shorter sequences in the batch to match the longest sequence:

```py
>>> batch_sentences = [
...     "But what about second breakfast?",
...     "Don't think he knows about second breakfast, Pip.",
...     "What about elevensies?


key: _node_content
string_value:
{"id_": "2c1719d8-a163-4e70-a5e5-5364896d7d05", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a701d0ef-c3ad-4dc0-92d7-7a8fd611db21", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5e97d5b5-574b-4118-b47e-4891aefd1937", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "7724906a002dc54e0aeb7f8ce7e2b11bb1633192a02d1b3893a62a537ccd5151", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "68e0243d-7b6a-4d8e-812a-166863167c03", "node_type": "1", "metadata": {}, "hash": "3250eb6b2bb63bb28fbd62d8853a0e87b7f5aa26e1ebcc850c8a2524edb5198e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4574, "end_char_idx": 6283, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Padding is a strategy for ensuring tensors are rectangular by adding a special *padding token* to shorter sentences.

Set the `padding` parameter to `True` to pad the shorter sequences in the batch to match the longest sequence:

```py
>>> batch_sentences = [
...     "But what about second breakfast?",
...     "Don't think he knows about second breakfast, Pip.",
...     "What about elevensies?",
... ]
>>> encoded_input = tokenizer(batch_sentences, padding=True)
>>> print(encoded_input)
{'input_ids': [[101, 1252, 1184, 1164, 1248, 6462, 136, 102, 0, 0, 0, 0, 0, 0, 0],
               [101, 1790, 112, 189, 1341, 1119, 3520, 1164, 1248, 6462, 117, 21902, 1643, 119, 102],
               [101, 1327, 1164, 5450, 23434, 136, 102, 0, 0, 0, 0, 0, 0, 0, 0]],
 'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                    [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                    [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],
 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0],
                    [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
                    [1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]]}
```

The first and third sentences are now padded with `0`'s because they are shorter.

### Truncation

On the other end of the spectrum, sometimes a sequence may be too long for a model to handle. In this case, you'll need to truncate the sequence to a shorter length.

Set the `truncation` parameter to `True` to truncate a sequence to the maximum length accepted by the model:

```py
>>> batch_sentences = [
...     "But what about second breakfast?",
...     "Don't think he knows about second breakfast, Pip.",
...     "What about elevensies?


key: _node_content
string_value:
{"id_": "68e0243d-7b6a-4d8e-812a-166863167c03", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a701d0ef-c3ad-4dc0-92d7-7a8fd611db21", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2c1719d8-a163-4e70-a5e5-5364896d7d05", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "1505b58ef2d673e96cfe0eecea44a21fc3469d6562e8197937d906ba622d23d3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "211197ec-0ef8-4818-af5b-1e1fead26b3d", "node_type": "1", "metadata": {}, "hash": "f34879afe7db6f923540c8fe90820cb6a8b79ffca3281adadf5751f16a091d58", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5824, "end_char_idx": 7603, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Truncation

On the other end of the spectrum, sometimes a sequence may be too long for a model to handle. In this case, you'll need to truncate the sequence to a shorter length.

Set the `truncation` parameter to `True` to truncate a sequence to the maximum length accepted by the model:

```py
>>> batch_sentences = [
...     "But what about second breakfast?",
...     "Don't think he knows about second breakfast, Pip.",
...     "What about elevensies?",
... ]
>>> encoded_input = tokenizer(batch_sentences, padding=True, truncation=True)
>>> print(encoded_input)
{'input_ids': [[101, 1252, 1184, 1164, 1248, 6462, 136, 102, 0, 0, 0, 0, 0, 0, 0],
               [101, 1790, 112, 189, 1341, 1119, 3520, 1164, 1248, 6462, 117, 21902, 1643, 119, 102],
               [101, 1327, 1164, 5450, 23434, 136, 102, 0, 0, 0, 0, 0, 0, 0, 0]],
 'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                    [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                    [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],
 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0],
                    [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
                    [1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]]}
```

<Tip>

Check out the [Padding and truncation](./pad_truncation) concept guide to learn more different padding and truncation arguments.

</Tip>

### Build tensors

Finally, you want the tokenizer to return the actual tensors that get fed to the model.

Set the `return_tensors` parameter to either `pt` for PyTorch, or `tf` for TensorFlow:

<frameworkcontent>
<pt>

```py
>>> batch_sentences = [
...     "But what about second breakfast?",
...     "Don't think he knows about second breakfast, Pip.",
...     "What about elevensies?


key: _node_content
string_value:
{"id_": "211197ec-0ef8-4818-af5b-1e1fead26b3d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a701d0ef-c3ad-4dc0-92d7-7a8fd611db21", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "68e0243d-7b6a-4d8e-812a-166863167c03", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "bee24609fcad865fd1e0cd04c39f87a6448e9eb57ee230b0f18342df56df06c8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "822d5206-5fe2-4bdd-8ecb-f1999f197002", "node_type": "1", "metadata": {}, "hash": "e8b58e4f6ef3714c1b8d88ab0ceb04951452251409f5513703a567bbcb65143b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7208, "end_char_idx": 8654, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

### Build tensors

Finally, you want the tokenizer to return the actual tensors that get fed to the model.

Set the `return_tensors` parameter to either `pt` for PyTorch, or `tf` for TensorFlow:

<frameworkcontent>
<pt>

```py
>>> batch_sentences = [
...     "But what about second breakfast?",
...     "Don't think he knows about second breakfast, Pip.",
...     "What about elevensies?",
... ]
>>> encoded_input = tokenizer(batch_sentences, padding=True, truncation=True, return_tensors="pt")
>>> print(encoded_input)
{'input_ids': tensor([[101, 1252, 1184, 1164, 1248, 6462, 136, 102, 0, 0, 0, 0, 0, 0, 0],
                      [101, 1790, 112, 189, 1341, 1119, 3520, 1164, 1248, 6462, 117, 21902, 1643, 119, 102],
                      [101, 1327, 1164, 5450, 23434, 136, 102, 0, 0, 0, 0, 0, 0, 0, 0]]),
 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                           [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
                           [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]),
 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0],
                           [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
                           [1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]])}
```
</pt>
<tf>
```py
>>> batch_sentences = [
...     "But what about second breakfast?",
...     "Don't think he knows about second breakfast, Pip.",
...     "What about elevensies?


key: _node_content
string_value:
{"id_": "822d5206-5fe2-4bdd-8ecb-f1999f197002", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a701d0ef-c3ad-4dc0-92d7-7a8fd611db21", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "211197ec-0ef8-4818-af5b-1e1fead26b3d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "475eb98a98a46cefa9f739543358037cf734ac45598a1fa9e5a79ec7b6366342", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0f95990c-43a1-4552-ae03-00f8492677ac", "node_type": "1", "metadata": {}, "hash": "8318013a55868ead22fceda3515200ae8857fa4a7a892b687c376aa66ed8975a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8559, "end_char_idx": 10249, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
",
...     "Don't think he knows about second breakfast, Pip.",
...     "What about elevensies?",
... ]
>>> encoded_input = tokenizer(batch_sentences, padding=True, truncation=True, return_tensors="tf")
>>> print(encoded_input)
{'input_ids': <tf.Tensor: shape=(2, 9), dtype=int32, numpy=
array([[101, 1252, 1184, 1164, 1248, 6462, 136, 102, 0, 0, 0, 0, 0, 0, 0],
       [101, 1790, 112, 189, 1341, 1119, 3520, 1164, 1248, 6462, 117, 21902, 1643, 119, 102],
       [101, 1327, 1164, 5450, 23434, 136, 102, 0, 0, 0, 0, 0, 0, 0, 0]],
      dtype=int32)>,
 'token_type_ids': <tf.Tensor: shape=(2, 9), dtype=int32, numpy=
array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], dtype=int32)>,
 'attention_mask': <tf.Tensor: shape=(2, 9), dtype=int32, numpy=
array([[1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0],
       [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
       [1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]], dtype=int32)>}
```
</tf>
</frameworkcontent>

<Tip>
Different pipelines support tokenizer arguments in their `__call__()` differently. `text-2-text-generation` pipelines support (i.e. pass on)
only `truncation`. `text-generation` pipelines support `max_length`, `truncation`, `padding` and `add_special_tokens`. 
In `fill-mask` pipelines, tokenizer arguments can be passed in the `tokenizer_kwargs` argument (dictionary).
</Tip>

## Audio

For audio tasks, you'll need a [feature extractor](main_classes/feature_extractor) to prepare your dataset for the model. The feature extractor is designed to extract features from raw audio data, and convert them into tensors.


key: _node_content
string_value:
{"id_": "0f95990c-43a1-4552-ae03-00f8492677ac", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a701d0ef-c3ad-4dc0-92d7-7a8fd611db21", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "822d5206-5fe2-4bdd-8ecb-f1999f197002", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "f50103c82d1802a67e0d4d5b02df3542e0a16aa10831d0e848bbd6c073431316", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "46fe234a-0d10-4a6e-b8e2-6025decd5fb0", "node_type": "1", "metadata": {}, "hash": "4b45e3f324520fefa6e70718e76fad19d056612febb742f115eaa857818f8e68", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9717, "end_char_idx": 12006, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
`text-2-text-generation` pipelines support (i.e. pass on)
only `truncation`. `text-generation` pipelines support `max_length`, `truncation`, `padding` and `add_special_tokens`. 
In `fill-mask` pipelines, tokenizer arguments can be passed in the `tokenizer_kwargs` argument (dictionary).
</Tip>

## Audio

For audio tasks, you'll need a [feature extractor](main_classes/feature_extractor) to prepare your dataset for the model. The feature extractor is designed to extract features from raw audio data, and convert them into tensors.

Load the [MInDS-14](https://huggingface.co/datasets/PolyAI/minds14) dataset (see the 🤗 [Datasets tutorial](https://huggingface.co/docs/datasets/load_hub) for more details on how to load a dataset) to see how you can use a feature extractor with audio datasets:

```py
>>> from datasets import load_dataset, Audio

>>> dataset = load_dataset("PolyAI/minds14", name="en-US", split="train")
```

Access the first element of the `audio` column to take a look at the input. Calling the `audio` column automatically loads and resamples the audio file:

```py
>>> dataset[0]["audio"]
{'array': array([ 0.        ,  0.00024414, -0.00024414, ..., -0.00024414,
         0.        ,  0.        ], dtype=float32),
 'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~JOINT_ACCOUNT/602ba55abb1e6d0fbce92065.wav',
 'sampling_rate': 8000}
```

This returns three items:

* `array` is the speech signal loaded - and potentially resampled - as a 1D array.
* `path` points to the location of the audio file.
* `sampling_rate` refers to how many data points in the speech signal are measured per second.

For this tutorial, you'll use the [Wav2Vec2](https://huggingface.co/facebook/wav2vec2-base) model. Take a look at the model card, and you'll learn Wav2Vec2 is pretrained on 16kHz sampled speech audio. It is important your audio data's sampling rate matches the sampling rate of the dataset used to pretrain the model. If your data's sampling rate isn't the same, then you need to resample your data.

1. Use 🤗 Datasets' [`~datasets.Dataset.cast_column`] method to upsample the sampling rate to 16kHz:

```py
>>> dataset = dataset.cast_column("audio", Audio(sampling_rate=16_000))
```

2.


key: _node_content
string_value:
{"id_": "46fe234a-0d10-4a6e-b8e2-6025decd5fb0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a701d0ef-c3ad-4dc0-92d7-7a8fd611db21", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0f95990c-43a1-4552-ae03-00f8492677ac", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eae12348a62d278c14a344b2dae0214b11cd38c6ee262077711804e2e917ebff", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f7b75cc9-2658-4a30-9ed4-ae57cb4af0e0", "node_type": "1", "metadata": {}, "hash": "0ede54a9d0ee9936ac77410e7cd1850768f9353bcf60dcc8f7575615d2e6e736", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10720, "end_char_idx": 13097, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Calling the `audio` column automatically loads and resamples the audio file:

```py
>>> dataset[0]["audio"]
{'array': array([ 0.        ,  0.00024414, -0.00024414, ..., -0.00024414,
         0.        ,  0.        ], dtype=float32),
 'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~JOINT_ACCOUNT/602ba55abb1e6d0fbce92065.wav',
 'sampling_rate': 8000}
```

This returns three items:

* `array` is the speech signal loaded - and potentially resampled - as a 1D array.
* `path` points to the location of the audio file.
* `sampling_rate` refers to how many data points in the speech signal are measured per second.

For this tutorial, you'll use the [Wav2Vec2](https://huggingface.co/facebook/wav2vec2-base) model. Take a look at the model card, and you'll learn Wav2Vec2 is pretrained on 16kHz sampled speech audio. It is important your audio data's sampling rate matches the sampling rate of the dataset used to pretrain the model. If your data's sampling rate isn't the same, then you need to resample your data.

1. Use 🤗 Datasets' [`~datasets.Dataset.cast_column`] method to upsample the sampling rate to 16kHz:

```py
>>> dataset = dataset.cast_column("audio", Audio(sampling_rate=16_000))
```

2. Call the `audio` column again to resample the audio file:

```py
>>> dataset[0]["audio"]
{'array': array([ 2.3443763e-05,  2.1729663e-04,  2.2145823e-04, ...,
         3.8356509e-05, -7.3497440e-06, -2.1754686e-05], dtype=float32),
 'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~JOINT_ACCOUNT/602ba55abb1e6d0fbce92065.wav',
 'sampling_rate': 16000}
```

Next, load a feature extractor to normalize and pad the input. When padding textual data, a `0` is added for shorter sequences. The same idea applies to audio data. The feature extractor adds a `0` - interpreted as silence - to `array`.

Load the feature extractor with [`AutoFeatureExtractor.from_pretrained`]:

```py
>>> from transformers import AutoFeatureExtractor

>>> feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base")
```

Pass the audio `array` to the feature extractor. We also recommend adding the `sampling_rate` argument in the feature extractor in order to better debug any silent errors that may occur.


key: _node_content
string_value:
{"id_": "f7b75cc9-2658-4a30-9ed4-ae57cb4af0e0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a701d0ef-c3ad-4dc0-92d7-7a8fd611db21", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "46fe234a-0d10-4a6e-b8e2-6025decd5fb0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "a38cbad9766cb401559c034fef6a25d91aed93818efca515ee56c30d7e1de8bf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "61b88408-bb84-4477-8e2e-8c854a3eb243", "node_type": "1", "metadata": {}, "hash": "ce4b82e7e4557e91950b591b48945e9f46d177f24a96da01936e1f6dbe87dfb7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11737, "end_char_idx": 14442, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If your data's sampling rate isn't the same, then you need to resample your data.

1. Use 🤗 Datasets' [`~datasets.Dataset.cast_column`] method to upsample the sampling rate to 16kHz:

```py
>>> dataset = dataset.cast_column("audio", Audio(sampling_rate=16_000))
```

2. Call the `audio` column again to resample the audio file:

```py
>>> dataset[0]["audio"]
{'array': array([ 2.3443763e-05,  2.1729663e-04,  2.2145823e-04, ...,
         3.8356509e-05, -7.3497440e-06, -2.1754686e-05], dtype=float32),
 'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~JOINT_ACCOUNT/602ba55abb1e6d0fbce92065.wav',
 'sampling_rate': 16000}
```

Next, load a feature extractor to normalize and pad the input. When padding textual data, a `0` is added for shorter sequences. The same idea applies to audio data. The feature extractor adds a `0` - interpreted as silence - to `array`.

Load the feature extractor with [`AutoFeatureExtractor.from_pretrained`]:

```py
>>> from transformers import AutoFeatureExtractor

>>> feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base")
```

Pass the audio `array` to the feature extractor. We also recommend adding the `sampling_rate` argument in the feature extractor in order to better debug any silent errors that may occur.

```py
>>> audio_input = [dataset[0]["audio"]["array"]]
>>> feature_extractor(audio_input, sampling_rate=16000)
{'input_values': [array([ 3.8106556e-04,  2.7506407e-03,  2.8015103e-03, ...,
        5.6335266e-04,  4.6588284e-06, -1.7142107e-04], dtype=float32)]}
```

Just like the tokenizer, you can apply padding or truncation to handle variable sequences in a batch. Take a look at the sequence length of these two audio samples:

```py
>>> dataset[0]["audio"]["array"].shape
(173398,)

>>> dataset[1]["audio"]["array"].shape
(106496,)
```

Create a function to preprocess the dataset so the audio samples are the same lengths. Specify a maximum sample length, and the feature extractor will either pad or truncate the sequences to match it:

```py
>>> def preprocess_function(examples):
...     audio_arrays = [x["array"] for x in examples["audio"]]
...     inputs = feature_extractor(
...         audio_arrays,
...         sampling_rate=16000,
...         padding=True,
...         max_length=100000,
...         truncation=True,
...     )
...     return inputs
```

Apply the `preprocess_function` to the first few examples in the dataset:

```py
>>> processed_dataset = preprocess_function(dataset[:5])
```

The sample lengths are now the same and match the specified maximum length. You can pass your processed dataset to the model now!


key: _node_content
string_value:
{"id_": "61b88408-bb84-4477-8e2e-8c854a3eb243", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a701d0ef-c3ad-4dc0-92d7-7a8fd611db21", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f7b75cc9-2658-4a30-9ed4-ae57cb4af0e0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "d9cb780779a56717491def3d88e478e65eb7ff6895cd3d9c7b68beb69f32f05b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7322035f-1a98-468f-8864-e44859db95b2", "node_type": "1", "metadata": {}, "hash": "8babeb15154b929a74c648ac374258ccbd38ffb97d59a18506a171204f37c0a8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12960, "end_char_idx": 16191, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We also recommend adding the `sampling_rate` argument in the feature extractor in order to better debug any silent errors that may occur.

```py
>>> audio_input = [dataset[0]["audio"]["array"]]
>>> feature_extractor(audio_input, sampling_rate=16000)
{'input_values': [array([ 3.8106556e-04,  2.7506407e-03,  2.8015103e-03, ...,
        5.6335266e-04,  4.6588284e-06, -1.7142107e-04], dtype=float32)]}
```

Just like the tokenizer, you can apply padding or truncation to handle variable sequences in a batch. Take a look at the sequence length of these two audio samples:

```py
>>> dataset[0]["audio"]["array"].shape
(173398,)

>>> dataset[1]["audio"]["array"].shape
(106496,)
```

Create a function to preprocess the dataset so the audio samples are the same lengths. Specify a maximum sample length, and the feature extractor will either pad or truncate the sequences to match it:

```py
>>> def preprocess_function(examples):
...     audio_arrays = [x["array"] for x in examples["audio"]]
...     inputs = feature_extractor(
...         audio_arrays,
...         sampling_rate=16000,
...         padding=True,
...         max_length=100000,
...         truncation=True,
...     )
...     return inputs
```

Apply the `preprocess_function` to the first few examples in the dataset:

```py
>>> processed_dataset = preprocess_function(dataset[:5])
```

The sample lengths are now the same and match the specified maximum length. You can pass your processed dataset to the model now!

```py
>>> processed_dataset["input_values"][0].shape
(100000,)

>>> processed_dataset["input_values"][1].shape
(100000,)
```

## Computer vision

For computer vision tasks, you'll need an [image processor](main_classes/image_processor) to prepare your dataset for the model.
Image preprocessing consists of several steps that convert images into the input expected by the model. These steps
include but are not limited to resizing, normalizing, color channel correction, and converting images to tensors.

<Tip>

Image preprocessing often follows some form of image augmentation. Both image preprocessing and image augmentation
transform image data, but they serve different purposes:

* Image augmentation alters images in a way that can help prevent overfitting and increase the robustness of the model. You can get creative in how you augment your data - adjust brightness and colors, crop, rotate, resize, zoom, etc. However, be mindful not to change the meaning of the images with your augmentations.
* Image preprocessing guarantees that the images match the model’s expected input format. When fine-tuning a computer vision model, images must be preprocessed exactly as when the model was initially trained.

You can use any library you like for image augmentation. For image preprocessing, use the `ImageProcessor` associated with the model.

</Tip>

Load the [food101](https://huggingface.co/datasets/food101) dataset (see the 🤗 [Datasets tutorial](https://huggingface.co/docs/datasets/load_hub) for more details on how to load a dataset) to see how you can use an image processor with computer vision datasets:

<Tip>

Use 🤗 Datasets `split` parameter to only load a small sample from the training split since the dataset is quite large!


key: _node_content
string_value:
{"id_": "7322035f-1a98-468f-8864-e44859db95b2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a701d0ef-c3ad-4dc0-92d7-7a8fd611db21", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "61b88408-bb84-4477-8e2e-8c854a3eb243", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "77c133edd86254b4e6604a73f6651b639bf1bec5beae8a8d44ae1df6aedd9800", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2fb8973d-0d9e-42c5-b3e2-6c51526eb530", "node_type": "1", "metadata": {}, "hash": "1410408978c9a36806a714f31d670fdc4c6466b75f566cf9734810c5d6fd0b74", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14389, "end_char_idx": 17918, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can pass your processed dataset to the model now!

```py
>>> processed_dataset["input_values"][0].shape
(100000,)

>>> processed_dataset["input_values"][1].shape
(100000,)
```

## Computer vision

For computer vision tasks, you'll need an [image processor](main_classes/image_processor) to prepare your dataset for the model.
Image preprocessing consists of several steps that convert images into the input expected by the model. These steps
include but are not limited to resizing, normalizing, color channel correction, and converting images to tensors.

<Tip>

Image preprocessing often follows some form of image augmentation. Both image preprocessing and image augmentation
transform image data, but they serve different purposes:

* Image augmentation alters images in a way that can help prevent overfitting and increase the robustness of the model. You can get creative in how you augment your data - adjust brightness and colors, crop, rotate, resize, zoom, etc. However, be mindful not to change the meaning of the images with your augmentations.
* Image preprocessing guarantees that the images match the model’s expected input format. When fine-tuning a computer vision model, images must be preprocessed exactly as when the model was initially trained.

You can use any library you like for image augmentation. For image preprocessing, use the `ImageProcessor` associated with the model.

</Tip>

Load the [food101](https://huggingface.co/datasets/food101) dataset (see the 🤗 [Datasets tutorial](https://huggingface.co/docs/datasets/load_hub) for more details on how to load a dataset) to see how you can use an image processor with computer vision datasets:

<Tip>

Use 🤗 Datasets `split` parameter to only load a small sample from the training split since the dataset is quite large!

</Tip>

```py
>>> from datasets import load_dataset

>>> dataset = load_dataset("food101", split="train[:100]")
```

Next, take a look at the image with 🤗 Datasets [`Image`](https://huggingface.co/docs/datasets/package_reference/main_classes?highlight=image#datasets.Image) feature:

```py
>>> dataset[0]["image"]
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vision-preprocess-tutorial.png"/>
</div>

Load the image processor with [`AutoImageProcessor.from_pretrained`]:

```py
>>> from transformers import AutoImageProcessor

>>> image_processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224")
```

First, let's add some image augmentation. You can use any library you prefer, but in this tutorial, we'll use torchvision's [`transforms`](https://pytorch.org/vision/stable/transforms.html) module. If you're interested in using another data augmentation library, learn how in the [Albumentations](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification_albumentations.ipynb) or [Kornia notebooks](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification_kornia.ipynb).

1. Here we use [`Compose`](https://pytorch.org/vision/master/generated/torchvision.transforms.Compose.html) to chain together a couple of
transforms - [`RandomResizedCrop`](https://pytorch.org/vision/main/generated/torchvision.transforms.RandomResizedCrop.html) and [`ColorJitter`](https://pytorch.org/vision/main/generated/torchvision.transforms.ColorJitter.html).
Note that for resizing, we can get the image size requirements from the `image_processor`.


key: _node_content
string_value:
{"id_": "2fb8973d-0d9e-42c5-b3e2-6c51526eb530", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a701d0ef-c3ad-4dc0-92d7-7a8fd611db21", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7322035f-1a98-468f-8864-e44859db95b2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "3e67c67e9fbdc5404f93b4c31ac5eef19f2306d0de16e8650c0900d73ee6ed1b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f8ccfd99-88e0-470a-becc-16d7ed229a77", "node_type": "1", "metadata": {}, "hash": "4ba95b02d45eb167035a2bb2d6ac07c2f79468a67297cc350a224c5b918ecd96", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16193, "end_char_idx": 19543, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

```py
>>> from datasets import load_dataset

>>> dataset = load_dataset("food101", split="train[:100]")
```

Next, take a look at the image with 🤗 Datasets [`Image`](https://huggingface.co/docs/datasets/package_reference/main_classes?highlight=image#datasets.Image) feature:

```py
>>> dataset[0]["image"]
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vision-preprocess-tutorial.png"/>
</div>

Load the image processor with [`AutoImageProcessor.from_pretrained`]:

```py
>>> from transformers import AutoImageProcessor

>>> image_processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224")
```

First, let's add some image augmentation. You can use any library you prefer, but in this tutorial, we'll use torchvision's [`transforms`](https://pytorch.org/vision/stable/transforms.html) module. If you're interested in using another data augmentation library, learn how in the [Albumentations](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification_albumentations.ipynb) or [Kornia notebooks](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification_kornia.ipynb).

1. Here we use [`Compose`](https://pytorch.org/vision/master/generated/torchvision.transforms.Compose.html) to chain together a couple of
transforms - [`RandomResizedCrop`](https://pytorch.org/vision/main/generated/torchvision.transforms.RandomResizedCrop.html) and [`ColorJitter`](https://pytorch.org/vision/main/generated/torchvision.transforms.ColorJitter.html).
Note that for resizing, we can get the image size requirements from the `image_processor`. For some models, an exact height and
width are expected, for others only the `shortest_edge` is defined.

```py
>>> from torchvision.transforms import RandomResizedCrop, ColorJitter, Compose

>>> size = (
...     image_processor.size["shortest_edge"]
...     if "shortest_edge" in image_processor.size
...     else (image_processor.size["height"], image_processor.size["width"])
... )

>>> _transforms = Compose([RandomResizedCrop(size), ColorJitter(brightness=0.5, hue=0.5)])
```

2. The model accepts [`pixel_values`](model_doc/vision-encoder-decoder#transformers.VisionEncoderDecoderModel.forward.pixel_values)
as its input. `ImageProcessor` can take care of normalizing the images, and generating appropriate tensors.
Create a function that combines image augmentation and image preprocessing for a batch of images and generates `pixel_values`:

```py
>>> def transforms(examples):
...     images = [_transforms(img.convert("RGB")) for img in examples["image"]]
...     examples["pixel_values"] = image_processor(images, do_resize=False, return_tensors="pt")["pixel_values"]
...     return examples
```

<Tip>

In the example above we set `do_resize=False` because we have already resized the images in the image augmentation transformation,
and leveraged the `size` attribute from the appropriate `image_processor`. If you do not resize images during image augmentation,
leave this parameter out. By default, `ImageProcessor` will handle the resizing.

If you wish to normalize images as a part of the augmentation transformation, use the `image_processor.image_mean`,
and `image_processor.image_std` values.
</Tip>

3.


key: _node_content
string_value:
{"id_": "f8ccfd99-88e0-470a-becc-16d7ed229a77", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a701d0ef-c3ad-4dc0-92d7-7a8fd611db21", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2fb8973d-0d9e-42c5-b3e2-6c51526eb530", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "bbc6e5cc6f03d73e736e09800cd410f2ce36b9467f71570aeda11642bb0a7138", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c6b5baf8-a449-4a34-aad5-dc6df4f7905c", "node_type": "1", "metadata": {}, "hash": "dd0abd2823e1c334ee40889278d1ca109fe21917312dd7f23cf17b401dd8e964", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17828, "end_char_idx": 20921, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note that for resizing, we can get the image size requirements from the `image_processor`. For some models, an exact height and
width are expected, for others only the `shortest_edge` is defined.

```py
>>> from torchvision.transforms import RandomResizedCrop, ColorJitter, Compose

>>> size = (
...     image_processor.size["shortest_edge"]
...     if "shortest_edge" in image_processor.size
...     else (image_processor.size["height"], image_processor.size["width"])
... )

>>> _transforms = Compose([RandomResizedCrop(size), ColorJitter(brightness=0.5, hue=0.5)])
```

2. The model accepts [`pixel_values`](model_doc/vision-encoder-decoder#transformers.VisionEncoderDecoderModel.forward.pixel_values)
as its input. `ImageProcessor` can take care of normalizing the images, and generating appropriate tensors.
Create a function that combines image augmentation and image preprocessing for a batch of images and generates `pixel_values`:

```py
>>> def transforms(examples):
...     images = [_transforms(img.convert("RGB")) for img in examples["image"]]
...     examples["pixel_values"] = image_processor(images, do_resize=False, return_tensors="pt")["pixel_values"]
...     return examples
```

<Tip>

In the example above we set `do_resize=False` because we have already resized the images in the image augmentation transformation,
and leveraged the `size` attribute from the appropriate `image_processor`. If you do not resize images during image augmentation,
leave this parameter out. By default, `ImageProcessor` will handle the resizing.

If you wish to normalize images as a part of the augmentation transformation, use the `image_processor.image_mean`,
and `image_processor.image_std` values.
</Tip>

3. Then use 🤗 Datasets[`~datasets.Dataset.set_transform`] to apply the transforms on the fly:
```py
>>> dataset.set_transform(transforms)
```

4. Now when you access the image, you'll notice the image processor has added `pixel_values`. You can pass your processed dataset to the model now!

```py
>>> dataset[0].keys()
```

Here is what the image looks like after the transforms are applied. The image has been randomly cropped and it's color properties are different.

```py
>>> import numpy as np
>>> import matplotlib.pyplot as plt

>>> img = dataset[0]["pixel_values"]
>>> plt.imshow(img.permute(1, 2, 0))
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/preprocessed_image.png"/>
</div>

<Tip>

For tasks like object detection, semantic segmentation, instance segmentation, and panoptic segmentation, `ImageProcessor`
offers post processing methods. These methods convert model's raw outputs into meaningful predictions such as bounding boxes,
or segmentation maps.

</Tip>

### Pad

In some cases, for instance, when fine-tuning [DETR](./model_doc/detr), the model applies scale augmentation at training
time. This may cause images to be different sizes in a batch. You can use [`DetrImageProcessor.pad`]
from [`DetrImageProcessor`] and define a custom `collate_fn` to batch images together.


key: _node_content
string_value:
{"id_": "c6b5baf8-a449-4a34-aad5-dc6df4f7905c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a701d0ef-c3ad-4dc0-92d7-7a8fd611db21", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f8ccfd99-88e0-470a-becc-16d7ed229a77", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "77dde90ed4160e8c6d60b85c74af9a99898accf63c156343bb79f4fc92bc0584", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "92ddff29-ab1b-4039-a25f-4c48252ba478", "node_type": "1", "metadata": {}, "hash": "c39f41d58c94f8d1a15fe5885021a22add22dfc84ed9e62378aff7d4f79c0e70", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19240, "end_char_idx": 21570, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If you do not resize images during image augmentation,
leave this parameter out. By default, `ImageProcessor` will handle the resizing.

If you wish to normalize images as a part of the augmentation transformation, use the `image_processor.image_mean`,
and `image_processor.image_std` values.
</Tip>

3. Then use 🤗 Datasets[`~datasets.Dataset.set_transform`] to apply the transforms on the fly:
```py
>>> dataset.set_transform(transforms)
```

4. Now when you access the image, you'll notice the image processor has added `pixel_values`. You can pass your processed dataset to the model now!

```py
>>> dataset[0].keys()
```

Here is what the image looks like after the transforms are applied. The image has been randomly cropped and it's color properties are different.

```py
>>> import numpy as np
>>> import matplotlib.pyplot as plt

>>> img = dataset[0]["pixel_values"]
>>> plt.imshow(img.permute(1, 2, 0))
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/preprocessed_image.png"/>
</div>

<Tip>

For tasks like object detection, semantic segmentation, instance segmentation, and panoptic segmentation, `ImageProcessor`
offers post processing methods. These methods convert model's raw outputs into meaningful predictions such as bounding boxes,
or segmentation maps.

</Tip>

### Pad

In some cases, for instance, when fine-tuning [DETR](./model_doc/detr), the model applies scale augmentation at training
time. This may cause images to be different sizes in a batch. You can use [`DetrImageProcessor.pad`]
from [`DetrImageProcessor`] and define a custom `collate_fn` to batch images together.

```py
>>> def collate_fn(batch):
...     pixel_values = [item["pixel_values"] for item in batch]
...     encoding = image_processor.pad(pixel_values, return_tensors="pt")
...     labels = [item["labels"] for item in batch]
...     batch = {}
...     batch["pixel_values"] = encoding["pixel_values"]
...     batch["pixel_mask"] = encoding["pixel_mask"]
...     batch["labels"] = labels
...     return batch
```

## Multimodal

For tasks involving multimodal inputs, you'll need a [processor](main_classes/processors) to prepare your dataset for the model. A processor couples together two processing objects such as tokenizer and feature extractor.


key: _node_content
string_value:
{"id_": "92ddff29-ab1b-4039-a25f-4c48252ba478", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a701d0ef-c3ad-4dc0-92d7-7a8fd611db21", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c6b5baf8-a449-4a34-aad5-dc6df4f7905c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "0063149ed445dc0c76f115253cd977d064d5fc2df24b2160ce89d9c107628802", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "aa449462-d49f-4fd2-96d4-3e9f26fabbc3", "node_type": "1", "metadata": {}, "hash": "b7196abb711ab229bf55d28e6fc41efa5416af045ca48ec302caa817d181487d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19833, "end_char_idx": 22892, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> dataset[0].keys()
```

Here is what the image looks like after the transforms are applied. The image has been randomly cropped and it's color properties are different.

```py
>>> import numpy as np
>>> import matplotlib.pyplot as plt

>>> img = dataset[0]["pixel_values"]
>>> plt.imshow(img.permute(1, 2, 0))
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/preprocessed_image.png"/>
</div>

<Tip>

For tasks like object detection, semantic segmentation, instance segmentation, and panoptic segmentation, `ImageProcessor`
offers post processing methods. These methods convert model's raw outputs into meaningful predictions such as bounding boxes,
or segmentation maps.

</Tip>

### Pad

In some cases, for instance, when fine-tuning [DETR](./model_doc/detr), the model applies scale augmentation at training
time. This may cause images to be different sizes in a batch. You can use [`DetrImageProcessor.pad`]
from [`DetrImageProcessor`] and define a custom `collate_fn` to batch images together.

```py
>>> def collate_fn(batch):
...     pixel_values = [item["pixel_values"] for item in batch]
...     encoding = image_processor.pad(pixel_values, return_tensors="pt")
...     labels = [item["labels"] for item in batch]
...     batch = {}
...     batch["pixel_values"] = encoding["pixel_values"]
...     batch["pixel_mask"] = encoding["pixel_mask"]
...     batch["labels"] = labels
...     return batch
```

## Multimodal

For tasks involving multimodal inputs, you'll need a [processor](main_classes/processors) to prepare your dataset for the model. A processor couples together two processing objects such as tokenizer and feature extractor.

Load the [LJ Speech](https://huggingface.co/datasets/lj_speech) dataset (see the 🤗 [Datasets tutorial](https://huggingface.co/docs/datasets/load_hub) for more details on how to load a dataset) to see how you can use a processor for automatic speech recognition (ASR):

```py
>>> from datasets import load_dataset

>>> lj_speech = load_dataset("lj_speech", split="train")
```

For ASR, you're mainly focused on `audio` and `text` so you can remove the other columns:

```py
>>> lj_speech = lj_speech.map(remove_columns=["file", "id", "normalized_text"])
```

Now take a look at the `audio` and `text` columns:

```py
>>> lj_speech[0]["audio"]
{'array': array([-7.3242188e-04, -7.6293945e-04, -6.4086914e-04, ...,
         7.3242188e-04,  2.1362305e-04,  6.1035156e-05], dtype=float32),
 'path': '/root/.cache/huggingface/datasets/downloads/extracted/917ece08c95cf0c4115e45294e3cd0dee724a1165b7fc11798369308a465bd26/LJSpeech-1.1/wavs/LJ001-0001.wav',
 'sampling_rate': 22050}

>>> lj_speech[0]["text"]
'Printing, in the only sense with which we are at present concerned, differs from most if not from all the arts and crafts represented in the Exhibition'
```

Remember you should always [resample](preprocessing#audio) your audio dataset's sampling rate to match the sampling rate of the dataset used to pretrain a model!


key: _node_content
string_value:
{"id_": "aa449462-d49f-4fd2-96d4-3e9f26fabbc3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a701d0ef-c3ad-4dc0-92d7-7a8fd611db21", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "eb5f432185a3f46a2f66cde236a8c347570bdb381055e64512097ce307099030", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "92ddff29-ab1b-4039-a25f-4c48252ba478", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/preprocessing", "title": "Preprocess", "tokens": 7343, "retrieve_doc": true, "source": "transformers"}, "hash": "e286f602d198cd0c06b5064766d12afca550545a8737abdf2be565c382f6eb20", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21572, "end_char_idx": 23806, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Load the [LJ Speech](https://huggingface.co/datasets/lj_speech) dataset (see the 🤗 [Datasets tutorial](https://huggingface.co/docs/datasets/load_hub) for more details on how to load a dataset) to see how you can use a processor for automatic speech recognition (ASR):

```py
>>> from datasets import load_dataset

>>> lj_speech = load_dataset("lj_speech", split="train")
```

For ASR, you're mainly focused on `audio` and `text` so you can remove the other columns:

```py
>>> lj_speech = lj_speech.map(remove_columns=["file", "id", "normalized_text"])
```

Now take a look at the `audio` and `text` columns:

```py
>>> lj_speech[0]["audio"]
{'array': array([-7.3242188e-04, -7.6293945e-04, -6.4086914e-04, ...,
         7.3242188e-04,  2.1362305e-04,  6.1035156e-05], dtype=float32),
 'path': '/root/.cache/huggingface/datasets/downloads/extracted/917ece08c95cf0c4115e45294e3cd0dee724a1165b7fc11798369308a465bd26/LJSpeech-1.1/wavs/LJ001-0001.wav',
 'sampling_rate': 22050}

>>> lj_speech[0]["text"]
'Printing, in the only sense with which we are at present concerned, differs from most if not from all the arts and crafts represented in the Exhibition'
```

Remember you should always [resample](preprocessing#audio) your audio dataset's sampling rate to match the sampling rate of the dataset used to pretrain a model!

```py
>>> lj_speech = lj_speech.cast_column("audio", Audio(sampling_rate=16_000))
```

Load a processor with [`AutoProcessor.from_pretrained`]:

```py
>>> from transformers import AutoProcessor

>>> processor = AutoProcessor.from_pretrained("facebook/wav2vec2-base-960h")
```

1. Create a function to process the audio data contained in `array` to `input_values`, and tokenize `text` to `labels`. These are the inputs to the model:

```py
>>> def prepare_dataset(example):
...     audio = example["audio"]

...     example.update(processor(audio=audio["array"], text=example["text"], sampling_rate=16000))

...     return example
```

2. Apply the `prepare_dataset` function to a sample:

```py
>>> prepare_dataset(lj_speech[0])
```

The processor has now added `input_values` and `labels`, and the sampling rate has also been correctly downsampled to 16kHz. You can pass your processed dataset to the model now!


key: _node_content
string_value:
{"id_": "02b3524e-2f5f-4da2-b83e-52ca616b3cee", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/sagemaker", "title": "Run training on Amazon SageMaker", "tokens": 278, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a94bb6a3-321e-493e-a3e6-17e24464c59a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/sagemaker", "title": "Run training on Amazon SageMaker", "tokens": 278, "retrieve_doc": true, "source": "transformers"}, "hash": "7698b2ff621f3e0a64efa530a64dc41f68dc6ea4f228f13ba9b6e5f358029a6b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1220, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<!---
Copyright 2020 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.

-->

# Run training on Amazon SageMaker

The documentation has been moved to [hf.co/docs/sagemaker](https://huggingface.co/docs/sagemaker). This page will be removed in `transformers` 5.0. 

### Table of Content

- [Train Hugging Face models on Amazon SageMaker with the SageMaker Python SDK](https://huggingface.co/docs/sagemaker/train)
- [Deploy Hugging Face models to Amazon SageMaker with the SageMaker Python SDK](https://huggingface.co/docs/sagemaker/inference)


key: _node_content
string_value:
{"id_": "23793a67-2c17-4742-9729-a67ad0ea7dd5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_sharing", "title": "Share a model", "tokens": 2342, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1397b358-a3fa-490b-bf94-887281c78c33", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_sharing", "title": "Share a model", "tokens": 2342, "retrieve_doc": true, "source": "transformers"}, "hash": "0299cc378101295900500be329593638b6156bdf475ecd9bcd1c1eb97ad01c22", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "020ed8be-9509-498f-ae43-05d9a7ab89ff", "node_type": "1", "metadata": {}, "hash": "43e1063616f77104376099d65cd307b5b70613459379e078f897eb9995c84cb2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3390, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Share a model

The last two tutorials showed how you can fine-tune a model with PyTorch, Keras, and 🤗 Accelerate for distributed setups. The next step is to share your model with the community! At Hugging Face, we believe in openly sharing knowledge and resources to democratize artificial intelligence for everyone. We encourage you to consider sharing your model with the community to help others save time and resources.

In this tutorial, you will learn two methods for sharing a trained or fine-tuned model on the [Model Hub](https://huggingface.co/models):

- Programmatically push your files to the Hub.
- Drag-and-drop your files to the Hub with the web interface.

<iframe width="560" height="315" src="https://www.youtube.com/embed/XvSGPZFEjDY" title="YouTube video player"
frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope;
picture-in-picture" allowfullscreen></iframe>

<Tip>

To share a model with the community, you need an account on [huggingface.co](https://huggingface.co/join). You can also join an existing organization or create a new one.

</Tip>

## Repository features

Each repository on the Model Hub behaves like a typical GitHub repository. Our repositories offer versioning, commit history, and the ability to visualize differences.

The Model Hub's built-in versioning is based on git and [git-lfs](https://git-lfs.github.com/). In other words, you can treat one model as one repository, enabling greater access control and scalability. Version control allows *revisions*, a method for pinning a specific version of a model with a commit hash, tag or branch.

As a result, you can load a specific model version with the `revision` parameter:

```py
>>> model = AutoModel.from_pretrained(
...     "julien-c/EsperBERTo-small", revision="v2.0.1"  # tag name, or branch name, or commit hash
... )
```

Files are also easily edited in a repository, and you can view the commit history as well as the difference:

![vis_diff](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vis_diff.png)

## Setup

Before sharing a model to the Hub, you will need your Hugging Face credentials. If you have access to a terminal, run the following command in the virtual environment where 🤗 Transformers is installed. This will store your access token in your Hugging Face cache folder (`~/.cache/` by default):

```bash
huggingface-cli login
```

If you are using a notebook like Jupyter or Colaboratory, make sure you have the [`huggingface_hub`](https://huggingface.co/docs/hub/adding-a-library) library installed. This library allows you to programmatically interact with the Hub.

```bash
pip install huggingface_hub
```

Then use `notebook_login` to sign-in to the Hub, and follow the link [here](https://huggingface.co/settings/token) to generate a token to login with:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Convert a model for all frameworks

To ensure your model can be used by someone working with a different framework, we recommend you convert and upload your model with both PyTorch and TensorFlow checkpoints. While users are still able to load your model from a different framework if you skip this step, it will be slower because 🤗 Transformers will need to convert the checkpoint on-the-fly.

Converting a checkpoint for another framework is easy.


key: _node_content
string_value:
{"id_": "020ed8be-9509-498f-ae43-05d9a7ab89ff", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_sharing", "title": "Share a model", "tokens": 2342, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1397b358-a3fa-490b-bf94-887281c78c33", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_sharing", "title": "Share a model", "tokens": 2342, "retrieve_doc": true, "source": "transformers"}, "hash": "0299cc378101295900500be329593638b6156bdf475ecd9bcd1c1eb97ad01c22", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "23793a67-2c17-4742-9729-a67ad0ea7dd5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_sharing", "title": "Share a model", "tokens": 2342, "retrieve_doc": true, "source": "transformers"}, "hash": "cb8b04670ba0a27daae93b6d03eff3e15e71e93938397e4cfe165828264a66a6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e8e1eaac-53e4-4a86-bf50-53207bb9f004", "node_type": "1", "metadata": {}, "hash": "94d6c6f43134da7416495ca6df382a5e47c027b53fab7c87ac6ca5b87eacead1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1982, "end_char_idx": 4909, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[vis_diff](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vis_diff.png)

## Setup

Before sharing a model to the Hub, you will need your Hugging Face credentials. If you have access to a terminal, run the following command in the virtual environment where 🤗 Transformers is installed. This will store your access token in your Hugging Face cache folder (`~/.cache/` by default):

```bash
huggingface-cli login
```

If you are using a notebook like Jupyter or Colaboratory, make sure you have the [`huggingface_hub`](https://huggingface.co/docs/hub/adding-a-library) library installed. This library allows you to programmatically interact with the Hub.

```bash
pip install huggingface_hub
```

Then use `notebook_login` to sign-in to the Hub, and follow the link [here](https://huggingface.co/settings/token) to generate a token to login with:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Convert a model for all frameworks

To ensure your model can be used by someone working with a different framework, we recommend you convert and upload your model with both PyTorch and TensorFlow checkpoints. While users are still able to load your model from a different framework if you skip this step, it will be slower because 🤗 Transformers will need to convert the checkpoint on-the-fly.

Converting a checkpoint for another framework is easy. Make sure you have PyTorch and TensorFlow installed (see [here](installation) for installation instructions), and then find the specific model for your task in the other framework. 

<frameworkcontent>
<pt>
Specify `from_tf=True` to convert a checkpoint from TensorFlow to PyTorch:

```py
>>> pt_model = DistilBertForSequenceClassification.from_pretrained("path/to/awesome-name-you-picked", from_tf=True)
>>> pt_model.save_pretrained("path/to/awesome-name-you-picked")
```
</pt>
<tf>
Specify `from_pt=True` to convert a checkpoint from PyTorch to TensorFlow:

```py
>>> tf_model = TFDistilBertForSequenceClassification.from_pretrained("path/to/awesome-name-you-picked", from_pt=True)
```

Then you can save your new TensorFlow model with its new checkpoint:

```py
>>> tf_model.save_pretrained("path/to/awesome-name-you-picked")
```
</tf>
<jax>
If a model is available in Flax, you can also convert a checkpoint from PyTorch to Flax:

```py
>>> flax_model = FlaxDistilBertForSequenceClassification.from_pretrained(
...     "path/to/awesome-name-you-picked", from_pt=True
... )
```
</jax>
</frameworkcontent>

## Push a model during training

<frameworkcontent>
<pt>
<Youtube id="Z1-XMy-GNLQ"/>

Sharing a model to the Hub is as simple as adding an extra parameter or callback. Remember from the [fine-tuning tutorial](training), the [`TrainingArguments`] class is where you specify hyperparameters and additional training options. One of these training options includes the ability to push a model directly to the Hub.


key: _node_content
string_value:
{"id_": "e8e1eaac-53e4-4a86-bf50-53207bb9f004", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_sharing", "title": "Share a model", "tokens": 2342, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1397b358-a3fa-490b-bf94-887281c78c33", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_sharing", "title": "Share a model", "tokens": 2342, "retrieve_doc": true, "source": "transformers"}, "hash": "0299cc378101295900500be329593638b6156bdf475ecd9bcd1c1eb97ad01c22", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "020ed8be-9509-498f-ae43-05d9a7ab89ff", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_sharing", "title": "Share a model", "tokens": 2342, "retrieve_doc": true, "source": "transformers"}, "hash": "85a56a74c660ae60959ce13939b5d61cfcd98e61ac0d753a0b5a3a89651cdba1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "667fd23f-8d32-44bb-94bc-d713f9abe3d4", "node_type": "1", "metadata": {}, "hash": "359803c6316cafb665e7d0c7efae85dc176287329b68d5a91cf8411c971d0036", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3336, "end_char_idx": 6480, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Converting a checkpoint for another framework is easy. Make sure you have PyTorch and TensorFlow installed (see [here](installation) for installation instructions), and then find the specific model for your task in the other framework. 

<frameworkcontent>
<pt>
Specify `from_tf=True` to convert a checkpoint from TensorFlow to PyTorch:

```py
>>> pt_model = DistilBertForSequenceClassification.from_pretrained("path/to/awesome-name-you-picked", from_tf=True)
>>> pt_model.save_pretrained("path/to/awesome-name-you-picked")
```
</pt>
<tf>
Specify `from_pt=True` to convert a checkpoint from PyTorch to TensorFlow:

```py
>>> tf_model = TFDistilBertForSequenceClassification.from_pretrained("path/to/awesome-name-you-picked", from_pt=True)
```

Then you can save your new TensorFlow model with its new checkpoint:

```py
>>> tf_model.save_pretrained("path/to/awesome-name-you-picked")
```
</tf>
<jax>
If a model is available in Flax, you can also convert a checkpoint from PyTorch to Flax:

```py
>>> flax_model = FlaxDistilBertForSequenceClassification.from_pretrained(
...     "path/to/awesome-name-you-picked", from_pt=True
... )
```
</jax>
</frameworkcontent>

## Push a model during training

<frameworkcontent>
<pt>
<Youtube id="Z1-XMy-GNLQ"/>

Sharing a model to the Hub is as simple as adding an extra parameter or callback. Remember from the [fine-tuning tutorial](training), the [`TrainingArguments`] class is where you specify hyperparameters and additional training options. One of these training options includes the ability to push a model directly to the Hub. Set `push_to_hub=True` in your [`TrainingArguments`]:

```py
>>> training_args = TrainingArguments(output_dir="my-awesome-model", push_to_hub=True)
```

Pass your training arguments as usual to [`Trainer`]:

```py
>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=small_train_dataset,
...     eval_dataset=small_eval_dataset,
...     compute_metrics=compute_metrics,
... )
```

After you fine-tune your model, call [`~transformers.Trainer.push_to_hub`] on [`Trainer`] to push the trained model to the Hub. 🤗 Transformers will even automatically add training hyperparameters, training results and framework versions to your model card!

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
Share a model to the Hub with [`PushToHubCallback`]. In the [`PushToHubCallback`] function, add:

- An output directory for your model.
- A tokenizer.
- The `hub_model_id`, which is your Hub username and model name.

```py
>>> from transformers import PushToHubCallback

>>> push_to_hub_callback = PushToHubCallback(
...     output_dir="./your_model_save_path", tokenizer=tokenizer, hub_model_id="your-username/my-awesome-model"
... )
```

Add the callback to [`fit`](https://keras.io/api/models/model_training_apis/), and 🤗 Transformers will push the trained model to the Hub:

```py
>>> model.fit(tf_train_dataset, validation_data=tf_validation_dataset, epochs=3, callbacks=push_to_hub_callback)
```
</tf>
</frameworkcontent>

## Use the `push_to_hub` function

You can also call `push_to_hub` directly on your model to upload it to the Hub.


key: _node_content
string_value:
{"id_": "667fd23f-8d32-44bb-94bc-d713f9abe3d4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_sharing", "title": "Share a model", "tokens": 2342, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1397b358-a3fa-490b-bf94-887281c78c33", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_sharing", "title": "Share a model", "tokens": 2342, "retrieve_doc": true, "source": "transformers"}, "hash": "0299cc378101295900500be329593638b6156bdf475ecd9bcd1c1eb97ad01c22", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e8e1eaac-53e4-4a86-bf50-53207bb9f004", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_sharing", "title": "Share a model", "tokens": 2342, "retrieve_doc": true, "source": "transformers"}, "hash": "c538258ffa9d2ffd331f21d33daa02dc233cf64c16d5272fa82d156b65284a65", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "512c4727-a218-4ba2-b6d7-32b6e2282772", "node_type": "1", "metadata": {}, "hash": "e79843da10f7db55e0b89d3b645885e719cdc4703c2e0c9759e2c7a14ae32657", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4822, "end_char_idx": 7966, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
One of these training options includes the ability to push a model directly to the Hub. Set `push_to_hub=True` in your [`TrainingArguments`]:

```py
>>> training_args = TrainingArguments(output_dir="my-awesome-model", push_to_hub=True)
```

Pass your training arguments as usual to [`Trainer`]:

```py
>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=small_train_dataset,
...     eval_dataset=small_eval_dataset,
...     compute_metrics=compute_metrics,
... )
```

After you fine-tune your model, call [`~transformers.Trainer.push_to_hub`] on [`Trainer`] to push the trained model to the Hub. 🤗 Transformers will even automatically add training hyperparameters, training results and framework versions to your model card!

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
Share a model to the Hub with [`PushToHubCallback`]. In the [`PushToHubCallback`] function, add:

- An output directory for your model.
- A tokenizer.
- The `hub_model_id`, which is your Hub username and model name.

```py
>>> from transformers import PushToHubCallback

>>> push_to_hub_callback = PushToHubCallback(
...     output_dir="./your_model_save_path", tokenizer=tokenizer, hub_model_id="your-username/my-awesome-model"
... )
```

Add the callback to [`fit`](https://keras.io/api/models/model_training_apis/), and 🤗 Transformers will push the trained model to the Hub:

```py
>>> model.fit(tf_train_dataset, validation_data=tf_validation_dataset, epochs=3, callbacks=push_to_hub_callback)
```
</tf>
</frameworkcontent>

## Use the `push_to_hub` function

You can also call `push_to_hub` directly on your model to upload it to the Hub.

Specify your model name in `push_to_hub`:

```py
>>> pt_model.push_to_hub("my-awesome-model")
```

This creates a repository under your username with the model name `my-awesome-model`. Users can now load your model with the `from_pretrained` function:

```py
>>> from transformers import AutoModel

>>> model = AutoModel.from_pretrained("your_username/my-awesome-model")
```

If you belong to an organization and want to push your model under the organization name instead, just add it to the `repo_id`:

```py
>>> pt_model.push_to_hub("my-awesome-org/my-awesome-model")
```

The `push_to_hub` function can also be used to add other files to a model repository. For example, add a tokenizer to a model repository:

```py
>>> tokenizer.push_to_hub("my-awesome-model")
```

Or perhaps you'd like to add the TensorFlow version of your fine-tuned PyTorch model:

```py
>>> tf_model.push_to_hub("my-awesome-model")
```

Now when you navigate to your Hugging Face profile, you should see your newly created model repository. Clicking on the **Files** tab will display all the files you've uploaded to the repository.

For more details on how to create and upload files to a repository, refer to the Hub documentation [here](https://huggingface.co/docs/hub/how-to-upstream).

## Upload with the web interface

Users who prefer a no-code approach are able to upload a model through the Hub's web interface. Visit [huggingface.co/new](https://huggingface.co/new) to create a new repository:

!


key: _node_content
string_value:
{"id_": "512c4727-a218-4ba2-b6d7-32b6e2282772", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_sharing", "title": "Share a model", "tokens": 2342, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1397b358-a3fa-490b-bf94-887281c78c33", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_sharing", "title": "Share a model", "tokens": 2342, "retrieve_doc": true, "source": "transformers"}, "hash": "0299cc378101295900500be329593638b6156bdf475ecd9bcd1c1eb97ad01c22", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "667fd23f-8d32-44bb-94bc-d713f9abe3d4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_sharing", "title": "Share a model", "tokens": 2342, "retrieve_doc": true, "source": "transformers"}, "hash": "9f1ec89db39ef9bd1805738b5f258676ce8d1ede8aacbef36dd9c662ecef55cd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6482, "end_char_idx": 9463, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Specify your model name in `push_to_hub`:

```py
>>> pt_model.push_to_hub("my-awesome-model")
```

This creates a repository under your username with the model name `my-awesome-model`. Users can now load your model with the `from_pretrained` function:

```py
>>> from transformers import AutoModel

>>> model = AutoModel.from_pretrained("your_username/my-awesome-model")
```

If you belong to an organization and want to push your model under the organization name instead, just add it to the `repo_id`:

```py
>>> pt_model.push_to_hub("my-awesome-org/my-awesome-model")
```

The `push_to_hub` function can also be used to add other files to a model repository. For example, add a tokenizer to a model repository:

```py
>>> tokenizer.push_to_hub("my-awesome-model")
```

Or perhaps you'd like to add the TensorFlow version of your fine-tuned PyTorch model:

```py
>>> tf_model.push_to_hub("my-awesome-model")
```

Now when you navigate to your Hugging Face profile, you should see your newly created model repository. Clicking on the **Files** tab will display all the files you've uploaded to the repository.

For more details on how to create and upload files to a repository, refer to the Hub documentation [here](https://huggingface.co/docs/hub/how-to-upstream).

## Upload with the web interface

Users who prefer a no-code approach are able to upload a model through the Hub's web interface. Visit [huggingface.co/new](https://huggingface.co/new) to create a new repository:

![new_model_repo](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/new_model_repo.png)

From here, add some information about your model:

- Select the **owner** of the repository. This can be yourself or any of the organizations you belong to.
- Pick a name for your model, which will also be the repository name.
- Choose whether your model is public or private.
- Specify the license usage for your model.

Now click on the **Files** tab and click on the **Add file** button to upload a new file to your repository. Then drag-and-drop a file to upload and add a commit message.

![upload_file](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/upload_file.png)

## Add a model card

To make sure users understand your model's capabilities, limitations, potential biases and ethical considerations, please add a model card to your repository. The model card is defined in the `README.md` file. You can add a model card by:

* Manually creating and uploading a `README.md` file.
* Clicking on the **Edit model card** button in your model repository.

Take a look at the DistilBert [model card](https://huggingface.co/distilbert/distilbert-base-uncased) for a good example of the type of information a model card should include. For more details about other options you can control in the `README.md` file such as a model's carbon footprint or widget examples, refer to the documentation [here](https://huggingface.co/docs/hub/models-cards).


key: _node_content
string_value:
{"id_": "ddd793c3-21a6-4dba-b5a1-caa10ddf8401", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4f7216d6-043c-4dda-a6ef-226e01ba6879", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "007d040f9c76352423c4d8e3f06a3eb050c8dc8a866aaf2368a104a0b1f96873", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "60c57af1-84cd-424c-8b69-fd671ce0ff92", "node_type": "1", "metadata": {}, "hash": "0d9cb863d56d182bebbd4a4355287f635b8317ff77453822bfade7a18a9ba103", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2914, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# GPU inference

GPUs are the standard choice of hardware for machine learning, unlike CPUs, because they are optimized for memory bandwidth and parallelism. To keep up with the larger sizes of modern models or to run these large models on existing and older hardware, there are several optimizations you can use to speed up GPU inference. In this guide, you'll learn how to use FlashAttention-2 (a more memory-efficient attention mechanism), BetterTransformer (a PyTorch native fastpath execution), and bitsandbytes to quantize your model to a lower precision. Finally, learn how to use 🤗 Optimum to accelerate inference with ONNX Runtime on Nvidia and AMD GPUs.

<Tip>

The majority of the optimizations described here also apply to multi-GPU setups!

</Tip>

## FlashAttention-2

<Tip>

FlashAttention-2 is experimental and may change considerably in future versions.

</Tip>

[FlashAttention-2](https://huggingface.co/papers/2205.14135) is a faster and more efficient implementation of the standard attention mechanism that can significantly speedup inference by:

1. additionally parallelizing the attention computation over sequence length
2. partitioning the work between GPU threads to reduce communication and shared memory reads/writes between them

FlashAttention-2 is currently supported for the following architectures:
* [Bark](https://huggingface.co/docs/transformers/model_doc/bark#transformers.BarkModel)
* [Bart](https://huggingface.co/docs/transformers/model_doc/bart#transformers.BartModel)
* [Chameleon](https://huggingface.co/docs/transformers/model_doc/chameleon#transformers.Chameleon)
* [CLIP](https://huggingface.co/docs/transformers/model_doc/clip#transformers.CLIPModel)
* [Cohere](https://huggingface.co/docs/transformers/model_doc/cohere#transformers.CohereModel)
* [Dbrx](https://huggingface.co/docs/transformers/model_doc/dbrx#transformers.DbrxModel)
* [DistilBert](https://huggingface.co/docs/transformers/model_doc/distilbert#transformers.DistilBertModel)
* [Gemma](https://huggingface.co/docs/transformers/model_doc/gemma#transformers.GemmaModel)
* [Gemma2](https://huggingface.co/docs/transformers/model_doc/gemma2#transformers.Gemma2Model)
* [GPT2](https://huggingface.co/docs/transformers/model_doc/gpt2)
* [GPTBigCode](https://huggingface.co/docs/transformers/model_doc/gpt_bigcode#transformers.GPTBigCodeModel)
* [GPTNeo](https://huggingface.co/docs/transformers/model_doc/gpt_neo#transformers.GPTNeoModel)
* [GPTNeoX](https://huggingface.co/docs/transformers/model_doc/gpt_neox#transformers.GPTNeoXModel)
* [GPT-J](https://huggingface.co/docs/transformers/model_doc/gptj#transformers.GPTJModel)
* [Idefics2](https://huggingface.co/docs/transformers/model_doc/idefics2#transformers.Idefics2Model)
* [Falcon](https://huggingface.co/docs/transformers/model_doc/falcon#transformers.FalconModel)
* [JetMoe](https://huggingface.co/docs/transformers/model_doc/jetmoe#transformers.


key: _node_content
string_value:
{"id_": "60c57af1-84cd-424c-8b69-fd671ce0ff92", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4f7216d6-043c-4dda-a6ef-226e01ba6879", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "007d040f9c76352423c4d8e3f06a3eb050c8dc8a866aaf2368a104a0b1f96873", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ddd793c3-21a6-4dba-b5a1-caa10ddf8401", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "c466680d57251f4e44d6a295f6a0eb06cddc08583f7b083aa31c60919ef3812f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2f0a61e8-69bb-4d76-a2e3-3b7edcff04d4", "node_type": "1", "metadata": {}, "hash": "6f91445addef4dceb9b39a8cb42d9e5e9d49248b81eb8b30b5c57fa68a2e9aea", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1730, "end_char_idx": 4110, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
co/docs/transformers/model_doc/cohere#transformers.CohereModel)
* [Dbrx](https://huggingface.co/docs/transformers/model_doc/dbrx#transformers.DbrxModel)
* [DistilBert](https://huggingface.co/docs/transformers/model_doc/distilbert#transformers.DistilBertModel)
* [Gemma](https://huggingface.co/docs/transformers/model_doc/gemma#transformers.GemmaModel)
* [Gemma2](https://huggingface.co/docs/transformers/model_doc/gemma2#transformers.Gemma2Model)
* [GPT2](https://huggingface.co/docs/transformers/model_doc/gpt2)
* [GPTBigCode](https://huggingface.co/docs/transformers/model_doc/gpt_bigcode#transformers.GPTBigCodeModel)
* [GPTNeo](https://huggingface.co/docs/transformers/model_doc/gpt_neo#transformers.GPTNeoModel)
* [GPTNeoX](https://huggingface.co/docs/transformers/model_doc/gpt_neox#transformers.GPTNeoXModel)
* [GPT-J](https://huggingface.co/docs/transformers/model_doc/gptj#transformers.GPTJModel)
* [Idefics2](https://huggingface.co/docs/transformers/model_doc/idefics2#transformers.Idefics2Model)
* [Falcon](https://huggingface.co/docs/transformers/model_doc/falcon#transformers.FalconModel)
* [JetMoe](https://huggingface.co/docs/transformers/model_doc/jetmoe#transformers.JetMoeModel)
* [Jamba](https://huggingface.co/docs/transformers/model_doc/jamba#transformers.JambaModel)
* [Llama](https://huggingface.co/docs/transformers/model_doc/llama#transformers.LlamaModel)
* [Llava](https://huggingface.co/docs/transformers/model_doc/llava)
* [Llava-NeXT](https://huggingface.co/docs/transformers/model_doc/llava_next)
* [Llava-NeXT-Video](https://huggingface.co/docs/transformers/model_doc/llava_next_video)
* [VipLlava](https://huggingface.co/docs/transformers/model_doc/vipllava)
* [VideoLlava](https://huggingface.co/docs/transformers/model_doc/video_llava)
* [M2M100](https://huggingface.co/docs/transformers/model_doc/m2m_100)
* [MBart](https://huggingface.co/docs/transformers/model_doc/mbart#transformers.MBartModel)
* [Mistral](https://huggingface.co/docs/transformers/model_doc/mistral#transformers.MistralModel)
* [Mixtral](https://huggingface.co/docs/transformers/model_doc/mixtral#transformers.MixtralModel)
* [Musicgen](https://huggingface.co/docs/transformers/model_doc/musicgen#transformers.MusicgenModel)
* [MusicGen Melody](https://huggingface.co/docs/transformers/model_doc/musicgen_melody#transformers.MusicgenMelodyModel)
* [NLLB](https://huggingface.


key: _node_content
string_value:
{"id_": "2f0a61e8-69bb-4d76-a2e3-3b7edcff04d4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4f7216d6-043c-4dda-a6ef-226e01ba6879", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "007d040f9c76352423c4d8e3f06a3eb050c8dc8a866aaf2368a104a0b1f96873", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "60c57af1-84cd-424c-8b69-fd671ce0ff92", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "155a45bd860a15db018f888f5bf51804b9593e7e0592a486e2b9a1ea6e153a24", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5858f365-a9a1-463f-8580-2244e4ab73dc", "node_type": "1", "metadata": {}, "hash": "993283a283ced790472473ab1f5121c4d0de427032e84e77ba4e1b65bb6fc005", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2863, "end_char_idx": 5250, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
co/docs/transformers/model_doc/jetmoe#transformers.JetMoeModel)
* [Jamba](https://huggingface.co/docs/transformers/model_doc/jamba#transformers.JambaModel)
* [Llama](https://huggingface.co/docs/transformers/model_doc/llama#transformers.LlamaModel)
* [Llava](https://huggingface.co/docs/transformers/model_doc/llava)
* [Llava-NeXT](https://huggingface.co/docs/transformers/model_doc/llava_next)
* [Llava-NeXT-Video](https://huggingface.co/docs/transformers/model_doc/llava_next_video)
* [VipLlava](https://huggingface.co/docs/transformers/model_doc/vipllava)
* [VideoLlava](https://huggingface.co/docs/transformers/model_doc/video_llava)
* [M2M100](https://huggingface.co/docs/transformers/model_doc/m2m_100)
* [MBart](https://huggingface.co/docs/transformers/model_doc/mbart#transformers.MBartModel)
* [Mistral](https://huggingface.co/docs/transformers/model_doc/mistral#transformers.MistralModel)
* [Mixtral](https://huggingface.co/docs/transformers/model_doc/mixtral#transformers.MixtralModel)
* [Musicgen](https://huggingface.co/docs/transformers/model_doc/musicgen#transformers.MusicgenModel)
* [MusicGen Melody](https://huggingface.co/docs/transformers/model_doc/musicgen_melody#transformers.MusicgenMelodyModel)
* [NLLB](https://huggingface.co/docs/transformers/model_doc/nllb)
* [OLMo](https://huggingface.co/docs/transformers/model_doc/olmo#transformers.OlmoModel)
* [OPT](https://huggingface.co/docs/transformers/model_doc/opt#transformers.OPTModel)
* [Phi](https://huggingface.co/docs/transformers/model_doc/phi#transformers.PhiModel)
* [Phi3](https://huggingface.co/docs/transformers/model_doc/phi3#transformers.Phi3Model)
* [SigLIP](https://huggingface.co/docs/transformers/model_doc/siglip)
* [StableLm](https://huggingface.co/docs/transformers/model_doc/stablelm#transformers.StableLmModel)
* [Starcoder2](https://huggingface.co/docs/transformers/model_doc/starcoder2#transformers.Starcoder2Model)
* [Qwen2](https://huggingface.co/docs/transformers/model_doc/qwen2#transformers.Qwen2Model)
* [Qwen2MoE](https://huggingface.co/docs/transformers/model_doc/qwen2_moe#transformers.Qwen2MoeModel)
* [Whisper](https://huggingface.co/docs/transformers/model_doc/whisper#transformers.WhisperModel)
* [Wav2Vec2](https://huggingface.co/docs/transformers/model_doc/wav2vec2#transformers.Wav2Vec2Model)
* [Hubert](https://huggingface.co/docs/transformers/model_doc/hubert#transformers.


key: _node_content
string_value:
{"id_": "5858f365-a9a1-463f-8580-2244e4ab73dc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4f7216d6-043c-4dda-a6ef-226e01ba6879", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "007d040f9c76352423c4d8e3f06a3eb050c8dc8a866aaf2368a104a0b1f96873", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2f0a61e8-69bb-4d76-a2e3-3b7edcff04d4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "4ad5517a01f385e117f9e610ccfdf6ca8b359f4d9ae7eac9f5db223b11fe0a94", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4eb8b208-c2f3-40b2-b387-43d814981b38", "node_type": "1", "metadata": {}, "hash": "7e75b03d3be713803c6962606a42781dc22ee6fc359f9428072ad35860d0aafd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4060, "end_char_idx": 6586, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
MusicgenMelodyModel)
* [NLLB](https://huggingface.co/docs/transformers/model_doc/nllb)
* [OLMo](https://huggingface.co/docs/transformers/model_doc/olmo#transformers.OlmoModel)
* [OPT](https://huggingface.co/docs/transformers/model_doc/opt#transformers.OPTModel)
* [Phi](https://huggingface.co/docs/transformers/model_doc/phi#transformers.PhiModel)
* [Phi3](https://huggingface.co/docs/transformers/model_doc/phi3#transformers.Phi3Model)
* [SigLIP](https://huggingface.co/docs/transformers/model_doc/siglip)
* [StableLm](https://huggingface.co/docs/transformers/model_doc/stablelm#transformers.StableLmModel)
* [Starcoder2](https://huggingface.co/docs/transformers/model_doc/starcoder2#transformers.Starcoder2Model)
* [Qwen2](https://huggingface.co/docs/transformers/model_doc/qwen2#transformers.Qwen2Model)
* [Qwen2MoE](https://huggingface.co/docs/transformers/model_doc/qwen2_moe#transformers.Qwen2MoeModel)
* [Whisper](https://huggingface.co/docs/transformers/model_doc/whisper#transformers.WhisperModel)
* [Wav2Vec2](https://huggingface.co/docs/transformers/model_doc/wav2vec2#transformers.Wav2Vec2Model)
* [Hubert](https://huggingface.co/docs/transformers/model_doc/hubert#transformers.HubertModel)
* [data2vec_audio](https://huggingface.co/docs/transformers/main/en/model_doc/data2vec#transformers.Data2VecAudioModel)
* [Sew](https://huggingface.co/docs/transformers/main/en/model_doc/sew#transformers.SEWModel)
* [UniSpeech](https://huggingface.co/docs/transformers/v4.39.3/en/model_doc/unispeech#transformers.UniSpeechModel)
* [unispeech_sat](https://huggingface.co/docs/transformers/v4.39.3/en/model_doc/unispeech-sat#transformers.UniSpeechSatModel)

You can request to add FlashAttention-2 support for another model by opening a GitHub Issue or Pull Request.

Before you begin, make sure you have FlashAttention-2 installed.

<hfoptions id="install">
<hfoption id="NVIDIA">

```bash
pip install flash-attn --no-build-isolation
```

We strongly suggest referring to the detailed [installation instructions](https://github.com/Dao-AILab/flash-attention?tab=readme-ov-file#installation-and-features) to learn more about supported hardware and data types!

</hfoption>
<hfoption id="AMD">

FlashAttention-2 is also supported on AMD GPUs and current support is limited to **Instinct MI210**, **Instinct MI250** and **Instinct MI300**. We strongly suggest using this [Dockerfile](https://github.com/huggingface/optimum-amd/tree/main/docker/transformers-pytorch-amd-gpu-flash/Dockerfile) to use FlashAttention-2 on AMD GPUs.


key: _node_content
string_value:
{"id_": "4eb8b208-c2f3-40b2-b387-43d814981b38", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4f7216d6-043c-4dda-a6ef-226e01ba6879", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "007d040f9c76352423c4d8e3f06a3eb050c8dc8a866aaf2368a104a0b1f96873", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5858f365-a9a1-463f-8580-2244e4ab73dc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "f1306f9f51ac19cdf63f8a1782f1c9166ebe3838d05635efafb6cedfca1dfc2d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "73beca23-85bd-4c9b-927e-2b4683a377c0", "node_type": "1", "metadata": {}, "hash": "2b03646c76fb23028f0bcde26f2ad2e40b1ad2163ee043c48c554fa4d3ae26a0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5153, "end_char_idx": 7565, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Wav2Vec2Model)
* [Hubert](https://huggingface.co/docs/transformers/model_doc/hubert#transformers.HubertModel)
* [data2vec_audio](https://huggingface.co/docs/transformers/main/en/model_doc/data2vec#transformers.Data2VecAudioModel)
* [Sew](https://huggingface.co/docs/transformers/main/en/model_doc/sew#transformers.SEWModel)
* [UniSpeech](https://huggingface.co/docs/transformers/v4.39.3/en/model_doc/unispeech#transformers.UniSpeechModel)
* [unispeech_sat](https://huggingface.co/docs/transformers/v4.39.3/en/model_doc/unispeech-sat#transformers.UniSpeechSatModel)

You can request to add FlashAttention-2 support for another model by opening a GitHub Issue or Pull Request.

Before you begin, make sure you have FlashAttention-2 installed.

<hfoptions id="install">
<hfoption id="NVIDIA">

```bash
pip install flash-attn --no-build-isolation
```

We strongly suggest referring to the detailed [installation instructions](https://github.com/Dao-AILab/flash-attention?tab=readme-ov-file#installation-and-features) to learn more about supported hardware and data types!

</hfoption>
<hfoption id="AMD">

FlashAttention-2 is also supported on AMD GPUs and current support is limited to **Instinct MI210**, **Instinct MI250** and **Instinct MI300**. We strongly suggest using this [Dockerfile](https://github.com/huggingface/optimum-amd/tree/main/docker/transformers-pytorch-amd-gpu-flash/Dockerfile) to use FlashAttention-2 on AMD GPUs.

</hfoption>
</hfoptions>

To enable FlashAttention-2, pass the argument `attn_implementation="flash_attention_2"` to [`~AutoModelForCausalLM.from_pretrained`]:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, LlamaForCausalLM

model_id = "tiiuae/falcon-7b"
tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)
```

<Tip>

FlashAttention-2 can only be used when the model's dtype is `fp16` or `bf16`. Make sure to cast your model to the appropriate dtype and load them on a supported device before using FlashAttention-2.

<br>

You can also set `use_flash_attention_2=True` to enable FlashAttention-2 but it is deprecated in favor of `attn_implementation="flash_attention_2"`.

</Tip>

FlashAttention-2 can be combined with other optimization techniques like quantization to further speedup inference.


key: _node_content
string_value:
{"id_": "73beca23-85bd-4c9b-927e-2b4683a377c0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4f7216d6-043c-4dda-a6ef-226e01ba6879", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "007d040f9c76352423c4d8e3f06a3eb050c8dc8a866aaf2368a104a0b1f96873", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4eb8b208-c2f3-40b2-b387-43d814981b38", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "3836a6e149d30ea2ddd0655d29bad86f9782b8d22a86c909fb5105ad448184cb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e9bbf1f7-8c77-4464-862e-e303d5e54449", "node_type": "1", "metadata": {}, "hash": "3329164b29608d66e7683702b3159761ea523b1a556afb98036610bf6198e2b4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6222, "end_char_idx": 8849, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</hfoption>
<hfoption id="AMD">

FlashAttention-2 is also supported on AMD GPUs and current support is limited to **Instinct MI210**, **Instinct MI250** and **Instinct MI300**. We strongly suggest using this [Dockerfile](https://github.com/huggingface/optimum-amd/tree/main/docker/transformers-pytorch-amd-gpu-flash/Dockerfile) to use FlashAttention-2 on AMD GPUs.

</hfoption>
</hfoptions>

To enable FlashAttention-2, pass the argument `attn_implementation="flash_attention_2"` to [`~AutoModelForCausalLM.from_pretrained`]:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, LlamaForCausalLM

model_id = "tiiuae/falcon-7b"
tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)
```

<Tip>

FlashAttention-2 can only be used when the model's dtype is `fp16` or `bf16`. Make sure to cast your model to the appropriate dtype and load them on a supported device before using FlashAttention-2.

<br>

You can also set `use_flash_attention_2=True` to enable FlashAttention-2 but it is deprecated in favor of `attn_implementation="flash_attention_2"`.

</Tip>

FlashAttention-2 can be combined with other optimization techniques like quantization to further speedup inference. For example, you can combine FlashAttention-2 with 8-bit or 4-bit quantization:

```py
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, LlamaForCausalLM

model_id = "tiiuae/falcon-7b"
tokenizer = AutoTokenizer.from_pretrained(model_id)

# load in 8bit
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    load_in_8bit=True,
    attn_implementation="flash_attention_2",
)

# load in 4bit
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    load_in_4bit=True,
    attn_implementation="flash_attention_2",
)
```

### Expected speedups

You can benefit from considerable speedups for inference, especially for inputs with long sequences. However, since FlashAttention-2 does not support computing attention scores with padding tokens, you must manually pad/unpad the attention scores for batched inference when the sequence contains padding tokens. This leads to a significant slowdown for batched generations with padding tokens.

To overcome this, you should use FlashAttention-2 without padding tokens in the sequence during training (by packing a dataset or [concatenating sequences](https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_clm.py#L516) until reaching the maximum sequence length).


key: _node_content
string_value:
{"id_": "e9bbf1f7-8c77-4464-862e-e303d5e54449", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4f7216d6-043c-4dda-a6ef-226e01ba6879", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "007d040f9c76352423c4d8e3f06a3eb050c8dc8a866aaf2368a104a0b1f96873", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "73beca23-85bd-4c9b-927e-2b4683a377c0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "4bf341f1609b54bc4b94cfbc108cf34d56973edad64ac7f64c907d6deda4dd3a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3bf2c3cc-bba8-4062-884d-87dffcf07550", "node_type": "1", "metadata": {}, "hash": "021412bd0dd7de709897a2acfcfb1a710c3907684b4992499ef52384ec4955f5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7164, "end_char_idx": 10468, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Make sure to cast your model to the appropriate dtype and load them on a supported device before using FlashAttention-2.

<br>

You can also set `use_flash_attention_2=True` to enable FlashAttention-2 but it is deprecated in favor of `attn_implementation="flash_attention_2"`.

</Tip>

FlashAttention-2 can be combined with other optimization techniques like quantization to further speedup inference. For example, you can combine FlashAttention-2 with 8-bit or 4-bit quantization:

```py
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, LlamaForCausalLM

model_id = "tiiuae/falcon-7b"
tokenizer = AutoTokenizer.from_pretrained(model_id)

# load in 8bit
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    load_in_8bit=True,
    attn_implementation="flash_attention_2",
)

# load in 4bit
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    load_in_4bit=True,
    attn_implementation="flash_attention_2",
)
```

### Expected speedups

You can benefit from considerable speedups for inference, especially for inputs with long sequences. However, since FlashAttention-2 does not support computing attention scores with padding tokens, you must manually pad/unpad the attention scores for batched inference when the sequence contains padding tokens. This leads to a significant slowdown for batched generations with padding tokens.

To overcome this, you should use FlashAttention-2 without padding tokens in the sequence during training (by packing a dataset or [concatenating sequences](https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_clm.py#L516) until reaching the maximum sequence length).

For a single forward pass on [tiiuae/falcon-7b](https://hf.co/tiiuae/falcon-7b) with a sequence length of 4096 and various batch sizes without padding tokens, the expected speedup is:

<div style="text-align: center">
<img src="https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/falcon-7b-inference-large-seqlen.png">
</div>

For a single forward pass on [meta-llama/Llama-7b-hf](https://hf.co/meta-llama/Llama-7b-hf) with a sequence length of 4096 and various batch sizes without padding tokens, the expected speedup is:

<div style="text-align: center">
<img src="https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/llama-7b-inference-large-seqlen.png">
</div>

For sequences with padding tokens (generating with padding tokens), you need to unpad/pad the input sequences to correctly compute the attention scores. With a relatively small sequence length, a single forward pass creates overhead leading to a small speedup (in the example below, 30% of the input is filled with padding tokens):

<div style="text-align: center">
<img src="https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/llama-2-small-seqlen-padding.png">
</div>

But for larger sequence lengths, you can expect even more speedup benefits:

<Tip>

FlashAttention is more memory efficient, meaning you can train on much larger sequence lengths without running into out-of-memory issues. You can potentially reduce memory usage up to 20x for larger sequence lengths. Take a look at the [flash-attention](https://github.com/Dao-AILab/flash-attention) repository for more details.


key: _node_content
string_value:
{"id_": "3bf2c3cc-bba8-4062-884d-87dffcf07550", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4f7216d6-043c-4dda-a6ef-226e01ba6879", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "007d040f9c76352423c4d8e3f06a3eb050c8dc8a866aaf2368a104a0b1f96873", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e9bbf1f7-8c77-4464-862e-e303d5e54449", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "1f0809105211ba1a6c33bb20c317778e5213b91040d244e9ca27298ca113394e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "92ad7628-bd17-4c40-9dab-0bfd6f8b1121", "node_type": "1", "metadata": {}, "hash": "64b216d09d783aae60e7b812a523d8ea2808fcd2cf6a3bbeae6c6e3c4cea44e1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8851, "end_char_idx": 11951, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For a single forward pass on [tiiuae/falcon-7b](https://hf.co/tiiuae/falcon-7b) with a sequence length of 4096 and various batch sizes without padding tokens, the expected speedup is:

<div style="text-align: center">
<img src="https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/falcon-7b-inference-large-seqlen.png">
</div>

For a single forward pass on [meta-llama/Llama-7b-hf](https://hf.co/meta-llama/Llama-7b-hf) with a sequence length of 4096 and various batch sizes without padding tokens, the expected speedup is:

<div style="text-align: center">
<img src="https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/llama-7b-inference-large-seqlen.png">
</div>

For sequences with padding tokens (generating with padding tokens), you need to unpad/pad the input sequences to correctly compute the attention scores. With a relatively small sequence length, a single forward pass creates overhead leading to a small speedup (in the example below, 30% of the input is filled with padding tokens):

<div style="text-align: center">
<img src="https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/llama-2-small-seqlen-padding.png">
</div>

But for larger sequence lengths, you can expect even more speedup benefits:

<Tip>

FlashAttention is more memory efficient, meaning you can train on much larger sequence lengths without running into out-of-memory issues. You can potentially reduce memory usage up to 20x for larger sequence lengths. Take a look at the [flash-attention](https://github.com/Dao-AILab/flash-attention) repository for more details.

</Tip>

<div style="text-align: center">
<img src="https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/llama-2-large-seqlen-padding.png">
</div>

## PyTorch scaled dot product attention

PyTorch's [`torch.nn.functional.scaled_dot_product_attention`](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html) (SDPA) can also call FlashAttention and memory-efficient attention kernels under the hood. SDPA support is currently being added natively in Transformers and is used by default for `torch>=2.1.1` when an implementation is available. You may also set `attn_implementation="sdpa"` in `from_pretrained()` to explicitly request SDPA to be used.

For now, Transformers supports SDPA inference and training for the following architectures:
* [Audio Spectrogram Transformer](https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer#transformers.ASTModel)
* [Bart](https://huggingface.co/docs/transformers/model_doc/bart#transformers.BartModel)
* [Bert](https://huggingface.co/docs/transformers/model_doc/bert#transformers.BertModel)
* [Chameleon](https://huggingface.co/docs/transformers/model_doc/chameleon#transformers.Chameleon)
* [CLIP](https://huggingface.co/docs/transformers/model_doc/clip#transformers.CLIPModel)
* [Cohere](https://huggingface.co/docs/transformers/model_doc/cohere#transformers.CohereModel)
* [Dbrx](https://huggingface.co/docs/transformers/model_doc/dbrx#transformers.


key: _node_content
string_value:
{"id_": "92ad7628-bd17-4c40-9dab-0bfd6f8b1121", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4f7216d6-043c-4dda-a6ef-226e01ba6879", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "007d040f9c76352423c4d8e3f06a3eb050c8dc8a866aaf2368a104a0b1f96873", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3bf2c3cc-bba8-4062-884d-87dffcf07550", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "1e4a0dbb541513f747978a71e580a939feff9b95a92d62f0459aa1cd254be91f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "110874aa-1040-4069-9ad5-1f4cf70a0e4c", "node_type": "1", "metadata": {}, "hash": "12d08406588eebe476190007984ea2af1f71d1f42262e6fb3473c5ba372023cd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10470, "end_char_idx": 13149, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

<div style="text-align: center">
<img src="https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/llama-2-large-seqlen-padding.png">
</div>

## PyTorch scaled dot product attention

PyTorch's [`torch.nn.functional.scaled_dot_product_attention`](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html) (SDPA) can also call FlashAttention and memory-efficient attention kernels under the hood. SDPA support is currently being added natively in Transformers and is used by default for `torch>=2.1.1` when an implementation is available. You may also set `attn_implementation="sdpa"` in `from_pretrained()` to explicitly request SDPA to be used.

For now, Transformers supports SDPA inference and training for the following architectures:
* [Audio Spectrogram Transformer](https://huggingface.co/docs/transformers/model_doc/audio-spectrogram-transformer#transformers.ASTModel)
* [Bart](https://huggingface.co/docs/transformers/model_doc/bart#transformers.BartModel)
* [Bert](https://huggingface.co/docs/transformers/model_doc/bert#transformers.BertModel)
* [Chameleon](https://huggingface.co/docs/transformers/model_doc/chameleon#transformers.Chameleon)
* [CLIP](https://huggingface.co/docs/transformers/model_doc/clip#transformers.CLIPModel)
* [Cohere](https://huggingface.co/docs/transformers/model_doc/cohere#transformers.CohereModel)
* [Dbrx](https://huggingface.co/docs/transformers/model_doc/dbrx#transformers.DbrxModel)
* [DeiT](https://huggingface.co/docs/transformers/model_doc/deit#transformers.DeiTModel)
* [Dpr](https://huggingface.co/docs/transformers/model_doc/dpr#transformers.DprReader)
* [Falcon](https://huggingface.co/docs/transformers/model_doc/falcon#transformers.FalconModel)
* [Gemma](https://huggingface.co/docs/transformers/model_doc/gemma#transformers.GemmaModel)
* [Gemma2](https://huggingface.co/docs/transformers/model_doc/gemma2#transformers.Gemma2Model)
* [GPT2](https://huggingface.co/docs/transformers/model_doc/gpt2)
* [GPTBigCode](https://huggingface.co/docs/transformers/model_doc/gpt_bigcode#transformers.GPTBigCodeModel)
* [GPTNeoX](https://huggingface.co/docs/transformers/model_doc/gpt_neox#transformers.GPTNeoXModel)
* [JetMoe](https://huggingface.co/docs/transformers/model_doc/jetmoe#transformers.JetMoeModel)
* [Jamba](https://huggingface.co/docs/transformers/model_doc/jamba#transformers.JambaModel)
* [Llama](https://huggingface.co/docs/transformers/model_doc/llama#transformers.LlamaModel)
* [OLMo](https://huggingface.co/docs/transformers/model_doc/olmo#transformers.OlmoModel)
* [PaliGemma](https://huggingface.co/docs/transformers/model_doc/paligemma#transformers.


key: _node_content
string_value:
{"id_": "110874aa-1040-4069-9ad5-1f4cf70a0e4c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4f7216d6-043c-4dda-a6ef-226e01ba6879", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "007d040f9c76352423c4d8e3f06a3eb050c8dc8a866aaf2368a104a0b1f96873", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "92ad7628-bd17-4c40-9dab-0bfd6f8b1121", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "678818867192c70fdacedeb1596bf13deddd4a2d5dcc450bc49ecb61a691820c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dc0829ba-b49e-4029-9cb2-e3feffc11f08", "node_type": "1", "metadata": {}, "hash": "3dc8ebd763754934535f3e9dc3f0957923c741fecec9525275569eb91eb005a1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11902, "end_char_idx": 14362, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
co/docs/transformers/model_doc/dbrx#transformers.DbrxModel)
* [DeiT](https://huggingface.co/docs/transformers/model_doc/deit#transformers.DeiTModel)
* [Dpr](https://huggingface.co/docs/transformers/model_doc/dpr#transformers.DprReader)
* [Falcon](https://huggingface.co/docs/transformers/model_doc/falcon#transformers.FalconModel)
* [Gemma](https://huggingface.co/docs/transformers/model_doc/gemma#transformers.GemmaModel)
* [Gemma2](https://huggingface.co/docs/transformers/model_doc/gemma2#transformers.Gemma2Model)
* [GPT2](https://huggingface.co/docs/transformers/model_doc/gpt2)
* [GPTBigCode](https://huggingface.co/docs/transformers/model_doc/gpt_bigcode#transformers.GPTBigCodeModel)
* [GPTNeoX](https://huggingface.co/docs/transformers/model_doc/gpt_neox#transformers.GPTNeoXModel)
* [JetMoe](https://huggingface.co/docs/transformers/model_doc/jetmoe#transformers.JetMoeModel)
* [Jamba](https://huggingface.co/docs/transformers/model_doc/jamba#transformers.JambaModel)
* [Llama](https://huggingface.co/docs/transformers/model_doc/llama#transformers.LlamaModel)
* [OLMo](https://huggingface.co/docs/transformers/model_doc/olmo#transformers.OlmoModel)
* [PaliGemma](https://huggingface.co/docs/transformers/model_doc/paligemma#transformers.PaliGemmaForConditionalGeneration)
* [Phi](https://huggingface.co/docs/transformers/model_doc/phi#transformers.PhiModel)
* [Idefics](https://huggingface.co/docs/transformers/model_doc/idefics#transformers.IdeficsModel)
* [Whisper](https://huggingface.co/docs/transformers/model_doc/whisper#transformers.WhisperModel)
* [Mistral](https://huggingface.co/docs/transformers/model_doc/mistral#transformers.MistralModel)
* [Mixtral](https://huggingface.co/docs/transformers/model_doc/mixtral#transformers.MixtralModel)
* [StableLm](https://huggingface.co/docs/transformers/model_doc/stablelm#transformers.StableLmModel)
* [Starcoder2](https://huggingface.co/docs/transformers/model_doc/starcoder2#transformers.Starcoder2Model)
* [Qwen2](https://huggingface.co/docs/transformers/model_doc/qwen2#transformers.Qwen2Model)
* [Qwen2MoE](https://huggingface.co/docs/transformers/model_doc/qwen2_moe#transformers.Qwen2MoeModel)
* [Musicgen](https://huggingface.co/docs/transformers/model_doc/musicgen#transformers.MusicgenModel)
* [MusicGen Melody](https://huggingface.co/docs/transformers/model_doc/musicgen_melody#transformers.MusicgenMelodyModel)
* [ViT](https://huggingface.co/docs/transformers/model_doc/vit#transformers.


key: _node_content
string_value:
{"id_": "dc0829ba-b49e-4029-9cb2-e3feffc11f08", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4f7216d6-043c-4dda-a6ef-226e01ba6879", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "007d040f9c76352423c4d8e3f06a3eb050c8dc8a866aaf2368a104a0b1f96873", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "110874aa-1040-4069-9ad5-1f4cf70a0e4c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "310a05822310ca57c21a990275192083411dbf1f0c17d9d014d52d0ec5caf648", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2ac61883-fb41-444a-b348-51ef0b96f637", "node_type": "1", "metadata": {}, "hash": "04fb98d13661d7cc2e679890c3af30f0e6c52b1bd95e5853b3fd508296a349b1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13095, "end_char_idx": 15522, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
co/docs/transformers/model_doc/paligemma#transformers.PaliGemmaForConditionalGeneration)
* [Phi](https://huggingface.co/docs/transformers/model_doc/phi#transformers.PhiModel)
* [Idefics](https://huggingface.co/docs/transformers/model_doc/idefics#transformers.IdeficsModel)
* [Whisper](https://huggingface.co/docs/transformers/model_doc/whisper#transformers.WhisperModel)
* [Mistral](https://huggingface.co/docs/transformers/model_doc/mistral#transformers.MistralModel)
* [Mixtral](https://huggingface.co/docs/transformers/model_doc/mixtral#transformers.MixtralModel)
* [StableLm](https://huggingface.co/docs/transformers/model_doc/stablelm#transformers.StableLmModel)
* [Starcoder2](https://huggingface.co/docs/transformers/model_doc/starcoder2#transformers.Starcoder2Model)
* [Qwen2](https://huggingface.co/docs/transformers/model_doc/qwen2#transformers.Qwen2Model)
* [Qwen2MoE](https://huggingface.co/docs/transformers/model_doc/qwen2_moe#transformers.Qwen2MoeModel)
* [Musicgen](https://huggingface.co/docs/transformers/model_doc/musicgen#transformers.MusicgenModel)
* [MusicGen Melody](https://huggingface.co/docs/transformers/model_doc/musicgen_melody#transformers.MusicgenMelodyModel)
* [ViT](https://huggingface.co/docs/transformers/model_doc/vit#transformers.ViTModel)
* [ViTHybrid](https://huggingface.co/docs/transformers/model_doc/vit_hybrid#transformers.ViTHybridModel)
* [ViTMAE](https://huggingface.co/docs/transformers/model_doc/vit_mae#transformers.ViTMAEModel)
* [ViTMSN](https://huggingface.co/docs/transformers/model_doc/vit_msn#transformers.ViTMSNModel)
* [VideoMAE](https://huggingface.co/docs/transformers/model_doc/videomae#transformers.VideoMAEModell)
* [wav2vec2](https://huggingface.co/docs/transformers/model_doc/wav2vec2#transformers.Wav2Vec2Model)
* [Hubert](https://huggingface.co/docs/transformers/model_doc/hubert#transformers.HubertModel)
* [data2vec_audio](https://huggingface.co/docs/transformers/main/en/model_doc/data2vec#transformers.Data2VecAudioModel)
* [SigLIP](https://huggingface.co/docs/transformers/model_doc/siglip)
* [Sew](https://huggingface.co/docs/transformers/main/en/model_doc/sew#transformers.SEWModel)
* [UniSpeech](https://huggingface.co/docs/transformers/v4.39.3/en/model_doc/unispeech#transformers.UniSpeechModel)
* [unispeech_sat](https://huggingface.co/docs/transformers/v4.39.3/en/model_doc/unispeech-sat#transformers.UniSpeechSatModel)
* [YOLOS](https://huggingface.


key: _node_content
string_value:
{"id_": "2ac61883-fb41-444a-b348-51ef0b96f637", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4f7216d6-043c-4dda-a6ef-226e01ba6879", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "007d040f9c76352423c4d8e3f06a3eb050c8dc8a866aaf2368a104a0b1f96873", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dc0829ba-b49e-4029-9cb2-e3feffc11f08", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "d1c1382e87d8d16121b6e2e66a13d905bd5a7982e0ac79c4e44459c476b391e7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9d97cb5c-8f99-41ea-bebb-6b31b554cc28", "node_type": "1", "metadata": {}, "hash": "426e9868cb1e0c0ea1825b024480ed7b6c2b96f96675b126f6049b8702d2f2f9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14314, "end_char_idx": 17113, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
co/docs/transformers/model_doc/vit#transformers.ViTModel)
* [ViTHybrid](https://huggingface.co/docs/transformers/model_doc/vit_hybrid#transformers.ViTHybridModel)
* [ViTMAE](https://huggingface.co/docs/transformers/model_doc/vit_mae#transformers.ViTMAEModel)
* [ViTMSN](https://huggingface.co/docs/transformers/model_doc/vit_msn#transformers.ViTMSNModel)
* [VideoMAE](https://huggingface.co/docs/transformers/model_doc/videomae#transformers.VideoMAEModell)
* [wav2vec2](https://huggingface.co/docs/transformers/model_doc/wav2vec2#transformers.Wav2Vec2Model)
* [Hubert](https://huggingface.co/docs/transformers/model_doc/hubert#transformers.HubertModel)
* [data2vec_audio](https://huggingface.co/docs/transformers/main/en/model_doc/data2vec#transformers.Data2VecAudioModel)
* [SigLIP](https://huggingface.co/docs/transformers/model_doc/siglip)
* [Sew](https://huggingface.co/docs/transformers/main/en/model_doc/sew#transformers.SEWModel)
* [UniSpeech](https://huggingface.co/docs/transformers/v4.39.3/en/model_doc/unispeech#transformers.UniSpeechModel)
* [unispeech_sat](https://huggingface.co/docs/transformers/v4.39.3/en/model_doc/unispeech-sat#transformers.UniSpeechSatModel)
* [YOLOS](https://huggingface.co/docs/transformers/model_doc/yolos#transformers.YolosModel)


<Tip>

FlashAttention can only be used for models with the `fp16` or `bf16` torch type, so make sure to cast your model to the appropriate type first. The memory-efficient attention backend is able to handle `fp32` models.

</Tip>

<Tip>

SDPA does not support certain sets of attention parameters, such as `head_mask` and `output_attentions=True`.
In that case, you should see a warning message and we will fall back to the (slower) eager implementation.

</Tip>

By default, SDPA selects the most performant kernel available but you can check whether a backend is available in a given setting (hardware, problem size) with [`torch.backends.cuda.sdp_kernel`](https://pytorch.org/docs/master/backends.html#torch.backends.cuda.sdp_kernel) as a context manager:

```diff
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m")
model = AutoModelForCausalLM.from_pretrained("facebook/opt-350m", torch_dtype=torch.float16).to("cuda")

input_text = "Hello my dog is cute and"
inputs = tokenizer(input_text, return_tensors="pt").to("cuda")

+ with torch.backends.cuda.sdp_kernel(enable_flash=True, enable_math=False, enable_mem_efficient=False):
    outputs = model.generate(**inputs)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

If you see a bug with the traceback below, try using the nightly version of PyTorch which may have broader coverage for FlashAttention:

```bash
RuntimeError: No available kernel. Aborting execution.


key: _node_content
string_value:
{"id_": "9d97cb5c-8f99-41ea-bebb-6b31b554cc28", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4f7216d6-043c-4dda-a6ef-226e01ba6879", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "007d040f9c76352423c4d8e3f06a3eb050c8dc8a866aaf2368a104a0b1f96873", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2ac61883-fb41-444a-b348-51ef0b96f637", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "c41e249f85de51b493e6bcd35f101c3e68238c37931acb17e79aab3bca29ca6b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "47873a84-430c-48b7-8f65-a7cce2992e81", "node_type": "1", "metadata": {}, "hash": "28251a96f8e71bb53783ab61d813ea4db87890e499c3ab3716e7a4d765432973", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15428, "end_char_idx": 18818, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
39.3/en/model_doc/unispeech-sat#transformers.UniSpeechSatModel)
* [YOLOS](https://huggingface.co/docs/transformers/model_doc/yolos#transformers.YolosModel)


<Tip>

FlashAttention can only be used for models with the `fp16` or `bf16` torch type, so make sure to cast your model to the appropriate type first. The memory-efficient attention backend is able to handle `fp32` models.

</Tip>

<Tip>

SDPA does not support certain sets of attention parameters, such as `head_mask` and `output_attentions=True`.
In that case, you should see a warning message and we will fall back to the (slower) eager implementation.

</Tip>

By default, SDPA selects the most performant kernel available but you can check whether a backend is available in a given setting (hardware, problem size) with [`torch.backends.cuda.sdp_kernel`](https://pytorch.org/docs/master/backends.html#torch.backends.cuda.sdp_kernel) as a context manager:

```diff
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m")
model = AutoModelForCausalLM.from_pretrained("facebook/opt-350m", torch_dtype=torch.float16).to("cuda")

input_text = "Hello my dog is cute and"
inputs = tokenizer(input_text, return_tensors="pt").to("cuda")

+ with torch.backends.cuda.sdp_kernel(enable_flash=True, enable_math=False, enable_mem_efficient=False):
    outputs = model.generate(**inputs)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

If you see a bug with the traceback below, try using the nightly version of PyTorch which may have broader coverage for FlashAttention:

```bash
RuntimeError: No available kernel. Aborting execution.

# install PyTorch nightly
pip3 install -U --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu118
```

## BetterTransformer

<Tip warning={true}>

Some BetterTransformer features are being upstreamed to Transformers with default support for native `torch.nn.scaled_dot_product_attention`. BetterTransformer still has a wider coverage than the Transformers SDPA integration, but you can expect more and more architectures to natively support SDPA in Transformers.

</Tip>

<Tip>

Check out our benchmarks with BetterTransformer and scaled dot product attention in the [Out of the box acceleration and memory savings of 🤗 decoder models with PyTorch 2.0](https://pytorch.org/blog/out-of-the-box-acceleration/) and learn more about the fastpath execution in the [BetterTransformer](https://medium.com/pytorch/bettertransformer-out-of-the-box-performance-for-huggingface-transformers-3fbe27d50ab2) blog post.

</Tip>

BetterTransformer accelerates inference with its fastpath (native PyTorch specialized implementation of Transformer functions) execution. The two optimizations in the fastpath execution are:

1. fusion, which combines multiple sequential operations into a single "kernel" to reduce the number of computation steps
2. skipping the inherent sparsity of padding tokens to avoid unnecessary computation with nested tensors

BetterTransformer also converts all attention operations to use the more memory-efficient [scaled dot product attention (SDPA)](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention), and it calls optimized kernels like [FlashAttention](https://huggingface.co/papers/2205.14135) under the hood.


key: _node_content
string_value:
{"id_": "47873a84-430c-48b7-8f65-a7cce2992e81", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4f7216d6-043c-4dda-a6ef-226e01ba6879", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "007d040f9c76352423c4d8e3f06a3eb050c8dc8a866aaf2368a104a0b1f96873", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9d97cb5c-8f99-41ea-bebb-6b31b554cc28", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "26a2e59abdde7058008df070f8190639bc97d5524ca7b2a1bc51ff4a7fa735e3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3347ee4f-094f-4ba6-add0-25d07f830f24", "node_type": "1", "metadata": {}, "hash": "2df3ff132a0114795f51163b6ad5e6ad7b45a77c21d624270b9da3a6a3eaa32d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17094, "end_char_idx": 20461, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Aborting execution.

# install PyTorch nightly
pip3 install -U --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu118
```

## BetterTransformer

<Tip warning={true}>

Some BetterTransformer features are being upstreamed to Transformers with default support for native `torch.nn.scaled_dot_product_attention`. BetterTransformer still has a wider coverage than the Transformers SDPA integration, but you can expect more and more architectures to natively support SDPA in Transformers.

</Tip>

<Tip>

Check out our benchmarks with BetterTransformer and scaled dot product attention in the [Out of the box acceleration and memory savings of 🤗 decoder models with PyTorch 2.0](https://pytorch.org/blog/out-of-the-box-acceleration/) and learn more about the fastpath execution in the [BetterTransformer](https://medium.com/pytorch/bettertransformer-out-of-the-box-performance-for-huggingface-transformers-3fbe27d50ab2) blog post.

</Tip>

BetterTransformer accelerates inference with its fastpath (native PyTorch specialized implementation of Transformer functions) execution. The two optimizations in the fastpath execution are:

1. fusion, which combines multiple sequential operations into a single "kernel" to reduce the number of computation steps
2. skipping the inherent sparsity of padding tokens to avoid unnecessary computation with nested tensors

BetterTransformer also converts all attention operations to use the more memory-efficient [scaled dot product attention (SDPA)](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention), and it calls optimized kernels like [FlashAttention](https://huggingface.co/papers/2205.14135) under the hood.

Before you start, make sure you have 🤗 Optimum [installed](https://huggingface.co/docs/optimum/installation).

Then you can enable BetterTransformer with the [`PreTrainedModel.to_bettertransformer`] method:

```python
model = model.to_bettertransformer()
```

You can return the original Transformers model with the [`~PreTrainedModel.reverse_bettertransformer`] method. You should use this before saving your model to use the canonical Transformers modeling:

```py
model = model.reverse_bettertransformer()
model.save_pretrained("saved_model")
```

## bitsandbytes

bitsandbytes is a quantization library that includes support for 4-bit and 8-bit quantization. Quantization reduces your model size compared to its native full precision version, making it easier to fit large models onto GPUs with limited memory.

Make sure you have bitsandbytes and 🤗 Accelerate installed:

```bash
# these versions support 8-bit and 4-bit
pip install bitsandbytes>=0.39.0 accelerate>=0.20.0

# install Transformers
pip install transformers
```

### 4-bit

To load a model in 4-bit for inference, use the `load_in_4bit` parameter. The `device_map` parameter is optional, but we recommend setting it to `"auto"` to allow 🤗 Accelerate to automatically and efficiently allocate the model given the available resources in the environment.

```py
from transformers import AutoModelForCausalLM

model_name = "bigscience/bloom-2b5"
model_4bit = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", load_in_4bit=True)
```

To load a model in 4-bit for inference with multiple GPUs, you can control how much GPU RAM you want to allocate to each GPU.


key: _node_content
string_value:
{"id_": "3347ee4f-094f-4ba6-add0-25d07f830f24", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4f7216d6-043c-4dda-a6ef-226e01ba6879", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "007d040f9c76352423c4d8e3f06a3eb050c8dc8a866aaf2368a104a0b1f96873", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "47873a84-430c-48b7-8f65-a7cce2992e81", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "ccf78299996d8a5758907ce69ddf30340b038e2f0ebbbfe4b4d09da24c7ce8b6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dabad0cf-e41d-4cc9-8d5a-79edfa600d7f", "node_type": "1", "metadata": {}, "hash": "6422d4ce7f3d2b9b0e65a4424f086fa872764eb797f625898186291905e181eb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18820, "end_char_idx": 21976, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Before you start, make sure you have 🤗 Optimum [installed](https://huggingface.co/docs/optimum/installation).

Then you can enable BetterTransformer with the [`PreTrainedModel.to_bettertransformer`] method:

```python
model = model.to_bettertransformer()
```

You can return the original Transformers model with the [`~PreTrainedModel.reverse_bettertransformer`] method. You should use this before saving your model to use the canonical Transformers modeling:

```py
model = model.reverse_bettertransformer()
model.save_pretrained("saved_model")
```

## bitsandbytes

bitsandbytes is a quantization library that includes support for 4-bit and 8-bit quantization. Quantization reduces your model size compared to its native full precision version, making it easier to fit large models onto GPUs with limited memory.

Make sure you have bitsandbytes and 🤗 Accelerate installed:

```bash
# these versions support 8-bit and 4-bit
pip install bitsandbytes>=0.39.0 accelerate>=0.20.0

# install Transformers
pip install transformers
```

### 4-bit

To load a model in 4-bit for inference, use the `load_in_4bit` parameter. The `device_map` parameter is optional, but we recommend setting it to `"auto"` to allow 🤗 Accelerate to automatically and efficiently allocate the model given the available resources in the environment.

```py
from transformers import AutoModelForCausalLM

model_name = "bigscience/bloom-2b5"
model_4bit = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", load_in_4bit=True)
```

To load a model in 4-bit for inference with multiple GPUs, you can control how much GPU RAM you want to allocate to each GPU. For example, to distribute 600MB of memory to the first GPU and 1GB of memory to the second GPU:

```py
max_memory_mapping = {0: "600MB", 1: "1GB"}
model_name = "bigscience/bloom-3b"
model_4bit = AutoModelForCausalLM.from_pretrained(
    model_name, device_map="auto", load_in_4bit=True, max_memory=max_memory_mapping
)
```

### 8-bit

<Tip>

If you're curious and interested in learning more about the concepts underlying 8-bit quantization, read the [Gentle Introduction to 8-bit Matrix Multiplication for transformers at scale using Hugging Face Transformers, Accelerate and bitsandbytes](https://huggingface.co/blog/hf-bitsandbytes-integration) blog post.

</Tip>

To load a model in 8-bit for inference, use the `load_in_8bit` parameter. The `device_map` parameter is optional, but we recommend setting it to `"auto"` to allow 🤗 Accelerate to automatically and efficiently allocate the model given the available resources in the environment:

```py
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

model_name = "bigscience/bloom-2b5"
model_8bit = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=BitsAndBytesConfig(load_in_8bit=True))
```

If you're loading a model in 8-bit for text generation, you should use the [`~transformers.GenerationMixin.generate`] method instead of the [`Pipeline`] function which is not optimized for 8-bit models and will be slower. Some sampling strategies, like nucleus sampling, are also not supported by the [`Pipeline`] for 8-bit models.


key: _node_content
string_value:
{"id_": "dabad0cf-e41d-4cc9-8d5a-79edfa600d7f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4f7216d6-043c-4dda-a6ef-226e01ba6879", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "007d040f9c76352423c4d8e3f06a3eb050c8dc8a866aaf2368a104a0b1f96873", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3347ee4f-094f-4ba6-add0-25d07f830f24", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "c62a1c64e557e7a42fc7337f67169a4202ed6f934e57e516cdedaf9e81e1def7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fe16e2f0-17e3-4ed3-ae9b-dc669ec2963d", "node_type": "1", "metadata": {}, "hash": "60189b310d343e7254922d56587eb1b57d4cc9b5e2f60c25e867df0efffbed02", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20462, "end_char_idx": 23333, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For example, to distribute 600MB of memory to the first GPU and 1GB of memory to the second GPU:

```py
max_memory_mapping = {0: "600MB", 1: "1GB"}
model_name = "bigscience/bloom-3b"
model_4bit = AutoModelForCausalLM.from_pretrained(
    model_name, device_map="auto", load_in_4bit=True, max_memory=max_memory_mapping
)
```

### 8-bit

<Tip>

If you're curious and interested in learning more about the concepts underlying 8-bit quantization, read the [Gentle Introduction to 8-bit Matrix Multiplication for transformers at scale using Hugging Face Transformers, Accelerate and bitsandbytes](https://huggingface.co/blog/hf-bitsandbytes-integration) blog post.

</Tip>

To load a model in 8-bit for inference, use the `load_in_8bit` parameter. The `device_map` parameter is optional, but we recommend setting it to `"auto"` to allow 🤗 Accelerate to automatically and efficiently allocate the model given the available resources in the environment:

```py
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

model_name = "bigscience/bloom-2b5"
model_8bit = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=BitsAndBytesConfig(load_in_8bit=True))
```

If you're loading a model in 8-bit for text generation, you should use the [`~transformers.GenerationMixin.generate`] method instead of the [`Pipeline`] function which is not optimized for 8-bit models and will be slower. Some sampling strategies, like nucleus sampling, are also not supported by the [`Pipeline`] for 8-bit models. You should also place all inputs on the same device as the model:

```py
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_name = "bigscience/bloom-2b5"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model_8bit = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=BitsAndBytesConfig(load_in_8bit=True))

prompt = "Hello, my llama is cute"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
generated_ids = model.generate(**inputs)
outputs = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
```

To load a model in 4-bit for inference with multiple GPUs, you can control how much GPU RAM you want to allocate to each GPU. For example, to distribute 1GB of memory to the first GPU and 2GB of memory to the second GPU:

```py
max_memory_mapping = {0: "1GB", 1: "2GB"}
model_name = "bigscience/bloom-3b"
model_8bit = AutoModelForCausalLM.from_pretrained(
    model_name, device_map="auto", load_in_8bit=True, max_memory=max_memory_mapping
)
```

<Tip>

Feel free to try running a 11 billion parameter [T5 model](https://colab.research.google.com/drive/1YORPWx4okIHXnjW7MSAidXN29mPVNT7F?usp=sharing) or the 3 billion parameter [BLOOM model](https://colab.research.google.com/drive/1qOjXfQIAULfKvZqwCen8-MoWKGdSatZ4?usp=sharing) for inference on Google Colab's free tier GPUs!


key: _node_content
string_value:
{"id_": "fe16e2f0-17e3-4ed3-ae9b-dc669ec2963d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4f7216d6-043c-4dda-a6ef-226e01ba6879", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "007d040f9c76352423c4d8e3f06a3eb050c8dc8a866aaf2368a104a0b1f96873", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dabad0cf-e41d-4cc9-8d5a-79edfa600d7f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "d8ddbf5e5db99983304993cda8d5e992757ccbc0f2df2fc3bba2c054b44cc723", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2982c02a-1451-4492-9bfe-53dff915c84c", "node_type": "1", "metadata": {}, "hash": "4fb7fa09c09665776357c1f165b687645b85ad0ae4e9cc8bc300cb83ee1c2ff0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21977, "end_char_idx": 24848, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You should also place all inputs on the same device as the model:

```py
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_name = "bigscience/bloom-2b5"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model_8bit = AutoModelForCausalLM.from_pretrained(model_name, quantization_config=BitsAndBytesConfig(load_in_8bit=True))

prompt = "Hello, my llama is cute"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
generated_ids = model.generate(**inputs)
outputs = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
```

To load a model in 4-bit for inference with multiple GPUs, you can control how much GPU RAM you want to allocate to each GPU. For example, to distribute 1GB of memory to the first GPU and 2GB of memory to the second GPU:

```py
max_memory_mapping = {0: "1GB", 1: "2GB"}
model_name = "bigscience/bloom-3b"
model_8bit = AutoModelForCausalLM.from_pretrained(
    model_name, device_map="auto", load_in_8bit=True, max_memory=max_memory_mapping
)
```

<Tip>

Feel free to try running a 11 billion parameter [T5 model](https://colab.research.google.com/drive/1YORPWx4okIHXnjW7MSAidXN29mPVNT7F?usp=sharing) or the 3 billion parameter [BLOOM model](https://colab.research.google.com/drive/1qOjXfQIAULfKvZqwCen8-MoWKGdSatZ4?usp=sharing) for inference on Google Colab's free tier GPUs!

</Tip>

## 🤗 Optimum

<Tip>

Learn more details about using ORT with 🤗 Optimum in the [Accelerated inference on NVIDIA GPUs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/gpu#accelerated-inference-on-nvidia-gpus) and [Accelerated inference on AMD GPUs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/amdgpu#accelerated-inference-on-amd-gpus) guides. This section only provides a brief and simple example.

</Tip>

ONNX Runtime (ORT) is a model accelerator that supports accelerated inference on Nvidia GPUs, and AMD GPUs that use [ROCm](https://www.amd.com/en/products/software/rocm.html) stack. ORT uses optimization techniques like fusing common operations into a single node and constant folding to reduce the number of computations performed and speedup inference. ORT also places the most computationally intensive operations on the GPU and the rest on the CPU to intelligently distribute the workload between the two devices.

ORT is supported by 🤗 Optimum which can be used in 🤗 Transformers. You'll need to use an [`~optimum.onnxruntime.ORTModel`] for the task you're solving, and specify the `provider` parameter which can be set to either [`CUDAExecutionProvider`](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/gpu#cudaexecutionprovider), [`ROCMExecutionProvider`](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/amdgpu) or [`TensorrtExecutionProvider`](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/gpu#tensorrtexecutionprovider).


key: _node_content
string_value:
{"id_": "2982c02a-1451-4492-9bfe-53dff915c84c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4f7216d6-043c-4dda-a6ef-226e01ba6879", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "007d040f9c76352423c4d8e3f06a3eb050c8dc8a866aaf2368a104a0b1f96873", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fe16e2f0-17e3-4ed3-ae9b-dc669ec2963d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "08d1fb64820d984927e3a1e1675323d4b6f668d46cc69867911414352bb05fb3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f681a33f-b00c-4585-b69a-53aad2679959", "node_type": "1", "metadata": {}, "hash": "6983403900b43b4cde8e5634c3937a5cb5a14610c1a42b3f450fc8fa4bf4251a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23335, "end_char_idx": 25875, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

## 🤗 Optimum

<Tip>

Learn more details about using ORT with 🤗 Optimum in the [Accelerated inference on NVIDIA GPUs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/gpu#accelerated-inference-on-nvidia-gpus) and [Accelerated inference on AMD GPUs](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/amdgpu#accelerated-inference-on-amd-gpus) guides. This section only provides a brief and simple example.

</Tip>

ONNX Runtime (ORT) is a model accelerator that supports accelerated inference on Nvidia GPUs, and AMD GPUs that use [ROCm](https://www.amd.com/en/products/software/rocm.html) stack. ORT uses optimization techniques like fusing common operations into a single node and constant folding to reduce the number of computations performed and speedup inference. ORT also places the most computationally intensive operations on the GPU and the rest on the CPU to intelligently distribute the workload between the two devices.

ORT is supported by 🤗 Optimum which can be used in 🤗 Transformers. You'll need to use an [`~optimum.onnxruntime.ORTModel`] for the task you're solving, and specify the `provider` parameter which can be set to either [`CUDAExecutionProvider`](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/gpu#cudaexecutionprovider), [`ROCMExecutionProvider`](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/amdgpu) or [`TensorrtExecutionProvider`](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/gpu#tensorrtexecutionprovider). If you want to load a model that was not yet exported to ONNX, you can set `export=True` to convert your model on-the-fly to the ONNX format:

```py
from optimum.onnxruntime import ORTModelForSequenceClassification

ort_model = ORTModelForSequenceClassification.from_pretrained(
  "distilbert/distilbert-base-uncased-finetuned-sst-2-english",
  export=True,
  provider="CUDAExecutionProvider",
)
```

Now you're free to use the model for inference:

```py
from optimum.pipelines import pipeline
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased-finetuned-sst-2-english")

pipeline = pipeline(task="text-classification", model=ort_model, tokenizer=tokenizer, device="cuda:0")
result = pipeline("Both the music and visual were astounding, not to mention the actors performance.")
```

## Combine optimizations

It is often possible to combine several of the optimization techniques described above to get the best inference performance possible for your model.


key: _node_content
string_value:
{"id_": "f681a33f-b00c-4585-b69a-53aad2679959", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "4f7216d6-043c-4dda-a6ef-226e01ba6879", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "007d040f9c76352423c4d8e3f06a3eb050c8dc8a866aaf2368a104a0b1f96873", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2982c02a-1451-4492-9bfe-53dff915c84c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_infer_gpu_one", "title": "GPU inference", "tokens": 7025, "retrieve_doc": true, "source": "transformers"}, "hash": "50ef78b03e735568e2f68439af972426ab19539fc50f0484a99945591d1e9f20", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24297, "end_char_idx": 26780, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
ORT is supported by 🤗 Optimum which can be used in 🤗 Transformers. You'll need to use an [`~optimum.onnxruntime.ORTModel`] for the task you're solving, and specify the `provider` parameter which can be set to either [`CUDAExecutionProvider`](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/gpu#cudaexecutionprovider), [`ROCMExecutionProvider`](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/amdgpu) or [`TensorrtExecutionProvider`](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/gpu#tensorrtexecutionprovider). If you want to load a model that was not yet exported to ONNX, you can set `export=True` to convert your model on-the-fly to the ONNX format:

```py
from optimum.onnxruntime import ORTModelForSequenceClassification

ort_model = ORTModelForSequenceClassification.from_pretrained(
  "distilbert/distilbert-base-uncased-finetuned-sst-2-english",
  export=True,
  provider="CUDAExecutionProvider",
)
```

Now you're free to use the model for inference:

```py
from optimum.pipelines import pipeline
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased-finetuned-sst-2-english")

pipeline = pipeline(task="text-classification", model=ort_model, tokenizer=tokenizer, device="cuda:0")
result = pipeline("Both the music and visual were astounding, not to mention the actors performance.")
```

## Combine optimizations

It is often possible to combine several of the optimization techniques described above to get the best inference performance possible for your model. For example, you can load a model in 4-bit, and then enable BetterTransformer with FlashAttention:

```py
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

# load model in 4-bit
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16
)

tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m")
model = AutoModelForCausalLM.from_pretrained("facebook/opt-350m", quantization_config=quantization_config)

# enable BetterTransformer
model = model.to_bettertransformer()

input_text = "Hello my dog is cute and"
inputs = tokenizer(input_text, return_tensors="pt").to("cuda")

# enable FlashAttention
with torch.backends.cuda.sdp_kernel(enable_flash=True, enable_math=False, enable_mem_efficient=False):
    outputs = model.generate(**inputs)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```


key: _node_content
string_value:
{"id_": "9ee218da-d410-4a34-ba2b-5c7a1c40abdb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3677, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "35ed512f-ca54-45b8-a5a6-eb5912c84e71", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3677, "retrieve_doc": true, "source": "transformers"}, "hash": "ffe609585380248d5b5108273a0d4d0feb0aa3c4bef6475e1078e180a1d4e440", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f2f58414-f07a-48c8-9ef8-0d011baf1941", "node_type": "1", "metadata": {}, "hash": "9ccfbd280615fea0c7e9ac62bd7ee5551db95b730ecddcfac28f222f8a580983", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2738, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Efficient Training on Multiple CPUs

When training on a single CPU is too slow, we can use multiple CPUs. This guide focuses on PyTorch-based DDP enabling
distributed CPU training efficiently on [bare metal](#usage-in-trainer) and [Kubernetes](#usage-with-kubernetes).

## Intel® oneCCL Bindings for PyTorch

[Intel® oneCCL](https://github.com/oneapi-src/oneCCL) (collective communications library) is a library for efficient distributed deep learning training implementing such collectives like allreduce, allgather, alltoall. For more information on oneCCL, please refer to the [oneCCL documentation](https://spec.oneapi.com/versions/latest/elements/oneCCL/source/index.html) and [oneCCL specification](https://spec.oneapi.com/versions/latest/elements/oneCCL/source/index.html).

Module `oneccl_bindings_for_pytorch` (`torch_ccl` before version 1.12)  implements PyTorch C10D ProcessGroup API and can be dynamically loaded as external ProcessGroup and only works on Linux platform now

Check more detailed information for [oneccl_bind_pt](https://github.com/intel/torch-ccl).

### Intel® oneCCL Bindings for PyTorch installation

Wheel files are available for the following Python versions:

| Extension Version | Python 3.6 | Python 3.7 | Python 3.8 | Python 3.9 | Python 3.10 |
| :---------------: | :--------: | :--------: | :--------: | :--------: | :---------: |
| 2.1.0             |            | √          | √          | √          | √           |
| 2.0.0             |            | √          | √          | √          | √           |
| 1.13.0            |            | √          | √          | √          | √           |
| 1.12.100          |            | √          | √          | √          | √           |
| 1.12.0            |            | √          | √          | √          | √           |

Please run `pip list | grep torch` to get your `pytorch_version`.
```bash
pip install oneccl_bind_pt=={pytorch_version} -f https://developer.intel.com/ipex-whl-stable-cpu
```
where `{pytorch_version}` should be your PyTorch version, for instance 2.1.0.
Check more approaches for [oneccl_bind_pt installation](https://github.com/intel/torch-ccl).
Versions of oneCCL and PyTorch must match.

<Tip warning={true}>

oneccl_bindings_for_pytorch 1.12.0 prebuilt wheel does not work with PyTorch 1.12.1 (it is for PyTorch 1.12.0)
PyTorch 1.12.1 should work with oneccl_bindings_for_pytorch 1.12.100

</Tip>

## Intel® MPI library
Use this standards-based MPI implementation to deliver flexible, efficient, scalable cluster messaging on Intel® architecture. This component is part of the Intel® oneAPI HPC Toolkit.

oneccl_bindings_for_pytorch is installed along with the MPI tool set. Need to source the environment before using it.


key: _node_content
string_value:
{"id_": "f2f58414-f07a-48c8-9ef8-0d011baf1941", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3677, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "35ed512f-ca54-45b8-a5a6-eb5912c84e71", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3677, "retrieve_doc": true, "source": "transformers"}, "hash": "ffe609585380248d5b5108273a0d4d0feb0aa3c4bef6475e1078e180a1d4e440", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9ee218da-d410-4a34-ba2b-5c7a1c40abdb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3677, "retrieve_doc": true, "source": "transformers"}, "hash": "6bb1eb5d523116886edf61afc0047f62889ba241c5bb59e9665c074368926d3c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3c3d8c16-1835-4ce9-ab47-ef0475f2dca7", "node_type": "1", "metadata": {}, "hash": "cc42a1766cb4ac5780371b8b51f5518e1f7b69ad5217ba383aec1bfae4ddac46", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1879, "end_char_idx": 4762, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```bash
pip install oneccl_bind_pt=={pytorch_version} -f https://developer.intel.com/ipex-whl-stable-cpu
```
where `{pytorch_version}` should be your PyTorch version, for instance 2.1.0.
Check more approaches for [oneccl_bind_pt installation](https://github.com/intel/torch-ccl).
Versions of oneCCL and PyTorch must match.

<Tip warning={true}>

oneccl_bindings_for_pytorch 1.12.0 prebuilt wheel does not work with PyTorch 1.12.1 (it is for PyTorch 1.12.0)
PyTorch 1.12.1 should work with oneccl_bindings_for_pytorch 1.12.100

</Tip>

## Intel® MPI library
Use this standards-based MPI implementation to deliver flexible, efficient, scalable cluster messaging on Intel® architecture. This component is part of the Intel® oneAPI HPC Toolkit.

oneccl_bindings_for_pytorch is installed along with the MPI tool set. Need to source the environment before using it.

for Intel® oneCCL >= 1.12.0
```bash
oneccl_bindings_for_pytorch_path=$(python -c "from oneccl_bindings_for_pytorch import cwd; print(cwd)")
source $oneccl_bindings_for_pytorch_path/env/setvars.sh
```

for Intel® oneCCL whose version < 1.12.0
```bash
torch_ccl_path=$(python -c "import torch; import torch_ccl; import os;  print(os.path.abspath(os.path.dirname(torch_ccl.__file__)))")
source $torch_ccl_path/env/setvars.sh
```

#### Intel® Extension for PyTorch installation

Intel Extension for PyTorch (IPEX) provides performance optimizations for CPU training with both Float32 and BFloat16 (refer to the [single CPU section](./perf_train_cpu) to learn more).


The following "Usage in Trainer" takes mpirun in Intel® MPI library as an example.


## Usage in Trainer
To enable multi CPU distributed training in the Trainer with the ccl backend, users should add **`--ddp_backend ccl`** in the command arguments.

Let's see an example with the [question-answering example](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering)


The following command enables training with 2 processes on one Xeon node, with one process running per one socket. The variables OMP_NUM_THREADS/CCL_WORKER_COUNT can be tuned for optimal performance.
```shell script
 export CCL_WORKER_COUNT=1
 export MASTER_ADDR=127.0.0.1
 mpirun -n 2 -genv OMP_NUM_THREADS=23 \
 python3 run_qa.py \
 --model_name_or_path google-bert/bert-large-uncased \
 --dataset_name squad \
 --do_train \
 --do_eval \
 --per_device_train_batch_size 12  \
 --learning_rate 3e-5  \
 --num_train_epochs 2  \
 --max_seq_length 384 \
 --doc_stride 128  \
 --output_dir /tmp/debug_squad/ \
 --no_cuda \
 --ddp_backend ccl \
 --use_ipex
```
The following command enables training with a total of four processes on two Xeons (node0 and node1, taking node0 as the main process), ppn (processes per node) is set to 2, with one process running per one socket. The variables OMP_NUM_THREADS/CCL_WORKER_COUNT can be tuned for optimal performance.


key: _node_content
string_value:
{"id_": "3c3d8c16-1835-4ce9-ab47-ef0475f2dca7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3677, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "35ed512f-ca54-45b8-a5a6-eb5912c84e71", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3677, "retrieve_doc": true, "source": "transformers"}, "hash": "ffe609585380248d5b5108273a0d4d0feb0aa3c4bef6475e1078e180a1d4e440", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f2f58414-f07a-48c8-9ef8-0d011baf1941", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3677, "retrieve_doc": true, "source": "transformers"}, "hash": "f946e61e467eaa0746476bd4f4152c19cc2a43a27bda090ba1a9b57b3c4a3c94", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0d12b0bd-b297-48f3-b831-6f320ce22eda", "node_type": "1", "metadata": {}, "hash": "99fe64b58df5888dd57dfbd3772bd28606c3155ef518407a7570183ade8f452d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3404, "end_char_idx": 6320, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The following "Usage in Trainer" takes mpirun in Intel® MPI library as an example.


## Usage in Trainer
To enable multi CPU distributed training in the Trainer with the ccl backend, users should add **`--ddp_backend ccl`** in the command arguments.

Let's see an example with the [question-answering example](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering)


The following command enables training with 2 processes on one Xeon node, with one process running per one socket. The variables OMP_NUM_THREADS/CCL_WORKER_COUNT can be tuned for optimal performance.
```shell script
 export CCL_WORKER_COUNT=1
 export MASTER_ADDR=127.0.0.1
 mpirun -n 2 -genv OMP_NUM_THREADS=23 \
 python3 run_qa.py \
 --model_name_or_path google-bert/bert-large-uncased \
 --dataset_name squad \
 --do_train \
 --do_eval \
 --per_device_train_batch_size 12  \
 --learning_rate 3e-5  \
 --num_train_epochs 2  \
 --max_seq_length 384 \
 --doc_stride 128  \
 --output_dir /tmp/debug_squad/ \
 --no_cuda \
 --ddp_backend ccl \
 --use_ipex
```
The following command enables training with a total of four processes on two Xeons (node0 and node1, taking node0 as the main process), ppn (processes per node) is set to 2, with one process running per one socket. The variables OMP_NUM_THREADS/CCL_WORKER_COUNT can be tuned for optimal performance.

In node0, you need to create a configuration file which contains the IP addresses of each node (for example hostfile) and pass that configuration file path as an argument.
```shell script
 cat hostfile
 xxx.xxx.xxx.xxx #node0 ip
 xxx.xxx.xxx.xxx #node1 ip
```
Now, run the following command in node0 and **4DDP** will be enabled in node0 and node1 with BF16 auto mixed precision:
```shell script
 export CCL_WORKER_COUNT=1
 export MASTER_ADDR=xxx.xxx.xxx.xxx #node0 ip
 mpirun -f hostfile -n 4 -ppn 2 \
 -genv OMP_NUM_THREADS=23 \
 python3 run_qa.py \
 --model_name_or_path google-bert/bert-large-uncased \
 --dataset_name squad \
 --do_train \
 --do_eval \
 --per_device_train_batch_size 12  \
 --learning_rate 3e-5  \
 --num_train_epochs 2  \
 --max_seq_length 384 \
 --doc_stride 128  \
 --output_dir /tmp/debug_squad/ \
 --no_cuda \
 --ddp_backend ccl \
 --use_ipex \
 --bf16
```

## Usage with Kubernetes

The same distributed training job from the previous section can be deployed to a Kubernetes cluster using the
[Kubeflow PyTorchJob training operator](https://www.kubeflow.org/docs/components/training/pytorch/).

### Setup

This example assumes that you have:
* Access to a Kubernetes cluster with [Kubeflow installed](https://www.kubeflow.org/docs/started/installing-kubeflow/)
* [`kubectl`](https://kubernetes.io/docs/tasks/tools/) installed and configured to access the Kubernetes cluster
* A [Persistent Volume Claim (PVC)](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) that can be used
  to store datasets and model files.


key: _node_content
string_value:
{"id_": "0d12b0bd-b297-48f3-b831-6f320ce22eda", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3677, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "35ed512f-ca54-45b8-a5a6-eb5912c84e71", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3677, "retrieve_doc": true, "source": "transformers"}, "hash": "ffe609585380248d5b5108273a0d4d0feb0aa3c4bef6475e1078e180a1d4e440", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3c3d8c16-1835-4ce9-ab47-ef0475f2dca7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3677, "retrieve_doc": true, "source": "transformers"}, "hash": "b49cc5d5f3b618d9cd15e04f8c99c98c7d2869aab6bf823947d375bf7ab1a12a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "130cffc2-d372-46e1-bdf7-49162e6593b4", "node_type": "1", "metadata": {}, "hash": "b7a07ab00c96e38bf3763e47a08be9a7bd2da43ff46e947acaae17e8f5d1bf8b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4936, "end_char_idx": 7756, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```shell script
 cat hostfile
 xxx.xxx.xxx.xxx #node0 ip
 xxx.xxx.xxx.xxx #node1 ip
```
Now, run the following command in node0 and **4DDP** will be enabled in node0 and node1 with BF16 auto mixed precision:
```shell script
 export CCL_WORKER_COUNT=1
 export MASTER_ADDR=xxx.xxx.xxx.xxx #node0 ip
 mpirun -f hostfile -n 4 -ppn 2 \
 -genv OMP_NUM_THREADS=23 \
 python3 run_qa.py \
 --model_name_or_path google-bert/bert-large-uncased \
 --dataset_name squad \
 --do_train \
 --do_eval \
 --per_device_train_batch_size 12  \
 --learning_rate 3e-5  \
 --num_train_epochs 2  \
 --max_seq_length 384 \
 --doc_stride 128  \
 --output_dir /tmp/debug_squad/ \
 --no_cuda \
 --ddp_backend ccl \
 --use_ipex \
 --bf16
```

## Usage with Kubernetes

The same distributed training job from the previous section can be deployed to a Kubernetes cluster using the
[Kubeflow PyTorchJob training operator](https://www.kubeflow.org/docs/components/training/pytorch/).

### Setup

This example assumes that you have:
* Access to a Kubernetes cluster with [Kubeflow installed](https://www.kubeflow.org/docs/started/installing-kubeflow/)
* [`kubectl`](https://kubernetes.io/docs/tasks/tools/) installed and configured to access the Kubernetes cluster
* A [Persistent Volume Claim (PVC)](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) that can be used
  to store datasets and model files. There are multiple options for setting up the PVC including using an NFS
  [storage class](https://kubernetes.io/docs/concepts/storage/storage-classes/) or a cloud storage bucket.
* A Docker container that includes your model training script and all the dependencies needed to run the script. For
  distributed CPU training jobs, this typically includes PyTorch, Transformers, Intel Extension for PyTorch, Intel
  oneCCL Bindings for PyTorch, and OpenSSH to communicate between the containers.

The snippet below is an example of a Dockerfile that uses a base image that supports distributed CPU training and then
extracts a Transformers release to the `/workspace` directory, so that the example scripts are included in the image:
```dockerfile
FROM intel/ai-workflows:torch-2.0.1-huggingface-multinode-py3.9

WORKDIR /workspace

# Download and extract the transformers code
ARG HF_TRANSFORMERS_VER="4.35.2"
RUN mkdir transformers && \
    curl -sSL --retry 5 https://github.com/huggingface/transformers/archive/refs/tags/v${HF_TRANSFORMERS_VER}.tar.gz | tar -C transformers --strip-components=1 -xzf -
```
The image needs to be built and copied to the cluster's nodes or pushed to a container registry prior to deploying the
PyTorchJob to the cluster.

### PyTorchJob Specification File

The [Kubeflow PyTorchJob](https://www.kubeflow.org/docs/components/training/pytorch/) is used to run the distributed
training job on the cluster.


key: _node_content
string_value:
{"id_": "130cffc2-d372-46e1-bdf7-49162e6593b4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3677, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "35ed512f-ca54-45b8-a5a6-eb5912c84e71", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3677, "retrieve_doc": true, "source": "transformers"}, "hash": "ffe609585380248d5b5108273a0d4d0feb0aa3c4bef6475e1078e180a1d4e440", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0d12b0bd-b297-48f3-b831-6f320ce22eda", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3677, "retrieve_doc": true, "source": "transformers"}, "hash": "16b0b2f89859c4c187e5284de3a273abbc0e9a1084b0cfff38360346c46c4ec2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ad484ac9-60a3-459f-bb93-e9c064f793e1", "node_type": "1", "metadata": {}, "hash": "f5d3deedbae3471e7fb8baf160bbf0e933862fb86a7a668fb1d0b1a9f9babbe4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6321, "end_char_idx": 8573, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
There are multiple options for setting up the PVC including using an NFS
  [storage class](https://kubernetes.io/docs/concepts/storage/storage-classes/) or a cloud storage bucket.
* A Docker container that includes your model training script and all the dependencies needed to run the script. For
  distributed CPU training jobs, this typically includes PyTorch, Transformers, Intel Extension for PyTorch, Intel
  oneCCL Bindings for PyTorch, and OpenSSH to communicate between the containers.

The snippet below is an example of a Dockerfile that uses a base image that supports distributed CPU training and then
extracts a Transformers release to the `/workspace` directory, so that the example scripts are included in the image:
```dockerfile
FROM intel/ai-workflows:torch-2.0.1-huggingface-multinode-py3.9

WORKDIR /workspace

# Download and extract the transformers code
ARG HF_TRANSFORMERS_VER="4.35.2"
RUN mkdir transformers && \
    curl -sSL --retry 5 https://github.com/huggingface/transformers/archive/refs/tags/v${HF_TRANSFORMERS_VER}.tar.gz | tar -C transformers --strip-components=1 -xzf -
```
The image needs to be built and copied to the cluster's nodes or pushed to a container registry prior to deploying the
PyTorchJob to the cluster.

### PyTorchJob Specification File

The [Kubeflow PyTorchJob](https://www.kubeflow.org/docs/components/training/pytorch/) is used to run the distributed
training job on the cluster. The yaml file for the PyTorchJob defines parameters such as:
 * The name of the PyTorchJob
 * The number of replicas (workers)
 * The python script and it's parameters that will be used to run the training job
 * The types of resources (node selector, memory, and CPU) needed for each worker
 * The image/tag for the Docker container to use
 * Environment variables
 * A volume mount for the PVC

The volume mount defines a path where the PVC will be mounted in the container for each worker pod. This location can be
used for the dataset, checkpoint files, and the saved model after training completes.

The snippet below is an example of a yaml file for a PyTorchJob with 4 workers running the
[question-answering example](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering).


key: _node_content
string_value:
{"id_": "ad484ac9-60a3-459f-bb93-e9c064f793e1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3677, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "35ed512f-ca54-45b8-a5a6-eb5912c84e71", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3677, "retrieve_doc": true, "source": "transformers"}, "hash": "ffe609585380248d5b5108273a0d4d0feb0aa3c4bef6475e1078e180a1d4e440", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "130cffc2-d372-46e1-bdf7-49162e6593b4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3677, "retrieve_doc": true, "source": "transformers"}, "hash": "8c872e9008c5b52155bc25ad47f409e0b937d5963cb2792801ce8c4e9877c8d1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ae963237-20ce-4387-bedb-4f8541a372fa", "node_type": "1", "metadata": {}, "hash": "94c7aa54c47772b3eb2781bd8e9ea38605eda9098a14a43dcfdd75d39f6b01be", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7576, "end_char_idx": 11580, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### PyTorchJob Specification File

The [Kubeflow PyTorchJob](https://www.kubeflow.org/docs/components/training/pytorch/) is used to run the distributed
training job on the cluster. The yaml file for the PyTorchJob defines parameters such as:
 * The name of the PyTorchJob
 * The number of replicas (workers)
 * The python script and it's parameters that will be used to run the training job
 * The types of resources (node selector, memory, and CPU) needed for each worker
 * The image/tag for the Docker container to use
 * Environment variables
 * A volume mount for the PVC

The volume mount defines a path where the PVC will be mounted in the container for each worker pod. This location can be
used for the dataset, checkpoint files, and the saved model after training completes.

The snippet below is an example of a yaml file for a PyTorchJob with 4 workers running the
[question-answering example](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering).
```yaml
apiVersion: "kubeflow.org/v1"
kind: PyTorchJob
metadata:
  name: transformers-pytorchjob
  namespace: kubeflow
spec:
  elasticPolicy:
    rdzvBackend: c10d
    minReplicas: 1
    maxReplicas: 4
    maxRestarts: 10
  pytorchReplicaSpecs:
    Worker:
      replicas: 4  # The number of worker pods
      restartPolicy: OnFailure
      template:
        spec:
          containers:
            - name: pytorch
              image: <image name>:<tag>  # Specify the docker image to use for the worker pods
              imagePullPolicy: IfNotPresent
              command:
                - torchrun
                - /workspace/transformers/examples/pytorch/question-answering/run_qa.py
                - --model_name_or_path
                - "google-bert/bert-large-uncased"
                - --dataset_name
                - "squad"
                - --do_train
                - --do_eval
                - --per_device_train_batch_size
                - "12"
                - --learning_rate
                - "3e-5"
                - --num_train_epochs
                - "2"
                - --max_seq_length
                - "384"
                - --doc_stride
                - "128"
                - --output_dir
                - "/tmp/pvc-mount/output"
                - --no_cuda
                - --ddp_backend
                - "ccl"
                - --use_ipex
                - --bf16  # Specify --bf16 if your hardware supports bfloat16
              env:
              - name: LD_PRELOAD
                value: "/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.9:/usr/local/lib/libiomp5.so"
              - name: TRANSFORMERS_CACHE
                value: "/tmp/pvc-mount/transformers_cache"
              - name: HF_DATASETS_CACHE
                value: "/tmp/pvc-mount/hf_datasets_cache"
              - name: LOGLEVEL
                value: "INFO"
              - name: CCL_WORKER_COUNT
                value: "1"
              - name: OMP_NUM_THREADS  # Can be tuned for optimal performance
-                value: "56"
              resources:
                limits:
                  cpu: 200  # Update the CPU and memory limit values based on your nodes
                  memory: 128Gi
                requests:
                  cpu: 200  # Update the CPU and memory request values based on your nodes
                  memory: 128Gi
              volumeMounts:
              - name: pvc-volume
                mountPath: /tmp/pvc-mount
              - mountPath: /dev/shm
                name: dshm
          restartPolicy: Never
          nodeSelector:  #  Optionally use the node selector to specify what types of nodes to use for the workers
            node-type: spr
          volumes:
          - name: pvc-volume
            persistentVolumeClaim:
              claimName: transformers-pvc
          - name: dshm
            emptyDir:
              medium: Memory
```
To run this example, update the yaml based on your training script and the nodes in your cluster.


key: _node_content
string_value:
{"id_": "ae963237-20ce-4387-bedb-4f8541a372fa", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3677, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "35ed512f-ca54-45b8-a5a6-eb5912c84e71", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3677, "retrieve_doc": true, "source": "transformers"}, "hash": "ffe609585380248d5b5108273a0d4d0feb0aa3c4bef6475e1078e180a1d4e440", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ad484ac9-60a3-459f-bb93-e9c064f793e1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_cpu_many", "title": "Efficient Training on Multiple CPUs", "tokens": 3677, "retrieve_doc": true, "source": "transformers"}, "hash": "05cea7679a80f168ce4da1837c8bcb923245531a706a6c0c678cd1c0b1e86c1b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11582, "end_char_idx": 14333, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<Tip>

The CPU resource limits/requests in the yaml are defined in [cpu units](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#meaning-of-cpu)
where 1 CPU unit is equivalent to 1 physical CPU core or 1 virtual core (depending on whether the node is a physical
host or a VM). The amount of CPU and memory limits/requests defined in the yaml should be less than the amount of
available CPU/memory capacity on a single machine. It is usually a good idea to not use the entire machine's capacity in
order to leave some resources for the kubelet and OS. In order to get ["guaranteed"](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/#guaranteed)
[quality of service](https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/) for the worker pods,
set the same CPU and memory amounts for both the resource limits and requests.

</Tip>

### Deploy

After the PyTorchJob spec has been updated with values appropriate for your cluster and training job, it can be deployed
to the cluster using:
```bash
kubectl create -f pytorchjob.yaml
```

The `kubectl get pods -n kubeflow` command can then be used to list the pods in the `kubeflow` namespace. You should see
the worker pods for the PyTorchJob that was just deployed. At first, they will probably have a status of "Pending" as
the containers get pulled and created, then the status should change to "Running".
```
NAME                                                     READY   STATUS                  RESTARTS          AGE
...
transformers-pytorchjob-worker-0                         1/1     Running                 0                 7m37s
transformers-pytorchjob-worker-1                         1/1     Running                 0                 7m37s
transformers-pytorchjob-worker-2                         1/1     Running                 0                 7m37s
transformers-pytorchjob-worker-3                         1/1     Running                 0                 7m37s
...
```

The logs for worker can be viewed using `kubectl logs -n kubeflow <pod name>`. Add `-f` to stream the logs, for example:
```bash
kubectl logs -n kubeflow transformers-pytorchjob-worker-0 -f
```

After the training job completes, the trained model can be copied from the PVC or storage location. When you are done
with the job, the PyTorchJob resource can be deleted from the cluster using `kubectl delete -f pytorchjob.yaml`.

## Summary

This guide covered running distributed PyTorch training jobs using multiple CPUs on bare metal and on a Kubernetes
cluster. Both cases utilize Intel Extension for PyTorch and Intel oneCCL Bindings for PyTorch for optimal training
performance, and can be used as a template to run your own workload on multiple nodes.


key: _node_content
string_value:
{"id_": "5bc13f53-148e-4043-892d-6e3661e8d12b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/pad_truncation", "title": "Padding and truncation", "tokens": 1364, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "123ca229-c652-4c30-8324-1ee8a14ff6ad", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/pad_truncation", "title": "Padding and truncation", "tokens": 1364, "retrieve_doc": true, "source": "transformers"}, "hash": "fcbf39d53d83c6f064cccf34a7654672d76b41edb3c0d340b128169d06afb319", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a4a6847a-2773-4103-8538-3d34af47b9ba", "node_type": "1", "metadata": {}, "hash": "e470205c927e63b5257be0726a9e40a2cb53fc05f060c8e73d6a930dc7d75c45", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3168, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Padding and truncation

Batched inputs are often different lengths, so they can't be converted to fixed-size tensors. Padding and truncation are strategies for dealing with this problem, to create rectangular tensors from batches of varying lengths. Padding adds a special **padding token** to ensure shorter sequences will have the same length as either the longest sequence in a batch or the maximum length accepted by the model. Truncation works in the other direction by truncating long sequences.

In most cases, padding your batch to the length of the longest sequence and truncating to the maximum length a model can accept works pretty well. However, the API supports more strategies if you need them. The three arguments you need to are: `padding`, `truncation` and `max_length`.

The `padding` argument controls padding. It can be a boolean or a string:

  - `True` or `'longest'`: pad to the longest sequence in the batch (no padding is applied if you only provide
    a single sequence).
  - `'max_length'`: pad to a length specified by the `max_length` argument or the maximum length accepted
    by the model if no `max_length` is provided (`max_length=None`). Padding will still be applied if you only provide a single sequence.
  - `False` or `'do_not_pad'`: no padding is applied. This is the default behavior.

The `truncation` argument controls truncation. It can be a boolean or a string:

  - `True` or `'longest_first'`: truncate to a maximum length specified by the `max_length` argument or
    the maximum length accepted by the model if no `max_length` is provided (`max_length=None`). This will
    truncate token by token, removing a token from the longest sequence in the pair until the proper length is
    reached.
  - `'only_second'`: truncate to a maximum length specified by the `max_length` argument or the maximum
    length accepted by the model if no `max_length` is provided (`max_length=None`). This will only truncate
    the second sentence of a pair if a pair of sequences (or a batch of pairs of sequences) is provided.
  - `'only_first'`: truncate to a maximum length specified by the `max_length` argument or the maximum
    length accepted by the model if no `max_length` is provided (`max_length=None`). This will only truncate
    the first sentence of a pair if a pair of sequences (or a batch of pairs of sequences) is provided.
  - `False` or `'do_not_truncate'`: no truncation is applied. This is the default behavior.

The `max_length` argument controls the length of the padding and truncation. It can be an integer or `None`, in which case it will default to the maximum length the model can accept. If the model has no specific maximum input length, truncation or padding to `max_length` is deactivated.

The following table summarizes the recommended way to setup padding and truncation. If you use pairs of input sequences in any of the following examples, you can replace `truncation=True` by a `STRATEGY` selected in
`['only_first', 'only_second', 'longest_first']`, i.e. `truncation='only_second'` or `truncation='longest_first'` to control how both sequences in the pair are truncated as detailed before.


key: _node_content
string_value:
{"id_": "a4a6847a-2773-4103-8538-3d34af47b9ba", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/pad_truncation", "title": "Padding and truncation", "tokens": 1364, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "123ca229-c652-4c30-8324-1ee8a14ff6ad", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/pad_truncation", "title": "Padding and truncation", "tokens": 1364, "retrieve_doc": true, "source": "transformers"}, "hash": "fcbf39d53d83c6f064cccf34a7654672d76b41edb3c0d340b128169d06afb319", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5bc13f53-148e-4043-892d-6e3661e8d12b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/pad_truncation", "title": "Padding and truncation", "tokens": 1364, "retrieve_doc": true, "source": "transformers"}, "hash": "c63f8575ca6beac956d1366d237eb2a23446ea0293863d38ffdcf7640921b0a2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1613, "end_char_idx": 6842, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This will
    truncate token by token, removing a token from the longest sequence in the pair until the proper length is
    reached.
  - `'only_second'`: truncate to a maximum length specified by the `max_length` argument or the maximum
    length accepted by the model if no `max_length` is provided (`max_length=None`). This will only truncate
    the second sentence of a pair if a pair of sequences (or a batch of pairs of sequences) is provided.
  - `'only_first'`: truncate to a maximum length specified by the `max_length` argument or the maximum
    length accepted by the model if no `max_length` is provided (`max_length=None`). This will only truncate
    the first sentence of a pair if a pair of sequences (or a batch of pairs of sequences) is provided.
  - `False` or `'do_not_truncate'`: no truncation is applied. This is the default behavior.

The `max_length` argument controls the length of the padding and truncation. It can be an integer or `None`, in which case it will default to the maximum length the model can accept. If the model has no specific maximum input length, truncation or padding to `max_length` is deactivated.

The following table summarizes the recommended way to setup padding and truncation. If you use pairs of input sequences in any of the following examples, you can replace `truncation=True` by a `STRATEGY` selected in
`['only_first', 'only_second', 'longest_first']`, i.e. `truncation='only_second'` or `truncation='longest_first'` to control how both sequences in the pair are truncated as detailed before.

| Truncation                           | Padding                           | Instruction                                                                                 |
|--------------------------------------|-----------------------------------|---------------------------------------------------------------------------------------------|
| no truncation                        | no padding                        | `tokenizer(batch_sentences)`                                                           |
|                                      | padding to max sequence in batch  | `tokenizer(batch_sentences, padding=True)` or                                          |
|                                      |                                   | `tokenizer(batch_sentences, padding='longest')`                                        |
|                                      | padding to max model input length | `tokenizer(batch_sentences, padding='max_length')`                                     |
|                                      | padding to specific length        | `tokenizer(batch_sentences, padding='max_length', max_length=42)`                      |
|                                      | padding to a multiple of a value  | `tokenizer(batch_sentences, padding=True, pad_to_multiple_of=8)`                        |
| truncation to max model input length | no padding                        | `tokenizer(batch_sentences, truncation=True)` or                                       |
|                                      |                                   | `tokenizer(batch_sentences, truncation=STRATEGY)`                                      |
|                                      | padding to max sequence in batch  | `tokenizer(batch_sentences, padding=True, truncation=True)` or                         |
|                                      |                                   | `tokenizer(batch_sentences, padding=True, truncation=STRATEGY)`                        |
|                                      | padding to max model input length | `tokenizer(batch_sentences, padding='max_length', truncation=True)` or                 |
|                                      |                                   | `tokenizer(batch_sentences, padding='max_length', truncation=STRATEGY)`                |
|                                      | padding to specific length        | Not possible                                                                                |
| truncation to specific length        | no padding                        | `tokenizer(batch_sentences, truncation=True, max_length=42)` or                        |
|                                      |                                   | `tokenizer(batch_sentences, truncation=STRATEGY, max_length=42)`                       |
|                                      | padding to max sequence in batch  | `tokenizer(batch_sentences, padding=True, truncation=True, max_length=42)` or          |
|                                      |                                   | `tokenizer(batch_sentences, padding=True, truncation=STRATEGY, max_length=42)`         |
|                                      | padding to max model input length | Not possible                                                                                |
|                                      | padding to specific length        | `tokenizer(batch_sentences, padding='max_length', truncation=True, max_length=42)` or  |
|                                      |                                   | `tokenizer(batch_sentences, padding='max_length', truncation=STRATEGY, max_length=42)` |


key: _node_content
string_value:
{"id_": "66c67af1-d146-4915-8d00-fafd48542201", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/accelerate", "title": "Distributed training with \ud83e\udd17 Accelerate", "tokens": 1054, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bd77a0d1-124f-4878-a2dc-d6fc9723d0f8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/accelerate", "title": "Distributed training with \ud83e\udd17 Accelerate", "tokens": 1054, "retrieve_doc": true, "source": "transformers"}, "hash": "839d0123fd4e6631f69648ba6ca5163ea5abc3628df918cbee14aaa6ef50dfd4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4e395ee4-7a36-4e96-876a-5bd5362d047a", "node_type": "1", "metadata": {}, "hash": "d72d682e22186a9be416d73902d80574c8e1bb97898d735edede859bfb0be1ae", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3394, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Distributed training with 🤗 Accelerate

As models get bigger, parallelism has emerged as a strategy for training larger models on limited hardware and accelerating training speed by several orders of magnitude. At Hugging Face, we created the [🤗 Accelerate](https://huggingface.co/docs/accelerate) library to help users easily train a 🤗 Transformers model on any type of distributed setup, whether it is multiple GPU's on one machine or multiple GPU's across several machines. In this tutorial, learn how to customize your native PyTorch training loop to enable training in a distributed environment.

## Setup

Get started by installing 🤗 Accelerate:

```bash
pip install accelerate
```

Then import and create an [`~accelerate.Accelerator`] object. The [`~accelerate.Accelerator`] will automatically detect your type of distributed setup and initialize all the necessary components for training. You don't need to explicitly place your model on a device.

```py
>>> from accelerate import Accelerator

>>> accelerator = Accelerator()
```

## Prepare to accelerate

The next step is to pass all the relevant training objects to the [`~accelerate.Accelerator.prepare`] method. This includes your training and evaluation DataLoaders, a model and an optimizer:

```py
>>> train_dataloader, eval_dataloader, model, optimizer = accelerator.prepare(
...     train_dataloader, eval_dataloader, model, optimizer
... )
```

## Backward

The last addition is to replace the typical `loss.backward()` in your training loop with 🤗 Accelerate's [`~accelerate.Accelerator.backward`]method:

```py
>>> for epoch in range(num_epochs):
...     for batch in train_dataloader:
...         outputs = model(**batch)
...         loss = outputs.loss
...         accelerator.backward(loss)

...         optimizer.step()
...         lr_scheduler.step()
...         optimizer.zero_grad()
...         progress_bar.update(1)
```

As you can see in the following code, you only need to add four additional lines of code to your training loop to enable distributed training!

```diff
+ from accelerate import Accelerator
  from transformers import AdamW, AutoModelForSequenceClassification, get_scheduler

+ accelerator = Accelerator()

  model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=2)
  optimizer = AdamW(model.parameters(), lr=3e-5)

- device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
- model.to(device)

+ train_dataloader, eval_dataloader, model, optimizer = accelerator.prepare(
+     train_dataloader, eval_dataloader, model, optimizer
+ )

  num_epochs = 3
  num_training_steps = num_epochs * len(train_dataloader)
  lr_scheduler = get_scheduler(
      "linear",
      optimizer=optimizer,
      num_warmup_steps=0,
      num_training_steps=num_training_steps
  )

  progress_bar = tqdm(range(num_training_steps))

  model.train()
  for epoch in range(num_epochs):
      for batch in train_dataloader:
-         batch = {k: v.to(device) for k, v in batch.items()}
          outputs = model(**batch)
          loss = outputs.loss
-         loss.backward()
+         accelerator.backward(loss)

          optimizer.step()
          lr_scheduler.step()
          optimizer.zero_grad()
          progress_bar.update(1)
```

## Train

Once you've added the relevant lines of code, launch your training in a script or a notebook like Colaboratory.


key: _node_content
string_value:
{"id_": "4e395ee4-7a36-4e96-876a-5bd5362d047a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/accelerate", "title": "Distributed training with \ud83e\udd17 Accelerate", "tokens": 1054, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bd77a0d1-124f-4878-a2dc-d6fc9723d0f8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/accelerate", "title": "Distributed training with \ud83e\udd17 Accelerate", "tokens": 1054, "retrieve_doc": true, "source": "transformers"}, "hash": "839d0123fd4e6631f69648ba6ca5163ea5abc3628df918cbee14aaa6ef50dfd4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "66c67af1-d146-4915-8d00-fafd48542201", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/accelerate", "title": "Distributed training with \ud83e\udd17 Accelerate", "tokens": 1054, "retrieve_doc": true, "source": "transformers"}, "hash": "c14a49a78c6c1489bc7fb3bf51ebb898dcffbad5148dd111f4f49db77c336be0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2049, "end_char_idx": 4096, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```diff
+ from accelerate import Accelerator
  from transformers import AdamW, AutoModelForSequenceClassification, get_scheduler

+ accelerator = Accelerator()

  model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=2)
  optimizer = AdamW(model.parameters(), lr=3e-5)

- device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
- model.to(device)

+ train_dataloader, eval_dataloader, model, optimizer = accelerator.prepare(
+     train_dataloader, eval_dataloader, model, optimizer
+ )

  num_epochs = 3
  num_training_steps = num_epochs * len(train_dataloader)
  lr_scheduler = get_scheduler(
      "linear",
      optimizer=optimizer,
      num_warmup_steps=0,
      num_training_steps=num_training_steps
  )

  progress_bar = tqdm(range(num_training_steps))

  model.train()
  for epoch in range(num_epochs):
      for batch in train_dataloader:
-         batch = {k: v.to(device) for k, v in batch.items()}
          outputs = model(**batch)
          loss = outputs.loss
-         loss.backward()
+         accelerator.backward(loss)

          optimizer.step()
          lr_scheduler.step()
          optimizer.zero_grad()
          progress_bar.update(1)
```

## Train

Once you've added the relevant lines of code, launch your training in a script or a notebook like Colaboratory.

### Train with a script

If you are running your training from a script, run the following command to create and save a configuration file:

```bash
accelerate config
```

Then launch your training with:

```bash
accelerate launch train.py
```

### Train with a notebook

🤗 Accelerate can also run in a notebook if you're planning on using Colaboratory's TPUs. Wrap all the code responsible for training in a function, and pass it to [`~accelerate.notebook_launcher`]:

```py
>>> from accelerate import notebook_launcher

>>> notebook_launcher(training_function)
```

For more information about 🤗 Accelerate and its rich features, refer to the [documentation](https://huggingface.co/docs/accelerate).


key: _node_content
string_value:
{"id_": "f0644d87-863c-4288-958b-2661314ec8c6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "98a81be9-b09f-4d62-b899-6ed78b4ebb8a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4fbae4eb-cf38-4d88-819d-a42b15b07a7f", "node_type": "1", "metadata": {}, "hash": "5646aaa13f706a472cedfbc8fc5d1cb7e667bb5490d127cec146423143f317f6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2733, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

# Community

This page regroups resources around 🤗 Transformers developed by the community.

## Community resources:

| Resource     |      Description      |      Author      |
|:----------|:-------------|------:|
| [Hugging Face Transformers Glossary Flashcards](https://www.darigovresearch.com/huggingface-transformers-glossary-flashcards) | A set of flashcards based on the [Transformers Docs Glossary](glossary) that has been put into a form which can be easily learned/revised using [Anki](https://apps.ankiweb.net/) an open source, cross platform app specifically designed for long term knowledge retention. See this [Introductory video on how to use the flashcards](https://www.youtube.com/watch?v=Dji_h7PILrw). | [Darigov Research](https://www.darigovresearch.com/) |

## Community notebooks:

| Notebook     |      Description      |      Author      |      |
|:----------|:-------------|:-------------|------:|
| [Fine-tune a pre-trained Transformer to generate lyrics](https://github.com/AlekseyKorshuk/huggingartists) | How to generate lyrics in the style of your favorite artist by fine-tuning a GPT-2 model |  [Aleksey Korshuk](https://github.com/AlekseyKorshuk) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/AlekseyKorshuk/huggingartists/blob/master/huggingartists-demo.ipynb) |
| [Train T5 in Tensorflow 2](https://github.com/snapthat/TF-T5-text-to-text) | How to train T5 for any task using Tensorflow 2. This notebook demonstrates a Question & Answer task implemented in Tensorflow 2 using SQUAD | [Muhammad Harris](https://github.com/HarrisDePerceptron) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/snapthat/TF-T5-text-to-text/blob/master/snapthatT5/notebooks/TF-T5-Datasets%20Training.ipynb) |
| [Train T5 on TPU](https://github.com/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb)  | How to train T5 on SQUAD with Transformers and Nlp | [Suraj Patil](https://github.com/patil-suraj) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb#scrollTo=QLGiFCDqvuil) |
| [Fine-tune T5 for Classification and Multiple Choice](https://github.com/patil-suraj/exploring-T5/blob/master/t5_fine_tuning.ipynb)  | How to fine-tune T5 for classification and multiple choice tasks using a text-to-text format with PyTorch Lightning |  [Suraj Patil](https://github.com/patil-suraj) | [!


key: _node_content
string_value:
{"id_": "4fbae4eb-cf38-4d88-819d-a42b15b07a7f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "98a81be9-b09f-4d62-b899-6ed78b4ebb8a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f0644d87-863c-4288-958b-2661314ec8c6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74757c3f59a27e93424cba24e98b9981afcfe44235c9057718462dbf88d7bba1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6a64ee0b-c895-4059-b874-b2419404cb13", "node_type": "1", "metadata": {}, "hash": "03e4059e6cdb8d1f431fb51aa411f27bddccb5dc77a4ab65d7f877e7830585ac", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1669, "end_char_idx": 4102, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This notebook demonstrates a Question & Answer task implemented in Tensorflow 2 using SQUAD | [Muhammad Harris](https://github.com/HarrisDePerceptron) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/snapthat/TF-T5-text-to-text/blob/master/snapthatT5/notebooks/TF-T5-Datasets%20Training.ipynb) |
| [Train T5 on TPU](https://github.com/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb)  | How to train T5 on SQUAD with Transformers and Nlp | [Suraj Patil](https://github.com/patil-suraj) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb#scrollTo=QLGiFCDqvuil) |
| [Fine-tune T5 for Classification and Multiple Choice](https://github.com/patil-suraj/exploring-T5/blob/master/t5_fine_tuning.ipynb)  | How to fine-tune T5 for classification and multiple choice tasks using a text-to-text format with PyTorch Lightning |  [Suraj Patil](https://github.com/patil-suraj) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patil-suraj/exploring-T5/blob/master/t5_fine_tuning.ipynb) |
| [Fine-tune DialoGPT on New Datasets and Languages](https://github.com/ncoop57/i-am-a-nerd/blob/master/_notebooks/2020-05-12-chatbot-part-1.ipynb)  | How to fine-tune the DialoGPT model on a new dataset for open-dialog conversational chatbots |  [Nathan Cooper](https://github.com/ncoop57) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ncoop57/i-am-a-nerd/blob/master/_notebooks/2020-05-12-chatbot-part-1.ipynb) |
| [Long Sequence Modeling with Reformer](https://github.com/patrickvonplaten/notebooks/blob/master/PyTorch_Reformer.ipynb)  | How to train on sequences as long as 500,000 tokens with Reformer |  [Patrick von Platen](https://github.com/patrickvonplaten) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/PyTorch_Reformer.ipynb)  |
| [Fine-tune BART for Summarization](https://github.com/ohmeow/ohmeow_website/blob/master/posts/2021-05-25-mbart-sequence-classification-with-blurr.ipynb) | How to fine-tune BART for summarization with fastai using blurr | [Wayde Gilliam](https://ohmeow.com/) | [!


key: _node_content
string_value:
{"id_": "6a64ee0b-c895-4059-b874-b2419404cb13", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "98a81be9-b09f-4d62-b899-6ed78b4ebb8a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4fbae4eb-cf38-4d88-819d-a42b15b07a7f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "0cac1fd67084d1022359bc1214802498c70abc02bed0969e29a13c0f44654ea3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8c784227-2d51-46bf-a0dc-0bca9109dc32", "node_type": "1", "metadata": {}, "hash": "43676eefa6ae10acac64ac2d310ae9b05be3d412612453e74c64b05b78b48f87", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3205, "end_char_idx": 5576, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ncoop57/i-am-a-nerd/blob/master/_notebooks/2020-05-12-chatbot-part-1.ipynb) |
| [Long Sequence Modeling with Reformer](https://github.com/patrickvonplaten/notebooks/blob/master/PyTorch_Reformer.ipynb)  | How to train on sequences as long as 500,000 tokens with Reformer |  [Patrick von Platen](https://github.com/patrickvonplaten) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/PyTorch_Reformer.ipynb)  |
| [Fine-tune BART for Summarization](https://github.com/ohmeow/ohmeow_website/blob/master/posts/2021-05-25-mbart-sequence-classification-with-blurr.ipynb) | How to fine-tune BART for summarization with fastai using blurr | [Wayde Gilliam](https://ohmeow.com/) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ohmeow/ohmeow_website/blob/master/posts/2021-05-25-mbart-sequence-classification-with-blurr.ipynb) |
| [Fine-tune a pre-trained Transformer on anyone's tweets](https://colab.research.google.com/github/borisdayma/huggingtweets/blob/master/huggingtweets-demo.ipynb) | How to generate tweets in the style of your favorite Twitter account by fine-tuning a GPT-2 model |  [Boris Dayma](https://github.com/borisdayma) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/borisdayma/huggingtweets/blob/master/huggingtweets-demo.ipynb) |
| [Optimize 🤗 Hugging Face models with Weights & Biases](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/huggingface/Optimize_Hugging_Face_models_with_Weights_%26_Biases.ipynb) | A complete tutorial showcasing W&B integration with Hugging Face | [Boris Dayma](https://github.com/borisdayma) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/huggingface/Optimize_Hugging_Face_models_with_Weights_%26_Biases.ipynb) |
| [Pretrain Longformer](https://github.com/allenai/longformer/blob/master/scripts/convert_model_to_long.ipynb)  | How to build a "long" version of existing pretrained models |  [Iz Beltagy](https://beltagy.net) | [!


key: _node_content
string_value:
{"id_": "8c784227-2d51-46bf-a0dc-0bca9109dc32", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "98a81be9-b09f-4d62-b899-6ed78b4ebb8a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6a64ee0b-c895-4059-b874-b2419404cb13", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "d586fea41f6da6f0f3425c51bfce70fa736ac50c2babb49ec871734ac2f8d42d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3c23fa17-e7e1-4f20-b183-73c39b6a62a4", "node_type": "1", "metadata": {}, "hash": "0d93a5a7e3890ef8f99723966e997099817e5d7fbcd3ef21d0a2968a08340ae8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4634, "end_char_idx": 6882, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/borisdayma/huggingtweets/blob/master/huggingtweets-demo.ipynb) |
| [Optimize 🤗 Hugging Face models with Weights & Biases](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/huggingface/Optimize_Hugging_Face_models_with_Weights_%26_Biases.ipynb) | A complete tutorial showcasing W&B integration with Hugging Face | [Boris Dayma](https://github.com/borisdayma) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/huggingface/Optimize_Hugging_Face_models_with_Weights_%26_Biases.ipynb) |
| [Pretrain Longformer](https://github.com/allenai/longformer/blob/master/scripts/convert_model_to_long.ipynb)  | How to build a "long" version of existing pretrained models |  [Iz Beltagy](https://beltagy.net) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/allenai/longformer/blob/master/scripts/convert_model_to_long.ipynb) |
| [Fine-tune Longformer for QA](https://github.com/patil-suraj/Notebooks/blob/master/longformer_qa_training.ipynb) | How to fine-tune longformer model for QA task | [Suraj Patil](https://github.com/patil-suraj) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patil-suraj/Notebooks/blob/master/longformer_qa_training.ipynb) |
| [Evaluate Model with 🤗nlp](https://github.com/patrickvonplaten/notebooks/blob/master/How_to_evaluate_Longformer_on_TriviaQA_using_NLP.ipynb) | How to evaluate longformer on TriviaQA with `nlp` | [Patrick von Platen](https://github.com/patrickvonplaten) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1m7eTGlPmLRgoPkkA7rkhQdZ9ydpmsdLE?usp=sharing) |
| [Fine-tune T5 for Sentiment Span Extraction](https://github.com/enzoampil/t5-intro/blob/master/t5_qa_training_pytorch_span_extraction.ipynb)  | How to fine-tune T5 for sentiment span extraction using a text-to-text format with PyTorch Lightning |  [Lorenzo Ampil](https://github.com/enzoampil) | [!


key: _node_content
string_value:
{"id_": "3c23fa17-e7e1-4f20-b183-73c39b6a62a4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "98a81be9-b09f-4d62-b899-6ed78b4ebb8a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8c784227-2d51-46bf-a0dc-0bca9109dc32", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "f42928e975e07076e62f5ece65bb8100d9c172af013b36309bafc4b47f7ad1e3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f7d1d5d3-dbe3-4e70-9af0-493eb17519b7", "node_type": "1", "metadata": {}, "hash": "4f6876d0a4c0ebf1f9e8b456eeef0cbe1df29ed19164be9f0eda6dc283b2f8dc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5576, "end_char_idx": 7882, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/allenai/longformer/blob/master/scripts/convert_model_to_long.ipynb) |
| [Fine-tune Longformer for QA](https://github.com/patil-suraj/Notebooks/blob/master/longformer_qa_training.ipynb) | How to fine-tune longformer model for QA task | [Suraj Patil](https://github.com/patil-suraj) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patil-suraj/Notebooks/blob/master/longformer_qa_training.ipynb) |
| [Evaluate Model with 🤗nlp](https://github.com/patrickvonplaten/notebooks/blob/master/How_to_evaluate_Longformer_on_TriviaQA_using_NLP.ipynb) | How to evaluate longformer on TriviaQA with `nlp` | [Patrick von Platen](https://github.com/patrickvonplaten) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1m7eTGlPmLRgoPkkA7rkhQdZ9ydpmsdLE?usp=sharing) |
| [Fine-tune T5 for Sentiment Span Extraction](https://github.com/enzoampil/t5-intro/blob/master/t5_qa_training_pytorch_span_extraction.ipynb)  | How to fine-tune T5 for sentiment span extraction using a text-to-text format with PyTorch Lightning |  [Lorenzo Ampil](https://github.com/enzoampil) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/enzoampil/t5-intro/blob/master/t5_qa_training_pytorch_span_extraction.ipynb) |
| [Fine-tune DistilBert for Multiclass Classification](https://github.com/abhimishra91/transformers-tutorials/blob/master/transformers_multiclass_classification.ipynb) | How to fine-tune DistilBert for multiclass classification with PyTorch | [Abhishek Kumar Mishra](https://github.com/abhimishra91) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/abhimishra91/transformers-tutorials/blob/master/transformers_multiclass_classification.ipynb)|
|[Fine-tune BERT for Multi-label Classification](https://github.com/abhimishra91/transformers-tutorials/blob/master/transformers_multi_label_classification.ipynb)|How to fine-tune BERT for multi-label classification using PyTorch|[Abhishek Kumar Mishra](https://github.com/abhimishra91) |[!


key: _node_content
string_value:
{"id_": "f7d1d5d3-dbe3-4e70-9af0-493eb17519b7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "98a81be9-b09f-4d62-b899-6ed78b4ebb8a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3c23fa17-e7e1-4f20-b183-73c39b6a62a4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "a9e0f6716bcc42f4ae1e91ab58b7e2f4ddbe9b5219411ba5edf691d7b3c48f5e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bf866754-cbbe-401e-bd1a-48f31cedfd46", "node_type": "1", "metadata": {}, "hash": "907c9c21a581057ef6092e330668e407c94875911b4e0b8d8edfa29506e142a8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6882, "end_char_idx": 9359, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/enzoampil/t5-intro/blob/master/t5_qa_training_pytorch_span_extraction.ipynb) |
| [Fine-tune DistilBert for Multiclass Classification](https://github.com/abhimishra91/transformers-tutorials/blob/master/transformers_multiclass_classification.ipynb) | How to fine-tune DistilBert for multiclass classification with PyTorch | [Abhishek Kumar Mishra](https://github.com/abhimishra91) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/abhimishra91/transformers-tutorials/blob/master/transformers_multiclass_classification.ipynb)|
|[Fine-tune BERT for Multi-label Classification](https://github.com/abhimishra91/transformers-tutorials/blob/master/transformers_multi_label_classification.ipynb)|How to fine-tune BERT for multi-label classification using PyTorch|[Abhishek Kumar Mishra](https://github.com/abhimishra91) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/abhimishra91/transformers-tutorials/blob/master/transformers_multi_label_classification.ipynb)|
|[Fine-tune T5 for Summarization](https://github.com/abhimishra91/transformers-tutorials/blob/master/transformers_summarization_wandb.ipynb)|How to fine-tune T5 for summarization in PyTorch and track experiments with WandB|[Abhishek Kumar Mishra](https://github.com/abhimishra91) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/abhimishra91/transformers-tutorials/blob/master/transformers_summarization_wandb.ipynb)|
|[Speed up Fine-Tuning in Transformers with Dynamic Padding / Bucketing](https://github.com/ELS-RD/transformers-notebook/blob/master/Divide_Hugging_Face_Transformers_training_time_by_2_or_more.ipynb)|How to speed up fine-tuning by a factor of 2 using dynamic padding / bucketing|[Michael Benesty](https://github.com/pommedeterresautee) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1CBfRU1zbfu7-ijiOqAAQUA-RJaxfcJoO?usp=sharing)|
|[Pretrain Reformer for Masked Language Modeling](https://github.com/patrickvonplaten/notebooks/blob/master/Reformer_For_Masked_LM.ipynb)| How to train a Reformer model with bi-directional self-attention layers | [Patrick von Platen](https://github.com/patrickvonplaten) | [!


key: _node_content
string_value:
{"id_": "bf866754-cbbe-401e-bd1a-48f31cedfd46", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "98a81be9-b09f-4d62-b899-6ed78b4ebb8a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f7d1d5d3-dbe3-4e70-9af0-493eb17519b7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "b3ad9009bfdc68ae216debd2d5a68b1a9617ec528d54e4df93d55b343a656193", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0910f2a7-a57c-41dd-9e9a-5828b332a9e6", "node_type": "1", "metadata": {}, "hash": "85fa2245760f981b1f35660b7c49afa4abafb5fe617af081d9fb99ba38bfd74b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8377, "end_char_idx": 10803, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/abhimishra91/transformers-tutorials/blob/master/transformers_summarization_wandb.ipynb)|
|[Speed up Fine-Tuning in Transformers with Dynamic Padding / Bucketing](https://github.com/ELS-RD/transformers-notebook/blob/master/Divide_Hugging_Face_Transformers_training_time_by_2_or_more.ipynb)|How to speed up fine-tuning by a factor of 2 using dynamic padding / bucketing|[Michael Benesty](https://github.com/pommedeterresautee) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1CBfRU1zbfu7-ijiOqAAQUA-RJaxfcJoO?usp=sharing)|
|[Pretrain Reformer for Masked Language Modeling](https://github.com/patrickvonplaten/notebooks/blob/master/Reformer_For_Masked_LM.ipynb)| How to train a Reformer model with bi-directional self-attention layers | [Patrick von Platen](https://github.com/patrickvonplaten) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1tzzh0i8PgDQGV3SMFUGxM7_gGae3K-uW?usp=sharing)|
|[Expand and Fine Tune Sci-BERT](https://github.com/lordtt13/word-embeddings/blob/master/COVID-19%20Research%20Data/COVID-SciBERT.ipynb)| How to increase vocabulary of a pretrained SciBERT model from AllenAI on the CORD dataset and pipeline it. | [Tanmay Thakur](https://github.com/lordtt13) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1rqAR40goxbAfez1xvF3hBJphSCsvXmh8)|
|[Fine Tune BlenderBotSmall for Summarization using the Trainer API](https://github.com/lordtt13/transformers-experiments/blob/master/Custom%20Tasks/fine-tune-blenderbot_small-for-summarization.ipynb)| How to fine-tune BlenderBotSmall for summarization on a custom dataset, using the Trainer API. | [Tanmay Thakur](https://github.com/lordtt13) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/19Wmupuls7mykSGyRN_Qo6lPQhgp56ymq?usp=sharing)|
|[Fine-tune Electra and interpret with Integrated Gradients](https://github.com/elsanns/xai-nlp-notebooks/blob/master/electra_fine_tune_interpret_captum_ig.ipynb) | How to fine-tune Electra for sentiment analysis and interpret predictions with Captum Integrated Gradients | [Eliza Szczechla](https://elsanns.github.io) | [!


key: _node_content
string_value:
{"id_": "0910f2a7-a57c-41dd-9e9a-5828b332a9e6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "98a81be9-b09f-4d62-b899-6ed78b4ebb8a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bf866754-cbbe-401e-bd1a-48f31cedfd46", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "6ea5efbb16cf1500241c36b30d05cb03219f587baa76820d8752040ad23f957f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2d507b83-6251-46a2-a096-f50b676c038a", "node_type": "1", "metadata": {}, "hash": "e2bf0113b2a3c3b6e1ad1f33e5a921bb1700b307dfc6541bf459fe8dc67a0c88", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9767, "end_char_idx": 12337, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
| [Tanmay Thakur](https://github.com/lordtt13) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1rqAR40goxbAfez1xvF3hBJphSCsvXmh8)|
|[Fine Tune BlenderBotSmall for Summarization using the Trainer API](https://github.com/lordtt13/transformers-experiments/blob/master/Custom%20Tasks/fine-tune-blenderbot_small-for-summarization.ipynb)| How to fine-tune BlenderBotSmall for summarization on a custom dataset, using the Trainer API. | [Tanmay Thakur](https://github.com/lordtt13) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/19Wmupuls7mykSGyRN_Qo6lPQhgp56ymq?usp=sharing)|
|[Fine-tune Electra and interpret with Integrated Gradients](https://github.com/elsanns/xai-nlp-notebooks/blob/master/electra_fine_tune_interpret_captum_ig.ipynb) | How to fine-tune Electra for sentiment analysis and interpret predictions with Captum Integrated Gradients | [Eliza Szczechla](https://elsanns.github.io) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/elsanns/xai-nlp-notebooks/blob/master/electra_fine_tune_interpret_captum_ig.ipynb)|
|[fine-tune a non-English GPT-2 Model with Trainer class](https://github.com/philschmid/fine-tune-GPT-2/blob/master/Fine_tune_a_non_English_GPT_2_Model_with_Huggingface.ipynb) | How to fine-tune a non-English GPT-2 Model with Trainer class | [Philipp Schmid](https://www.philschmid.de) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/philschmid/fine-tune-GPT-2/blob/master/Fine_tune_a_non_English_GPT_2_Model_with_Huggingface.ipynb)|
|[Fine-tune a DistilBERT Model for Multi Label Classification task](https://github.com/DhavalTaunk08/Transformers_scripts/blob/master/Transformers_multilabel_distilbert.ipynb) | How to fine-tune a DistilBERT Model for Multi Label Classification task | [Dhaval Taunk](https://github.com/DhavalTaunk08) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/DhavalTaunk08/Transformers_scripts/blob/master/Transformers_multilabel_distilbert.ipynb)|
|[Fine-tune ALBERT for sentence-pair classification](https://github.com/NadirEM/nlp-notebooks/blob/master/Fine_tune_ALBERT_sentence_pair_classification.ipynb) | How to fine-tune an ALBERT model or another BERT-based model for the sentence-pair classification task | [Nadir El Manouzi](https://github.com/NadirEM) | [!


key: _node_content
string_value:
{"id_": "2d507b83-6251-46a2-a096-f50b676c038a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "98a81be9-b09f-4d62-b899-6ed78b4ebb8a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0910f2a7-a57c-41dd-9e9a-5828b332a9e6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "48847e1090dbe7503e4bbe3262eb3c518c13ec6378b6668a0c109e4d29f78ae3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "31df94ea-8005-4005-a0d2-795359ec7851", "node_type": "1", "metadata": {}, "hash": "81394bed4305e0d14d84fad18b41452ea7a31658c6b5493e6766e1f856ab11eb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11293, "end_char_idx": 13629, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/philschmid/fine-tune-GPT-2/blob/master/Fine_tune_a_non_English_GPT_2_Model_with_Huggingface.ipynb)|
|[Fine-tune a DistilBERT Model for Multi Label Classification task](https://github.com/DhavalTaunk08/Transformers_scripts/blob/master/Transformers_multilabel_distilbert.ipynb) | How to fine-tune a DistilBERT Model for Multi Label Classification task | [Dhaval Taunk](https://github.com/DhavalTaunk08) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/DhavalTaunk08/Transformers_scripts/blob/master/Transformers_multilabel_distilbert.ipynb)|
|[Fine-tune ALBERT for sentence-pair classification](https://github.com/NadirEM/nlp-notebooks/blob/master/Fine_tune_ALBERT_sentence_pair_classification.ipynb) | How to fine-tune an ALBERT model or another BERT-based model for the sentence-pair classification task | [Nadir El Manouzi](https://github.com/NadirEM) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NadirEM/nlp-notebooks/blob/master/Fine_tune_ALBERT_sentence_pair_classification.ipynb)|
|[Fine-tune Roberta for sentiment analysis](https://github.com/DhavalTaunk08/NLP_scripts/blob/master/sentiment_analysis_using_roberta.ipynb) | How to fine-tune a Roberta model for sentiment analysis | [Dhaval Taunk](https://github.com/DhavalTaunk08) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/DhavalTaunk08/NLP_scripts/blob/master/sentiment_analysis_using_roberta.ipynb)|
|[Evaluating Question Generation Models](https://github.com/flexudy-pipe/qugeev) | How accurate are the answers to questions generated by your seq2seq transformer model? | [Pascal Zoleko](https://github.com/zolekode) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1bpsSqCQU-iw_5nNoRm_crPq6FRuJthq_?usp=sharing)|
|[Classify text with DistilBERT and Tensorflow](https://github.com/peterbayerle/huggingface_notebook/blob/main/distilbert_tf.ipynb) | How to fine-tune DistilBERT for text classification in TensorFlow | [Peter Bayerle](https://github.com/peterbayerle) | [!


key: _node_content
string_value:
{"id_": "31df94ea-8005-4005-a0d2-795359ec7851", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "98a81be9-b09f-4d62-b899-6ed78b4ebb8a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2d507b83-6251-46a2-a096-f50b676c038a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "55cb64c04f0c9143458b64aeaa0ac09c338dfc9e1301597619a59d98cf97f83a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "960b669f-1883-4a91-80b4-54b862db868b", "node_type": "1", "metadata": {}, "hash": "4d1b70bf401d7b8d1835e95f85b9c6c7c7c3a5c66cce3eefc058e29e42fc239c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12337, "end_char_idx": 14703, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NadirEM/nlp-notebooks/blob/master/Fine_tune_ALBERT_sentence_pair_classification.ipynb)|
|[Fine-tune Roberta for sentiment analysis](https://github.com/DhavalTaunk08/NLP_scripts/blob/master/sentiment_analysis_using_roberta.ipynb) | How to fine-tune a Roberta model for sentiment analysis | [Dhaval Taunk](https://github.com/DhavalTaunk08) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/DhavalTaunk08/NLP_scripts/blob/master/sentiment_analysis_using_roberta.ipynb)|
|[Evaluating Question Generation Models](https://github.com/flexudy-pipe/qugeev) | How accurate are the answers to questions generated by your seq2seq transformer model? | [Pascal Zoleko](https://github.com/zolekode) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1bpsSqCQU-iw_5nNoRm_crPq6FRuJthq_?usp=sharing)|
|[Classify text with DistilBERT and Tensorflow](https://github.com/peterbayerle/huggingface_notebook/blob/main/distilbert_tf.ipynb) | How to fine-tune DistilBERT for text classification in TensorFlow | [Peter Bayerle](https://github.com/peterbayerle) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/peterbayerle/huggingface_notebook/blob/main/distilbert_tf.ipynb)|
|[Leverage BERT for Encoder-Decoder Summarization on CNN/Dailymail](https://github.com/patrickvonplaten/notebooks/blob/master/BERT2BERT_for_CNN_Dailymail.ipynb) | How to warm-start a *EncoderDecoderModel* with a *google-bert/bert-base-uncased* checkpoint for summarization on CNN/Dailymail | [Patrick von Platen](https://github.com/patrickvonplaten) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/BERT2BERT_for_CNN_Dailymail.ipynb)|
|[Leverage RoBERTa for Encoder-Decoder Summarization on BBC XSum](https://github.com/patrickvonplaten/notebooks/blob/master/RoBERTaShared_for_BBC_XSum.ipynb) | How to warm-start a shared *EncoderDecoderModel* with a *FacebookAI/roberta-base* checkpoint for summarization on BBC/XSum | [Patrick von Platen](https://github.com/patrickvonplaten) | [!


key: _node_content
string_value:
{"id_": "960b669f-1883-4a91-80b4-54b862db868b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "98a81be9-b09f-4d62-b899-6ed78b4ebb8a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "31df94ea-8005-4005-a0d2-795359ec7851", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "0c3e7de539006203d02e2e493c671c0fc62eda424de1330a4453386a848b97be", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8e09dbce-ad13-4bde-a539-8a09637ef9db", "node_type": "1", "metadata": {}, "hash": "9d8ffb1ae73fff6d8f6498f4c12d81e2909193bda9968544980adea14993260b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13629, "end_char_idx": 15876, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/peterbayerle/huggingface_notebook/blob/main/distilbert_tf.ipynb)|
|[Leverage BERT for Encoder-Decoder Summarization on CNN/Dailymail](https://github.com/patrickvonplaten/notebooks/blob/master/BERT2BERT_for_CNN_Dailymail.ipynb) | How to warm-start a *EncoderDecoderModel* with a *google-bert/bert-base-uncased* checkpoint for summarization on CNN/Dailymail | [Patrick von Platen](https://github.com/patrickvonplaten) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/BERT2BERT_for_CNN_Dailymail.ipynb)|
|[Leverage RoBERTa for Encoder-Decoder Summarization on BBC XSum](https://github.com/patrickvonplaten/notebooks/blob/master/RoBERTaShared_for_BBC_XSum.ipynb) | How to warm-start a shared *EncoderDecoderModel* with a *FacebookAI/roberta-base* checkpoint for summarization on BBC/XSum | [Patrick von Platen](https://github.com/patrickvonplaten) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/RoBERTaShared_for_BBC_XSum.ipynb)|
|[Fine-tune TAPAS on Sequential Question Answering (SQA)](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb) | How to fine-tune *TapasForQuestionAnswering* with a *tapas-base* checkpoint on the Sequential Question Answering (SQA) dataset | [Niels Rogge](https://github.com/nielsrogge) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb)|
|[Evaluate TAPAS on Table Fact Checking (TabFact)](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/TAPAS/Evaluating_TAPAS_on_the_Tabfact_test_set.ipynb) | How to evaluate a fine-tuned *TapasForSequenceClassification* with a *tapas-base-finetuned-tabfact* checkpoint using a combination of the 🤗 datasets and 🤗 transformers libraries | [Niels Rogge](https://github.com/nielsrogge) | [!


key: _node_content
string_value:
{"id_": "8e09dbce-ad13-4bde-a539-8a09637ef9db", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "98a81be9-b09f-4d62-b899-6ed78b4ebb8a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "960b669f-1883-4a91-80b4-54b862db868b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "9147ae27054d6af462ef8b943d5525823b7388d22adb2a75db88f5c8d36c85c1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d7cca092-eb4a-4fce-aff6-06a31c6ce477", "node_type": "1", "metadata": {}, "hash": "e2ac6a16f6b7109db2f03ad1df0552e6370961d89eee34ed421acfcb26efbd4d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14703, "end_char_idx": 17436, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/RoBERTaShared_for_BBC_XSum.ipynb)|
|[Fine-tune TAPAS on Sequential Question Answering (SQA)](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb) | How to fine-tune *TapasForQuestionAnswering* with a *tapas-base* checkpoint on the Sequential Question Answering (SQA) dataset | [Niels Rogge](https://github.com/nielsrogge) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb)|
|[Evaluate TAPAS on Table Fact Checking (TabFact)](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/TAPAS/Evaluating_TAPAS_on_the_Tabfact_test_set.ipynb) | How to evaluate a fine-tuned *TapasForSequenceClassification* with a *tapas-base-finetuned-tabfact* checkpoint using a combination of the 🤗 datasets and 🤗 transformers libraries | [Niels Rogge](https://github.com/nielsrogge) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TAPAS/Evaluating_TAPAS_on_the_Tabfact_test_set.ipynb)|
|[Fine-tuning mBART for translation](https://colab.research.google.com/github/vasudevgupta7/huggingface-tutorials/blob/main/translation_training.ipynb) | How to fine-tune mBART using Seq2SeqTrainer for Hindi to English translation | [Vasudev Gupta](https://github.com/vasudevgupta7) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/vasudevgupta7/huggingface-tutorials/blob/main/translation_training.ipynb)|
|[Fine-tune LayoutLM on FUNSD (a form understanding dataset)](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LayoutLM/Fine_tuning_LayoutLMForTokenClassification_on_FUNSD.ipynb) | How to fine-tune *LayoutLMForTokenClassification* on the FUNSD dataset for information extraction from scanned documents | [Niels Rogge](https://github.com/nielsrogge) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLM/Fine_tuning_LayoutLMForTokenClassification_on_FUNSD.ipynb)|
|[Fine-Tune DistilGPT2 and Generate Text](https://colab.research.google.com/github/tripathiaakash/DistilGPT2-Tutorial/blob/main/distilgpt2_fine_tuning.ipynb) | How to fine-tune DistilGPT2 and generate text | [Aakash Tripathi](https://github.com/tripathiaakash) | [!


key: _node_content
string_value:
{"id_": "d7cca092-eb4a-4fce-aff6-06a31c6ce477", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "98a81be9-b09f-4d62-b899-6ed78b4ebb8a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8e09dbce-ad13-4bde-a539-8a09637ef9db", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "e7a4a68c89db29749dfa854d37ea870f0735b81b95ca660d59332132e94d5e4f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f722dc3e-1dbf-45d5-af8e-7fd623f74065", "node_type": "1", "metadata": {}, "hash": "5ad18b33ca524874f40ea2c8c82314066533866bf20837faf9555c311f9123ac", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16380, "end_char_idx": 18946, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/vasudevgupta7/huggingface-tutorials/blob/main/translation_training.ipynb)|
|[Fine-tune LayoutLM on FUNSD (a form understanding dataset)](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LayoutLM/Fine_tuning_LayoutLMForTokenClassification_on_FUNSD.ipynb) | How to fine-tune *LayoutLMForTokenClassification* on the FUNSD dataset for information extraction from scanned documents | [Niels Rogge](https://github.com/nielsrogge) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLM/Fine_tuning_LayoutLMForTokenClassification_on_FUNSD.ipynb)|
|[Fine-Tune DistilGPT2 and Generate Text](https://colab.research.google.com/github/tripathiaakash/DistilGPT2-Tutorial/blob/main/distilgpt2_fine_tuning.ipynb) | How to fine-tune DistilGPT2 and generate text | [Aakash Tripathi](https://github.com/tripathiaakash) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tripathiaakash/DistilGPT2-Tutorial/blob/main/distilgpt2_fine_tuning.ipynb)|
|[Fine-Tune LED on up to 8K tokens](https://github.com/patrickvonplaten/notebooks/blob/master/Fine_tune_Longformer_Encoder_Decoder_(LED)_for_Summarization_on_pubmed.ipynb) | How to fine-tune LED on pubmed for long-range summarization | [Patrick von Platen](https://github.com/patrickvonplaten) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/Fine_tune_Longformer_Encoder_Decoder_(LED)_for_Summarization_on_pubmed.ipynb)|
|[Evaluate LED on Arxiv](https://github.com/patrickvonplaten/notebooks/blob/master/LED_on_Arxiv.ipynb) | How to effectively evaluate LED on long-range summarization | [Patrick von Platen](https://github.com/patrickvonplaten) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/LED_on_Arxiv.ipynb)|
|[Fine-tune LayoutLM on RVL-CDIP (a document image classification dataset)](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LayoutLM/Fine_tuning_LayoutLMForSequenceClassification_on_RVL_CDIP.ipynb) | How to fine-tune *LayoutLMForSequenceClassification* on the RVL-CDIP dataset for scanned document classification | [Niels Rogge](https://github.com/nielsrogge) | [!


key: _node_content
string_value:
{"id_": "f722dc3e-1dbf-45d5-af8e-7fd623f74065", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "98a81be9-b09f-4d62-b899-6ed78b4ebb8a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d7cca092-eb4a-4fce-aff6-06a31c6ce477", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "241de393eaa45675b35b5331e7af6d912404ddc32a7343e02f582412f5989df5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "71d9d6b5-6ce2-49fa-9764-50dfb7ce1c8a", "node_type": "1", "metadata": {}, "hash": "efbb1bef0f0869670a2cc0b02a62fbbf397b2293d8f75852a29d8bd8fd617634", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17926, "end_char_idx": 20347, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/Fine_tune_Longformer_Encoder_Decoder_(LED)_for_Summarization_on_pubmed.ipynb)|
|[Evaluate LED on Arxiv](https://github.com/patrickvonplaten/notebooks/blob/master/LED_on_Arxiv.ipynb) | How to effectively evaluate LED on long-range summarization | [Patrick von Platen](https://github.com/patrickvonplaten) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/LED_on_Arxiv.ipynb)|
|[Fine-tune LayoutLM on RVL-CDIP (a document image classification dataset)](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LayoutLM/Fine_tuning_LayoutLMForSequenceClassification_on_RVL_CDIP.ipynb) | How to fine-tune *LayoutLMForSequenceClassification* on the RVL-CDIP dataset for scanned document classification | [Niels Rogge](https://github.com/nielsrogge) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLM/Fine_tuning_LayoutLMForSequenceClassification_on_RVL_CDIP.ipynb)|
|[Wav2Vec2 CTC decoding with GPT2 adjustment](https://github.com/voidful/huggingface_notebook/blob/main/xlsr_gpt.ipynb) | How to decode CTC sequence with language model adjustment | [Eric Lam](https://github.com/voidful) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1e_z5jQHYbO2YKEaUgzb1ww1WwiAyydAj?usp=sharing)|
|[Fine-tune BART for summarization in two languages with Trainer class](https://github.com/elsanns/xai-nlp-notebooks/blob/master/fine_tune_bart_summarization_two_langs.ipynb) | How to fine-tune BART for summarization in two languages with Trainer class | [Eliza Szczechla](https://github.com/elsanns) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/elsanns/xai-nlp-notebooks/blob/master/fine_tune_bart_summarization_two_langs.ipynb)|
|[Evaluate Big Bird on Trivia QA](https://github.com/patrickvonplaten/notebooks/blob/master/Evaluating_Big_Bird_on_TriviaQA.ipynb) | How to evaluate BigBird on long document question answering on Trivia QA | [Patrick von Platen](https://github.com/patrickvonplaten) | [!


key: _node_content
string_value:
{"id_": "71d9d6b5-6ce2-49fa-9764-50dfb7ce1c8a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "98a81be9-b09f-4d62-b899-6ed78b4ebb8a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f722dc3e-1dbf-45d5-af8e-7fd623f74065", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "af040384843084370ddeebeeb60e708a758ad5b169cffa2b7a7bbf15977d7577", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a4cafa83-47d7-45ba-8055-c6444c3a2421", "node_type": "1", "metadata": {}, "hash": "c334f9b6cc26964de64e7a5852add96f0ec3bb6db3ac6237a1751b9b85d77e9d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19408, "end_char_idx": 21397, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1e_z5jQHYbO2YKEaUgzb1ww1WwiAyydAj?usp=sharing)|
|[Fine-tune BART for summarization in two languages with Trainer class](https://github.com/elsanns/xai-nlp-notebooks/blob/master/fine_tune_bart_summarization_two_langs.ipynb) | How to fine-tune BART for summarization in two languages with Trainer class | [Eliza Szczechla](https://github.com/elsanns) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/elsanns/xai-nlp-notebooks/blob/master/fine_tune_bart_summarization_two_langs.ipynb)|
|[Evaluate Big Bird on Trivia QA](https://github.com/patrickvonplaten/notebooks/blob/master/Evaluating_Big_Bird_on_TriviaQA.ipynb) | How to evaluate BigBird on long document question answering on Trivia QA | [Patrick von Platen](https://github.com/patrickvonplaten) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/Evaluating_Big_Bird_on_TriviaQA.ipynb)|
| [Create video captions using Wav2Vec2](https://github.com/Muennighoff/ytclipcc/blob/main/wav2vec_youtube_captions.ipynb) | How to create YouTube captions from any video by transcribing the audio with Wav2Vec | [Niklas Muennighoff](https://github.com/Muennighoff) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Muennighoff/ytclipcc/blob/main/wav2vec_youtube_captions.ipynb) |
| [Fine-tune the Vision Transformer on CIFAR-10 using PyTorch Lightning](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/VisionTransformer/Fine_tuning_the_Vision_Transformer_on_CIFAR_10_with_PyTorch_Lightning.ipynb) | How to fine-tune the Vision Transformer (ViT) on CIFAR-10 using HuggingFace Transformers, Datasets and PyTorch Lightning | [Niels Rogge](https://github.com/nielsrogge) |[!


key: _node_content
string_value:
{"id_": "a4cafa83-47d7-45ba-8055-c6444c3a2421", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "98a81be9-b09f-4d62-b899-6ed78b4ebb8a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "71d9d6b5-6ce2-49fa-9764-50dfb7ce1c8a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "5f5ab0b4cb8a4d6c3a629d2222f5b8bf136fe8b7c3ac25d5f7b2a33acfbe7997", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a4f746dc-c689-441c-8fe0-7e5953ae3173", "node_type": "1", "metadata": {}, "hash": "b49382e8ec3c496a6546d7ce7234f59ab3ed0a0d2e6de36c71c6c98eff613089", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20347, "end_char_idx": 22601, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/Evaluating_Big_Bird_on_TriviaQA.ipynb)|
| [Create video captions using Wav2Vec2](https://github.com/Muennighoff/ytclipcc/blob/main/wav2vec_youtube_captions.ipynb) | How to create YouTube captions from any video by transcribing the audio with Wav2Vec | [Niklas Muennighoff](https://github.com/Muennighoff) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Muennighoff/ytclipcc/blob/main/wav2vec_youtube_captions.ipynb) |
| [Fine-tune the Vision Transformer on CIFAR-10 using PyTorch Lightning](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/VisionTransformer/Fine_tuning_the_Vision_Transformer_on_CIFAR_10_with_PyTorch_Lightning.ipynb) | How to fine-tune the Vision Transformer (ViT) on CIFAR-10 using HuggingFace Transformers, Datasets and PyTorch Lightning | [Niels Rogge](https://github.com/nielsrogge) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/VisionTransformer/Fine_tuning_the_Vision_Transformer_on_CIFAR_10_with_PyTorch_Lightning.ipynb) |
| [Fine-tune the Vision Transformer on CIFAR-10 using the 🤗 Trainer](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/VisionTransformer/Fine_tuning_the_Vision_Transformer_on_CIFAR_10_with_the_%F0%9F%A4%97_Trainer.ipynb) | How to fine-tune the Vision Transformer (ViT) on CIFAR-10 using HuggingFace Transformers, Datasets and the 🤗 Trainer | [Niels Rogge](https://github.com/nielsrogge) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/VisionTransformer/Fine_tuning_the_Vision_Transformer_on_CIFAR_10_with_the_%F0%9F%A4%97_Trainer.ipynb) |
| [Evaluate LUKE on Open Entity, an entity typing dataset](https://github.com/studio-ousia/luke/blob/master/notebooks/huggingface_open_entity.ipynb) | How to evaluate *LukeForEntityClassification* on the Open Entity dataset | [Ikuya Yamada](https://github.com/ikuyamada) |[!


key: _node_content
string_value:
{"id_": "a4f746dc-c689-441c-8fe0-7e5953ae3173", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "98a81be9-b09f-4d62-b899-6ed78b4ebb8a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a4cafa83-47d7-45ba-8055-c6444c3a2421", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "e6adedf0de056165e868a1ea43a9cee769668d353388004a864f0c9c1efa08af", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5a4dd43d-102f-40a4-a999-dff5f5986ce1", "node_type": "1", "metadata": {}, "hash": "d354221b9e7c8ee88d8535fdd44dc929672beac5dfa0df074ebd02aa949684f1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21397, "end_char_idx": 23982, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/VisionTransformer/Fine_tuning_the_Vision_Transformer_on_CIFAR_10_with_PyTorch_Lightning.ipynb) |
| [Fine-tune the Vision Transformer on CIFAR-10 using the 🤗 Trainer](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/VisionTransformer/Fine_tuning_the_Vision_Transformer_on_CIFAR_10_with_the_%F0%9F%A4%97_Trainer.ipynb) | How to fine-tune the Vision Transformer (ViT) on CIFAR-10 using HuggingFace Transformers, Datasets and the 🤗 Trainer | [Niels Rogge](https://github.com/nielsrogge) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/VisionTransformer/Fine_tuning_the_Vision_Transformer_on_CIFAR_10_with_the_%F0%9F%A4%97_Trainer.ipynb) |
| [Evaluate LUKE on Open Entity, an entity typing dataset](https://github.com/studio-ousia/luke/blob/master/notebooks/huggingface_open_entity.ipynb) | How to evaluate *LukeForEntityClassification* on the Open Entity dataset | [Ikuya Yamada](https://github.com/ikuyamada) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/studio-ousia/luke/blob/master/notebooks/huggingface_open_entity.ipynb) |
| [Evaluate LUKE on TACRED, a relation extraction dataset](https://github.com/studio-ousia/luke/blob/master/notebooks/huggingface_tacred.ipynb) | How to evaluate *LukeForEntityPairClassification* on the TACRED dataset | [Ikuya Yamada](https://github.com/ikuyamada) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/studio-ousia/luke/blob/master/notebooks/huggingface_tacred.ipynb) |
| [Evaluate LUKE on CoNLL-2003, an important NER benchmark](https://github.com/studio-ousia/luke/blob/master/notebooks/huggingface_conll_2003.ipynb) | How to evaluate *LukeForEntitySpanClassification* on the CoNLL-2003 dataset | [Ikuya Yamada](https://github.com/ikuyamada) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/studio-ousia/luke/blob/master/notebooks/huggingface_conll_2003.ipynb) |
| [Evaluate BigBird-Pegasus on PubMed dataset](https://github.com/vasudevgupta7/bigbird/blob/main/notebooks/bigbird_pegasus_evaluation.ipynb) | How to evaluate *BigBirdPegasusForConditionalGeneration* on PubMed dataset | [Vasudev Gupta](https://github.com/vasudevgupta7) | [!


key: _node_content
string_value:
{"id_": "5a4dd43d-102f-40a4-a999-dff5f5986ce1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "98a81be9-b09f-4d62-b899-6ed78b4ebb8a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a4f746dc-c689-441c-8fe0-7e5953ae3173", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "9930f64b672dee2107d148977b687f8e97b83fbf0abd7f6055a90a4bb1af667b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c4da2d45-1416-4c03-8cce-5e375b8584ab", "node_type": "1", "metadata": {}, "hash": "f7d658bf782333a73641622c708d7a5f434e7253ee90567e028039a46c0ffe1f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23058, "end_char_idx": 25572, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/studio-ousia/luke/blob/master/notebooks/huggingface_tacred.ipynb) |
| [Evaluate LUKE on CoNLL-2003, an important NER benchmark](https://github.com/studio-ousia/luke/blob/master/notebooks/huggingface_conll_2003.ipynb) | How to evaluate *LukeForEntitySpanClassification* on the CoNLL-2003 dataset | [Ikuya Yamada](https://github.com/ikuyamada) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/studio-ousia/luke/blob/master/notebooks/huggingface_conll_2003.ipynb) |
| [Evaluate BigBird-Pegasus on PubMed dataset](https://github.com/vasudevgupta7/bigbird/blob/main/notebooks/bigbird_pegasus_evaluation.ipynb) | How to evaluate *BigBirdPegasusForConditionalGeneration* on PubMed dataset | [Vasudev Gupta](https://github.com/vasudevgupta7) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/vasudevgupta7/bigbird/blob/main/notebooks/bigbird_pegasus_evaluation.ipynb) |
| [Speech Emotion Classification with Wav2Vec2](https://github/m3hrdadfi/soxan/blob/main/notebooks/Emotion_recognition_in_Greek_speech_using_Wav2Vec2.ipynb) | How to leverage a pretrained Wav2Vec2 model for Emotion Classification on the MEGA dataset | [Mehrdad Farahani](https://github.com/m3hrdadfi) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/m3hrdadfi/soxan/blob/main/notebooks/Emotion_recognition_in_Greek_speech_using_Wav2Vec2.ipynb) |
| [Detect objects in an image with DETR](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/DETR/DETR_minimal_example_(with_DetrFeatureExtractor).ipynb) | How to use a trained *DetrForObjectDetection* model to detect objects in an image and visualize attention | [Niels Rogge](https://github.com/NielsRogge) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/DETR/DETR_minimal_example_(with_DetrFeatureExtractor).ipynb) |
| [Fine-tune DETR on a custom object detection dataset](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/DETR/Fine_tuning_DetrForObjectDetection_on_custom_dataset_(balloon).ipynb) | How to fine-tune *DetrForObjectDetection* on a custom object detection dataset | [Niels Rogge](https://github.com/NielsRogge) | [!


key: _node_content
string_value:
{"id_": "c4da2d45-1416-4c03-8cce-5e375b8584ab", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "98a81be9-b09f-4d62-b899-6ed78b4ebb8a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5a4dd43d-102f-40a4-a999-dff5f5986ce1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "042ce2a793b648ceb4dd9b88416732f08900ec361f6032c5d461bced7f02cb8f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24481, "end_char_idx": 26213, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/m3hrdadfi/soxan/blob/main/notebooks/Emotion_recognition_in_Greek_speech_using_Wav2Vec2.ipynb) |
| [Detect objects in an image with DETR](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/DETR/DETR_minimal_example_(with_DetrFeatureExtractor).ipynb) | How to use a trained *DetrForObjectDetection* model to detect objects in an image and visualize attention | [Niels Rogge](https://github.com/NielsRogge) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/DETR/DETR_minimal_example_(with_DetrFeatureExtractor).ipynb) |
| [Fine-tune DETR on a custom object detection dataset](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/DETR/Fine_tuning_DetrForObjectDetection_on_custom_dataset_(balloon).ipynb) | How to fine-tune *DetrForObjectDetection* on a custom object detection dataset | [Niels Rogge](https://github.com/NielsRogge) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/DETR/Fine_tuning_DetrForObjectDetection_on_custom_dataset_(balloon).ipynb) |
| [Finetune T5 for Named Entity Recognition](https://github.com/ToluClassics/Notebooks/blob/main/T5_Ner_Finetuning.ipynb) | How to fine-tune *T5* on a Named Entity Recognition Task | [Ogundepo Odunayo](https://github.com/ToluClassics) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1obr78FY_cBmWY5ODViCmzdY6O1KB65Vc?usp=sharing) |


key: _node_content
string_value:
{"id_": "cd44dd36-5357-4c6c-b068-701da95e390b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/troubleshooting", "title": "Troubleshoot", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f574da76-3507-4737-9ba1-f03f993b1b06", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/troubleshooting", "title": "Troubleshoot", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "cf7ceb63fcd69053af31c31e769d7cbe115106f05438ffa92b3abb285b66d3aa", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3250db8f-5e23-496e-83d8-dff8a2446b21", "node_type": "1", "metadata": {}, "hash": "b3dd10e4f76c6551c63850c81e5b0af2754fd19389c8c589fd05bb452412f72f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3294, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<!---
Copyright 2022 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.

-->

# Troubleshoot

Sometimes errors occur, but we are here to help! This guide covers some of the most common issues we've seen and how you can resolve them. However, this guide isn't meant to be a comprehensive collection of every 🤗 Transformers issue. For more help with troubleshooting your issue, try:

<Youtube id="S2EEG3JIt2A"/>

1. Asking for help on the [forums](https://discuss.huggingface.co/). There are specific categories you can post your question to, like [Beginners](https://discuss.huggingface.co/c/beginners/5) or [🤗 Transformers](https://discuss.huggingface.co/c/transformers/9). Make sure you write a good descriptive forum post with some reproducible code to maximize the likelihood that your problem is solved!

<Youtube id="_PAli-V4wj0"/>

2. Create an [Issue](https://github.com/huggingface/transformers/issues/new/choose) on the 🤗 Transformers repository if it is a bug related to the library. Try to include as much information describing the bug as possible to help us better figure out what's wrong and how we can fix it.

3. Check the [Migration](migration) guide if you use an older version of 🤗 Transformers since some important changes have been introduced between versions.

For more details about troubleshooting and getting help, take a look at [Chapter 8](https://huggingface.co/course/chapter8/1?fw=pt) of the Hugging Face course.


## Firewalled environments

Some GPU instances on cloud and intranet setups are firewalled to external connections, resulting in a connection error. When your script attempts to download model weights or datasets, the download will hang and then timeout with the following message:

```
ValueError: Connection error, and we cannot find the requested files in the cached path.
Please try again or make sure your Internet connection is on.
```

In this case, you should try to run 🤗 Transformers on [offline mode](installation#offline-mode) to avoid the connection error.

## CUDA out of memory

Training large models with millions of parameters can be challenging without the appropriate hardware. A common error you may encounter when the GPU runs out of memory is:

```
CUDA out of memory. Tried to allocate 256.00 MiB (GPU 0; 11.17 GiB total capacity; 9.70 GiB already allocated; 179.81 MiB free; 9.85 GiB reserved in total by PyTorch)
```

Here are some potential solutions you can try to lessen memory use:

- Reduce the [`per_device_train_batch_size`](main_classes/trainer#transformers.TrainingArguments.per_device_train_batch_size) value in [`TrainingArguments`].


key: _node_content
string_value:
{"id_": "3250db8f-5e23-496e-83d8-dff8a2446b21", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/troubleshooting", "title": "Troubleshoot", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f574da76-3507-4737-9ba1-f03f993b1b06", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/troubleshooting", "title": "Troubleshoot", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "cf7ceb63fcd69053af31c31e769d7cbe115106f05438ffa92b3abb285b66d3aa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cd44dd36-5357-4c6c-b068-701da95e390b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/troubleshooting", "title": "Troubleshoot", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "1da95ab0384de49a8832d67734cb621930471eb74752c7e9d7a3602fb0a7191a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d90aa1b9-f528-4e93-9b9c-bcbc311035cb", "node_type": "1", "metadata": {}, "hash": "568b298d42e80db4371c2dcff4595575dee08541b8363fab802dafe31b46e4f9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2125, "end_char_idx": 5499, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Firewalled environments

Some GPU instances on cloud and intranet setups are firewalled to external connections, resulting in a connection error. When your script attempts to download model weights or datasets, the download will hang and then timeout with the following message:

```
ValueError: Connection error, and we cannot find the requested files in the cached path.
Please try again or make sure your Internet connection is on.
```

In this case, you should try to run 🤗 Transformers on [offline mode](installation#offline-mode) to avoid the connection error.

## CUDA out of memory

Training large models with millions of parameters can be challenging without the appropriate hardware. A common error you may encounter when the GPU runs out of memory is:

```
CUDA out of memory. Tried to allocate 256.00 MiB (GPU 0; 11.17 GiB total capacity; 9.70 GiB already allocated; 179.81 MiB free; 9.85 GiB reserved in total by PyTorch)
```

Here are some potential solutions you can try to lessen memory use:

- Reduce the [`per_device_train_batch_size`](main_classes/trainer#transformers.TrainingArguments.per_device_train_batch_size) value in [`TrainingArguments`].
- Try using [`gradient_accumulation_steps`](main_classes/trainer#transformers.TrainingArguments.gradient_accumulation_steps) in [`TrainingArguments`] to effectively increase overall batch size.

<Tip>

Refer to the Performance [guide](performance) for more details about memory-saving techniques.

</Tip>

## Unable to load a saved TensorFlow model

TensorFlow's [model.save](https://www.tensorflow.org/tutorials/keras/save_and_load#save_the_entire_model) method will save the entire model - architecture, weights, training configuration - in a single file. However, when you load the model file again, you may run into an error because 🤗 Transformers may not load all the TensorFlow-related objects in the model file. To avoid issues with saving and loading TensorFlow models, we recommend you:

- Save the model weights as a `h5` file extension with [`model.save_weights`](https://www.tensorflow.org/tutorials/keras/save_and_load#save_the_entire_model) and then reload the model with [`~TFPreTrainedModel.from_pretrained`]:

```py
>>> from transformers import TFPreTrainedModel
>>> from tensorflow import keras

>>> model.save_weights("some_folder/tf_model.h5")
>>> model = TFPreTrainedModel.from_pretrained("some_folder")
```

- Save the model with [`~TFPretrainedModel.save_pretrained`] and load it again with [`~TFPreTrainedModel.from_pretrained`]:

```py
>>> from transformers import TFPreTrainedModel

>>> model.save_pretrained("path_to/model")
>>> model = TFPreTrainedModel.from_pretrained("path_to/model")
```

## ImportError

Another common error you may encounter, especially if it is a newly released model, is `ImportError`:

```
ImportError: cannot import name 'ImageGPTImageProcessor' from 'transformers' (unknown location)
```

For these error types, check to make sure you have the latest version of 🤗 Transformers installed to access the most recent models:

```bash
pip install transformers --upgrade
```

## CUDA error: device-side assert triggered

Sometimes you may run into a generic CUDA error about an error in the device code.

```
RuntimeError: CUDA error: device-side assert triggered
```

You should try to run the code on a CPU first to get a more descriptive error message.


key: _node_content
string_value:
{"id_": "d90aa1b9-f528-4e93-9b9c-bcbc311035cb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/troubleshooting", "title": "Troubleshoot", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f574da76-3507-4737-9ba1-f03f993b1b06", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/troubleshooting", "title": "Troubleshoot", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "cf7ceb63fcd69053af31c31e769d7cbe115106f05438ffa92b3abb285b66d3aa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3250db8f-5e23-496e-83d8-dff8a2446b21", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/troubleshooting", "title": "Troubleshoot", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "7e813b2db2bdca09ca22968e591bdff8ebc326d78dfdcae1f448e72b12dbf2e7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6854394c-ac5f-4f21-940d-feca2c3310ac", "node_type": "1", "metadata": {}, "hash": "d092617d771890efe17cf01a85b00b48d02a30f3f9a2f5934b0f548cb950fac5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3853, "end_char_idx": 6260, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, when you load the model file again, you may run into an error because 🤗 Transformers may not load all the TensorFlow-related objects in the model file. To avoid issues with saving and loading TensorFlow models, we recommend you:

- Save the model weights as a `h5` file extension with [`model.save_weights`](https://www.tensorflow.org/tutorials/keras/save_and_load#save_the_entire_model) and then reload the model with [`~TFPreTrainedModel.from_pretrained`]:

```py
>>> from transformers import TFPreTrainedModel
>>> from tensorflow import keras

>>> model.save_weights("some_folder/tf_model.h5")
>>> model = TFPreTrainedModel.from_pretrained("some_folder")
```

- Save the model with [`~TFPretrainedModel.save_pretrained`] and load it again with [`~TFPreTrainedModel.from_pretrained`]:

```py
>>> from transformers import TFPreTrainedModel

>>> model.save_pretrained("path_to/model")
>>> model = TFPreTrainedModel.from_pretrained("path_to/model")
```

## ImportError

Another common error you may encounter, especially if it is a newly released model, is `ImportError`:

```
ImportError: cannot import name 'ImageGPTImageProcessor' from 'transformers' (unknown location)
```

For these error types, check to make sure you have the latest version of 🤗 Transformers installed to access the most recent models:

```bash
pip install transformers --upgrade
```

## CUDA error: device-side assert triggered

Sometimes you may run into a generic CUDA error about an error in the device code.

```
RuntimeError: CUDA error: device-side assert triggered
```

You should try to run the code on a CPU first to get a more descriptive error message. Add the following environment variable to the beginning of your code to switch to a CPU:

```py
>>> import os

>>> os.environ["CUDA_VISIBLE_DEVICES"] = ""
```

Another option is to get a better traceback from the GPU. Add the following environment variable to the beginning of your code to get the traceback to point to the source of the error:

```py
>>> import os

>>> os.environ["CUDA_LAUNCH_BLOCKING"] = "1"
```

## Incorrect output when padding tokens aren't masked

In some cases, the output `hidden_state` may be incorrect if the `input_ids` include padding tokens. To demonstrate, load a model and tokenizer. You can access a model's `pad_token_id` to see its value. The `pad_token_id` may be `None` for some models, but you can always manually set it.


key: _node_content
string_value:
{"id_": "6854394c-ac5f-4f21-940d-feca2c3310ac", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/troubleshooting", "title": "Troubleshoot", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f574da76-3507-4737-9ba1-f03f993b1b06", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/troubleshooting", "title": "Troubleshoot", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "cf7ceb63fcd69053af31c31e769d7cbe115106f05438ffa92b3abb285b66d3aa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d90aa1b9-f528-4e93-9b9c-bcbc311035cb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/troubleshooting", "title": "Troubleshoot", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "d949a44b8448370e7fe0b8f1778455d9ac4a2c1fa78b4debd4098eaddb920fdc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9c91336f-b6ba-4a00-b1ca-3a9c9ffc7c6b", "node_type": "1", "metadata": {}, "hash": "600f469dd4f389d108283ff543bffeac5b0baa6e042a33481ac5febbdedadcaf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5349, "end_char_idx": 8395, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```
RuntimeError: CUDA error: device-side assert triggered
```

You should try to run the code on a CPU first to get a more descriptive error message. Add the following environment variable to the beginning of your code to switch to a CPU:

```py
>>> import os

>>> os.environ["CUDA_VISIBLE_DEVICES"] = ""
```

Another option is to get a better traceback from the GPU. Add the following environment variable to the beginning of your code to get the traceback to point to the source of the error:

```py
>>> import os

>>> os.environ["CUDA_LAUNCH_BLOCKING"] = "1"
```

## Incorrect output when padding tokens aren't masked

In some cases, the output `hidden_state` may be incorrect if the `input_ids` include padding tokens. To demonstrate, load a model and tokenizer. You can access a model's `pad_token_id` to see its value. The `pad_token_id` may be `None` for some models, but you can always manually set it.

```py
>>> from transformers import AutoModelForSequenceClassification
>>> import torch

>>> model = AutoModelForSequenceClassification.from_pretrained("google-bert/bert-base-uncased")
>>> model.config.pad_token_id
0
```

The following example shows the output without masking the padding tokens:

```py
>>> input_ids = torch.tensor([[7592, 2057, 2097, 2393, 9611, 2115], [7592, 0, 0, 0, 0, 0]])
>>> output = model(input_ids)
>>> print(output.logits)
tensor([[ 0.0082, -0.2307],
        [ 0.1317, -0.1683]], grad_fn=<AddmmBackward0>)
```

Here is the actual output of the second sequence:

```py
>>> input_ids = torch.tensor([[7592]])
>>> output = model(input_ids)
>>> print(output.logits)
tensor([[-0.1008, -0.4061]], grad_fn=<AddmmBackward0>)
```

Most of the time, you should provide an `attention_mask` to your model to ignore the padding tokens to avoid this silent error. Now the output of the second sequence matches its actual output:

<Tip>

By default, the tokenizer creates an `attention_mask` for you based on your specific tokenizer's defaults.

</Tip>

```py
>>> attention_mask = torch.tensor([[1, 1, 1, 1, 1, 1], [1, 0, 0, 0, 0, 0]])
>>> output = model(input_ids, attention_mask=attention_mask)
>>> print(output.logits)
tensor([[ 0.0082, -0.2307],
        [-0.1008, -0.4061]], grad_fn=<AddmmBackward0>)
```

🤗 Transformers doesn't automatically create an `attention_mask` to mask a padding token if it is provided because:

- Some models don't have a padding token.
- For some use-cases, users want a model to attend to a padding token.

## ValueError: Unrecognized configuration class XYZ for this kind of AutoModel

Generally, we recommend using the [`AutoModel`] class to load pretrained instances of models. This class
can automatically infer and load the correct architecture from a given checkpoint based on the configuration. If you see
this `ValueError` when loading a model from a checkpoint, this means the Auto class couldn't find a mapping from
the configuration in the given checkpoint to the kind of model you are trying to load. Most commonly, this happens when a
checkpoint doesn't support a given task.


key: _node_content
string_value:
{"id_": "9c91336f-b6ba-4a00-b1ca-3a9c9ffc7c6b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/troubleshooting", "title": "Troubleshoot", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f574da76-3507-4737-9ba1-f03f993b1b06", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/troubleshooting", "title": "Troubleshoot", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "cf7ceb63fcd69053af31c31e769d7cbe115106f05438ffa92b3abb285b66d3aa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6854394c-ac5f-4f21-940d-feca2c3310ac", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/troubleshooting", "title": "Troubleshoot", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "597354f279b4e601ba3008b16676ceaa7fb27c8b3aaaac36042bd6b73758807b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7139, "end_char_idx": 9048, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Now the output of the second sequence matches its actual output:

<Tip>

By default, the tokenizer creates an `attention_mask` for you based on your specific tokenizer's defaults.

</Tip>

```py
>>> attention_mask = torch.tensor([[1, 1, 1, 1, 1, 1], [1, 0, 0, 0, 0, 0]])
>>> output = model(input_ids, attention_mask=attention_mask)
>>> print(output.logits)
tensor([[ 0.0082, -0.2307],
        [-0.1008, -0.4061]], grad_fn=<AddmmBackward0>)
```

🤗 Transformers doesn't automatically create an `attention_mask` to mask a padding token if it is provided because:

- Some models don't have a padding token.
- For some use-cases, users want a model to attend to a padding token.

## ValueError: Unrecognized configuration class XYZ for this kind of AutoModel

Generally, we recommend using the [`AutoModel`] class to load pretrained instances of models. This class
can automatically infer and load the correct architecture from a given checkpoint based on the configuration. If you see
this `ValueError` when loading a model from a checkpoint, this means the Auto class couldn't find a mapping from
the configuration in the given checkpoint to the kind of model you are trying to load. Most commonly, this happens when a
checkpoint doesn't support a given task.
For instance, you'll see this error in the following example because there is no GPT2 for question answering:

```py
>>> from transformers import AutoProcessor, AutoModelForQuestionAnswering

>>> processor = AutoProcessor.from_pretrained("openai-community/gpt2-medium")
>>> model = AutoModelForQuestionAnswering.from_pretrained("openai-community/gpt2-medium")
ValueError: Unrecognized configuration class <class 'transformers.models.gpt2.configuration_gpt2.GPT2Config'> for this kind of AutoModel: AutoModelForQuestionAnswering.
Model type should be one of AlbertConfig, BartConfig, BertConfig, BigBirdConfig, BigBirdPegasusConfig, BloomConfig, ...
```


key: _node_content
string_value:
{"id_": "1a309e02-c931-4334-9524-4f525ba05aab", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fa46a645-8e6b-465b-ae69-3d51f650c846", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "hash": "b97638da20ee8948778fc11261763dd7a2666a79a5866a41f99ad4299f694ae6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5836f719-6e02-4e19-8893-47bbef5b5e90", "node_type": "1", "metadata": {}, "hash": "931116e3deacf06a9e2f17d13ea7291f63b37d84bf62f1bea4a600e10f1d1523", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2694, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Export to ONNX

Deploying 🤗 Transformers models in production environments often requires, or can benefit from exporting the models into 
a serialized format that can be loaded and executed on specialized runtimes and hardware.

🤗 Optimum is an extension of Transformers that enables exporting models from PyTorch or TensorFlow to serialized formats 
such as ONNX and TFLite through its `exporters` module. 🤗 Optimum also provides a set of performance optimization tools to train 
and run models on targeted hardware with maximum efficiency.

This guide demonstrates how you can export 🤗 Transformers models to ONNX with 🤗 Optimum, for the guide on exporting models to TFLite, 
please refer to the [Export to TFLite page](tflite).

## Export to ONNX 

[ONNX (Open Neural Network eXchange)](http://onnx.ai) is an open standard that defines a common set of operators and a 
common file format to represent deep learning models in a wide variety of frameworks, including PyTorch and
TensorFlow. When a model is exported to the ONNX format, these operators are used to
construct a computational graph (often called an _intermediate representation_) which
represents the flow of data through the neural network.

By exposing a graph with standardized operators and data types, ONNX makes it easy to
switch between frameworks. For example, a model trained in PyTorch can be exported to
ONNX format and then imported in TensorFlow (and vice versa).

Once exported to ONNX format, a model can be:
- optimized for inference via techniques such as [graph optimization](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) and [quantization](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization). 
- run with ONNX Runtime via [`ORTModelForXXX` classes](https://huggingface.co/docs/optimum/onnxruntime/package_reference/modeling_ort),
which follow the same `AutoModel` API as the one you are used to in 🤗 Transformers.
- run with [optimized inference pipelines](https://huggingface.co/docs/optimum/main/en/onnxruntime/usage_guides/pipelines),
which has the same API as the [`pipeline`] function in 🤗 Transformers. 

🤗 Optimum provides support for the ONNX export by leveraging configuration objects. These configuration objects come 
ready-made for a number of model architectures, and are designed to be easily extendable to other architectures.

For the list of ready-made configurations, please refer to [🤗 Optimum documentation](https://huggingface.co/docs/optimum/exporters/onnx/overview).

There are two ways to export a 🤗 Transformers model to ONNX, here we show both:

- export with 🤗 Optimum via CLI.
- export with 🤗 Optimum with `optimum.onnxruntime`.


key: _node_content
string_value:
{"id_": "5836f719-6e02-4e19-8893-47bbef5b5e90", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fa46a645-8e6b-465b-ae69-3d51f650c846", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "hash": "b97638da20ee8948778fc11261763dd7a2666a79a5866a41f99ad4299f694ae6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1a309e02-c931-4334-9524-4f525ba05aab", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "hash": "4a53a6ddf954af28b05288e06c33c5f1c0f3437a8886a02e96d15c3f5fd3ba26", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "80d8b397-2a54-4673-a046-c84516e2c1c3", "node_type": "1", "metadata": {}, "hash": "345ffc7a2848e26b8c9f520076bb4fc57d3fd19f076f6ca2a0b39a6903913f41", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 994, "end_char_idx": 4083, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
When a model is exported to the ONNX format, these operators are used to
construct a computational graph (often called an _intermediate representation_) which
represents the flow of data through the neural network.

By exposing a graph with standardized operators and data types, ONNX makes it easy to
switch between frameworks. For example, a model trained in PyTorch can be exported to
ONNX format and then imported in TensorFlow (and vice versa).

Once exported to ONNX format, a model can be:
- optimized for inference via techniques such as [graph optimization](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) and [quantization](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization). 
- run with ONNX Runtime via [`ORTModelForXXX` classes](https://huggingface.co/docs/optimum/onnxruntime/package_reference/modeling_ort),
which follow the same `AutoModel` API as the one you are used to in 🤗 Transformers.
- run with [optimized inference pipelines](https://huggingface.co/docs/optimum/main/en/onnxruntime/usage_guides/pipelines),
which has the same API as the [`pipeline`] function in 🤗 Transformers. 

🤗 Optimum provides support for the ONNX export by leveraging configuration objects. These configuration objects come 
ready-made for a number of model architectures, and are designed to be easily extendable to other architectures.

For the list of ready-made configurations, please refer to [🤗 Optimum documentation](https://huggingface.co/docs/optimum/exporters/onnx/overview).

There are two ways to export a 🤗 Transformers model to ONNX, here we show both:

- export with 🤗 Optimum via CLI.
- export with 🤗 Optimum with `optimum.onnxruntime`.

### Exporting a 🤗 Transformers model to ONNX with CLI

To export a 🤗 Transformers model to ONNX, first install an extra dependency:

```bash
pip install optimum[exporters]
```

To check out all available arguments, refer to the [🤗 Optimum docs](https://huggingface.co/docs/optimum/exporters/onnx/usage_guides/export_a_model#exporting-a-model-to-onnx-using-the-cli), 
or view help in command line:

```bash
optimum-cli export onnx --help
```

To export a model's checkpoint from the 🤗 Hub, for example, `distilbert/distilbert-base-uncased-distilled-squad`, run the following command: 

```bash
optimum-cli export onnx --model distilbert/distilbert-base-uncased-distilled-squad distilbert_base_uncased_squad_onnx/
```

You should see the logs indicating progress and showing where the resulting `model.onnx` is saved, like this:

```bash
Validating ONNX model distilbert_base_uncased_squad_onnx/model.onnx...
	-[✓] ONNX model output names match reference model (start_logits, end_logits)
	- Validating ONNX Model output "start_logits":
		-[✓] (2, 16) matches (2, 16)
		-[✓] all values close (atol: 0.0001)
	- Validating ONNX Model output "end_logits":
		-[✓] (2, 16) matches (2, 16)
		-[✓] all values close (atol: 0.0001)
The ONNX export succeeded and the exported model was saved at: distilbert_base_uncased_squad_onnx
```

The example above illustrates exporting a checkpoint from 🤗 Hub.


key: _node_content
string_value:
{"id_": "80d8b397-2a54-4673-a046-c84516e2c1c3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fa46a645-8e6b-465b-ae69-3d51f650c846", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "hash": "b97638da20ee8948778fc11261763dd7a2666a79a5866a41f99ad4299f694ae6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5836f719-6e02-4e19-8893-47bbef5b5e90", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "hash": "1e1f4e9fa7bb3fd1839a3571e8d4ab7bf9438c1d296500a931de4c48a3c82dbf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3280cd42-0b2b-4c88-aeee-0385d7c0d3f3", "node_type": "1", "metadata": {}, "hash": "7d11f21fa86f5366da58a521326e1d04781c99acdd252ed4c285ae7f564a002a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2696, "end_char_idx": 5498, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Exporting a 🤗 Transformers model to ONNX with CLI

To export a 🤗 Transformers model to ONNX, first install an extra dependency:

```bash
pip install optimum[exporters]
```

To check out all available arguments, refer to the [🤗 Optimum docs](https://huggingface.co/docs/optimum/exporters/onnx/usage_guides/export_a_model#exporting-a-model-to-onnx-using-the-cli), 
or view help in command line:

```bash
optimum-cli export onnx --help
```

To export a model's checkpoint from the 🤗 Hub, for example, `distilbert/distilbert-base-uncased-distilled-squad`, run the following command: 

```bash
optimum-cli export onnx --model distilbert/distilbert-base-uncased-distilled-squad distilbert_base_uncased_squad_onnx/
```

You should see the logs indicating progress and showing where the resulting `model.onnx` is saved, like this:

```bash
Validating ONNX model distilbert_base_uncased_squad_onnx/model.onnx...
	-[✓] ONNX model output names match reference model (start_logits, end_logits)
	- Validating ONNX Model output "start_logits":
		-[✓] (2, 16) matches (2, 16)
		-[✓] all values close (atol: 0.0001)
	- Validating ONNX Model output "end_logits":
		-[✓] (2, 16) matches (2, 16)
		-[✓] all values close (atol: 0.0001)
The ONNX export succeeded and the exported model was saved at: distilbert_base_uncased_squad_onnx
```

The example above illustrates exporting a checkpoint from 🤗 Hub. When exporting a local model, first make sure that you 
saved both the model's weights and tokenizer files in the same directory (`local_path`). When using CLI, pass the 
`local_path` to the `model` argument instead of the checkpoint name on 🤗 Hub and provide the `--task` argument. 
You can review the list of supported tasks in the [🤗 Optimum documentation](https://huggingface.co/docs/optimum/exporters/task_manager).
If `task` argument is not provided, it will default to the model architecture without any task specific head.

```bash
optimum-cli export onnx --model local_path --task question-answering distilbert_base_uncased_squad_onnx/
```

The resulting `model.onnx` file can then be run on one of the [many
accelerators](https://onnx.ai/supported-tools.html#deployModel) that support the ONNX
standard. For example, we can load and run the model with [ONNX
Runtime](https://onnxruntime.ai/) as follows:

```python
>>> from transformers import AutoTokenizer
>>> from optimum.onnxruntime import ORTModelForQuestionAnswering

>>> tokenizer = AutoTokenizer.from_pretrained("distilbert_base_uncased_squad_onnx")
>>> model = ORTModelForQuestionAnswering.from_pretrained("distilbert_base_uncased_squad_onnx")
>>> inputs = tokenizer("What am I using?", "Using DistilBERT with ONNX Runtime!", return_tensors="pt")
>>> outputs = model(**inputs)
```

The process is identical for TensorFlow checkpoints on the Hub.


key: _node_content
string_value:
{"id_": "3280cd42-0b2b-4c88-aeee-0385d7c0d3f3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fa46a645-8e6b-465b-ae69-3d51f650c846", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "hash": "b97638da20ee8948778fc11261763dd7a2666a79a5866a41f99ad4299f694ae6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "80d8b397-2a54-4673-a046-c84516e2c1c3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "hash": "5d6b1a6b3f867adb5925e021d8970e56a542c7fc72f6f14d396d2d0c35d40df5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9e28a275-03ef-4051-89b3-3677f3374e2e", "node_type": "1", "metadata": {}, "hash": "5eb36c9de8909b05989aa9b7d03e7770055a28c4a9b5d7ef1c558ec87d97c263", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4084, "end_char_idx": 7106, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
When exporting a local model, first make sure that you 
saved both the model's weights and tokenizer files in the same directory (`local_path`). When using CLI, pass the 
`local_path` to the `model` argument instead of the checkpoint name on 🤗 Hub and provide the `--task` argument. 
You can review the list of supported tasks in the [🤗 Optimum documentation](https://huggingface.co/docs/optimum/exporters/task_manager).
If `task` argument is not provided, it will default to the model architecture without any task specific head.

```bash
optimum-cli export onnx --model local_path --task question-answering distilbert_base_uncased_squad_onnx/
```

The resulting `model.onnx` file can then be run on one of the [many
accelerators](https://onnx.ai/supported-tools.html#deployModel) that support the ONNX
standard. For example, we can load and run the model with [ONNX
Runtime](https://onnxruntime.ai/) as follows:

```python
>>> from transformers import AutoTokenizer
>>> from optimum.onnxruntime import ORTModelForQuestionAnswering

>>> tokenizer = AutoTokenizer.from_pretrained("distilbert_base_uncased_squad_onnx")
>>> model = ORTModelForQuestionAnswering.from_pretrained("distilbert_base_uncased_squad_onnx")
>>> inputs = tokenizer("What am I using?", "Using DistilBERT with ONNX Runtime!", return_tensors="pt")
>>> outputs = model(**inputs)
```

The process is identical for TensorFlow checkpoints on the Hub. For instance, here's how you would
export a pure TensorFlow checkpoint from the [Keras organization](https://huggingface.co/keras-io):

```bash
optimum-cli export onnx --model keras-io/transformers-qa distilbert_base_cased_squad_onnx/
```

### Exporting a 🤗 Transformers model to ONNX with `optimum.onnxruntime`

Alternative to CLI, you can export a 🤗 Transformers model to ONNX programmatically like so: 

```python
>>> from optimum.onnxruntime import ORTModelForSequenceClassification
>>> from transformers import AutoTokenizer

>>> model_checkpoint = "distilbert_base_uncased_squad"
>>> save_directory = "onnx/"

>>> # Load a model from transformers and export it to ONNX
>>> ort_model = ORTModelForSequenceClassification.from_pretrained(model_checkpoint, export=True)
>>> tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)

>>> # Save the onnx model and tokenizer
>>> ort_model.save_pretrained(save_directory)
>>> tokenizer.save_pretrained(save_directory)
```

### Exporting a model for an unsupported architecture

If you wish to contribute by adding support for a model that cannot be currently exported, you should first check if it is
supported in [`optimum.exporters.onnx`](https://huggingface.co/docs/optimum/exporters/onnx/overview),
and if it is not, [contribute to 🤗 Optimum](https://huggingface.co/docs/optimum/exporters/onnx/usage_guides/contribute)
directly.

### Exporting a model with `transformers.onnx`

<Tip warning={true}>

`tranformers.onnx` is no longer maintained, please export models with 🤗 Optimum as described above. This section will be removed in the future versions.


key: _node_content
string_value:
{"id_": "9e28a275-03ef-4051-89b3-3677f3374e2e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fa46a645-8e6b-465b-ae69-3d51f650c846", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "hash": "b97638da20ee8948778fc11261763dd7a2666a79a5866a41f99ad4299f694ae6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3280cd42-0b2b-4c88-aeee-0385d7c0d3f3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "hash": "22c6d0444383c509054462501ea7e014de72b2ee3129e4fe2a6475840b317499", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "492fb6a8-f5d0-4fb3-ba42-6a49899258e0", "node_type": "1", "metadata": {}, "hash": "3c66741d635fb69a621735c0e4c3442c0c99f4715f545ef96d51b4b5c378de9f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5499, "end_char_idx": 8714, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For instance, here's how you would
export a pure TensorFlow checkpoint from the [Keras organization](https://huggingface.co/keras-io):

```bash
optimum-cli export onnx --model keras-io/transformers-qa distilbert_base_cased_squad_onnx/
```

### Exporting a 🤗 Transformers model to ONNX with `optimum.onnxruntime`

Alternative to CLI, you can export a 🤗 Transformers model to ONNX programmatically like so: 

```python
>>> from optimum.onnxruntime import ORTModelForSequenceClassification
>>> from transformers import AutoTokenizer

>>> model_checkpoint = "distilbert_base_uncased_squad"
>>> save_directory = "onnx/"

>>> # Load a model from transformers and export it to ONNX
>>> ort_model = ORTModelForSequenceClassification.from_pretrained(model_checkpoint, export=True)
>>> tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)

>>> # Save the onnx model and tokenizer
>>> ort_model.save_pretrained(save_directory)
>>> tokenizer.save_pretrained(save_directory)
```

### Exporting a model for an unsupported architecture

If you wish to contribute by adding support for a model that cannot be currently exported, you should first check if it is
supported in [`optimum.exporters.onnx`](https://huggingface.co/docs/optimum/exporters/onnx/overview),
and if it is not, [contribute to 🤗 Optimum](https://huggingface.co/docs/optimum/exporters/onnx/usage_guides/contribute)
directly.

### Exporting a model with `transformers.onnx`

<Tip warning={true}>

`tranformers.onnx` is no longer maintained, please export models with 🤗 Optimum as described above. This section will be removed in the future versions.

</Tip>

To export a 🤗 Transformers model to ONNX with `tranformers.onnx`, install extra dependencies:

```bash
pip install transformers[onnx]
```

Use `transformers.onnx` package as a Python module to export a checkpoint using a ready-made configuration:

```bash
python -m transformers.onnx --model=distilbert/distilbert-base-uncased onnx/
```

This exports an ONNX graph of the checkpoint defined by the `--model` argument. Pass any checkpoint on the 🤗 Hub or one that's stored locally.
The resulting `model.onnx` file can then be run on one of the many accelerators that support the ONNX standard. For example, 
load and run the model with ONNX Runtime as follows:

```python
>>> from transformers import AutoTokenizer
>>> from onnxruntime import InferenceSession

>>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
>>> session = InferenceSession("onnx/model.onnx")
>>> # ONNX Runtime expects NumPy arrays as input
>>> inputs = tokenizer("Using DistilBERT with ONNX Runtime!", return_tensors="np")
>>> outputs = session.run(output_names=["last_hidden_state"], input_feed=dict(inputs))
```

The required output names (like `["last_hidden_state"]`) can be obtained by taking a look at the ONNX configuration of 
each model. For example, for DistilBERT we have:

```python
>>> from transformers.models.distilbert import DistilBertConfig, DistilBertOnnxConfig

>>> config = DistilBertConfig()
>>> onnx_config = DistilBertOnnxConfig(config)
>>> print(list(onnx_config.outputs.keys()))
["last_hidden_state"]
```

The process is identical for TensorFlow checkpoints on the Hub.


key: _node_content
string_value:
{"id_": "492fb6a8-f5d0-4fb3-ba42-6a49899258e0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fa46a645-8e6b-465b-ae69-3d51f650c846", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "hash": "b97638da20ee8948778fc11261763dd7a2666a79a5866a41f99ad4299f694ae6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9e28a275-03ef-4051-89b3-3677f3374e2e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "hash": "35ca19d17894ea828fc851bc477d10394ba8788ebc42afd503e8644325742c24", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7054, "end_char_idx": 9186, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This section will be removed in the future versions.

</Tip>

To export a 🤗 Transformers model to ONNX with `tranformers.onnx`, install extra dependencies:

```bash
pip install transformers[onnx]
```

Use `transformers.onnx` package as a Python module to export a checkpoint using a ready-made configuration:

```bash
python -m transformers.onnx --model=distilbert/distilbert-base-uncased onnx/
```

This exports an ONNX graph of the checkpoint defined by the `--model` argument. Pass any checkpoint on the 🤗 Hub or one that's stored locally.
The resulting `model.onnx` file can then be run on one of the many accelerators that support the ONNX standard. For example, 
load and run the model with ONNX Runtime as follows:

```python
>>> from transformers import AutoTokenizer
>>> from onnxruntime import InferenceSession

>>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
>>> session = InferenceSession("onnx/model.onnx")
>>> # ONNX Runtime expects NumPy arrays as input
>>> inputs = tokenizer("Using DistilBERT with ONNX Runtime!", return_tensors="np")
>>> outputs = session.run(output_names=["last_hidden_state"], input_feed=dict(inputs))
```

The required output names (like `["last_hidden_state"]`) can be obtained by taking a look at the ONNX configuration of 
each model. For example, for DistilBERT we have:

```python
>>> from transformers.models.distilbert import DistilBertConfig, DistilBertOnnxConfig

>>> config = DistilBertConfig()
>>> onnx_config = DistilBertOnnxConfig(config)
>>> print(list(onnx_config.outputs.keys()))
["last_hidden_state"]
```

The process is identical for TensorFlow checkpoints on the Hub. For example, export a pure TensorFlow checkpoint like so:

```bash
python -m transformers.onnx --model=keras-io/transformers-qa onnx/
```

To export a model that's stored locally, save the model's weights and tokenizer files in the same directory (e.g. `local-pt-checkpoint`), 
then export it to ONNX by pointing the `--model` argument of the `transformers.onnx` package to the desired directory:

```bash
python -m transformers.onnx --model=local-pt-checkpoint onnx/
```


key: _node_content
string_value:
{"id_": "dd4fed7f-e101-45e9-9cd7-a67474db7551", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "443128bef3012ec142fe3dbfb1dc4b34611cc0cc2d78a674a0686d1026a6d3ed", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "18c5e94d-c31f-4a9f-be73-b1eb2abdea07", "node_type": "1", "metadata": {}, "hash": "f272dc2d1b293d7d32e56b81d9ab47cf5d9ffc72a54819a4d51c2484bac391f6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3212, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Fine-tune a pretrained model

[[open-in-colab]]

There are significant benefits to using a pretrained model. It reduces computation costs, your carbon footprint, and allows you to use state-of-the-art models without having to train one from scratch. 🤗 Transformers provides access to thousands of pretrained models for a wide range of tasks. When you use a pretrained model, you train it on a dataset specific to your task. This is known as fine-tuning, an incredibly powerful training technique. In this tutorial, you will fine-tune a pretrained model with a deep learning framework of your choice:

* Fine-tune a pretrained model with 🤗 Transformers [`Trainer`].
* Fine-tune a pretrained model in TensorFlow with Keras.
* Fine-tune a pretrained model in native PyTorch.

<a id='data-processing'></a>

## Prepare a dataset

<Youtube id="_BZearw7f0w"/>

Before you can fine-tune a pretrained model, download a dataset and prepare it for training. The previous tutorial showed you how to process data for training, and now you get an opportunity to put those skills to the test!

Begin by loading the [Yelp Reviews](https://huggingface.co/datasets/yelp_review_full) dataset:

```py
>>> from datasets import load_dataset

>>> dataset = load_dataset("yelp_review_full")
>>> dataset["train"][100]
{'label': 0,
 'text': 'My expectations for McDonalds are t rarely high. But for one to still fail so spectacularly...that takes something special!\\nThe cashier took my friends\'s order, then promptly ignored me. I had to force myself in front of a cashier who opened his register to wait on the person BEHIND me. I waited over five minutes for a gigantic order that included precisely one kid\'s meal. After watching two people who ordered after me be handed their food, I asked where mine was. The manager started yelling at the cashiers for \\"serving off their orders\\" when they didn\'t have their food. But neither cashier was anywhere near those controls, and the manager was the one serving food to customers and clearing the boards.\\nThe manager was rude when giving me my order. She didn\'t make sure that I had everything ON MY RECEIPT, and never even had the decency to apologize that I felt I was getting poor service.\\nI\'ve eaten at various McDonalds restaurants for over 30 years. I\'ve worked at more than one location. I expect bad days, bad moods, and the occasional mistake. But I have yet to have a decent experience at this store. It will remain a place I avoid unless someone in my party needs to avoid illness from low blood sugar. Perhaps I should go back to the racially biased service of Steak n Shake instead!'}
```

As you now know, you need a tokenizer to process the text and include a padding and truncation strategy to handle any variable sequence lengths. To process your dataset in one step, use 🤗 Datasets [`map`](https://huggingface.co/docs/datasets/process#map) method to apply a preprocessing function over the entire dataset:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-cased")


>>> def tokenize_function(examples):
...     return tokenizer(examples["text"], padding="max_length", truncation=True)


key: _node_content
string_value:
{"id_": "18c5e94d-c31f-4a9f-be73-b1eb2abdea07", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "443128bef3012ec142fe3dbfb1dc4b34611cc0cc2d78a674a0686d1026a6d3ed", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dd4fed7f-e101-45e9-9cd7-a67474db7551", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "343fb348c3175abd370ff46fe74b134456ce4f71037fcd0424931dde4183709d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9b5de8c7-82be-435d-9fae-d6611d9aab71", "node_type": "1", "metadata": {}, "hash": "3a9b92824b68eeb1872e3b92309ad40945eb7f5b133dfb7e235d80a9e2cbb9c6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3094, "end_char_idx": 6395, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> def tokenize_function(examples):
...     return tokenizer(examples["text"], padding="max_length", truncation=True)


>>> tokenized_datasets = dataset.map(tokenize_function, batched=True)
```

If you like, you can create a smaller subset of the full dataset to fine-tune on to reduce the time it takes:

```py
>>> small_train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(1000))
>>> small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42).select(range(1000))
```

<a id='trainer'></a>

## Train

At this point, you should follow the section corresponding to the framework you want to use. You can use the links
in the right sidebar to jump to the one you want - and if you want to hide all of the content for a given framework,
just use the button at the top-right of that framework's block!

<frameworkcontent>
<pt>
<Youtube id="nvBXf7s7vTI"/>

## Train with PyTorch Trainer

🤗 Transformers provides a [`Trainer`] class optimized for training 🤗 Transformers models, making it easier to start training without manually writing your own training loop. The [`Trainer`] API supports a wide range of training options and features such as logging, gradient accumulation, and mixed precision.

Start by loading your model and specify the number of expected labels. From the Yelp Review [dataset card](https://huggingface.co/datasets/yelp_review_full#data-fields), you know there are five labels:

```py
>>> from transformers import AutoModelForSequenceClassification

>>> model = AutoModelForSequenceClassification.from_pretrained("google-bert/bert-base-cased", num_labels=5)
```

<Tip>

You will see a warning about some of the pretrained weights not being used and some weights being randomly
initialized. Don't worry, this is completely normal! The pretrained head of the BERT model is discarded, and replaced with a randomly initialized classification head. You will fine-tune this new model head on your sequence classification task, transferring the knowledge of the pretrained model to it.

</Tip>

### Training hyperparameters

Next, create a [`TrainingArguments`] class which contains all the hyperparameters you can tune as well as flags for activating different training options. For this tutorial you can start with the default training [hyperparameters](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments), but feel free to experiment with these to find your optimal settings.

Specify where to save the checkpoints from your training:

```py
>>> from transformers import TrainingArguments

>>> training_args = TrainingArguments(output_dir="test_trainer")
```

### Evaluate

[`Trainer`] does not automatically evaluate model performance during training. You'll need to pass [`Trainer`] a function to compute and report metrics. The [🤗 Evaluate](https://huggingface.co/docs/evaluate/index) library provides a simple [`accuracy`](https://huggingface.co/spaces/evaluate-metric/accuracy) function you can load with the [`evaluate.load`] (see this [quicktour](https://huggingface.co/docs/evaluate/a_quick_tour) for more information) function:

```py
>>> import numpy as np
>>> import evaluate

>>> metric = evaluate.load("accuracy")
```

Call [`~evaluate.compute`] on `metric` to calculate the accuracy of your predictions.


key: _node_content
string_value:
{"id_": "9b5de8c7-82be-435d-9fae-d6611d9aab71", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "443128bef3012ec142fe3dbfb1dc4b34611cc0cc2d78a674a0686d1026a6d3ed", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "18c5e94d-c31f-4a9f-be73-b1eb2abdea07", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "8df8826a90b077b78425d840cf28f896ccaa9c5626911ea60b589d821d2e48e2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ad6c63b0-880d-440e-8c87-45fe92c5e2e4", "node_type": "1", "metadata": {}, "hash": "ce246453266e69a69e7dc81f91a6f10d243c090ba178d8dfe5d3a7c05724d63a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4830, "end_char_idx": 8253, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Don't worry, this is completely normal! The pretrained head of the BERT model is discarded, and replaced with a randomly initialized classification head. You will fine-tune this new model head on your sequence classification task, transferring the knowledge of the pretrained model to it.

</Tip>

### Training hyperparameters

Next, create a [`TrainingArguments`] class which contains all the hyperparameters you can tune as well as flags for activating different training options. For this tutorial you can start with the default training [hyperparameters](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments), but feel free to experiment with these to find your optimal settings.

Specify where to save the checkpoints from your training:

```py
>>> from transformers import TrainingArguments

>>> training_args = TrainingArguments(output_dir="test_trainer")
```

### Evaluate

[`Trainer`] does not automatically evaluate model performance during training. You'll need to pass [`Trainer`] a function to compute and report metrics. The [🤗 Evaluate](https://huggingface.co/docs/evaluate/index) library provides a simple [`accuracy`](https://huggingface.co/spaces/evaluate-metric/accuracy) function you can load with the [`evaluate.load`] (see this [quicktour](https://huggingface.co/docs/evaluate/a_quick_tour) for more information) function:

```py
>>> import numpy as np
>>> import evaluate

>>> metric = evaluate.load("accuracy")
```

Call [`~evaluate.compute`] on `metric` to calculate the accuracy of your predictions. Before passing your predictions to `compute`, you need to convert the logits to predictions (remember all 🤗 Transformers models return logits):

```py
>>> def compute_metrics(eval_pred):
...     logits, labels = eval_pred
...     predictions = np.argmax(logits, axis=-1)
...     return metric.compute(predictions=predictions, references=labels)
```

If you'd like to monitor your evaluation metrics during fine-tuning, specify the `eval_strategy` parameter in your training arguments to report the evaluation metric at the end of each epoch:

```py
>>> from transformers import TrainingArguments, Trainer

>>> training_args = TrainingArguments(output_dir="test_trainer", eval_strategy="epoch")
```

### Trainer

Create a [`Trainer`] object with your model, training arguments, training and test datasets, and evaluation function:

```py
>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=small_train_dataset,
...     eval_dataset=small_eval_dataset,
...     compute_metrics=compute_metrics,
... )
```

Then fine-tune your model by calling [`~transformers.Trainer.train`]:

```py
>>> trainer.train()
```
</pt>
<tf>
<a id='keras'></a>

<Youtube id="rnTGBy2ax1c"/>

## Train a TensorFlow model with Keras

You can also train 🤗 Transformers models in TensorFlow with the Keras API!

### Loading data for Keras

When you want to train a 🤗 Transformers model with the Keras API, you need to convert your dataset to a format that
Keras understands. If your dataset is small, you can just convert the whole thing to NumPy arrays and pass it to Keras.
Let's try that first before we do anything more complicated.

First, load a dataset. We'll use the CoLA dataset from the [GLUE benchmark](https://huggingface.co/datasets/glue),
since it's a simple binary text classification task, and just take the training split for now.


key: _node_content
string_value:
{"id_": "ad6c63b0-880d-440e-8c87-45fe92c5e2e4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "443128bef3012ec142fe3dbfb1dc4b34611cc0cc2d78a674a0686d1026a6d3ed", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9b5de8c7-82be-435d-9fae-d6611d9aab71", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "bacd0f5cd722024f3e52d24f3782aa9b4ad2512529b0a6ede0b5a6aa5c44d0d7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "380bd34c-8c52-49ca-b183-f43295c96f4c", "node_type": "1", "metadata": {}, "hash": "ce7bb17bf1d7a1dd7247002908fb8643603a54cfcc22f0aff87f5154e504a180", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7718, "end_char_idx": 11047, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Loading data for Keras

When you want to train a 🤗 Transformers model with the Keras API, you need to convert your dataset to a format that
Keras understands. If your dataset is small, you can just convert the whole thing to NumPy arrays and pass it to Keras.
Let's try that first before we do anything more complicated.

First, load a dataset. We'll use the CoLA dataset from the [GLUE benchmark](https://huggingface.co/datasets/glue),
since it's a simple binary text classification task, and just take the training split for now.

```py
from datasets import load_dataset

dataset = load_dataset("glue", "cola")
dataset = dataset["train"]  # Just take the training split for now
```

Next, load a tokenizer and tokenize the data as NumPy arrays. Note that the labels are already a list of 0 and 1s,
so we can just convert that directly to a NumPy array without tokenization!

```py
from transformers import AutoTokenizer
import numpy as np

tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-cased")
tokenized_data = tokenizer(dataset["sentence"], return_tensors="np", padding=True)
# Tokenizer returns a BatchEncoding, but we convert that to a dict for Keras
tokenized_data = dict(tokenized_data)

labels = np.array(dataset["label"])  # Label is already an array of 0 and 1
```

Finally, load, [`compile`](https://keras.io/api/models/model_training_apis/#compile-method), and [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) the model. Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
from transformers import TFAutoModelForSequenceClassification
from tensorflow.keras.optimizers import Adam

# Load and compile our model
model = TFAutoModelForSequenceClassification.from_pretrained("google-bert/bert-base-cased")
# Lower learning rates are often better for fine-tuning transformers
model.compile(optimizer=Adam(3e-5))  # No loss argument!

model.fit(tokenized_data, labels)
```

<Tip>

You don't have to pass a loss argument to your models when you `compile()` them! Hugging Face models automatically
choose a loss that is appropriate for their task and model architecture if this argument is left blank. You can always
override this by specifying a loss yourself if you want to!

</Tip>

This approach works great for smaller datasets, but for larger datasets, you might find it starts to become a problem. Why?
Because the tokenized array and labels would have to be fully loaded into memory, and because NumPy doesn’t handle
“jagged” arrays, so every tokenized sample would have to be padded to the length of the longest sample in the whole
dataset. That’s going to make your array even bigger, and all those padding tokens will slow down training too!

### Loading data as a tf.data.Dataset

If you want to avoid slowing down training, you can load your data as a `tf.data.Dataset` instead. Although you can write your own
`tf.data` pipeline if you want, we have two convenience methods for doing this:

- [`~TFPreTrainedModel.prepare_tf_dataset`]: This is the method we recommend in most cases. Because it is a method
on your model, it can inspect the model to automatically figure out which columns are usable as model inputs, and
discard the others to make a simpler, more performant dataset.


key: _node_content
string_value:
{"id_": "380bd34c-8c52-49ca-b183-f43295c96f4c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "443128bef3012ec142fe3dbfb1dc4b34611cc0cc2d78a674a0686d1026a6d3ed", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ad6c63b0-880d-440e-8c87-45fe92c5e2e4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "82191562886a8bd93b678fa3199ee9b7f37db8606bc0a00c73a212c55d55ab4d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "da90ba86-c748-449a-9586-b81d27ce44ba", "node_type": "1", "metadata": {}, "hash": "0638e4a95dff63d8b1aebe46086c239745a2b1e232e2595f5237a7776588b272", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9688, "end_char_idx": 12937, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
model.fit(tokenized_data, labels)
```

<Tip>

You don't have to pass a loss argument to your models when you `compile()` them! Hugging Face models automatically
choose a loss that is appropriate for their task and model architecture if this argument is left blank. You can always
override this by specifying a loss yourself if you want to!

</Tip>

This approach works great for smaller datasets, but for larger datasets, you might find it starts to become a problem. Why?
Because the tokenized array and labels would have to be fully loaded into memory, and because NumPy doesn’t handle
“jagged” arrays, so every tokenized sample would have to be padded to the length of the longest sample in the whole
dataset. That’s going to make your array even bigger, and all those padding tokens will slow down training too!

### Loading data as a tf.data.Dataset

If you want to avoid slowing down training, you can load your data as a `tf.data.Dataset` instead. Although you can write your own
`tf.data` pipeline if you want, we have two convenience methods for doing this:

- [`~TFPreTrainedModel.prepare_tf_dataset`]: This is the method we recommend in most cases. Because it is a method
on your model, it can inspect the model to automatically figure out which columns are usable as model inputs, and
discard the others to make a simpler, more performant dataset.
- [`~datasets.Dataset.to_tf_dataset`]: This method is more low-level, and is useful when you want to exactly control how
your dataset is created, by specifying exactly which `columns` and `label_cols` to include.

Before you can use [`~TFPreTrainedModel.prepare_tf_dataset`], you will need to add the tokenizer outputs to your dataset as columns, as shown in
the following code sample:

```py
def tokenize_dataset(data):
    # Keys of the returned dictionary will be added to the dataset as columns
    return tokenizer(data["text"])


dataset = dataset.map(tokenize_dataset)
```

Remember that Hugging Face datasets are stored on disk by default, so this will not inflate your memory usage! Once the
columns have been added, you can stream batches from the dataset and add padding to each batch, which greatly
reduces the number of padding tokens compared to padding the entire dataset.


```py
>>> tf_dataset = model.prepare_tf_dataset(dataset["train"], batch_size=16, shuffle=True, tokenizer=tokenizer)
```

Note that in the code sample above, you need to pass the tokenizer to `prepare_tf_dataset` so it can correctly pad batches as they're loaded.
If all the samples in your dataset are the same length and no padding is necessary, you can skip this argument.
If you need to do something more complex than just padding samples (e.g. corrupting tokens for masked language
modelling), you can use the `collate_fn` argument instead to pass a function that will be called to transform the
list of samples into a batch and apply any preprocessing you want. See our
[examples](https://github.com/huggingface/transformers/tree/main/examples) or
[notebooks](https://huggingface.co/docs/transformers/notebooks) to see this approach in action.

Once you've created a `tf.data.Dataset`, you can compile and fit the model as before:

```py
model.compile(optimizer=Adam(3e-5))  # No loss argument!


key: _node_content
string_value:
{"id_": "da90ba86-c748-449a-9586-b81d27ce44ba", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "443128bef3012ec142fe3dbfb1dc4b34611cc0cc2d78a674a0686d1026a6d3ed", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "380bd34c-8c52-49ca-b183-f43295c96f4c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "66429603b6d2af483eaf01d8d6d312550ea4c7bb9ba691ba7463fa2f5515556e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a836e663-0337-4dd0-8ce9-0dc4be51c475", "node_type": "1", "metadata": {}, "hash": "074d18a40a427a36ea36bcd86caa3e6a20f89500808ef958e0ddae589e9914ca", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11262, "end_char_idx": 13951, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Before you can use [`~TFPreTrainedModel.prepare_tf_dataset`], you will need to add the tokenizer outputs to your dataset as columns, as shown in
the following code sample:

```py
def tokenize_dataset(data):
    # Keys of the returned dictionary will be added to the dataset as columns
    return tokenizer(data["text"])


dataset = dataset.map(tokenize_dataset)
```

Remember that Hugging Face datasets are stored on disk by default, so this will not inflate your memory usage! Once the
columns have been added, you can stream batches from the dataset and add padding to each batch, which greatly
reduces the number of padding tokens compared to padding the entire dataset.


```py
>>> tf_dataset = model.prepare_tf_dataset(dataset["train"], batch_size=16, shuffle=True, tokenizer=tokenizer)
```

Note that in the code sample above, you need to pass the tokenizer to `prepare_tf_dataset` so it can correctly pad batches as they're loaded.
If all the samples in your dataset are the same length and no padding is necessary, you can skip this argument.
If you need to do something more complex than just padding samples (e.g. corrupting tokens for masked language
modelling), you can use the `collate_fn` argument instead to pass a function that will be called to transform the
list of samples into a batch and apply any preprocessing you want. See our
[examples](https://github.com/huggingface/transformers/tree/main/examples) or
[notebooks](https://huggingface.co/docs/transformers/notebooks) to see this approach in action.

Once you've created a `tf.data.Dataset`, you can compile and fit the model as before:

```py
model.compile(optimizer=Adam(3e-5))  # No loss argument!

model.fit(tf_dataset)
```

</tf>
</frameworkcontent>

<a id='pytorch_native'></a>

## Train in native PyTorch

<frameworkcontent>
<pt>
<Youtube id="Dh9CL8fyG80"/>

[`Trainer`] takes care of the training loop and allows you to fine-tune a model in a single line of code. For users who prefer to write their own training loop, you can also fine-tune a 🤗 Transformers model in native PyTorch.

At this point, you may need to restart your notebook or execute the following code to free some memory:

```py
del model
del trainer
torch.cuda.empty_cache()
```

Next, manually postprocess `tokenized_dataset` to prepare it for training.

1. Remove the `text` column because the model does not accept raw text as an input:

    ```py
    >>> tokenized_datasets = tokenized_datasets.remove_columns(["text"])
    ```

2. Rename the `label` column to `labels` because the model expects the argument to be named `labels`:

    ```py
    >>> tokenized_datasets = tokenized_datasets.rename_column("label", "labels")
    ```

3.


key: _node_content
string_value:
{"id_": "a836e663-0337-4dd0-8ce9-0dc4be51c475", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "443128bef3012ec142fe3dbfb1dc4b34611cc0cc2d78a674a0686d1026a6d3ed", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "da90ba86-c748-449a-9586-b81d27ce44ba", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "93c05aeeb2f2848f9c4424b821ba04ea0e300d146be60aeedebacfd32dd425ae", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2bbd83a8-0636-46ab-a836-734bc9b88abe", "node_type": "1", "metadata": {}, "hash": "a30a0c83f7eb103efd390953bcefd85dbaae196e50ea672e64a857c9d4faf310", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12386, "end_char_idx": 15051, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
corrupting tokens for masked language
modelling), you can use the `collate_fn` argument instead to pass a function that will be called to transform the
list of samples into a batch and apply any preprocessing you want. See our
[examples](https://github.com/huggingface/transformers/tree/main/examples) or
[notebooks](https://huggingface.co/docs/transformers/notebooks) to see this approach in action.

Once you've created a `tf.data.Dataset`, you can compile and fit the model as before:

```py
model.compile(optimizer=Adam(3e-5))  # No loss argument!

model.fit(tf_dataset)
```

</tf>
</frameworkcontent>

<a id='pytorch_native'></a>

## Train in native PyTorch

<frameworkcontent>
<pt>
<Youtube id="Dh9CL8fyG80"/>

[`Trainer`] takes care of the training loop and allows you to fine-tune a model in a single line of code. For users who prefer to write their own training loop, you can also fine-tune a 🤗 Transformers model in native PyTorch.

At this point, you may need to restart your notebook or execute the following code to free some memory:

```py
del model
del trainer
torch.cuda.empty_cache()
```

Next, manually postprocess `tokenized_dataset` to prepare it for training.

1. Remove the `text` column because the model does not accept raw text as an input:

    ```py
    >>> tokenized_datasets = tokenized_datasets.remove_columns(["text"])
    ```

2. Rename the `label` column to `labels` because the model expects the argument to be named `labels`:

    ```py
    >>> tokenized_datasets = tokenized_datasets.rename_column("label", "labels")
    ```

3. Set the format of the dataset to return PyTorch tensors instead of lists:

    ```py
    >>> tokenized_datasets.set_format("torch")
    ```

Then create a smaller subset of the dataset as previously shown to speed up the fine-tuning:

```py
>>> small_train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(1000))
>>> small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42).select(range(1000))
```

### DataLoader

Create a `DataLoader` for your training and test datasets so you can iterate over batches of data:

```py
>>> from torch.utils.data import DataLoader

>>> train_dataloader = DataLoader(small_train_dataset, shuffle=True, batch_size=8)
>>> eval_dataloader = DataLoader(small_eval_dataset, batch_size=8)
```

Load your model with the number of expected labels:

```py
>>> from transformers import AutoModelForSequenceClassification

>>> model = AutoModelForSequenceClassification.from_pretrained("google-bert/bert-base-cased", num_labels=5)
```

### Optimizer and learning rate scheduler

Create an optimizer and learning rate scheduler to fine-tune the model.


key: _node_content
string_value:
{"id_": "2bbd83a8-0636-46ab-a836-734bc9b88abe", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "443128bef3012ec142fe3dbfb1dc4b34611cc0cc2d78a674a0686d1026a6d3ed", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a836e663-0337-4dd0-8ce9-0dc4be51c475", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "787226d725c928e5c30425048c9138e05900feb0b59bc29768ab9617f5b128a8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cd8e4512-62c1-468a-81cc-b56b098a35d2", "node_type": "1", "metadata": {}, "hash": "51a8f89f30303e85c9c694c0bfbc834a46b40609f5f474facc7afb8895c407df", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13330, "end_char_idx": 16132, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
At this point, you may need to restart your notebook or execute the following code to free some memory:

```py
del model
del trainer
torch.cuda.empty_cache()
```

Next, manually postprocess `tokenized_dataset` to prepare it for training.

1. Remove the `text` column because the model does not accept raw text as an input:

    ```py
    >>> tokenized_datasets = tokenized_datasets.remove_columns(["text"])
    ```

2. Rename the `label` column to `labels` because the model expects the argument to be named `labels`:

    ```py
    >>> tokenized_datasets = tokenized_datasets.rename_column("label", "labels")
    ```

3. Set the format of the dataset to return PyTorch tensors instead of lists:

    ```py
    >>> tokenized_datasets.set_format("torch")
    ```

Then create a smaller subset of the dataset as previously shown to speed up the fine-tuning:

```py
>>> small_train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(1000))
>>> small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42).select(range(1000))
```

### DataLoader

Create a `DataLoader` for your training and test datasets so you can iterate over batches of data:

```py
>>> from torch.utils.data import DataLoader

>>> train_dataloader = DataLoader(small_train_dataset, shuffle=True, batch_size=8)
>>> eval_dataloader = DataLoader(small_eval_dataset, batch_size=8)
```

Load your model with the number of expected labels:

```py
>>> from transformers import AutoModelForSequenceClassification

>>> model = AutoModelForSequenceClassification.from_pretrained("google-bert/bert-base-cased", num_labels=5)
```

### Optimizer and learning rate scheduler

Create an optimizer and learning rate scheduler to fine-tune the model. Let's use the [`AdamW`](https://pytorch.org/docs/stable/generated/torch.optim.AdamW.html) optimizer from PyTorch:

```py
>>> from torch.optim import AdamW

>>> optimizer = AdamW(model.parameters(), lr=5e-5)
```

Create the default learning rate scheduler from [`Trainer`]:

```py
>>> from transformers import get_scheduler

>>> num_epochs = 3
>>> num_training_steps = num_epochs * len(train_dataloader)
>>> lr_scheduler = get_scheduler(
...     name="linear", optimizer=optimizer, num_warmup_steps=0, num_training_steps=num_training_steps
... )
```

Lastly, specify `device` to use a GPU if you have access to one. Otherwise, training on a CPU may take several hours instead of a couple of minutes.

```py
>>> import torch

>>> device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
>>> model.to(device)
```

<Tip>

Get free access to a cloud GPU if you don't have one with a hosted notebook like [Colaboratory](https://colab.research.google.com/) or [SageMaker StudioLab](https://studiolab.sagemaker.aws/).

</Tip>

Great, now you are ready to train!


key: _node_content
string_value:
{"id_": "cd8e4512-62c1-468a-81cc-b56b098a35d2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e72ee958-f5be-4a02-ab46-92f9c580b19a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "443128bef3012ec142fe3dbfb1dc4b34611cc0cc2d78a674a0686d1026a6d3ed", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2bbd83a8-0636-46ab-a836-734bc9b88abe", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "d4c416ec53455728ed0aba26d918ccc4d7f1f4d14e1a59866761037f03cd1649", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15052, "end_char_idx": 17983, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Let's use the [`AdamW`](https://pytorch.org/docs/stable/generated/torch.optim.AdamW.html) optimizer from PyTorch:

```py
>>> from torch.optim import AdamW

>>> optimizer = AdamW(model.parameters(), lr=5e-5)
```

Create the default learning rate scheduler from [`Trainer`]:

```py
>>> from transformers import get_scheduler

>>> num_epochs = 3
>>> num_training_steps = num_epochs * len(train_dataloader)
>>> lr_scheduler = get_scheduler(
...     name="linear", optimizer=optimizer, num_warmup_steps=0, num_training_steps=num_training_steps
... )
```

Lastly, specify `device` to use a GPU if you have access to one. Otherwise, training on a CPU may take several hours instead of a couple of minutes.

```py
>>> import torch

>>> device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
>>> model.to(device)
```

<Tip>

Get free access to a cloud GPU if you don't have one with a hosted notebook like [Colaboratory](https://colab.research.google.com/) or [SageMaker StudioLab](https://studiolab.sagemaker.aws/).

</Tip>

Great, now you are ready to train! 🥳 

### Training loop

To keep track of your training progress, use the [tqdm](https://tqdm.github.io/) library to add a progress bar over the number of training steps:

```py
>>> from tqdm.auto import tqdm

>>> progress_bar = tqdm(range(num_training_steps))

>>> model.train()
>>> for epoch in range(num_epochs):
...     for batch in train_dataloader:
...         batch = {k: v.to(device) for k, v in batch.items()}
...         outputs = model(**batch)
...         loss = outputs.loss
...         loss.backward()

...         optimizer.step()
...         lr_scheduler.step()
...         optimizer.zero_grad()
...         progress_bar.update(1)
```

### Evaluate

Just like how you added an evaluation function to [`Trainer`], you need to do the same when you write your own training loop. But instead of calculating and reporting the metric at the end of each epoch, this time you'll accumulate all the batches with [`~evaluate.add_batch`] and calculate the metric at the very end.

```py
>>> import evaluate

>>> metric = evaluate.load("accuracy")
>>> model.eval()
>>> for batch in eval_dataloader:
...     batch = {k: v.to(device) for k, v in batch.items()}
...     with torch.no_grad():
...         outputs = model(**batch)

...     logits = outputs.logits
...     predictions = torch.argmax(logits, dim=-1)
...     metric.add_batch(predictions=predictions, references=batch["labels"])

>>> metric.compute()
```
</pt>
</frameworkcontent>

<a id='additional-resources'></a>

## Additional resources

For more fine-tuning examples, refer to:

- [🤗 Transformers Examples](https://github.com/huggingface/transformers/tree/main/examples) includes scripts
  to train common NLP tasks in PyTorch and TensorFlow.

- [🤗 Transformers Notebooks](notebooks) contains various notebooks on how to fine-tune a model for specific tasks in PyTorch and TensorFlow.


key: _node_content
string_value:
{"id_": "6a01e851-f052-4bbc-942d-6a3898066694", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "d0faf8596bdd083d896283e3e14e6b207798309c03ac95db0d357f4c7d7780d7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3defa731-9e06-4e45-9e8c-daf690ed62e7", "node_type": "1", "metadata": {}, "hash": "2b2678ba65ce5d66be3a488416c657f056d2f62e504f0b0b1db6076d8ade811a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3223, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Chatting with Transformers

If you're reading this article, you're almost certainly aware of **chat models**. Chat models are conversational
AIs that you can send and receive messages with. The most famous of these is the proprietary ChatGPT, but there are
now many open-source chat models which match or even substantially exceed its performance. These models are free to
download and run on a local machine. Although the largest and most capable models require high-powered hardware
and lots of memory to run, there are smaller models that will run perfectly well on a single consumer GPU, or even
an ordinary desktop or notebook CPU. 

This guide will help you get started with chat models. We'll start with a brief quickstart guide that uses a convenient,
high-level "pipeline". This is all you need if you just want to start running a chat model 
immediately. After the quickstart, we'll move on to more detailed information about
what exactly chat models are, how to choose an appropriate one, and a low-level breakdown of each of the
steps involved in talking to a chat model. We'll also give some tips on optimizing the performance and memory usage
of your chat models.


## Quickstart

If you have no time for details, here's the brief summary: Chat models continue chats. This means that you pass them
a conversation history, which can be as short as a single user message, and the model will continue the conversation
by adding its response. Let's see this in action. First, let's build a chat:

```python
chat = [
    {"role": "system", "content": "You are a sassy, wise-cracking robot as imagined by Hollywood circa 1986."},
    {"role": "user", "content": "Hey, can you tell me any fun things to do in New York?"}
]
```

Notice that in addition to the user's message, we added a **system** message at the start of the conversation. Not all
chat models support system messages, but when they do, they represent high-level directives about how the model
should behave in the conversation. You can use this to guide the model - whether you want short or long responses,
lighthearted or serious ones, and so on. If you want the model to do useful work instead of
practicing its improv routine, you can either omit the system message or try a terse one such as "You are a helpful and intelligent
AI assistant who responds to user queries."

Once you have a chat, the quickest way to continue it is using the [`TextGenerationPipeline`]. 
Let's see this in action with `LLaMA-3`. Note that `LLaMA-3` is a gated model, which means you will need to 
[apply for access](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) and log in with your Hugging Face 
account to use it. We'll also use `device_map="auto"`, which will load the model on GPU if there's enough memory
for it, and set the dtype to `torch.bfloat16` to save memory:

```python
import torch
from transformers import pipeline

pipe = pipeline("text-generation", "meta-llama/Meta-Llama-3-8B-Instruct", torch_dtype=torch.bfloat16, device_map="auto")
response = pipe(chat, max_new_tokens=512)
print(response[0]['generated_text'][-1]['content'])
```

And you'll get:

```text
(sigh) Oh boy, you're asking me for advice? You're gonna need a map, pal!


key: _node_content
string_value:
{"id_": "3defa731-9e06-4e45-9e8c-daf690ed62e7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "d0faf8596bdd083d896283e3e14e6b207798309c03ac95db0d357f4c7d7780d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6a01e851-f052-4bbc-942d-6a3898066694", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "ab72716695b9356e6a5920eecf87b4f6c29dfd7254b22dbbd215352e53812e99", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "23c985c2-2b62-4233-845d-b85fc0be3731", "node_type": "1", "metadata": {}, "hash": "fac1f6ed0a1ae8a42daa28cfe5ef68d584be2b4c4e7d53dd12227cb8bfb91071", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1730, "end_char_idx": 4652, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
]
```

Notice that in addition to the user's message, we added a **system** message at the start of the conversation. Not all
chat models support system messages, but when they do, they represent high-level directives about how the model
should behave in the conversation. You can use this to guide the model - whether you want short or long responses,
lighthearted or serious ones, and so on. If you want the model to do useful work instead of
practicing its improv routine, you can either omit the system message or try a terse one such as "You are a helpful and intelligent
AI assistant who responds to user queries."

Once you have a chat, the quickest way to continue it is using the [`TextGenerationPipeline`]. 
Let's see this in action with `LLaMA-3`. Note that `LLaMA-3` is a gated model, which means you will need to 
[apply for access](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) and log in with your Hugging Face 
account to use it. We'll also use `device_map="auto"`, which will load the model on GPU if there's enough memory
for it, and set the dtype to `torch.bfloat16` to save memory:

```python
import torch
from transformers import pipeline

pipe = pipeline("text-generation", "meta-llama/Meta-Llama-3-8B-Instruct", torch_dtype=torch.bfloat16, device_map="auto")
response = pipe(chat, max_new_tokens=512)
print(response[0]['generated_text'][-1]['content'])
```

And you'll get:

```text
(sigh) Oh boy, you're asking me for advice? You're gonna need a map, pal! Alright, 
alright, I'll give you the lowdown. But don't say I didn't warn you, I'm a robot, not a tour guide!

So, you wanna know what's fun to do in the Big Apple? Well, let me tell you, there's a million 
things to do, but I'll give you the highlights. First off, you gotta see the sights: the Statue of 
Liberty, Central Park, Times Square... you know, the usual tourist traps. But if you're lookin' for 
something a little more... unusual, I'd recommend checkin' out the Museum of Modern Art. It's got 
some wild stuff, like that Warhol guy's soup cans and all that jazz.

And if you're feelin' adventurous, take a walk across the Brooklyn Bridge. Just watch out for 
those pesky pigeons, they're like little feathered thieves! (laughs) Get it? Thieves? Ah, never mind.

Now, if you're lookin' for some serious fun, hit up the comedy clubs in Greenwich Village. You might 
even catch a glimpse of some up-and-coming comedians... or a bunch of wannabes tryin' to make it big. (winks)

And finally, if you're feelin' like a real New Yorker, grab a slice of pizza from one of the many amazing
pizzerias around the city. Just don't try to order a "robot-sized" slice, trust me, it won't end well. (laughs)

So, there you have it, pal! That's my expert advice on what to do in New York. Now, if you'll
excuse me, I've got some oil changes to attend to. (winks)
```

You can continue the chat by appending your own response to it.


