# embedding_metadata

key: _node_content
string_value:
{"id_": "25c9a4fb-136b-42a3-a9ed-821772ffb7b8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e9f71c25-71f0-4b18-be11-c33b6fcecac6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "d7c025cad61c09f008bde35b4154c5dc0fe13b268c8411677421925cfff7b247", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0ccfd617-bbe4-461f-b2b5-0ed978f399b9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "8fdb84429b0008c2c1a6c7a630578fa84906a696a8b73f0dba3c53d89b5ded7b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8378f642-196f-45e0-8a21-62b1541be889", "node_type": "1", "metadata": {}, "hash": "a89cde1b24bc423d65409d5a5c2fc2ce3cbb0bf91e5584574f5dd69db0942bce", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23430, "end_char_idx": 26144, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, in assisted decoding, reducing the temperature may help improve the latency.

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed
>>> set_seed(42)  # For reproducibility

>>> prompt = "Alice and Bob"
>>> checkpoint = "EleutherAI/pythia-1.4b-deduped"
>>> assistant_checkpoint = "EleutherAI/pythia-160m-deduped"

>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> inputs = tokenizer(prompt, return_tensors="pt")

>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)
>>> assistant_model = AutoModelForCausalLM.from_pretrained(assistant_checkpoint)
>>> outputs = model.generate(**inputs, assistant_model=assistant_model, do_sample=True, temperature=0.5)
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['Alice and Bob, a couple of friends of mine, who are both in the same office as']
```

Alternativelly, you can also set the `prompt_lookup_num_tokens` to trigger n-gram based assisted decoding, as opposed
to model based assisted decoding. You can read more about it [here](https://twitter.com/joao_gante/status/1747322413006643259).
### DoLa Decoding

**D**ecoding by C**o**ntrasting **La**yers (DoLa) is a contrastive decoding strategy to improve the factuality and reduce the
hallucinations of LLMs, as described in this paper of ICLR 2024 [DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models](https://arxiv.org/abs/2309.03883).

DoLa is achieved by contrasting the differences in logits obtained from final
layers versus earlier layers, thus amplify the factual knowledge localized to particular part of transformer layers.

Do the following two steps to activate DoLa decoding when calling the `model.generate` function:
1. Set the `dola_layers` argument, which can be either a string or a list of integers.
    - If set to a string, it can be one of `low`, `high`.
    - If set to a list of integers, it should be a list of layer indices between 0 and the total number of layers in the model. The 0-th layer is word embedding, and the 1st layer is the first transformer layer, and so on.
2. Set `repetition_penalty = 1.2` is suggested to reduce repetition in DoLa decoding.

See the following examples for DoLa decoding with the 32-layer LLaMA-7B model.

```python
>>> from transformers import AutoTokenizer, AutoModelForCausalLM, set_seed
>>> import torch

>>> tokenizer = AutoTokenizer.from_pretrained("huggyllama/llama-7b")
>>> model = AutoModelForCausalLM.from_pretrained("huggyllama/llama-7b", torch_dtype=torch.float16)
>>> device = 'cuda' if torch.cuda.is_available() else 'cpu'
>>> model.to(device)
>>> set_seed(42)

>>> text = "On what date was the Declaration of Independence officially signed?"


key: _node_content
string_value:
{"id_": "8378f642-196f-45e0-8a21-62b1541be889", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e9f71c25-71f0-4b18-be11-c33b6fcecac6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "d7c025cad61c09f008bde35b4154c5dc0fe13b268c8411677421925cfff7b247", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "25c9a4fb-136b-42a3-a9ed-821772ffb7b8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "162dda793dd91dd3167935700bf88888eafb965025b8a703c3205dfac0660a13", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2689e292-1167-452c-9647-1fffc98d9c23", "node_type": "1", "metadata": {}, "hash": "0a672838ba810f7e6bf629f46d9f8e8ac75a6332b442431bf7cbf99cdea803c9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24866, "end_char_idx": 27935, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
DoLa is achieved by contrasting the differences in logits obtained from final
layers versus earlier layers, thus amplify the factual knowledge localized to particular part of transformer layers.

Do the following two steps to activate DoLa decoding when calling the `model.generate` function:
1. Set the `dola_layers` argument, which can be either a string or a list of integers.
    - If set to a string, it can be one of `low`, `high`.
    - If set to a list of integers, it should be a list of layer indices between 0 and the total number of layers in the model. The 0-th layer is word embedding, and the 1st layer is the first transformer layer, and so on.
2. Set `repetition_penalty = 1.2` is suggested to reduce repetition in DoLa decoding.

See the following examples for DoLa decoding with the 32-layer LLaMA-7B model.

```python
>>> from transformers import AutoTokenizer, AutoModelForCausalLM, set_seed
>>> import torch

>>> tokenizer = AutoTokenizer.from_pretrained("huggyllama/llama-7b")
>>> model = AutoModelForCausalLM.from_pretrained("huggyllama/llama-7b", torch_dtype=torch.float16)
>>> device = 'cuda' if torch.cuda.is_available() else 'cpu'
>>> model.to(device)
>>> set_seed(42)

>>> text = "On what date was the Declaration of Independence officially signed?"
>>> inputs = tokenizer(text, return_tensors="pt").to(device)

# Vanilla greddy decoding
>>> vanilla_output = model.generate(**inputs, do_sample=False, max_new_tokens=50)
>>> tokenizer.batch_decode(vanilla_output[:, inputs.input_ids.shape[-1]:], skip_special_tokens=True)
['\nThe Declaration of Independence was signed on July 4, 1776.\nWhat was the date of the signing of the Declaration of Independence?\nThe Declaration of Independence was signed on July 4,']

# DoLa decoding with contrasting higher part of layers (layers 16,18,...,30)
>>> dola_high_output = model.generate(**inputs, do_sample=False, max_new_tokens=50, dola_layers='high')
>>> tokenizer.batch_decode(dola_high_output[:, inputs.input_ids.shape[-1]:], skip_special_tokens=True)
['\nJuly 4, 1776, when the Continental Congress voted to separate from Great Britain. The 56 delegates to the Continental Congress signed the Declaration on August 2, 1776.']

# DoLa decoding with contrasting specific layers (layers 28 and 30)
>>> dola_custom_output = model.generate(**inputs, do_sample=False, max_new_tokens=50, dola_layers=[28,30], repetition_penalty=1.2)
>>> tokenizer.batch_decode(dola_custom_output[:, inputs.input_ids.shape[-1]:], skip_special_tokens=True)
['\nIt was officially signed on 2 August 1776, when 56 members of the Second Continental Congress, representing the original 13 American colonies, voted unanimously for the resolution for independence. The 2']
```

#### Understanding the `dola_layers` argument

`dola_layers` stands for the candidate layers in premature layer selection, as described in the DoLa paper. The selected premature layer will be contrasted with the final layer.

Setting `dola_layers` to `'low'` or `'high'` will select the lower or higher part of the layers to contrast, respectively.


key: _node_content
string_value:
{"id_": "2689e292-1167-452c-9647-1fffc98d9c23", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e9f71c25-71f0-4b18-be11-c33b6fcecac6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "d7c025cad61c09f008bde35b4154c5dc0fe13b268c8411677421925cfff7b247", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8378f642-196f-45e0-8a21-62b1541be889", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "8d06c62dd38aefda03df00e79d426f8e8ecbe16ac808436db4d2f9befd5d4d5e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 26978, "end_char_idx": 29004, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The 56 delegates to the Continental Congress signed the Declaration on August 2, 1776.']

# DoLa decoding with contrasting specific layers (layers 28 and 30)
>>> dola_custom_output = model.generate(**inputs, do_sample=False, max_new_tokens=50, dola_layers=[28,30], repetition_penalty=1.2)
>>> tokenizer.batch_decode(dola_custom_output[:, inputs.input_ids.shape[-1]:], skip_special_tokens=True)
['\nIt was officially signed on 2 August 1776, when 56 members of the Second Continental Congress, representing the original 13 American colonies, voted unanimously for the resolution for independence. The 2']
```

#### Understanding the `dola_layers` argument

`dola_layers` stands for the candidate layers in premature layer selection, as described in the DoLa paper. The selected premature layer will be contrasted with the final layer.

Setting `dola_layers` to `'low'` or `'high'` will select the lower or higher part of the layers to contrast, respectively.
- For `N`-layer models with `N <= 40` layers, the layers of `range(0, N // 2, 2)` and `range(N // 2, N, 2)` are used for `'low'` and `'high'` layers, respectively.
- For models with `N > 40` layers, the layers of `range(0, 20, 2)` and `range(N - 20, N, 2)` are used for `'low'` and `'high'` layers, respectively.
- If the model has tied word embeddings, we skip the word embeddings (0-th) layer and start from the 2nd layer, as the early exit from word embeddings will become identity function.
- Set the `dola_layers` to a list of integers for layer indices to contrast manually specified layers. For example, setting `dola_layers=[28,30]` will contrast the final layer (32-th layer) with the 28-th and 30-th layers.

The paper suggested that contrasting `'high'` layers to improve short-answer tasks like TruthfulQA, and contrasting `'low'` layers to improve all the other long-answer reasoning tasks, such as GSM8K, StrategyQA, FACTOR, and VicunaQA. Applying DoLa to smaller models like GPT-2 is not recommended, as the results shown in the Appendix N of the paper.


key: _node_content
string_value:
{"id_": "c4b8bda6-39f9-43b4-8848-22af017b5f4d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fb71e9b1-65b0-49d0-8378-be659acc24b0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "909eeb49-fb97-43be-9249-ecab67c2beb7", "node_type": "1", "metadata": {}, "hash": "544bbdf4780eebd34942a774eafb30a2ddac2e3559a77cfab2cbf38622b332d9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2682, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Glossary

This glossary defines general machine learning and 🤗 Transformers terms to help you better understand the
documentation.

## A

### attention mask

The attention mask is an optional argument used when batching sequences together.

<Youtube id="M6adb1j2jPI"/>

This argument indicates to the model which tokens should be attended to, and which should not.

For example, consider these two sequences:

```python
>>> from transformers import BertTokenizer

>>> tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-cased")

>>> sequence_a = "This is a short sequence."
>>> sequence_b = "This is a rather long sequence. It is at least longer than the sequence A."

>>> encoded_sequence_a = tokenizer(sequence_a)["input_ids"]
>>> encoded_sequence_b = tokenizer(sequence_b)["input_ids"]
```

The encoded versions have different lengths:

```python
>>> len(encoded_sequence_a), len(encoded_sequence_b)
(8, 19)
```

Therefore, we can't put them together in the same tensor as-is. The first sequence needs to be padded up to the length
of the second one, or the second one needs to be truncated down to the length of the first one.

In the first case, the list of IDs will be extended by the padding indices. We can pass a list to the tokenizer and ask
it to pad like this:

```python
>>> padded_sequences = tokenizer([sequence_a, sequence_b], padding=True)
```

We can see that 0s have been added on the right of the first sentence to make it the same length as the second one:

```python
>>> padded_sequences["input_ids"]
[[101, 1188, 1110, 170, 1603, 4954, 119, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [101, 1188, 1110, 170, 1897, 1263, 4954, 119, 1135, 1110, 1120, 1655, 2039, 1190, 1103, 4954, 138, 119, 102]]
```

This can then be converted into a tensor in PyTorch or TensorFlow. The attention mask is a binary tensor indicating the
position of the padded indices so that the model does not attend to them. For the [`BertTokenizer`], `1` indicates a
value that should be attended to, while `0` indicates a padded value. This attention mask is in the dictionary returned
by the tokenizer under the key "attention_mask":

```python
>>> padded_sequences["attention_mask"]
[[1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]
```

### autoencoding models

See [encoder models](#encoder-models) and [masked language modeling](#masked-language-modeling-mlm)

### autoregressive models

See [causal language modeling](#causal-language-modeling) and [decoder models](#decoder-models)

## B

### backbone

The backbone is the network (embeddings and layers) that outputs the raw hidden states or features.


key: _node_content
string_value:
{"id_": "909eeb49-fb97-43be-9249-ecab67c2beb7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fb71e9b1-65b0-49d0-8378-be659acc24b0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c4b8bda6-39f9-43b4-8848-22af017b5f4d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "d50d2bbc0fa78ec35e75e159d92ad0cf1c169b4269062722a0b5fb2da5761eca", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f1c3d6a2-67f6-4b04-a519-96978ef0caad", "node_type": "1", "metadata": {}, "hash": "724e55a11efb5d580233e2ee07613afcf8e36cc3d6d11d6dfe70f31fd3d954e8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1800, "end_char_idx": 4934, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The attention mask is a binary tensor indicating the
position of the padded indices so that the model does not attend to them. For the [`BertTokenizer`], `1` indicates a
value that should be attended to, while `0` indicates a padded value. This attention mask is in the dictionary returned
by the tokenizer under the key "attention_mask":

```python
>>> padded_sequences["attention_mask"]
[[1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]
```

### autoencoding models

See [encoder models](#encoder-models) and [masked language modeling](#masked-language-modeling-mlm)

### autoregressive models

See [causal language modeling](#causal-language-modeling) and [decoder models](#decoder-models)

## B

### backbone

The backbone is the network (embeddings and layers) that outputs the raw hidden states or features. It is usually connected to a [head](#head) which accepts the features as its input to make a prediction. For example, [`ViTModel`] is a backbone without a specific head on top. Other models can also use [`VitModel`] as a backbone such as [DPT](model_doc/dpt).

## C

### causal language modeling

A pretraining task where the model reads the texts in order and has to predict the next word. It's usually done by
reading the whole sentence but using a mask inside the model to hide the future tokens at a certain timestep.

### channel

Color images are made up of some combination of values in three channels: red, green, and blue (RGB) and grayscale images only have one channel. In 🤗 Transformers, the channel can be the first or last dimension of an image's tensor: [`n_channels`, `height`, `width`] or [`height`, `width`, `n_channels`].

### connectionist temporal classification (CTC)

An algorithm which allows a model to learn without knowing exactly how the input and output are aligned; CTC calculates the distribution of all possible outputs for a given input and chooses the most likely output from it. CTC is commonly used in speech recognition tasks because speech doesn't always cleanly align with the transcript for a variety of reasons such as a speaker's different speech rates.

### convolution

A type of layer in a neural network where the input matrix is multiplied element-wise by a smaller matrix (kernel or filter) and the values are summed up in a new matrix. This is known as a convolutional operation which is repeated over the entire input matrix. Each operation is applied to a different segment of the input matrix. Convolutional neural networks (CNNs) are commonly used in computer vision.

## D

### DataParallel (DP)

Parallelism technique for training on multiple GPUs where the same setup is replicated multiple times, with each instance 
receiving a distinct data slice. The processing is done in parallel and all setups are synchronized at the end of each training step.

Learn more about how DataParallel works [here](perf_train_gpu_many#dataparallel-vs-distributeddataparallel).

### decoder input IDs

This input is specific to encoder-decoder models, and contains the input IDs that will be fed to the decoder.


key: _node_content
string_value:
{"id_": "f1c3d6a2-67f6-4b04-a519-96978ef0caad", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fb71e9b1-65b0-49d0-8378-be659acc24b0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "909eeb49-fb97-43be-9249-ecab67c2beb7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "8d3dbc685199699a6a99795d3e4200189233ee685ef8a2330b7c126480d34e90", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ef754210-08ae-4a63-8495-849b847caf68", "node_type": "1", "metadata": {}, "hash": "d9a6c8657631fc50e8689ef751a97b0bb3966f8c4fdec5d1e8574e7595cf4a95", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3074, "end_char_idx": 6770, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
It's usually done by
reading the whole sentence but using a mask inside the model to hide the future tokens at a certain timestep.

### channel

Color images are made up of some combination of values in three channels: red, green, and blue (RGB) and grayscale images only have one channel. In 🤗 Transformers, the channel can be the first or last dimension of an image's tensor: [`n_channels`, `height`, `width`] or [`height`, `width`, `n_channels`].

### connectionist temporal classification (CTC)

An algorithm which allows a model to learn without knowing exactly how the input and output are aligned; CTC calculates the distribution of all possible outputs for a given input and chooses the most likely output from it. CTC is commonly used in speech recognition tasks because speech doesn't always cleanly align with the transcript for a variety of reasons such as a speaker's different speech rates.

### convolution

A type of layer in a neural network where the input matrix is multiplied element-wise by a smaller matrix (kernel or filter) and the values are summed up in a new matrix. This is known as a convolutional operation which is repeated over the entire input matrix. Each operation is applied to a different segment of the input matrix. Convolutional neural networks (CNNs) are commonly used in computer vision.

## D

### DataParallel (DP)

Parallelism technique for training on multiple GPUs where the same setup is replicated multiple times, with each instance 
receiving a distinct data slice. The processing is done in parallel and all setups are synchronized at the end of each training step.

Learn more about how DataParallel works [here](perf_train_gpu_many#dataparallel-vs-distributeddataparallel).

### decoder input IDs

This input is specific to encoder-decoder models, and contains the input IDs that will be fed to the decoder. These
inputs should be used for sequence to sequence tasks, such as translation or summarization, and are usually built in a
way specific to each model.

Most encoder-decoder models (BART, T5) create their `decoder_input_ids` on their own from the `labels`. In such models,
passing the `labels` is the preferred way to handle training.

Please check each model's docs to see how they handle these input IDs for sequence to sequence training.

### decoder models

Also referred to as autoregressive models, decoder models involve a pretraining task (called causal language modeling) where the model reads the texts in order and has to predict the next word. It's usually done by
reading the whole sentence with a mask to hide future tokens at a certain timestep.

<Youtube id="d_ixlCubqQw"/>

### deep learning (DL)

Machine learning algorithms which use neural networks with several layers.

## E

### encoder models

Also known as autoencoding models, encoder models take an input (such as text or images) and transform them into a condensed numerical representation called an embedding. Oftentimes, encoder models are pretrained using techniques like [masked language modeling](#masked-language-modeling-mlm), which masks parts of the input sequence and forces the model to create more meaningful representations.

<Youtube id="H39Z_720T5s"/>

## F

### feature extraction

The process of selecting and transforming raw data into a set of features that are more informative and useful for machine learning algorithms. Some examples of feature extraction include transforming raw text into word embeddings and extracting important features such as edges or shapes from image/video data.

### feed forward chunking

In each residual attention block in transformers the self-attention layer is usually followed by 2 feed forward layers.


key: _node_content
string_value:
{"id_": "ef754210-08ae-4a63-8495-849b847caf68", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fb71e9b1-65b0-49d0-8378-be659acc24b0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f1c3d6a2-67f6-4b04-a519-96978ef0caad", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "7c30f61cefcc05b2f9e95617a1d2ea6e59d939daf84116d46f81d95e4065ec80", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fac617bb-cb3a-421a-b177-07a2d9c7e996", "node_type": "1", "metadata": {}, "hash": "15a043b5d160ad96a20bb0c6851e88fdbfa41a9f0652116c56d57e630b2ff971", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4935, "end_char_idx": 8450, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
These
inputs should be used for sequence to sequence tasks, such as translation or summarization, and are usually built in a
way specific to each model.

Most encoder-decoder models (BART, T5) create their `decoder_input_ids` on their own from the `labels`. In such models,
passing the `labels` is the preferred way to handle training.

Please check each model's docs to see how they handle these input IDs for sequence to sequence training.

### decoder models

Also referred to as autoregressive models, decoder models involve a pretraining task (called causal language modeling) where the model reads the texts in order and has to predict the next word. It's usually done by
reading the whole sentence with a mask to hide future tokens at a certain timestep.

<Youtube id="d_ixlCubqQw"/>

### deep learning (DL)

Machine learning algorithms which use neural networks with several layers.

## E

### encoder models

Also known as autoencoding models, encoder models take an input (such as text or images) and transform them into a condensed numerical representation called an embedding. Oftentimes, encoder models are pretrained using techniques like [masked language modeling](#masked-language-modeling-mlm), which masks parts of the input sequence and forces the model to create more meaningful representations.

<Youtube id="H39Z_720T5s"/>

## F

### feature extraction

The process of selecting and transforming raw data into a set of features that are more informative and useful for machine learning algorithms. Some examples of feature extraction include transforming raw text into word embeddings and extracting important features such as edges or shapes from image/video data.

### feed forward chunking

In each residual attention block in transformers the self-attention layer is usually followed by 2 feed forward layers.
The intermediate embedding size of the feed forward layers is often bigger than the hidden size of the model (e.g., for
`google-bert/bert-base-uncased`).

For an input of size `[batch_size, sequence_length]`, the memory required to store the intermediate feed forward
embeddings `[batch_size, sequence_length, config.intermediate_size]` can account for a large fraction of the memory
use. The authors of [Reformer: The Efficient Transformer](https://arxiv.org/abs/2001.04451) noticed that since the
computation is independent of the `sequence_length` dimension, it is mathematically equivalent to compute the output
embeddings of both feed forward layers `[batch_size, config.hidden_size]_0, ..., [batch_size, config.hidden_size]_n`
individually and concat them afterward to `[batch_size, sequence_length, config.hidden_size]` with `n = sequence_length`, which trades increased computation time against reduced memory use, but yields a mathematically
**equivalent** result.

For models employing the function [`apply_chunking_to_forward`], the `chunk_size` defines the number of output
embeddings that are computed in parallel and thus defines the trade-off between memory and time complexity. If
`chunk_size` is set to 0, no feed forward chunking is done.

### finetuned models

Finetuning is a form of transfer learning which involves taking a pretrained model, freezing its weights, and replacing the output layer with a newly added [model head](#head). The model head is trained on your target dataset.

See the [Fine-tune a pretrained model](https://huggingface.co/docs/transformers/training) tutorial for more details, and learn how to fine-tune models with 🤗 Transformers.


key: _node_content
string_value:
{"id_": "fac617bb-cb3a-421a-b177-07a2d9c7e996", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fb71e9b1-65b0-49d0-8378-be659acc24b0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ef754210-08ae-4a63-8495-849b847caf68", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "ec2ac1d172a8f79577956475146d6b37a9fff6f96f1f772ec492c91611d86553", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c86ae43b-42b2-4584-b3bd-bce104778b18", "node_type": "1", "metadata": {}, "hash": "6df752eec3d8dbb429d09288e9786890f9091a8faa09054fa713e908fc9c8ad4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6771, "end_char_idx": 9956, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The intermediate embedding size of the feed forward layers is often bigger than the hidden size of the model (e.g., for
`google-bert/bert-base-uncased`).

For an input of size `[batch_size, sequence_length]`, the memory required to store the intermediate feed forward
embeddings `[batch_size, sequence_length, config.intermediate_size]` can account for a large fraction of the memory
use. The authors of [Reformer: The Efficient Transformer](https://arxiv.org/abs/2001.04451) noticed that since the
computation is independent of the `sequence_length` dimension, it is mathematically equivalent to compute the output
embeddings of both feed forward layers `[batch_size, config.hidden_size]_0, ..., [batch_size, config.hidden_size]_n`
individually and concat them afterward to `[batch_size, sequence_length, config.hidden_size]` with `n = sequence_length`, which trades increased computation time against reduced memory use, but yields a mathematically
**equivalent** result.

For models employing the function [`apply_chunking_to_forward`], the `chunk_size` defines the number of output
embeddings that are computed in parallel and thus defines the trade-off between memory and time complexity. If
`chunk_size` is set to 0, no feed forward chunking is done.

### finetuned models

Finetuning is a form of transfer learning which involves taking a pretrained model, freezing its weights, and replacing the output layer with a newly added [model head](#head). The model head is trained on your target dataset.

See the [Fine-tune a pretrained model](https://huggingface.co/docs/transformers/training) tutorial for more details, and learn how to fine-tune models with 🤗 Transformers.

## H

### head

The model head refers to the last layer of a neural network that accepts the raw hidden states and projects them onto a different dimension. There is a different model head for each task. For example:

  * [`GPT2ForSequenceClassification`] is a sequence classification head - a linear layer - on top of the base [`GPT2Model`].
  * [`ViTForImageClassification`] is an image classification head - a linear layer on top of the final hidden state of the `CLS` token - on top of the base [`ViTModel`].
  * [`Wav2Vec2ForCTC`] is a language modeling head with [CTC](#connectionist-temporal-classification-ctc) on top of the base [`Wav2Vec2Model`].

## I

### image patch

Vision-based Transformers models split an image into smaller patches which are linearly embedded, and then passed as a sequence to the model. You can find the `patch_size` - or resolution - of the model in its configuration.

### inference

Inference is the process of evaluating a model on new data after training is complete. See the [Pipeline for inference](https://huggingface.co/docs/transformers/pipeline_tutorial) tutorial to learn how to perform inference with 🤗 Transformers.

### input IDs

The input ids are often the only required parameters to be passed to the model as input. They are token indices,
numerical representations of tokens building the sequences that will be used as input by the model.

<Youtube id="VFp38yj8h3A"/>

Each tokenizer works differently but the underlying mechanism remains the same.


key: _node_content
string_value:
{"id_": "c86ae43b-42b2-4584-b3bd-bce104778b18", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fb71e9b1-65b0-49d0-8378-be659acc24b0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fac617bb-cb3a-421a-b177-07a2d9c7e996", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "dd8f820a11d1195ab3b7a90b535fbaaf376ae407cf7137254cc15ec4c4d8d580", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e3874ee7-9a7a-4092-ad87-7020e5ab6a96", "node_type": "1", "metadata": {}, "hash": "eca63cd2b900dfb0620eaa2d0865e268c5a14fbecf1d51dc0d8f52eddcad1941", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8279, "end_char_idx": 11256, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
See the [Fine-tune a pretrained model](https://huggingface.co/docs/transformers/training) tutorial for more details, and learn how to fine-tune models with 🤗 Transformers.

## H

### head

The model head refers to the last layer of a neural network that accepts the raw hidden states and projects them onto a different dimension. There is a different model head for each task. For example:

  * [`GPT2ForSequenceClassification`] is a sequence classification head - a linear layer - on top of the base [`GPT2Model`].
  * [`ViTForImageClassification`] is an image classification head - a linear layer on top of the final hidden state of the `CLS` token - on top of the base [`ViTModel`].
  * [`Wav2Vec2ForCTC`] is a language modeling head with [CTC](#connectionist-temporal-classification-ctc) on top of the base [`Wav2Vec2Model`].

## I

### image patch

Vision-based Transformers models split an image into smaller patches which are linearly embedded, and then passed as a sequence to the model. You can find the `patch_size` - or resolution - of the model in its configuration.

### inference

Inference is the process of evaluating a model on new data after training is complete. See the [Pipeline for inference](https://huggingface.co/docs/transformers/pipeline_tutorial) tutorial to learn how to perform inference with 🤗 Transformers.

### input IDs

The input ids are often the only required parameters to be passed to the model as input. They are token indices,
numerical representations of tokens building the sequences that will be used as input by the model.

<Youtube id="VFp38yj8h3A"/>

Each tokenizer works differently but the underlying mechanism remains the same. Here's an example using the BERT
tokenizer, which is a [WordPiece](https://arxiv.org/pdf/1609.08144.pdf) tokenizer:

```python
>>> from transformers import BertTokenizer

>>> tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-cased")

>>> sequence = "A Titan RTX has 24GB of VRAM"
```

The tokenizer takes care of splitting the sequence into tokens available in the tokenizer vocabulary.

```python
>>> tokenized_sequence = tokenizer.tokenize(sequence)
```

The tokens are either words or subwords. Here for instance, "VRAM" wasn't in the model vocabulary, so it's been split
in "V", "RA" and "M". To indicate those tokens are not separate words but parts of the same word, a double-hash prefix
is added for "RA" and "M":

```python
>>> print(tokenized_sequence)
['A', 'Titan', 'R', '##T', '##X', 'has', '24', '##GB', 'of', 'V', '##RA', '##M']
```

These tokens can then be converted into IDs which are understandable by the model. This can be done by directly feeding the sentence to the tokenizer, which leverages the Rust implementation of [🤗 Tokenizers](https://github.com/huggingface/tokenizers) for peak performance.

```python
>>> inputs = tokenizer(sequence)
```

The tokenizer returns a dictionary with all the arguments necessary for its corresponding model to work properly.


key: _node_content
string_value:
{"id_": "e3874ee7-9a7a-4092-ad87-7020e5ab6a96", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fb71e9b1-65b0-49d0-8378-be659acc24b0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c86ae43b-42b2-4584-b3bd-bce104778b18", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "df0e83b77324df0369bf3a0e8366210d4970df17e05ef3b7f78e5312a29593ab", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8c268d0f-0407-4456-bd9f-cd9f5bcfd915", "node_type": "1", "metadata": {}, "hash": "fdac5ab48bed5bcc837fabca6cdd2170943cf020784648ccf6a7fd9936b481c4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9619, "end_char_idx": 12723, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### input IDs

The input ids are often the only required parameters to be passed to the model as input. They are token indices,
numerical representations of tokens building the sequences that will be used as input by the model.

<Youtube id="VFp38yj8h3A"/>

Each tokenizer works differently but the underlying mechanism remains the same. Here's an example using the BERT
tokenizer, which is a [WordPiece](https://arxiv.org/pdf/1609.08144.pdf) tokenizer:

```python
>>> from transformers import BertTokenizer

>>> tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-cased")

>>> sequence = "A Titan RTX has 24GB of VRAM"
```

The tokenizer takes care of splitting the sequence into tokens available in the tokenizer vocabulary.

```python
>>> tokenized_sequence = tokenizer.tokenize(sequence)
```

The tokens are either words or subwords. Here for instance, "VRAM" wasn't in the model vocabulary, so it's been split
in "V", "RA" and "M". To indicate those tokens are not separate words but parts of the same word, a double-hash prefix
is added for "RA" and "M":

```python
>>> print(tokenized_sequence)
['A', 'Titan', 'R', '##T', '##X', 'has', '24', '##GB', 'of', 'V', '##RA', '##M']
```

These tokens can then be converted into IDs which are understandable by the model. This can be done by directly feeding the sentence to the tokenizer, which leverages the Rust implementation of [🤗 Tokenizers](https://github.com/huggingface/tokenizers) for peak performance.

```python
>>> inputs = tokenizer(sequence)
```

The tokenizer returns a dictionary with all the arguments necessary for its corresponding model to work properly. The
token indices are under the key `input_ids`:

```python
>>> encoded_sequence = inputs["input_ids"]
>>> print(encoded_sequence)
[101, 138, 18696, 155, 1942, 3190, 1144, 1572, 13745, 1104, 159, 9664, 2107, 102]
```

Note that the tokenizer automatically adds "special tokens" (if the associated model relies on them) which are special
IDs the model sometimes uses.

If we decode the previous sequence of ids,

```python
>>> decoded_sequence = tokenizer.decode(encoded_sequence)
```

we will see

```python
>>> print(decoded_sequence)
[CLS] A Titan RTX has 24GB of VRAM [SEP]
```

because this is the way a [`BertModel`] is going to expect its inputs.

## L

### labels

The labels are an optional argument which can be passed in order for the model to compute the loss itself. These labels
should be the expected prediction of the model: it will use the standard loss in order to compute the loss between its
predictions and the expected value (the label).

These labels are different according to the model head, for example:

- For sequence classification models, ([`BertForSequenceClassification`]), the model expects a tensor of dimension
  `(batch_size)` with each value of the batch corresponding to the expected label of the entire sequence.
- For token classification models, ([`BertForTokenClassification`]), the model expects a tensor of dimension
  `(batch_size, seq_length)` with each value corresponding to the expected label of each individual token.


key: _node_content
string_value:
{"id_": "8c268d0f-0407-4456-bd9f-cd9f5bcfd915", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fb71e9b1-65b0-49d0-8378-be659acc24b0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e3874ee7-9a7a-4092-ad87-7020e5ab6a96", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "1933c18c61867bbce6edd33e542e7fb8ac5bb0ecfbd7a4307a50265113b15e7a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c70cb79d-3758-4fde-8242-eec030a277a9", "node_type": "1", "metadata": {}, "hash": "afe19266d1444271f74feff15a3fa592247d9bf7376625116a291f1313006444", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11095, "end_char_idx": 14472, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
>>> inputs = tokenizer(sequence)
```

The tokenizer returns a dictionary with all the arguments necessary for its corresponding model to work properly. The
token indices are under the key `input_ids`:

```python
>>> encoded_sequence = inputs["input_ids"]
>>> print(encoded_sequence)
[101, 138, 18696, 155, 1942, 3190, 1144, 1572, 13745, 1104, 159, 9664, 2107, 102]
```

Note that the tokenizer automatically adds "special tokens" (if the associated model relies on them) which are special
IDs the model sometimes uses.

If we decode the previous sequence of ids,

```python
>>> decoded_sequence = tokenizer.decode(encoded_sequence)
```

we will see

```python
>>> print(decoded_sequence)
[CLS] A Titan RTX has 24GB of VRAM [SEP]
```

because this is the way a [`BertModel`] is going to expect its inputs.

## L

### labels

The labels are an optional argument which can be passed in order for the model to compute the loss itself. These labels
should be the expected prediction of the model: it will use the standard loss in order to compute the loss between its
predictions and the expected value (the label).

These labels are different according to the model head, for example:

- For sequence classification models, ([`BertForSequenceClassification`]), the model expects a tensor of dimension
  `(batch_size)` with each value of the batch corresponding to the expected label of the entire sequence.
- For token classification models, ([`BertForTokenClassification`]), the model expects a tensor of dimension
  `(batch_size, seq_length)` with each value corresponding to the expected label of each individual token.
- For masked language modeling, ([`BertForMaskedLM`]), the model expects a tensor of dimension `(batch_size,
  seq_length)` with each value corresponding to the expected label of each individual token: the labels being the token
  ID for the masked token, and values to be ignored for the rest (usually -100).
- For sequence to sequence tasks, ([`BartForConditionalGeneration`], [`MBartForConditionalGeneration`]), the model
  expects a tensor of dimension `(batch_size, tgt_seq_length)` with each value corresponding to the target sequences
  associated with each input sequence. During training, both BART and T5 will make the appropriate
  `decoder_input_ids` and decoder attention masks internally. They usually do not need to be supplied. This does not
  apply to models leveraging the Encoder-Decoder framework.
- For image classification models, ([`ViTForImageClassification`]), the model expects a tensor of dimension
  `(batch_size)` with each value of the batch corresponding to the expected label of each individual image.
- For semantic segmentation models, ([`SegformerForSemanticSegmentation`]), the model expects a tensor of dimension
  `(batch_size, height, width)` with each value of the batch corresponding to the expected label of each individual pixel.
- For object detection models, ([`DetrForObjectDetection`]), the model expects a list of dictionaries with a
  `class_labels` and `boxes` key where each value of the batch corresponds to the expected label and number of bounding boxes of each individual image.
- For automatic speech recognition models, ([`Wav2Vec2ForCTC`]), the model expects a tensor of dimension `(batch_size,
  target_length)` with each value corresponding to the expected label of each individual token.


key: _node_content
string_value:
{"id_": "c70cb79d-3758-4fde-8242-eec030a277a9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fb71e9b1-65b0-49d0-8378-be659acc24b0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8c268d0f-0407-4456-bd9f-cd9f5bcfd915", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "b76b06068b5d3a6828bbf3020e823848c23b7ada1dd1accfc406ca46bd0fe592", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3f843d6e-9299-49ce-adf5-1be27337d98c", "node_type": "1", "metadata": {}, "hash": "fc97bdac04d667f2f250bad3d94e1b66467153e4c48013d289ab0eb0399d1542", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12724, "end_char_idx": 16156, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- For masked language modeling, ([`BertForMaskedLM`]), the model expects a tensor of dimension `(batch_size,
  seq_length)` with each value corresponding to the expected label of each individual token: the labels being the token
  ID for the masked token, and values to be ignored for the rest (usually -100).
- For sequence to sequence tasks, ([`BartForConditionalGeneration`], [`MBartForConditionalGeneration`]), the model
  expects a tensor of dimension `(batch_size, tgt_seq_length)` with each value corresponding to the target sequences
  associated with each input sequence. During training, both BART and T5 will make the appropriate
  `decoder_input_ids` and decoder attention masks internally. They usually do not need to be supplied. This does not
  apply to models leveraging the Encoder-Decoder framework.
- For image classification models, ([`ViTForImageClassification`]), the model expects a tensor of dimension
  `(batch_size)` with each value of the batch corresponding to the expected label of each individual image.
- For semantic segmentation models, ([`SegformerForSemanticSegmentation`]), the model expects a tensor of dimension
  `(batch_size, height, width)` with each value of the batch corresponding to the expected label of each individual pixel.
- For object detection models, ([`DetrForObjectDetection`]), the model expects a list of dictionaries with a
  `class_labels` and `boxes` key where each value of the batch corresponds to the expected label and number of bounding boxes of each individual image.
- For automatic speech recognition models, ([`Wav2Vec2ForCTC`]), the model expects a tensor of dimension `(batch_size,
  target_length)` with each value corresponding to the expected label of each individual token.
  
<Tip>

Each model's labels may be different, so be sure to always check the documentation of each model for more information
about their specific labels!

</Tip>

The base models ([`BertModel`]) do not accept labels, as these are the base transformer models, simply outputting
features.

### large language models (LLM)

A generic term that refers to transformer language models (GPT-3, BLOOM, OPT) that were trained on a large quantity of data. These models also tend to have a large number of learnable parameters (e.g. 175 billion for GPT-3).

## M

### masked language modeling (MLM)

A pretraining task where the model sees a corrupted version of the texts, usually done by
masking some tokens randomly, and has to predict the original text.

### multimodal

A task that combines texts with another kind of inputs (for instance images).

## N

### Natural language generation (NLG)

All tasks related to generating text (for instance, [Write With Transformers](https://transformer.huggingface.co/), translation).

### Natural language processing (NLP)

A generic way to say "deal with texts".

### Natural language understanding (NLU)

All tasks related to understanding what is in a text (for instance classifying the
whole text, individual words).

## P

### pipeline

A pipeline in 🤗 Transformers is an abstraction referring to a series of steps that are executed in a specific order to preprocess and transform data and return a prediction from a model. Some example stages found in a pipeline might be data preprocessing, feature extraction, and normalization.

For more details, see [Pipelines for inference](https://huggingface.co/docs/transformers/pipeline_tutorial).


key: _node_content
string_value:
{"id_": "3f843d6e-9299-49ce-adf5-1be27337d98c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fb71e9b1-65b0-49d0-8378-be659acc24b0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c70cb79d-3758-4fde-8242-eec030a277a9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "299fc7f9cfe3d7d6eac8eb834ae577d1bab5dde5f15b84184221376cf1eeccc9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e46ee55c-130f-41f6-b671-0203de499c0c", "node_type": "1", "metadata": {}, "hash": "9087e322d1352cb5d100a624420de265143a8cc5cd06d843cebdbe240a9b3f75", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14476, "end_char_idx": 18172, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<Tip>

Each model's labels may be different, so be sure to always check the documentation of each model for more information
about their specific labels!

</Tip>

The base models ([`BertModel`]) do not accept labels, as these are the base transformer models, simply outputting
features.

### large language models (LLM)

A generic term that refers to transformer language models (GPT-3, BLOOM, OPT) that were trained on a large quantity of data. These models also tend to have a large number of learnable parameters (e.g. 175 billion for GPT-3).

## M

### masked language modeling (MLM)

A pretraining task where the model sees a corrupted version of the texts, usually done by
masking some tokens randomly, and has to predict the original text.

### multimodal

A task that combines texts with another kind of inputs (for instance images).

## N

### Natural language generation (NLG)

All tasks related to generating text (for instance, [Write With Transformers](https://transformer.huggingface.co/), translation).

### Natural language processing (NLP)

A generic way to say "deal with texts".

### Natural language understanding (NLU)

All tasks related to understanding what is in a text (for instance classifying the
whole text, individual words).

## P

### pipeline

A pipeline in 🤗 Transformers is an abstraction referring to a series of steps that are executed in a specific order to preprocess and transform data and return a prediction from a model. Some example stages found in a pipeline might be data preprocessing, feature extraction, and normalization.

For more details, see [Pipelines for inference](https://huggingface.co/docs/transformers/pipeline_tutorial).

### PipelineParallel (PP)

Parallelism technique in which the model is split up vertically (layer-level) across multiple GPUs, so that only one or 
several layers of the model are placed on a single GPU. Each GPU processes in parallel different stages of the pipeline 
and working on a small chunk of the batch. Learn more about how PipelineParallel works [here](perf_train_gpu_many#from-naive-model-parallelism-to-pipeline-parallelism).

### pixel values

A tensor of the numerical representations of an image that is passed to a model. The pixel values have a shape of [`batch_size`, `num_channels`, `height`, `width`], and are generated from an image processor.

### pooling

An operation that reduces a matrix into a smaller matrix, either by taking the maximum or average of the pooled dimension(s). Pooling layers are commonly found between convolutional layers to downsample the feature representation.

### position IDs

Contrary to RNNs that have the position of each token embedded within them, transformers are unaware of the position of
each token. Therefore, the position IDs (`position_ids`) are used by the model to identify each token's position in the
list of tokens.

They are an optional parameter. If no `position_ids` are passed to the model, the IDs are automatically created as
absolute positional embeddings.

Absolute positional embeddings are selected in the range `[0, config.max_position_embeddings - 1]`. Some models use
other types of positional embeddings, such as sinusoidal position embeddings or relative position embeddings.

### preprocessing

The task of preparing raw data into a format that can be easily consumed by machine learning models. For example, text is typically preprocessed by tokenization. To gain a better idea of what preprocessing looks like for other input types, check out the [Preprocess](https://huggingface.co/docs/transformers/preprocessing) tutorial.

### pretrained model

A model that has been pretrained on some data (for instance all of Wikipedia).


key: _node_content
string_value:
{"id_": "e46ee55c-130f-41f6-b671-0203de499c0c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fb71e9b1-65b0-49d0-8378-be659acc24b0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3f843d6e-9299-49ce-adf5-1be27337d98c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "0062f1c99fa507afd9a6d60f5a01084332677be9e09bb1cbb24894d2c0a6dfb5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "05d74600-f788-40d2-9971-54d0c7cc7cb9", "node_type": "1", "metadata": {}, "hash": "c3a7a4e15a3260f545c0a7a29a610fb2c510e14501dff3997f8930c8ac5ddd62", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16362, "end_char_idx": 19966, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Each GPU processes in parallel different stages of the pipeline 
and working on a small chunk of the batch. Learn more about how PipelineParallel works [here](perf_train_gpu_many#from-naive-model-parallelism-to-pipeline-parallelism).

### pixel values

A tensor of the numerical representations of an image that is passed to a model. The pixel values have a shape of [`batch_size`, `num_channels`, `height`, `width`], and are generated from an image processor.

### pooling

An operation that reduces a matrix into a smaller matrix, either by taking the maximum or average of the pooled dimension(s). Pooling layers are commonly found between convolutional layers to downsample the feature representation.

### position IDs

Contrary to RNNs that have the position of each token embedded within them, transformers are unaware of the position of
each token. Therefore, the position IDs (`position_ids`) are used by the model to identify each token's position in the
list of tokens.

They are an optional parameter. If no `position_ids` are passed to the model, the IDs are automatically created as
absolute positional embeddings.

Absolute positional embeddings are selected in the range `[0, config.max_position_embeddings - 1]`. Some models use
other types of positional embeddings, such as sinusoidal position embeddings or relative position embeddings.

### preprocessing

The task of preparing raw data into a format that can be easily consumed by machine learning models. For example, text is typically preprocessed by tokenization. To gain a better idea of what preprocessing looks like for other input types, check out the [Preprocess](https://huggingface.co/docs/transformers/preprocessing) tutorial.

### pretrained model

A model that has been pretrained on some data (for instance all of Wikipedia). Pretraining methods involve a
self-supervised objective, which can be reading the text and trying to predict the next word (see [causal language
modeling](#causal-language-modeling)) or masking some words and trying to predict them (see [masked language
modeling](#masked-language-modeling-mlm)). 

Speech and vision models have their own pretraining objectives. For example, Wav2Vec2 is a speech model pretrained on a contrastive task which requires the model to identify the "true" speech representation from a set of "false" speech representations. On the other hand, BEiT is a vision model pretrained on a masked image modeling task which masks some of the image patches and requires the model to predict the masked patches (similar to the masked language modeling objective).

## R

### recurrent neural network (RNN)

A type of model that uses a loop over a layer to process texts.

### representation learning

A subfield of machine learning which focuses on learning meaningful representations of raw data. Some examples of representation learning techniques include word embeddings, autoencoders, and Generative Adversarial Networks (GANs).

## S

### sampling rate

A measurement in hertz of the number of samples (the audio signal) taken per second. The sampling rate is a result of discretizing a continuous signal such as speech.

### self-attention

Each element of the input finds out which other elements of the input they should attend to.

### self-supervised learning 

A category of machine learning techniques in which a model creates its own learning objective from unlabeled data. It differs from [unsupervised learning](#unsupervised-learning) and [supervised learning](#supervised-learning) in that the learning process is supervised, but not explicitly from the user.


key: _node_content
string_value:
{"id_": "05d74600-f788-40d2-9971-54d0c7cc7cb9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fb71e9b1-65b0-49d0-8378-be659acc24b0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e46ee55c-130f-41f6-b671-0203de499c0c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "4cac900d19ea22b9bba8005f58227656ef0fee0f3ad877650ccd9656603c3cc7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8b4f44b4-499c-4ef6-8b8a-5c6ff0aace8e", "node_type": "1", "metadata": {}, "hash": "84cf497223feeb5b11ef3bc6a703d790e376496aed1cab97791ff4b9be76a373", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18072, "end_char_idx": 21661, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### pretrained model

A model that has been pretrained on some data (for instance all of Wikipedia). Pretraining methods involve a
self-supervised objective, which can be reading the text and trying to predict the next word (see [causal language
modeling](#causal-language-modeling)) or masking some words and trying to predict them (see [masked language
modeling](#masked-language-modeling-mlm)). 

Speech and vision models have their own pretraining objectives. For example, Wav2Vec2 is a speech model pretrained on a contrastive task which requires the model to identify the "true" speech representation from a set of "false" speech representations. On the other hand, BEiT is a vision model pretrained on a masked image modeling task which masks some of the image patches and requires the model to predict the masked patches (similar to the masked language modeling objective).

## R

### recurrent neural network (RNN)

A type of model that uses a loop over a layer to process texts.

### representation learning

A subfield of machine learning which focuses on learning meaningful representations of raw data. Some examples of representation learning techniques include word embeddings, autoencoders, and Generative Adversarial Networks (GANs).

## S

### sampling rate

A measurement in hertz of the number of samples (the audio signal) taken per second. The sampling rate is a result of discretizing a continuous signal such as speech.

### self-attention

Each element of the input finds out which other elements of the input they should attend to.

### self-supervised learning 

A category of machine learning techniques in which a model creates its own learning objective from unlabeled data. It differs from [unsupervised learning](#unsupervised-learning) and [supervised learning](#supervised-learning) in that the learning process is supervised, but not explicitly from the user. 

One example of self-supervised learning is [masked language modeling](#masked-language-modeling-mlm), where a model is passed sentences with a proportion of its tokens removed and learns to predict the missing tokens.

### semi-supervised learning

A broad category of machine learning training techniques that leverages a small amount of labeled data with a larger quantity of unlabeled data to improve the accuracy of a model, unlike [supervised learning](#supervised-learning) and [unsupervised learning](#unsupervised-learning).

An example of a semi-supervised learning approach is "self-training", in which a model is trained on labeled data, and then used to make predictions on the unlabeled data. The portion of the unlabeled data that the model predicts with the most confidence gets added to the labeled dataset and used to retrain the model.

### sequence-to-sequence (seq2seq)

Models that generate a new sequence from an input, like translation models, or summarization models (such as
[Bart](model_doc/bart) or [T5](model_doc/t5)).

### Sharded DDP

Another name for the foundational [ZeRO](#zero-redundancy-optimizer-zero) concept as used by various other implementations of ZeRO.

### stride

In [convolution](#convolution) or [pooling](#pooling), the stride refers to the distance the kernel is moved over a matrix. A stride of 1 means the kernel is moved one pixel over at a time, and a stride of 2 means the kernel is moved two pixels over at a time.

### supervised learning

A form of model training that directly uses labeled data to correct and instruct model performance. Data is fed into the model being trained, and its predictions are compared to the known labels.


key: _node_content
string_value:
{"id_": "8b4f44b4-499c-4ef6-8b8a-5c6ff0aace8e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fb71e9b1-65b0-49d0-8378-be659acc24b0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "05d74600-f788-40d2-9971-54d0c7cc7cb9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "c0e605d22eb79396eb7f63a3f4d8468d051dbc8860751bfcbd4c6d2cc2be8bdb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "42bb6e52-655a-4a82-ac53-f260a8e4c7b7", "node_type": "1", "metadata": {}, "hash": "8bb2e16160716b62be4ddb3f60bd6c01d50330634fa4e43a5753e4b5497d2c3e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19969, "end_char_idx": 23395, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
One example of self-supervised learning is [masked language modeling](#masked-language-modeling-mlm), where a model is passed sentences with a proportion of its tokens removed and learns to predict the missing tokens.

### semi-supervised learning

A broad category of machine learning training techniques that leverages a small amount of labeled data with a larger quantity of unlabeled data to improve the accuracy of a model, unlike [supervised learning](#supervised-learning) and [unsupervised learning](#unsupervised-learning).

An example of a semi-supervised learning approach is "self-training", in which a model is trained on labeled data, and then used to make predictions on the unlabeled data. The portion of the unlabeled data that the model predicts with the most confidence gets added to the labeled dataset and used to retrain the model.

### sequence-to-sequence (seq2seq)

Models that generate a new sequence from an input, like translation models, or summarization models (such as
[Bart](model_doc/bart) or [T5](model_doc/t5)).

### Sharded DDP

Another name for the foundational [ZeRO](#zero-redundancy-optimizer-zero) concept as used by various other implementations of ZeRO.

### stride

In [convolution](#convolution) or [pooling](#pooling), the stride refers to the distance the kernel is moved over a matrix. A stride of 1 means the kernel is moved one pixel over at a time, and a stride of 2 means the kernel is moved two pixels over at a time.

### supervised learning

A form of model training that directly uses labeled data to correct and instruct model performance. Data is fed into the model being trained, and its predictions are compared to the known labels. The model updates its weights based on how incorrect its predictions were, and the process is repeated to optimize model performance.

## T

### Tensor Parallelism (TP)

Parallelism technique for training on multiple GPUs in which each tensor is split up into multiple chunks, so instead of 
having the whole tensor reside on a single GPU, each shard of the tensor resides on its designated GPU. Shards gets 
processed separately and in parallel on different GPUs and the results are synced at the end of the processing step. 
This is what is sometimes called horizontal parallelism, as the splitting happens on horizontal level.
Learn more about Tensor Parallelism [here](perf_train_gpu_many#tensor-parallelism).

### token

A part of a sentence, usually a word, but can also be a subword (non-common words are often split in subwords) or a
punctuation symbol.

### token Type IDs

Some models' purpose is to do classification on pairs of sentences or question answering.

<Youtube id="0u3ioSwev3s"/>

These require two different sequences to be joined in a single "input_ids" entry, which usually is performed with the
help of special tokens, such as the classifier (`[CLS]`) and separator (`[SEP]`) tokens. For example, the BERT model
builds its two sequence input as such:

```python
>>> # [CLS] SEQUENCE_A [SEP] SEQUENCE_B [SEP]
```

We can use our tokenizer to automatically generate such a sentence by passing the two sequences to `tokenizer` as two
arguments (and not a list, like before) like this:

```python
>>> from transformers import BertTokenizer

>>> tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-cased")
>>> sequence_a = "HuggingFace is based in NYC"
>>> sequence_b = "Where is HuggingFace based?"


key: _node_content
string_value:
{"id_": "42bb6e52-655a-4a82-ac53-f260a8e4c7b7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fb71e9b1-65b0-49d0-8378-be659acc24b0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8b4f44b4-499c-4ef6-8b8a-5c6ff0aace8e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "40ff71a4608ac570415e381da7957498ab1cf92c44f2b0f0220d92ca4700ac2e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "72d151f8-d3c9-4b92-b041-b47beafe5bd3", "node_type": "1", "metadata": {}, "hash": "fde4c4570d7f72e7333237d6ca27730b8b3d4e54b6069a8bd84acfac5f481a8f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21662, "end_char_idx": 25025, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The model updates its weights based on how incorrect its predictions were, and the process is repeated to optimize model performance.

## T

### Tensor Parallelism (TP)

Parallelism technique for training on multiple GPUs in which each tensor is split up into multiple chunks, so instead of 
having the whole tensor reside on a single GPU, each shard of the tensor resides on its designated GPU. Shards gets 
processed separately and in parallel on different GPUs and the results are synced at the end of the processing step. 
This is what is sometimes called horizontal parallelism, as the splitting happens on horizontal level.
Learn more about Tensor Parallelism [here](perf_train_gpu_many#tensor-parallelism).

### token

A part of a sentence, usually a word, but can also be a subword (non-common words are often split in subwords) or a
punctuation symbol.

### token Type IDs

Some models' purpose is to do classification on pairs of sentences or question answering.

<Youtube id="0u3ioSwev3s"/>

These require two different sequences to be joined in a single "input_ids" entry, which usually is performed with the
help of special tokens, such as the classifier (`[CLS]`) and separator (`[SEP]`) tokens. For example, the BERT model
builds its two sequence input as such:

```python
>>> # [CLS] SEQUENCE_A [SEP] SEQUENCE_B [SEP]
```

We can use our tokenizer to automatically generate such a sentence by passing the two sequences to `tokenizer` as two
arguments (and not a list, like before) like this:

```python
>>> from transformers import BertTokenizer

>>> tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-cased")
>>> sequence_a = "HuggingFace is based in NYC"
>>> sequence_b = "Where is HuggingFace based?"

>>> encoded_dict = tokenizer(sequence_a, sequence_b)
>>> decoded = tokenizer.decode(encoded_dict["input_ids"])
```

which will return:

```python
>>> print(decoded)
[CLS] HuggingFace is based in NYC [SEP] Where is HuggingFace based? [SEP]
```

This is enough for some models to understand where one sequence ends and where another begins. However, other models,
such as BERT, also deploy token type IDs (also called segment IDs). They are represented as a binary mask identifying
the two types of sequence in the model.

The tokenizer returns this mask as the "token_type_ids" entry:

```python
>>> encoded_dict["token_type_ids"]
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1]
```

The first sequence, the "context" used for the question, has all its tokens represented by a `0`, whereas the second
sequence, corresponding to the "question", has all its tokens represented by a `1`.

Some models, like [`XLNetModel`] use an additional token represented by a `2`.

### transfer learning

A technique that involves taking a pretrained model and adapting it to a dataset specific to your task. Instead of training a model from scratch, you can leverage knowledge obtained from an existing model as a starting point. This speeds up the learning process and reduces the amount of training data needed.

### transformer

Self-attention based deep learning model architecture.

## U

### unsupervised learning

A form of model training in which data provided to the model is not labeled. Unsupervised learning techniques leverage statistical information of the data distribution to find patterns useful for the task at hand.


key: _node_content
string_value:
{"id_": "72d151f8-d3c9-4b92-b041-b47beafe5bd3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fb71e9b1-65b0-49d0-8378-be659acc24b0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "42bb6e52-655a-4a82-ac53-f260a8e4c7b7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "d1a5e8fa2adf75b8c851275efc29474c809f3daeb549529ec7e674870d69989d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23397, "end_char_idx": 25502, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> encoded_dict = tokenizer(sequence_a, sequence_b)
>>> decoded = tokenizer.decode(encoded_dict["input_ids"])
```

which will return:

```python
>>> print(decoded)
[CLS] HuggingFace is based in NYC [SEP] Where is HuggingFace based? [SEP]
```

This is enough for some models to understand where one sequence ends and where another begins. However, other models,
such as BERT, also deploy token type IDs (also called segment IDs). They are represented as a binary mask identifying
the two types of sequence in the model.

The tokenizer returns this mask as the "token_type_ids" entry:

```python
>>> encoded_dict["token_type_ids"]
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1]
```

The first sequence, the "context" used for the question, has all its tokens represented by a `0`, whereas the second
sequence, corresponding to the "question", has all its tokens represented by a `1`.

Some models, like [`XLNetModel`] use an additional token represented by a `2`.

### transfer learning

A technique that involves taking a pretrained model and adapting it to a dataset specific to your task. Instead of training a model from scratch, you can leverage knowledge obtained from an existing model as a starting point. This speeds up the learning process and reduces the amount of training data needed.

### transformer

Self-attention based deep learning model architecture.

## U

### unsupervised learning

A form of model training in which data provided to the model is not labeled. Unsupervised learning techniques leverage statistical information of the data distribution to find patterns useful for the task at hand.

## Z

### Zero Redundancy Optimizer (ZeRO)

Parallelism technique which performs sharding of the tensors somewhat similar to [TensorParallel](#tensor-parallelism-tp), 
except the whole tensor gets reconstructed in time for a forward or backward computation, therefore the model doesn't need 
to be modified. This method also supports various offloading techniques to compensate for limited GPU memory. 
Learn more about ZeRO [here](perf_train_gpu_many#zero-data-parallelism).


key: _node_content
string_value:
{"id_": "98efd096-9250-4a6c-92a1-63a1808fbd70", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tf_xla", "title": "XLA Integration for TensorFlow Models", "tokens": 2214, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b2fa9d41-8b2a-427a-8592-a612587167e7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tf_xla", "title": "XLA Integration for TensorFlow Models", "tokens": 2214, "retrieve_doc": true, "source": "transformers"}, "hash": "23b27be6b7bfc01e4419837b98502ee0f1a5c474524c60ca30bb03dbf2dc67a2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "263922ae-c4b1-4c2a-9293-c06291e16379", "node_type": "1", "metadata": {}, "hash": "cb1acb123801198053de1788131088b2385d44d17b82393016640fdedd49f18c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3326, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# XLA Integration for TensorFlow Models

[[open-in-colab]]

Accelerated Linear Algebra, dubbed XLA, is a compiler for accelerating the runtime of TensorFlow Models. From the [official documentation](https://www.tensorflow.org/xla):

XLA (Accelerated Linear Algebra) is a domain-specific compiler for linear algebra that can accelerate TensorFlow models with potentially no source code changes.

Using XLA in TensorFlow is simple – it comes packaged inside the `tensorflow` library, and it can be triggered with the `jit_compile` argument in any graph-creating function such as [`tf.function`](https://www.tensorflow.org/guide/intro_to_graphs). When using Keras methods like `fit()` and `predict()`, you can enable XLA simply by passing the `jit_compile` argument to `model.compile()`. However, XLA is not limited to these methods - it can also be used to accelerate any arbitrary `tf.function`.

Several TensorFlow methods in 🤗 Transformers have been rewritten to be XLA-compatible, including text generation for models such as [GPT2](https://huggingface.co/docs/transformers/model_doc/gpt2), [T5](https://huggingface.co/docs/transformers/model_doc/t5) and [OPT](https://huggingface.co/docs/transformers/model_doc/opt), as well as speech processing for models such as [Whisper](https://huggingface.co/docs/transformers/model_doc/whisper).

While the exact amount of speed-up is very much model-dependent, for TensorFlow text generation models inside 🤗 Transformers, we noticed a speed-up of ~100x. This document will explain how you can use XLA for these models to get the maximum amount of performance. We’ll also provide links to additional resources if you’re interested to learn more about the benchmarks and our design philosophy behind the XLA integration.

## Running TF functions with XLA

Let us consider the following model in TensorFlow:

```py
import tensorflow as tf

model = tf.keras.Sequential(
    [tf.keras.layers.Dense(10, input_shape=(10,), activation="relu"), tf.keras.layers.Dense(5, activation="softmax")]
)
```

The above model accepts inputs having a dimension of `(10, )`. We can use the model for running a forward pass like so:

```py
# Generate random inputs for the model.
batch_size = 16
input_vector_dim = 10
random_inputs = tf.random.normal((batch_size, input_vector_dim))

# Run a forward pass.
_ = model(random_inputs)
```

In order to run the forward pass with an XLA-compiled function, we’d need to do:

```py
xla_fn = tf.function(model, jit_compile=True)
_ = xla_fn(random_inputs)
```

The default `call()` function of the `model` is used for compiling the XLA graph. But if there’s any other model function you want to compile into XLA that’s also possible with:

```py
my_xla_fn = tf.function(model.my_xla_fn, jit_compile=True)
```

## Running a TF text generation model with XLA from 🤗 Transformers

To enable XLA-accelerated generation within 🤗 Transformers, you need to have a recent version of `transformers` installed. You can install it by running:

```bash
pip install transformers --upgrade
```

And then you can run the following code:

```py
import tensorflow as tf
from transformers import AutoTokenizer, TFAutoModelForCausalLM

# Will error if the minimal version of Transformers is not installed.
from transformers.utils import check_min_version

check_min_version("4.21.0")


key: _node_content
string_value:
{"id_": "263922ae-c4b1-4c2a-9293-c06291e16379", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tf_xla", "title": "XLA Integration for TensorFlow Models", "tokens": 2214, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b2fa9d41-8b2a-427a-8592-a612587167e7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tf_xla", "title": "XLA Integration for TensorFlow Models", "tokens": 2214, "retrieve_doc": true, "source": "transformers"}, "hash": "23b27be6b7bfc01e4419837b98502ee0f1a5c474524c60ca30bb03dbf2dc67a2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "98efd096-9250-4a6c-92a1-63a1808fbd70", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tf_xla", "title": "XLA Integration for TensorFlow Models", "tokens": 2214, "retrieve_doc": true, "source": "transformers"}, "hash": "4eb426f65174b1c9f2292d882e46ca166016bdcbac877b827054473a931eb8b5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b6469f91-7b42-4df3-81e6-59663d304fa9", "node_type": "1", "metadata": {}, "hash": "b1e6b252b90f2def1c917fe7ecd7d78aaa3358d6e598a8229458511e4a80f6f7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3329, "end_char_idx": 6144, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2", padding_side="left", pad_token="</s>")
model = TFAutoModelForCausalLM.from_pretrained("openai-community/gpt2")
input_string = ["TensorFlow is"]

# One line to create an XLA generation function
xla_generate = tf.function(model.generate, jit_compile=True)

tokenized_input = tokenizer(input_string, return_tensors="tf")
generated_tokens = xla_generate(**tokenized_input, num_beams=2)

decoded_text = tokenizer.decode(generated_tokens[0], skip_special_tokens=True)
print(f"Generated -- {decoded_text}")
# Generated -- TensorFlow is an open-source, open-source, distributed-source application # framework for the
```

As you can notice, enabling XLA on `generate()` is just a single line of code. The rest of the code remains unchanged. However, there are a couple of gotchas in the above code snippet that are specific to XLA. You need to be aware of those to realize the speed-ups that XLA can bring in. We discuss these in the following section. 

## Gotchas to be aware of

When you are executing an XLA-enabled function (like `xla_generate()` above) for the first time, it will internally try to infer the computation graph, which is time-consuming.  This process is known as [“tracing”](https://www.tensorflow.org/guide/intro_to_graphs#when_is_a_function_tracing). 

You might notice that the generation time is not fast. Successive calls of `xla_generate()` (or any other XLA-enabled function) won’t have to infer the computation graph, given the inputs to the function follow the same shape with which the computation graph was initially built. While this is not a problem for modalities with fixed input shapes (e.g., images), you must pay attention if you are working with variable input shape modalities (e.g., text).

To ensure `xla_generate()` always operates with the same input shapes, you can specify the `padding` arguments when calling the tokenizer. 

```py
import tensorflow as tf
from transformers import AutoTokenizer, TFAutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2", padding_side="left", pad_token="</s>")
model = TFAutoModelForCausalLM.from_pretrained("openai-community/gpt2")
input_string = ["TensorFlow is"]

xla_generate = tf.function(model.generate, jit_compile=True)

# Here, we call the tokenizer with padding options.
tokenized_input = tokenizer(input_string, pad_to_multiple_of=8, padding=True, return_tensors="tf")

generated_tokens = xla_generate(**tokenized_input, num_beams=2)
decoded_text = tokenizer.decode(generated_tokens[0], skip_special_tokens=True)
print(f"Generated -- {decoded_text}")
```

This way, you can ensure that the inputs to `xla_generate()` will always receive inputs with the shape it was traced with and thus leading to speed-ups in the generation time.


key: _node_content
string_value:
{"id_": "b6469f91-7b42-4df3-81e6-59663d304fa9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tf_xla", "title": "XLA Integration for TensorFlow Models", "tokens": 2214, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b2fa9d41-8b2a-427a-8592-a612587167e7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tf_xla", "title": "XLA Integration for TensorFlow Models", "tokens": 2214, "retrieve_doc": true, "source": "transformers"}, "hash": "23b27be6b7bfc01e4419837b98502ee0f1a5c474524c60ca30bb03dbf2dc67a2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "263922ae-c4b1-4c2a-9293-c06291e16379", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tf_xla", "title": "XLA Integration for TensorFlow Models", "tokens": 2214, "retrieve_doc": true, "source": "transformers"}, "hash": "e063f35516d708c2ef0fd575d6021b00244959933d9d2fa98a070b746f51f5b3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "91688198-cbf1-4f5c-80b6-77438b142d79", "node_type": "1", "metadata": {}, "hash": "216e335be07ea691533df35710f26cf33dcb4d826c4df5cb1251e9aa8001f19d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4548, "end_char_idx": 7631, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This process is known as [“tracing”](https://www.tensorflow.org/guide/intro_to_graphs#when_is_a_function_tracing). 

You might notice that the generation time is not fast. Successive calls of `xla_generate()` (or any other XLA-enabled function) won’t have to infer the computation graph, given the inputs to the function follow the same shape with which the computation graph was initially built. While this is not a problem for modalities with fixed input shapes (e.g., images), you must pay attention if you are working with variable input shape modalities (e.g., text).

To ensure `xla_generate()` always operates with the same input shapes, you can specify the `padding` arguments when calling the tokenizer. 

```py
import tensorflow as tf
from transformers import AutoTokenizer, TFAutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2", padding_side="left", pad_token="</s>")
model = TFAutoModelForCausalLM.from_pretrained("openai-community/gpt2")
input_string = ["TensorFlow is"]

xla_generate = tf.function(model.generate, jit_compile=True)

# Here, we call the tokenizer with padding options.
tokenized_input = tokenizer(input_string, pad_to_multiple_of=8, padding=True, return_tensors="tf")

generated_tokens = xla_generate(**tokenized_input, num_beams=2)
decoded_text = tokenizer.decode(generated_tokens[0], skip_special_tokens=True)
print(f"Generated -- {decoded_text}")
```

This way, you can ensure that the inputs to `xla_generate()` will always receive inputs with the shape it was traced with and thus leading to speed-ups in the generation time. You can verify this with the code below:

```py
import time
import tensorflow as tf
from transformers import AutoTokenizer, TFAutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2", padding_side="left", pad_token="</s>")
model = TFAutoModelForCausalLM.from_pretrained("openai-community/gpt2")

xla_generate = tf.function(model.generate, jit_compile=True)

for input_string in ["TensorFlow is", "TensorFlow is a", "TFLite is a"]:
    tokenized_input = tokenizer(input_string, pad_to_multiple_of=8, padding=True, return_tensors="tf")
    start = time.time_ns()
    generated_tokens = xla_generate(**tokenized_input, num_beams=2)
    end = time.time_ns()
    print(f"Execution time -- {(end - start) / 1e6:.1f} ms\n")
```

On a Tesla T4 GPU, you can expect the outputs like so:

```bash
Execution time -- 30819.6 ms

Execution time -- 79.0 ms

Execution time -- 78.9 ms
```
The first call to `xla_generate()` is time-consuming because of tracing, but the successive calls are orders of magnitude faster. Keep in mind that any change in the generation options at any point will trigger re-tracing and thus leading to slow-downs in the generation time. 

We didn’t cover all the text generation options 🤗 Transformers provides in this document. We encourage you to read the documentation for advanced use cases.

## Additional Resources

Here, we leave you with some additional resources if you want to delve deeper into XLA in 🤗 Transformers and in general.


key: _node_content
string_value:
{"id_": "91688198-cbf1-4f5c-80b6-77438b142d79", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tf_xla", "title": "XLA Integration for TensorFlow Models", "tokens": 2214, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b2fa9d41-8b2a-427a-8592-a612587167e7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tf_xla", "title": "XLA Integration for TensorFlow Models", "tokens": 2214, "retrieve_doc": true, "source": "transformers"}, "hash": "23b27be6b7bfc01e4419837b98502ee0f1a5c474524c60ca30bb03dbf2dc67a2", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b6469f91-7b42-4df3-81e6-59663d304fa9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tf_xla", "title": "XLA Integration for TensorFlow Models", "tokens": 2214, "retrieve_doc": true, "source": "transformers"}, "hash": "e705f913a760b288bad1b8828b010909e2c50068f24b84ffc8d96929c1e4f04c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6145, "end_char_idx": 8819, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can verify this with the code below:

```py
import time
import tensorflow as tf
from transformers import AutoTokenizer, TFAutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2", padding_side="left", pad_token="</s>")
model = TFAutoModelForCausalLM.from_pretrained("openai-community/gpt2")

xla_generate = tf.function(model.generate, jit_compile=True)

for input_string in ["TensorFlow is", "TensorFlow is a", "TFLite is a"]:
    tokenized_input = tokenizer(input_string, pad_to_multiple_of=8, padding=True, return_tensors="tf")
    start = time.time_ns()
    generated_tokens = xla_generate(**tokenized_input, num_beams=2)
    end = time.time_ns()
    print(f"Execution time -- {(end - start) / 1e6:.1f} ms\n")
```

On a Tesla T4 GPU, you can expect the outputs like so:

```bash
Execution time -- 30819.6 ms

Execution time -- 79.0 ms

Execution time -- 78.9 ms
```
The first call to `xla_generate()` is time-consuming because of tracing, but the successive calls are orders of magnitude faster. Keep in mind that any change in the generation options at any point will trigger re-tracing and thus leading to slow-downs in the generation time. 

We didn’t cover all the text generation options 🤗 Transformers provides in this document. We encourage you to read the documentation for advanced use cases.

## Additional Resources

Here, we leave you with some additional resources if you want to delve deeper into XLA in 🤗 Transformers and in general. 
 
* [This Colab Notebook](https://colab.research.google.com/github/huggingface/blog/blob/main/notebooks/91_tf_xla_generate.ipynb) provides an interactive demonstration if you want to fiddle with the XLA-compatible encoder-decoder (like [T5](https://huggingface.co/docs/transformers/model_doc/t5)) and decoder-only (like [GPT2](https://huggingface.co/docs/transformers/model_doc/gpt2)) text generation models. 
* [This blog post](https://huggingface.co/blog/tf-xla-generate) provides an overview of the comparison benchmarks for XLA-compatible models along with a friendly introduction to XLA in TensorFlow. 
* [This blog post](https://blog.tensorflow.org/2022/11/how-hugging-face-improved-text-generation-performance-with-xla.html) discusses our design philosophy behind adding XLA support to the TensorFlow models in 🤗 Transformers. 
* Recommended posts for learning more about XLA and TensorFlow graphs in general:
    * [XLA: Optimizing Compiler for Machine Learning](https://www.tensorflow.org/xla)
    * [Introduction to graphs and tf.function](https://www.tensorflow.org/guide/intro_to_graphs)
    * [Better performance with tf.function](https://www.tensorflow.org/guide/function)


key: _node_content
string_value:
{"id_": "72b66dbc-cb17-47b9-a70c-160719532158", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7310b771-9506-4834-8b0c-51199b9e3546", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "hash": "68a458ed8c65861d8af91639f073c49a64d8a0e1217f62736bc11f914aed38be", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0757c5a3-73a5-45e6-88eb-9e988cc65c30", "node_type": "1", "metadata": {}, "hash": "ddaacc45968e786951487bfdb97fcf5e661b19e0f957354b550e9f5998ebd8a0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3468, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# LLM inference optimization

Large language models (LLMs) have pushed text generation applications, such as chat and code completion models, to the next level by producing text that displays a high level of understanding and fluency. But what makes LLMs so powerful - namely their size - also presents challenges for inference.

Basic inference is slow because LLMs have to be called repeatedly to generate the next token. The input sequence increases as generation progresses, which takes longer and longer for the LLM to process. LLMs also have billions of parameters, making it a challenge to store and handle all those weights in memory.

This guide will show you how to use the optimization techniques available in Transformers to accelerate LLM inference.

> [!TIP]
> Hugging Face also provides [Text Generation Inference (TGI)](https://hf.co/docs/text-generation-inference), a library dedicated to deploying and serving highly optimized LLMs for inference. It includes deployment-oriented optimization features not included in Transformers, such as continuous batching for increasing throughput and tensor parallelism for multi-GPU inference.

## Static kv-cache and `torch.compile`

During decoding, a LLM computes the key-value (kv) values for each input token and since it is autoregressive, it computes the same kv values each time because the generated output becomes part of the input now. This is not very efficient because you're recomputing the same kv values each time.

To optimize this, you can use a kv-cache to store the past keys and values instead of recomputing them each time. However, since the kv-cache grows with each generation step and is dynamic, it prevents you from taking advantage of [`torch.compile`](./perf_torch_compile), a powerful optimization tool that fuses PyTorch code into fast and optimized kernels.

The *static kv-cache* solves this issue by pre-allocating the kv-cache size to a maximum value which allows you to combine it with `torch.compile` for up to a 4x speed up. Your speed up may vary depending on the model size (larger models have a smaller speed up) and hardware.

> [!WARNING]
> Currently, only [Llama](./model_doc/llama2) and a few other models support static kv-cache and `torch.compile`. Check [this issue](https://github.com/huggingface/transformers/issues/28981) for a live model compatibility list.

There are three flavors of static kv-cache usage, depending on the complexity of your task:
1. Basic usage: simply set a flag in `generation_config` (recommended);
2. Advanced usage: handle a cache object for multi-turn generation or a custom generation loop;
3. Advanced usage: compile the entire `generate` function into a single graph, if having a single graph is relevant for you.

Select the correct tab below for further instructions on each of these flavors.

> [!TIP]
> Regardless of the strategy used with `torch.compile`, you can avoid shape-related recompilations if you left-pad your LLM inputs to a limited set of values. The [`pad_to_multiple_of` tokenizer flag](https://huggingface.co/docs/transformers/main_classes/tokenizer#transformers.PreTrainedTokenizer.__call__.pad_to_multiple_of) is your friend!

<hfoptions id="static-kv">
<hfoption id="basic usage: generation_config">

For this example, let's use the [Gemma](https://hf.co/google/gemma-2b) model. All we need to do is to:
1. Access the model's `generation_config` attribute and set the `cache_implementation` to "static";
2.


key: _node_content
string_value:
{"id_": "0757c5a3-73a5-45e6-88eb-9e988cc65c30", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7310b771-9506-4834-8b0c-51199b9e3546", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "hash": "68a458ed8c65861d8af91639f073c49a64d8a0e1217f62736bc11f914aed38be", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "72b66dbc-cb17-47b9-a70c-160719532158", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "hash": "bb2b271ddf5e4a4d18a9952520298adf9a9da704ffd4957c6d7773ab8b846254", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "87b1144d-3465-44b2-94c5-fc46603b0e9a", "node_type": "1", "metadata": {}, "hash": "1db3f623357e2768acd53769cd8a752b904644b1a134f7d049d57a0831fbfa83", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1848, "end_char_idx": 5085, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The *static kv-cache* solves this issue by pre-allocating the kv-cache size to a maximum value which allows you to combine it with `torch.compile` for up to a 4x speed up. Your speed up may vary depending on the model size (larger models have a smaller speed up) and hardware.

> [!WARNING]
> Currently, only [Llama](./model_doc/llama2) and a few other models support static kv-cache and `torch.compile`. Check [this issue](https://github.com/huggingface/transformers/issues/28981) for a live model compatibility list.

There are three flavors of static kv-cache usage, depending on the complexity of your task:
1. Basic usage: simply set a flag in `generation_config` (recommended);
2. Advanced usage: handle a cache object for multi-turn generation or a custom generation loop;
3. Advanced usage: compile the entire `generate` function into a single graph, if having a single graph is relevant for you.

Select the correct tab below for further instructions on each of these flavors.

> [!TIP]
> Regardless of the strategy used with `torch.compile`, you can avoid shape-related recompilations if you left-pad your LLM inputs to a limited set of values. The [`pad_to_multiple_of` tokenizer flag](https://huggingface.co/docs/transformers/main_classes/tokenizer#transformers.PreTrainedTokenizer.__call__.pad_to_multiple_of) is your friend!

<hfoptions id="static-kv">
<hfoption id="basic usage: generation_config">

For this example, let's use the [Gemma](https://hf.co/google/gemma-2b) model. All we need to do is to:
1. Access the model's `generation_config` attribute and set the `cache_implementation` to "static";
2. Call `torch.compile` on the model to compile the forward pass with the static kv-cache.

And that's it!

```py
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
import os
os.environ["TOKENIZERS_PARALLELISM"] = "false"  # To prevent long warnings :)

tokenizer = AutoTokenizer.from_pretrained("google/gemma-2b")
model = AutoModelForCausalLM.from_pretrained("google/gemma-2b", device_map="auto")

model.generation_config.cache_implementation = "static"

model.forward = torch.compile(model.forward, mode="reduce-overhead", fullgraph=True)
input_text = "The theory of special relativity states "
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")

outputs = model.generate(**input_ids)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
['The theory of special relativity states 1. The speed of light is constant in all inertial reference']
```

Under the hood, `generate` will attempt to reuse the same cache object, removing the need for re-compilation at each call. Avoiding re-compilation is critical to get the most out of `torch.compile`, and you should be aware of the following:
1. If the batch size changes or the maximum output length increases between calls, the cache will have to be reinitialized, triggering a new compilation;
2. The first couple of calls of the compiled function are slower, as the function is being compiled.

> [!WARNING]
> For a more advanced usage of the static cache, such as multi-turn conversations, we recommend instantiating and manipulating the cache object outside [`~GenerationMixin.generate`]. See the advanced usage tab.


key: _node_content
string_value:
{"id_": "87b1144d-3465-44b2-94c5-fc46603b0e9a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7310b771-9506-4834-8b0c-51199b9e3546", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "hash": "68a458ed8c65861d8af91639f073c49a64d8a0e1217f62736bc11f914aed38be", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0757c5a3-73a5-45e6-88eb-9e988cc65c30", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "hash": "121ef23c31d6e766626936c530158b5fe7b3b981653073d1b71d8277f886e362", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7c1995ef-1844-4c21-85ef-8e94eabc849f", "node_type": "1", "metadata": {}, "hash": "e4c84fcfc0906743311ac55b0b14125eb2507966337d4b6473f10e18b0b7d72b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3341, "end_char_idx": 6716, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
All we need to do is to:
1. Access the model's `generation_config` attribute and set the `cache_implementation` to "static";
2. Call `torch.compile` on the model to compile the forward pass with the static kv-cache.

And that's it!

```py
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
import os
os.environ["TOKENIZERS_PARALLELISM"] = "false"  # To prevent long warnings :)

tokenizer = AutoTokenizer.from_pretrained("google/gemma-2b")
model = AutoModelForCausalLM.from_pretrained("google/gemma-2b", device_map="auto")

model.generation_config.cache_implementation = "static"

model.forward = torch.compile(model.forward, mode="reduce-overhead", fullgraph=True)
input_text = "The theory of special relativity states "
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")

outputs = model.generate(**input_ids)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
['The theory of special relativity states 1. The speed of light is constant in all inertial reference']
```

Under the hood, `generate` will attempt to reuse the same cache object, removing the need for re-compilation at each call. Avoiding re-compilation is critical to get the most out of `torch.compile`, and you should be aware of the following:
1. If the batch size changes or the maximum output length increases between calls, the cache will have to be reinitialized, triggering a new compilation;
2. The first couple of calls of the compiled function are slower, as the function is being compiled.

> [!WARNING]
> For a more advanced usage of the static cache, such as multi-turn conversations, we recommend instantiating and manipulating the cache object outside [`~GenerationMixin.generate`]. See the advanced usage tab.

</hfoption>
<hfoption id="advanced usage: control Static Cache">

A [`StaticCache`] object can be passed to the model's [`~GenerationMixin.generate`] under the `past_key_values` argument. The object will retain the cache contents, so you can pass it to a new [`~GenerationMixin.generate`] call to continue generation, like you would do with a dynamic cache.

```py
from transformers import AutoTokenizer, AutoModelForCausalLM, StaticCache
import torch
import os
os.environ["TOKENIZERS_PARALLELISM"] = "false"  # To prevent long warnings :)

tokenizer = AutoTokenizer.from_pretrained("google/gemma-2b")
model = AutoModelForCausalLM.from_pretrained("google/gemma-2b", device_map="auto")

model.forward = torch.compile(model.forward, mode="reduce-overhead", fullgraph=True)
input_text = "The theory of special relativity states "
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
prompt_length = input_ids.input_ids.shape[1]
model.generation_config.max_new_tokens = 16

past_key_values = StaticCache(
    config=model.config,
    batch_size=1,
    # If you plan to reuse the cache, make sure the cache length is large enough for all cases
    max_cache_len=prompt_length+(model.generation_config.max_new_tokens*2),
    device=model.device,
    dtype=model.dtype
)
outputs = model.generate(**input_ids, past_key_values=past_key_values)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
['The theory of special relativity states 1. The speed of light is constant in all inertial reference frames. 2']

# pass in the generated text and the same cache object to continue generation from where it left off.


key: _node_content
string_value:
{"id_": "7c1995ef-1844-4c21-85ef-8e94eabc849f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7310b771-9506-4834-8b0c-51199b9e3546", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "hash": "68a458ed8c65861d8af91639f073c49a64d8a0e1217f62736bc11f914aed38be", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "87b1144d-3465-44b2-94c5-fc46603b0e9a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "hash": "66b92725ffe96b1bb8687fdaf95ab5ab17b39b74da7e94bed8f8a959cdccca58", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a7224bc1-f4cb-45e4-8a20-55a4dbeaad87", "node_type": "1", "metadata": {}, "hash": "9da2e8142ccea58c9b41c1ef33921f062cdccd983cc5d33646df57200c8dffcc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5058, "end_char_idx": 7915, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
See the advanced usage tab.

</hfoption>
<hfoption id="advanced usage: control Static Cache">

A [`StaticCache`] object can be passed to the model's [`~GenerationMixin.generate`] under the `past_key_values` argument. The object will retain the cache contents, so you can pass it to a new [`~GenerationMixin.generate`] call to continue generation, like you would do with a dynamic cache.

```py
from transformers import AutoTokenizer, AutoModelForCausalLM, StaticCache
import torch
import os
os.environ["TOKENIZERS_PARALLELISM"] = "false"  # To prevent long warnings :)

tokenizer = AutoTokenizer.from_pretrained("google/gemma-2b")
model = AutoModelForCausalLM.from_pretrained("google/gemma-2b", device_map="auto")

model.forward = torch.compile(model.forward, mode="reduce-overhead", fullgraph=True)
input_text = "The theory of special relativity states "
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
prompt_length = input_ids.input_ids.shape[1]
model.generation_config.max_new_tokens = 16

past_key_values = StaticCache(
    config=model.config,
    batch_size=1,
    # If you plan to reuse the cache, make sure the cache length is large enough for all cases
    max_cache_len=prompt_length+(model.generation_config.max_new_tokens*2),
    device=model.device,
    dtype=model.dtype
)
outputs = model.generate(**input_ids, past_key_values=past_key_values)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
['The theory of special relativity states 1. The speed of light is constant in all inertial reference frames. 2']

# pass in the generated text and the same cache object to continue generation from where it left off. Optionally, in a
# multi-turn conversation, append the new user input to the generated text.
new_input_ids = outputs
outputs = model.generate(new_input_ids, past_key_values=past_key_values)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
['The theory of special relativity states 1. The speed of light is constant in all inertial reference frames. 2. The speed of light is constant in all inertial reference frames. 3.']
```

> [!TIP]
> If you want to reuse the same [`StaticCache`] object on a new prompt, be sure to reset its contents with the `.reset()` method between calls

If you want to go further down a level, the [`StaticCache`] object can also be passed to the model's forward pass under the same `past_key_values` argument. Using this strategy, you can write your own function to decode the next token given the current token and position and cache position of previously generated tokens.

```py
from transformers import LlamaTokenizer, LlamaForCausalLM, StaticCache, logging
from transformers.testing_utils import CaptureLogger
import torch

prompts = [
    "Simply put, the theory of relativity states that ",
    "My favorite all time favorite condiment is ketchup.


key: _node_content
string_value:
{"id_": "a7224bc1-f4cb-45e4-8a20-55a4dbeaad87", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7310b771-9506-4834-8b0c-51199b9e3546", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "hash": "68a458ed8c65861d8af91639f073c49a64d8a0e1217f62736bc11f914aed38be", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7c1995ef-1844-4c21-85ef-8e94eabc849f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "hash": "ca3b1816e947d5afcd2fc18c2d8e9dfe7ba26fe7185f47c4dae3d9b5c58f6b15", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "94732132-bc7a-4724-8ab5-b9a2e0856638", "node_type": "1", "metadata": {}, "hash": "4774fd35fcf319198c646786524715fdf3966f33bdecd4f651f42f4df89ea67b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6545, "end_char_idx": 9309, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The speed of light is constant in all inertial reference frames. 2']

# pass in the generated text and the same cache object to continue generation from where it left off. Optionally, in a
# multi-turn conversation, append the new user input to the generated text.
new_input_ids = outputs
outputs = model.generate(new_input_ids, past_key_values=past_key_values)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
['The theory of special relativity states 1. The speed of light is constant in all inertial reference frames. 2. The speed of light is constant in all inertial reference frames. 3.']
```

> [!TIP]
> If you want to reuse the same [`StaticCache`] object on a new prompt, be sure to reset its contents with the `.reset()` method between calls

If you want to go further down a level, the [`StaticCache`] object can also be passed to the model's forward pass under the same `past_key_values` argument. Using this strategy, you can write your own function to decode the next token given the current token and position and cache position of previously generated tokens.

```py
from transformers import LlamaTokenizer, LlamaForCausalLM, StaticCache, logging
from transformers.testing_utils import CaptureLogger
import torch

prompts = [
    "Simply put, the theory of relativity states that ",
    "My favorite all time favorite condiment is ketchup.",
]

NUM_TOKENS_TO_GENERATE = 40
torch_device = "cuda"

tokenizer = LlamaTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf", pad_token="</s>", padding_side="right")
model = LlamaForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf", device_map="sequential")
inputs = tokenizer(prompts, return_tensors="pt", padding=True).to(model.device)

def decode_one_tokens(model, cur_token, input_pos, cache_position, past_key_values):
    logits = model(
        cur_token,
        position_ids=input_pos,
        cache_position=cache_position,
        past_key_values=past_key_values,
        return_dict=False,
        use_cache=True
    )[0]
    new_token = torch.argmax(logits[:, -1], dim=-1)[:, None]
    return new_token
```

There are a few important things you must do to enable static kv-cache and `torch.compile` with the `StaticCache` method:
1. Initialize the [`StaticCache`] instance before using the model for inference. There you can configure parameters like the maximum batch size and sequence length.
2. Call `torch.compile` on the model to compile the forward pass with the static kv-cache.
3. Set `enable_math=True` in the [torch.backends.cuda.sdp_kernel](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html) context manager to enable the native PyTorch C++ implementation of scaled dot product attention to speed up inference even more.


key: _node_content
string_value:
{"id_": "94732132-bc7a-4724-8ab5-b9a2e0856638", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7310b771-9506-4834-8b0c-51199b9e3546", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "hash": "68a458ed8c65861d8af91639f073c49a64d8a0e1217f62736bc11f914aed38be", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a7224bc1-f4cb-45e4-8a20-55a4dbeaad87", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "hash": "27a7d89d358016afd2617bf4a0e70df79a9ed1d8945cf8690f52b3db7531bc8f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "68fbeaf1-76d9-4da2-a4b0-2d3f1f0bda4c", "node_type": "1", "metadata": {}, "hash": "ddd4322fe07698b1deee1bda9e84b7920bec713ec47b618ed6b1d2be320cd4ab", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7637, "end_char_idx": 10890, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
from transformers import LlamaTokenizer, LlamaForCausalLM, StaticCache, logging
from transformers.testing_utils import CaptureLogger
import torch

prompts = [
    "Simply put, the theory of relativity states that ",
    "My favorite all time favorite condiment is ketchup.",
]

NUM_TOKENS_TO_GENERATE = 40
torch_device = "cuda"

tokenizer = LlamaTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf", pad_token="</s>", padding_side="right")
model = LlamaForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf", device_map="sequential")
inputs = tokenizer(prompts, return_tensors="pt", padding=True).to(model.device)

def decode_one_tokens(model, cur_token, input_pos, cache_position, past_key_values):
    logits = model(
        cur_token,
        position_ids=input_pos,
        cache_position=cache_position,
        past_key_values=past_key_values,
        return_dict=False,
        use_cache=True
    )[0]
    new_token = torch.argmax(logits[:, -1], dim=-1)[:, None]
    return new_token
```

There are a few important things you must do to enable static kv-cache and `torch.compile` with the `StaticCache` method:
1. Initialize the [`StaticCache`] instance before using the model for inference. There you can configure parameters like the maximum batch size and sequence length.
2. Call `torch.compile` on the model to compile the forward pass with the static kv-cache.
3. Set `enable_math=True` in the [torch.backends.cuda.sdp_kernel](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html) context manager to enable the native PyTorch C++ implementation of scaled dot product attention to speed up inference even more.

```py
batch_size, seq_length = inputs["input_ids"].shape
with torch.no_grad():
    past_key_values = StaticCache(
        config=model.config, batch_size=2, max_cache_len=4096, device=torch_device, dtype=model.dtype
    )
    cache_position = torch.arange(seq_length, device=torch_device)
    generated_ids = torch.zeros(
        batch_size, seq_length + NUM_TOKENS_TO_GENERATE + 1, dtype=torch.int, device=torch_device
    )
    generated_ids[:, cache_position] = inputs["input_ids"].to(torch_device).to(torch.int)

    logits = model(
        **inputs, cache_position=cache_position, past_key_values=past_key_values,return_dict=False, use_cache=True
    )[0]
    next_token = torch.argmax(logits[:, -1], dim=-1)[:, None]
    generated_ids[:, seq_length] = next_token[:, 0]

    decode_one_tokens = torch.compile(decode_one_tokens, mode="reduce-overhead", fullgraph=True)
    cache_position = torch.tensor([seq_length + 1], device=torch_device)
    for _ in range(1, NUM_TOKENS_TO_GENERATE):
        with torch.backends.cuda.sdp_kernel(enable_flash=False, enable_mem_efficient=False, enable_math=True):
            next_token = decode_one_tokens(model, next_token.clone(), None, cache_position, past_key_values)
            generated_ids[:, cache_position] = next_token.int()
        cache_position += 1

text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
text
['Simply put, the theory of relativity states that 1) the speed of light is constant, 2) the speed of light is the same for all observers, and 3) the laws of physics are the same for all observers.


key: _node_content
string_value:
{"id_": "68fbeaf1-76d9-4da2-a4b0-2d3f1f0bda4c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7310b771-9506-4834-8b0c-51199b9e3546", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "hash": "68a458ed8c65861d8af91639f073c49a64d8a0e1217f62736bc11f914aed38be", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "94732132-bc7a-4724-8ab5-b9a2e0856638", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "hash": "6ec372cc86f016bc7d7c5798c16f4e0b48f00602c82f8520c86834de445b7d90", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "542df6e7-1ccb-4fe4-84e8-701b65f6fa6a", "node_type": "1", "metadata": {}, "hash": "184b03bd4c391e6fa51566c3c6db9f8fba0a6b3a0b77cc8125df8e0ba9646a62", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9311, "end_char_idx": 12587, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
batch_size, seq_length = inputs["input_ids"].shape
with torch.no_grad():
    past_key_values = StaticCache(
        config=model.config, batch_size=2, max_cache_len=4096, device=torch_device, dtype=model.dtype
    )
    cache_position = torch.arange(seq_length, device=torch_device)
    generated_ids = torch.zeros(
        batch_size, seq_length + NUM_TOKENS_TO_GENERATE + 1, dtype=torch.int, device=torch_device
    )
    generated_ids[:, cache_position] = inputs["input_ids"].to(torch_device).to(torch.int)

    logits = model(
        **inputs, cache_position=cache_position, past_key_values=past_key_values,return_dict=False, use_cache=True
    )[0]
    next_token = torch.argmax(logits[:, -1], dim=-1)[:, None]
    generated_ids[:, seq_length] = next_token[:, 0]

    decode_one_tokens = torch.compile(decode_one_tokens, mode="reduce-overhead", fullgraph=True)
    cache_position = torch.tensor([seq_length + 1], device=torch_device)
    for _ in range(1, NUM_TOKENS_TO_GENERATE):
        with torch.backends.cuda.sdp_kernel(enable_flash=False, enable_mem_efficient=False, enable_math=True):
            next_token = decode_one_tokens(model, next_token.clone(), None, cache_position, past_key_values)
            generated_ids[:, cache_position] = next_token.int()
        cache_position += 1

text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
text
['Simply put, the theory of relativity states that 1) the speed of light is constant, 2) the speed of light is the same for all observers, and 3) the laws of physics are the same for all observers.',
 'My favorite all time favorite condiment is ketchup. I love it on everything. I love it on my eggs, my fries, my chicken, my burgers, my hot dogs, my sandwiches, my salads, my p']
```

</hfoption>
<hfoption id="advanced usage: end-to-end generate compilation">

Compiling the entire `generate` function, in terms of code, is even simpler than in the basic usage: call `torch.compile` on `generate` to compile the entire function. No need to specify the use of the static cache: although it is compatible, dynamic cache (default) was faster in our benchmarks.

```py
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
import os
os.environ["TOKENIZERS_PARALLELISM"] = "false"  # To prevent long warnings :)

tokenizer = AutoTokenizer.from_pretrained("google/gemma-2b")
model = AutoModelForCausalLM.from_pretrained("google/gemma-2b", device_map="auto")

model.generate = torch.compile(model.generate, mode="reduce-overhead", fullgraph=True)
input_text = "The theory of special relativity states "
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")

outputs = model.generate(**input_ids)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
['The theory of special relativity states 1. The speed of light is constant in all inertial reference']
```

As a result, we compile not only the model forward pass, but also all input preparation, logit processor operations, and so on. The result should be a slightly `generate` call, compared to the basic usage example, and the compiled graph may be better suited to more exotic hardware devices or use cases. However, there are severe drawbacks in using this approach:
1. Compilation is much slower;
2.


key: _node_content
string_value:
{"id_": "542df6e7-1ccb-4fe4-84e8-701b65f6fa6a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7310b771-9506-4834-8b0c-51199b9e3546", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "hash": "68a458ed8c65861d8af91639f073c49a64d8a0e1217f62736bc11f914aed38be", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "68fbeaf1-76d9-4da2-a4b0-2d3f1f0bda4c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "hash": "998d83125a70b6ea81cd6fe506127d0de3b2e86e209701b574d9300b0679eedf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ba69fa22-d9d3-4060-8996-b6a8d8ad695a", "node_type": "1", "metadata": {}, "hash": "381b25edcb7292addf769260c52e45f4816662e1b448e5444b3a0fa9095cf5f1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10890, "end_char_idx": 14343, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
',
 'My favorite all time favorite condiment is ketchup. I love it on everything. I love it on my eggs, my fries, my chicken, my burgers, my hot dogs, my sandwiches, my salads, my p']
```

</hfoption>
<hfoption id="advanced usage: end-to-end generate compilation">

Compiling the entire `generate` function, in terms of code, is even simpler than in the basic usage: call `torch.compile` on `generate` to compile the entire function. No need to specify the use of the static cache: although it is compatible, dynamic cache (default) was faster in our benchmarks.

```py
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
import os
os.environ["TOKENIZERS_PARALLELISM"] = "false"  # To prevent long warnings :)

tokenizer = AutoTokenizer.from_pretrained("google/gemma-2b")
model = AutoModelForCausalLM.from_pretrained("google/gemma-2b", device_map="auto")

model.generate = torch.compile(model.generate, mode="reduce-overhead", fullgraph=True)
input_text = "The theory of special relativity states "
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")

outputs = model.generate(**input_ids)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
['The theory of special relativity states 1. The speed of light is constant in all inertial reference']
```

As a result, we compile not only the model forward pass, but also all input preparation, logit processor operations, and so on. The result should be a slightly `generate` call, compared to the basic usage example, and the compiled graph may be better suited to more exotic hardware devices or use cases. However, there are severe drawbacks in using this approach:
1. Compilation is much slower;
2. All parameterization of `generate` must be done through `generation_config`;
3. Many warnings and exceptions are suppressed -- we suggest testing with its uncompiled form first;
4. Although we are working on it, it is heavily feature restricted (for instance, at the time of writing, generation does not stop if an EOS token is selected).

</hfoption>
</hfoptions>

## Speculative decoding

> [!TIP]
> For a more in-depth explanation, take a look at the [Assisted Generation: a new direction toward low-latency text generation](https://hf.co/blog/assisted-generation) blog post!

Another issue with autoregression is that for each input token you need to load the model weights each time during the forward pass. This is slow and cumbersome for LLMs which have billions of parameters. Speculative decoding alleviates this slowdown by using a second smaller and faster assistant model to generate candidate tokens that are verified by the larger LLM in a single forward pass. If the verified tokens are correct, the LLM essentially gets them for "free" without having to generate them itself. There is no degradation in accuracy because the verification forward pass ensures the same outputs are generated as if the LLM had generated them on its own.

To get the largest speed up, the assistant model should be a lot smaller than the LLM so that it can generate tokens quickly. The assistant and LLM model must also share the same tokenizer to avoid re-encoding and decoding tokens.

> [!WARNING]
> Speculative decoding is only supported for the greedy search and sampling decoding strategies, and it also doesn't support batched inputs.

Enable speculative decoding by loading an assistant model and passing it to the [`~GenerationMixin.generate`] method.


key: _node_content
string_value:
{"id_": "ba69fa22-d9d3-4060-8996-b6a8d8ad695a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7310b771-9506-4834-8b0c-51199b9e3546", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "hash": "68a458ed8c65861d8af91639f073c49a64d8a0e1217f62736bc11f914aed38be", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "542df6e7-1ccb-4fe4-84e8-701b65f6fa6a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "hash": "11c75ec85aa79b8eb3818feaf71d06fa151a3f6e8f0697f7ea7d962aef480efc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ad0cb13e-f301-4646-aa9a-928cb9aad7a3", "node_type": "1", "metadata": {}, "hash": "76f67fe6a8a479472994d93f390addc7e3fb3746a26ee2951655e09a39eab1a2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12494, "end_char_idx": 15282, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, there are severe drawbacks in using this approach:
1. Compilation is much slower;
2. All parameterization of `generate` must be done through `generation_config`;
3. Many warnings and exceptions are suppressed -- we suggest testing with its uncompiled form first;
4. Although we are working on it, it is heavily feature restricted (for instance, at the time of writing, generation does not stop if an EOS token is selected).

</hfoption>
</hfoptions>

## Speculative decoding

> [!TIP]
> For a more in-depth explanation, take a look at the [Assisted Generation: a new direction toward low-latency text generation](https://hf.co/blog/assisted-generation) blog post!

Another issue with autoregression is that for each input token you need to load the model weights each time during the forward pass. This is slow and cumbersome for LLMs which have billions of parameters. Speculative decoding alleviates this slowdown by using a second smaller and faster assistant model to generate candidate tokens that are verified by the larger LLM in a single forward pass. If the verified tokens are correct, the LLM essentially gets them for "free" without having to generate them itself. There is no degradation in accuracy because the verification forward pass ensures the same outputs are generated as if the LLM had generated them on its own.

To get the largest speed up, the assistant model should be a lot smaller than the LLM so that it can generate tokens quickly. The assistant and LLM model must also share the same tokenizer to avoid re-encoding and decoding tokens.

> [!WARNING]
> Speculative decoding is only supported for the greedy search and sampling decoding strategies, and it also doesn't support batched inputs.

Enable speculative decoding by loading an assistant model and passing it to the [`~GenerationMixin.generate`] method.

<hfoptions id="spec-decoding">
<hfoption id="greedy search">

```py
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained("facebook/opt-1.3b")
inputs = tokenizer("Einstein's theory of relativity states", return_tensors="pt").to(device)

model = AutoModelForCausalLM.from_pretrained("facebook/opt-1.3b").to(device)
assistant_model = AutoModelForCausalLM.from_pretrained("facebook/opt-125m").to(device)
outputs = model.generate(**inputs, assistant_model=assistant_model)
tokenizer.batch_decode(outputs, skip_special_tokens=True)
["Einstein's theory of relativity states that the speed of light is constant.    "]
```

</hfoption>
<hfoption id="sampling">

For speculative sampling decoding, add the `do_sample` and `temperature` parameters to the [`~GenerationMixin.generate`] method in addition to the assistant model.


key: _node_content
string_value:
{"id_": "ad0cb13e-f301-4646-aa9a-928cb9aad7a3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7310b771-9506-4834-8b0c-51199b9e3546", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "hash": "68a458ed8c65861d8af91639f073c49a64d8a0e1217f62736bc11f914aed38be", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ba69fa22-d9d3-4060-8996-b6a8d8ad695a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "hash": "16e43673e9ff8085e944e7f7cb3c0860a9cd4b484d0003a629accdd983d37a38", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "81f72a5a-6d7b-4ee1-83da-392b45d51e01", "node_type": "1", "metadata": {}, "hash": "0248ab152570f78fe982fae2b54ae145f941996d0e9198c5bc89fb8b8a4f81e0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13563, "end_char_idx": 16616, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If the verified tokens are correct, the LLM essentially gets them for "free" without having to generate them itself. There is no degradation in accuracy because the verification forward pass ensures the same outputs are generated as if the LLM had generated them on its own.

To get the largest speed up, the assistant model should be a lot smaller than the LLM so that it can generate tokens quickly. The assistant and LLM model must also share the same tokenizer to avoid re-encoding and decoding tokens.

> [!WARNING]
> Speculative decoding is only supported for the greedy search and sampling decoding strategies, and it also doesn't support batched inputs.

Enable speculative decoding by loading an assistant model and passing it to the [`~GenerationMixin.generate`] method.

<hfoptions id="spec-decoding">
<hfoption id="greedy search">

```py
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained("facebook/opt-1.3b")
inputs = tokenizer("Einstein's theory of relativity states", return_tensors="pt").to(device)

model = AutoModelForCausalLM.from_pretrained("facebook/opt-1.3b").to(device)
assistant_model = AutoModelForCausalLM.from_pretrained("facebook/opt-125m").to(device)
outputs = model.generate(**inputs, assistant_model=assistant_model)
tokenizer.batch_decode(outputs, skip_special_tokens=True)
["Einstein's theory of relativity states that the speed of light is constant.    "]
```

</hfoption>
<hfoption id="sampling">

For speculative sampling decoding, add the `do_sample` and `temperature` parameters to the [`~GenerationMixin.generate`] method in addition to the assistant model.

```py
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained("facebook/opt-1.3b")
inputs = tokenizer("Einstein's theory of relativity states", return_tensors="pt").to(device)

model = AutoModelForCausalLM.from_pretrained("facebook/opt-1.3b").to(device)
assistant_model = AutoModelForCausalLM.from_pretrained("facebook/opt-125m").to(device)
outputs = model.generate(**inputs, assistant_model=assistant_model, do_sample=True, temperature=0.7)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
["Einstein's theory of relativity states that motion in the universe is not a straight line.\n"]
```

</hfoption>
</hfoptions>

### Prompt lookup decoding

Prompt lookup decoding is a variant of speculative decoding that is also compatible with greedy search and sampling. Prompt lookup works especially well for input-grounded tasks - such as summarization - where there is often overlapping words between the prompt and output. These overlapping n-grams are used as the LLM candidate tokens.

To enable prompt lookup decoding, specify the number of tokens that should be overlapping in the `prompt_lookup_num_tokens` parameter. Then you can pass this parameter to the [`~GenerationMixin.generate`] method.


key: _node_content
string_value:
{"id_": "81f72a5a-6d7b-4ee1-83da-392b45d51e01", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7310b771-9506-4834-8b0c-51199b9e3546", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "hash": "68a458ed8c65861d8af91639f073c49a64d8a0e1217f62736bc11f914aed38be", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ad0cb13e-f301-4646-aa9a-928cb9aad7a3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "hash": "b6af82ae002276126432bbf34002092508c9b57dc5d61f4ec727075e0c4240f3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0b1023e3-c35b-484b-9e30-fb98dbd1dff8", "node_type": "1", "metadata": {}, "hash": "b56d33337d8165730f160893853b895275fe35aaf5fc8e09916092fe532a7212", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15076, "end_char_idx": 18452, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```

</hfoption>
<hfoption id="sampling">

For speculative sampling decoding, add the `do_sample` and `temperature` parameters to the [`~GenerationMixin.generate`] method in addition to the assistant model.

```py
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained("facebook/opt-1.3b")
inputs = tokenizer("Einstein's theory of relativity states", return_tensors="pt").to(device)

model = AutoModelForCausalLM.from_pretrained("facebook/opt-1.3b").to(device)
assistant_model = AutoModelForCausalLM.from_pretrained("facebook/opt-125m").to(device)
outputs = model.generate(**inputs, assistant_model=assistant_model, do_sample=True, temperature=0.7)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
["Einstein's theory of relativity states that motion in the universe is not a straight line.\n"]
```

</hfoption>
</hfoptions>

### Prompt lookup decoding

Prompt lookup decoding is a variant of speculative decoding that is also compatible with greedy search and sampling. Prompt lookup works especially well for input-grounded tasks - such as summarization - where there is often overlapping words between the prompt and output. These overlapping n-grams are used as the LLM candidate tokens.

To enable prompt lookup decoding, specify the number of tokens that should be overlapping in the `prompt_lookup_num_tokens` parameter. Then you can pass this parameter to the [`~GenerationMixin.generate`] method.

<hfoptions id="pld">
<hfoption id="greedy decoding">

```py
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained("facebook/opt-1.3b")
inputs = tokenizer("The second law of thermodynamics states", return_tensors="pt").to(device)

model = AutoModelForCausalLM.from_pretrained("facebook/opt-1.3b").to(device)
assistant_model = AutoModelForCausalLM.from_pretrained("facebook/opt-125m").to(device)
outputs = model.generate(**inputs, prompt_lookup_num_tokens=3)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
['The second law of thermodynamics states that entropy increases with temperature.      ']
```

</hfoption>
<hfoption id="sampling">

For prompt lookup decoding with sampling, add the `do_sample` and `temperature` parameters to the [`~GenerationMixin.generate`] method.

```py
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained("facebook/opt-1.3b")
inputs = tokenizer("The second law of thermodynamics states", return_tensors="pt").to(device)

model = AutoModelForCausalLM.from_pretrained("facebook/opt-1.3b").to(device)
outputs = model.generate(**inputs, prompt_lookup_num_tokens=3, do_sample=True, temperature=0.7)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
["The second law of thermodynamics states that energy cannot be created nor destroyed. It's not a"]
```

</hfoption>
</hfoptions>

## Attention optimizations

A known issue with transformer models is that the self-attention mechanism grows quadratically in compute and memory with the number of input tokens. This limitation is only magnified in LLMs which handles much longer sequences.


key: _node_content
string_value:
{"id_": "0b1023e3-c35b-484b-9e30-fb98dbd1dff8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7310b771-9506-4834-8b0c-51199b9e3546", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "hash": "68a458ed8c65861d8af91639f073c49a64d8a0e1217f62736bc11f914aed38be", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "81f72a5a-6d7b-4ee1-83da-392b45d51e01", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "hash": "ab6136ef180c6ac74f3e88af63ad92531339f84c95b0932c1bfff356adf05f10", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5c1ce314-b410-4978-9978-c70d6d10464f", "node_type": "1", "metadata": {}, "hash": "9e2ed46688312f8ca138d030807a7268e277b8174b71ebe64bc144ba1f0264da", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17351, "end_char_idx": 20835, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```

</hfoption>
<hfoption id="sampling">

For prompt lookup decoding with sampling, add the `do_sample` and `temperature` parameters to the [`~GenerationMixin.generate`] method.

```py
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained("facebook/opt-1.3b")
inputs = tokenizer("The second law of thermodynamics states", return_tensors="pt").to(device)

model = AutoModelForCausalLM.from_pretrained("facebook/opt-1.3b").to(device)
outputs = model.generate(**inputs, prompt_lookup_num_tokens=3, do_sample=True, temperature=0.7)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
["The second law of thermodynamics states that energy cannot be created nor destroyed. It's not a"]
```

</hfoption>
</hfoptions>

## Attention optimizations

A known issue with transformer models is that the self-attention mechanism grows quadratically in compute and memory with the number of input tokens. This limitation is only magnified in LLMs which handles much longer sequences. To address this, try FlashAttention2 or PyTorch's scaled dot product attention (SDPA), which are more memory efficient attention implementations and can accelerate inference.

### FlashAttention-2

FlashAttention and [FlashAttention-2](./perf_infer_gpu_one#flashattention-2) break up the attention computation into smaller chunks and reduces the number of intermediate read/write operations to GPU memory to speed up inference. FlashAttention-2 improves on the original FlashAttention algorithm by also parallelizing over sequence length dimension and better partitioning work on the hardware to reduce synchronization and communication overhead.

To use FlashAttention-2, set `attn_implementation="flash_attention_2"` in the [`~PreTrainedModel.from_pretrained`] method.

```py
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

quant_config = BitsAndBytesConfig(load_in_8bit=True)
model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-2b",
    quantization_config=quant_config,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)
```

### PyTorch scaled dot product attention

Scaled dot product attention (SDPA) is automatically enabled in PyTorch 2.0 and it supports FlashAttention, xFormers, and PyTorch's C++ implementation. SDPA chooses the most performant attention algorithm if you're using a CUDA backend. For other backends, SDPA defaults to the PyTorch C++ implementation.

> [!TIP]
> SDPA supports FlashAttention-2 as long as you have the latest PyTorch version installed.

Use the [torch.backends.cuda.sdp_kernel](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html) context manager to explicitly enable or disable any of the three attention algorithms. For example, set `enable_flash=True` to enable FlashAttention.

```py
import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-2b",
    torch_dtype=torch.bfloat16,
)

with torch.backends.cuda.sdp_kernel(enable_flash=True, enable_math=False, enable_mem_efficient=False):
    outputs = model.generate(**inputs)
```

## Quantization

Quantization reduces the size of the LLM weights by storing them in a lower precision. This translates to lower memory usage and makes loading LLMs for inference more accessible if you're constrained by your GPUs memory.


key: _node_content
string_value:
{"id_": "5c1ce314-b410-4978-9978-c70d6d10464f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7310b771-9506-4834-8b0c-51199b9e3546", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "hash": "68a458ed8c65861d8af91639f073c49a64d8a0e1217f62736bc11f914aed38be", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0b1023e3-c35b-484b-9e30-fb98dbd1dff8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "hash": "9e9a31708f3b6f363ace8fc8437d8ce65471d2797414da11f23a07c2941b2805", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b9117e67-2587-4371-bf81-884e37b7057c", "node_type": "1", "metadata": {}, "hash": "5e16265a671d5180fb197170a1fdd86732e0b971c90b35594dc39477ad678e0d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19225, "end_char_idx": 22079, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

quant_config = BitsAndBytesConfig(load_in_8bit=True)
model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-2b",
    quantization_config=quant_config,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)
```

### PyTorch scaled dot product attention

Scaled dot product attention (SDPA) is automatically enabled in PyTorch 2.0 and it supports FlashAttention, xFormers, and PyTorch's C++ implementation. SDPA chooses the most performant attention algorithm if you're using a CUDA backend. For other backends, SDPA defaults to the PyTorch C++ implementation.

> [!TIP]
> SDPA supports FlashAttention-2 as long as you have the latest PyTorch version installed.

Use the [torch.backends.cuda.sdp_kernel](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html) context manager to explicitly enable or disable any of the three attention algorithms. For example, set `enable_flash=True` to enable FlashAttention.

```py
import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-2b",
    torch_dtype=torch.bfloat16,
)

with torch.backends.cuda.sdp_kernel(enable_flash=True, enable_math=False, enable_mem_efficient=False):
    outputs = model.generate(**inputs)
```

## Quantization

Quantization reduces the size of the LLM weights by storing them in a lower precision. This translates to lower memory usage and makes loading LLMs for inference more accessible if you're constrained by your GPUs memory. If you aren't limited by your GPU, you don't necessarily need to quantize your model because it can incur a small latency cost (except for AWQ and fused AWQ modules) due to the extra step required to quantize and dequantize the weights.

> [!TIP]
> There are many quantization libraries (see the [Quantization](./quantization) guide for more details) available, such as Quanto, AQLM, AWQ, and AutoGPTQ. Feel free to try them out and see which one works best for your use case. We also recommend reading the [Overview of natively supported quantization schemes in 🤗 Transformers](https://hf.co/blog/overview-quantization-transformers) blog post which compares AutoGPTQ and bitsandbytes.

Use the Model Memory Calculator below to estimate and compare how much memory is required to load a model. For example, try estimating how much memory it costs to load [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1).

<iframe
	src="https://hf-accelerate-model-memory-usage.hf.space"
	frameborder="0"
	width="850"
	height="450"
></iframe>

To load Mistral-7B-v0.1 in half-precision, set the `torch_dtype` parameter in the [`~transformers.AutoModelForCausalLM.from_pretrained`] method to `torch.bfloat16`. This requires 13.74GB of memory.


key: _node_content
string_value:
{"id_": "b9117e67-2587-4371-bf81-884e37b7057c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7310b771-9506-4834-8b0c-51199b9e3546", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "hash": "68a458ed8c65861d8af91639f073c49a64d8a0e1217f62736bc11f914aed38be", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5c1ce314-b410-4978-9978-c70d6d10464f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 5371, "retrieve_doc": true, "source": "transformers"}, "hash": "a06ead91ca8fba172eb27c4144fac1f3456169385cd7e2b796301ddee44dcd07", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20702, "end_char_idx": 22824, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This translates to lower memory usage and makes loading LLMs for inference more accessible if you're constrained by your GPUs memory. If you aren't limited by your GPU, you don't necessarily need to quantize your model because it can incur a small latency cost (except for AWQ and fused AWQ modules) due to the extra step required to quantize and dequantize the weights.

> [!TIP]
> There are many quantization libraries (see the [Quantization](./quantization) guide for more details) available, such as Quanto, AQLM, AWQ, and AutoGPTQ. Feel free to try them out and see which one works best for your use case. We also recommend reading the [Overview of natively supported quantization schemes in 🤗 Transformers](https://hf.co/blog/overview-quantization-transformers) blog post which compares AutoGPTQ and bitsandbytes.

Use the Model Memory Calculator below to estimate and compare how much memory is required to load a model. For example, try estimating how much memory it costs to load [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1).

<iframe
	src="https://hf-accelerate-model-memory-usage.hf.space"
	frameborder="0"
	width="850"
	height="450"
></iframe>

To load Mistral-7B-v0.1 in half-precision, set the `torch_dtype` parameter in the [`~transformers.AutoModelForCausalLM.from_pretrained`] method to `torch.bfloat16`. This requires 13.74GB of memory.

```py
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-v0.1", torch_dtype=torch.bfloat16, device_map="auto",
)
```

To load a quantized model (8-bit or 4-bit) for inference, try [bitsandbytes](https://hf.co/docs/bitsandbytes) and set the `load_in_4bit` or `load_in_8bit` parameters to `True`. Loading the model in 8-bits only requires 6.87 GB of memory.

```py
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
import torch

quant_config = BitsAndBytesConfig(load_in_8bit=True)
model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-v0.1", quantization_config=quant_config, device_map="auto"
)
```


key: _node_content
string_value:
{"id_": "e42b2841-6af7-457e-bb28-7dc8e231be21", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3633, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8a0d6a0b-7fe7-4d8d-9cad-0a44c95338b9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3633, "retrieve_doc": true, "source": "transformers"}, "hash": "68e788e16b16e5bd99336fbb45baed5162caec9be33f7bc2fdbbf6082c67e10b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "92219761-126a-4ec3-bc9b-3d11046d5dd6", "node_type": "1", "metadata": {}, "hash": "8ade258cbf1c0901b8893588a0fa93d16296686d236c09f3f7400d435c838a07", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3615, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Generation with LLMs

[[open-in-colab]]

LLMs, or Large Language Models, are the key component behind text generation. In a nutshell, they consist of large pretrained transformer models trained to predict the next word (or, more precisely, token) given some input text. Since they predict one token at a time, you need to do something more elaborate to generate new sentences other than just calling the model -- you need to do autoregressive generation.

Autoregressive generation is the inference-time procedure of iteratively calling a model with its own generated outputs, given a few initial inputs. In 🤗 Transformers, this is handled by the [`~generation.GenerationMixin.generate`] method, which is available to all models with generative capabilities.

This tutorial will show you how to:

* Generate text with an LLM
* Avoid common pitfalls
* Next steps to help you get the most out of your LLM

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install transformers bitsandbytes>=0.39.0 -q
```


## Generate text

A language model trained for [causal language modeling](tasks/language_modeling) takes a sequence of text tokens as input and returns the probability distribution for the next token.

<!-- [GIF 1 -- FWD PASS] -->
<figure class="image table text-center m-0 w-full">
    <video
        style="max-width: 90%; margin: auto;"
        autoplay loop muted playsinline
        src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/assisted-generation/gif_1_1080p.mov"
    ></video>
    <figcaption>"Forward pass of an LLM"</figcaption>
</figure>

A critical aspect of autoregressive generation with LLMs is how to select the next token from this probability distribution. Anything goes in this step as long as you end up with a token for the next iteration. This means it can be as simple as selecting the most likely token from the probability distribution or as complex as applying a dozen transformations before sampling from the resulting distribution.

<!-- [GIF 2 -- TEXT GENERATION] -->
<figure class="image table text-center m-0 w-full">
    <video
        style="max-width: 90%; margin: auto;"
        autoplay loop muted playsinline
        src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/assisted-generation/gif_2_1080p.mov"
    ></video>
    <figcaption>"Autoregressive generation iteratively selects the next token from a probability distribution to generate text"</figcaption>
</figure>

The process depicted above is repeated iteratively until some stopping condition is reached. Ideally, the stopping condition is dictated by the model, which should learn when to output an end-of-sequence (`EOS`) token. If this is not the case, generation stops when some predefined maximum length is reached.

Properly setting up the token selection step and the stopping condition is essential to make your model behave as you'd expect on your task. That is why we have a [`~generation.GenerationConfig`] file associated with each model, which contains a good default generative parameterization and is loaded alongside your model.

Let's talk code!

<Tip>

If you're interested in basic LLM usage, our high-level [`Pipeline`](pipeline_tutorial) interface is a great starting point. However, LLMs often require advanced features like quantization and fine control of the token selection step, which is best done through [`~generation.GenerationMixin.generate`]. Autoregressive generation with LLMs is also resource-intensive and should be executed on a GPU for adequate throughput.


key: _node_content
string_value:
{"id_": "92219761-126a-4ec3-bc9b-3d11046d5dd6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3633, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8a0d6a0b-7fe7-4d8d-9cad-0a44c95338b9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3633, "retrieve_doc": true, "source": "transformers"}, "hash": "68e788e16b16e5bd99336fbb45baed5162caec9be33f7bc2fdbbf6082c67e10b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e42b2841-6af7-457e-bb28-7dc8e231be21", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3633, "retrieve_doc": true, "source": "transformers"}, "hash": "4d4b4c7d595b1f38aad073a84d6544af8f9ececb2ee1a64ff7253151eedd124d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "42a742c1-4a71-4758-a7b1-ffc688b350bd", "node_type": "1", "metadata": {}, "hash": "ed92f45d2f6f0334ecc0a16537d491eb128a24b5c930acab9e2960e54f0f23e1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1763, "end_char_idx": 5251, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Anything goes in this step as long as you end up with a token for the next iteration. This means it can be as simple as selecting the most likely token from the probability distribution or as complex as applying a dozen transformations before sampling from the resulting distribution.

<!-- [GIF 2 -- TEXT GENERATION] -->
<figure class="image table text-center m-0 w-full">
    <video
        style="max-width: 90%; margin: auto;"
        autoplay loop muted playsinline
        src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/assisted-generation/gif_2_1080p.mov"
    ></video>
    <figcaption>"Autoregressive generation iteratively selects the next token from a probability distribution to generate text"</figcaption>
</figure>

The process depicted above is repeated iteratively until some stopping condition is reached. Ideally, the stopping condition is dictated by the model, which should learn when to output an end-of-sequence (`EOS`) token. If this is not the case, generation stops when some predefined maximum length is reached.

Properly setting up the token selection step and the stopping condition is essential to make your model behave as you'd expect on your task. That is why we have a [`~generation.GenerationConfig`] file associated with each model, which contains a good default generative parameterization and is loaded alongside your model.

Let's talk code!

<Tip>

If you're interested in basic LLM usage, our high-level [`Pipeline`](pipeline_tutorial) interface is a great starting point. However, LLMs often require advanced features like quantization and fine control of the token selection step, which is best done through [`~generation.GenerationMixin.generate`]. Autoregressive generation with LLMs is also resource-intensive and should be executed on a GPU for adequate throughput.

</Tip>

First, you need to load the model.

```py
>>> from transformers import AutoModelForCausalLM

>>> model = AutoModelForCausalLM.from_pretrained(
...     "mistralai/Mistral-7B-v0.1", device_map="auto", load_in_4bit=True
... )
```

You'll notice two flags in the `from_pretrained` call:

 - `device_map` ensures the model is moved to your GPU(s)
 - `load_in_4bit` applies [4-bit dynamic quantization](main_classes/quantization) to massively reduce the resource requirements

There are other ways to initialize a model, but this is a good baseline to begin with an LLM.

Next, you need to preprocess your text input with a [tokenizer](tokenizer_summary).

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1", padding_side="left")
>>> model_inputs = tokenizer(["A list of colors: red, blue"], return_tensors="pt").to("cuda")
```

The `model_inputs` variable holds the tokenized text input, as well as the attention mask. While [`~generation.GenerationMixin.generate`] does its best effort to infer the attention mask when it is not passed, we recommend passing it whenever possible for optimal results.

After tokenizing the inputs, you can call the [`~generation.GenerationMixin.generate`] method to returns the generated tokens. The generated tokens then should be converted to text before printing.

```py
>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'A list of colors: red, blue, green, yellow, orange, purple, pink,'
```

Finally, you don't need to do it one sequence at a time!


key: _node_content
string_value:
{"id_": "42a742c1-4a71-4758-a7b1-ffc688b350bd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3633, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8a0d6a0b-7fe7-4d8d-9cad-0a44c95338b9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3633, "retrieve_doc": true, "source": "transformers"}, "hash": "68e788e16b16e5bd99336fbb45baed5162caec9be33f7bc2fdbbf6082c67e10b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "92219761-126a-4ec3-bc9b-3d11046d5dd6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3633, "retrieve_doc": true, "source": "transformers"}, "hash": "1d8e51a1764cf212cfafdf4325089d91a6cb6cd43494344b235f90750f02095c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4a5dcc13-dc93-41d9-a01a-da33b903d942", "node_type": "1", "metadata": {}, "hash": "a11e967a82ca12cf79fcdc2c1c41f470651fe99f7ac274a0492d888249a6c76a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3617, "end_char_idx": 6010, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

First, you need to load the model.

```py
>>> from transformers import AutoModelForCausalLM

>>> model = AutoModelForCausalLM.from_pretrained(
...     "mistralai/Mistral-7B-v0.1", device_map="auto", load_in_4bit=True
... )
```

You'll notice two flags in the `from_pretrained` call:

 - `device_map` ensures the model is moved to your GPU(s)
 - `load_in_4bit` applies [4-bit dynamic quantization](main_classes/quantization) to massively reduce the resource requirements

There are other ways to initialize a model, but this is a good baseline to begin with an LLM.

Next, you need to preprocess your text input with a [tokenizer](tokenizer_summary).

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1", padding_side="left")
>>> model_inputs = tokenizer(["A list of colors: red, blue"], return_tensors="pt").to("cuda")
```

The `model_inputs` variable holds the tokenized text input, as well as the attention mask. While [`~generation.GenerationMixin.generate`] does its best effort to infer the attention mask when it is not passed, we recommend passing it whenever possible for optimal results.

After tokenizing the inputs, you can call the [`~generation.GenerationMixin.generate`] method to returns the generated tokens. The generated tokens then should be converted to text before printing.

```py
>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'A list of colors: red, blue, green, yellow, orange, purple, pink,'
```

Finally, you don't need to do it one sequence at a time! You can batch your inputs, which will greatly improve the throughput at a small latency and memory cost. All you need to do is to make sure you pad your inputs properly (more on that below).

```py
>>> tokenizer.pad_token = tokenizer.eos_token  # Most LLMs don't have a pad token by default
>>> model_inputs = tokenizer(
...     ["A list of colors: red, blue", "Portugal is"], return_tensors="pt", padding=True
... ).to("cuda")
>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
['A list of colors: red, blue, green, yellow, orange, purple, pink,',
'Portugal is a country in southwestern Europe, on the Iber']
```

And that's it! In a few lines of code, you can harness the power of an LLM.


key: _node_content
string_value:
{"id_": "4a5dcc13-dc93-41d9-a01a-da33b903d942", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3633, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8a0d6a0b-7fe7-4d8d-9cad-0a44c95338b9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3633, "retrieve_doc": true, "source": "transformers"}, "hash": "68e788e16b16e5bd99336fbb45baed5162caec9be33f7bc2fdbbf6082c67e10b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "42a742c1-4a71-4758-a7b1-ffc688b350bd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3633, "retrieve_doc": true, "source": "transformers"}, "hash": "be46d2c327094c3844b0f22f31691ce976dc919fa05e0b1d02515ad406fb2f87", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "192e571a-a0bd-4be1-8de4-684686d344eb", "node_type": "1", "metadata": {}, "hash": "53d392ad922752b7299d8be8c25a38dd47f1c6dd0db6ee72ace1baf2fa6b2d34", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4612, "end_char_idx": 7132, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
While [`~generation.GenerationMixin.generate`] does its best effort to infer the attention mask when it is not passed, we recommend passing it whenever possible for optimal results.

After tokenizing the inputs, you can call the [`~generation.GenerationMixin.generate`] method to returns the generated tokens. The generated tokens then should be converted to text before printing.

```py
>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'A list of colors: red, blue, green, yellow, orange, purple, pink,'
```

Finally, you don't need to do it one sequence at a time! You can batch your inputs, which will greatly improve the throughput at a small latency and memory cost. All you need to do is to make sure you pad your inputs properly (more on that below).

```py
>>> tokenizer.pad_token = tokenizer.eos_token  # Most LLMs don't have a pad token by default
>>> model_inputs = tokenizer(
...     ["A list of colors: red, blue", "Portugal is"], return_tensors="pt", padding=True
... ).to("cuda")
>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
['A list of colors: red, blue, green, yellow, orange, purple, pink,',
'Portugal is a country in southwestern Europe, on the Iber']
```

And that's it! In a few lines of code, you can harness the power of an LLM.


## Common pitfalls

There are many [generation strategies](generation_strategies), and sometimes the default values may not be appropriate for your use case. If your outputs aren't aligned with what you're expecting, we've created a list of the most common pitfalls and how to avoid them.

```py
>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")
>>> tokenizer.pad_token = tokenizer.eos_token  # Most LLMs don't have a pad token by default
>>> model = AutoModelForCausalLM.from_pretrained(
...     "mistralai/Mistral-7B-v0.1", device_map="auto", load_in_4bit=True
... )
```

### Generated output is too short/long

If not specified in the [`~generation.GenerationConfig`] file, `generate` returns up to 20 tokens by default. We highly recommend manually setting `max_new_tokens` in your `generate` call to control the maximum number of new tokens it can return. Keep in mind LLMs (more precisely, [decoder-only models](https://huggingface.co/learn/nlp-course/chapter1/6?fw=pt)) also return the input prompt as part of the output.


key: _node_content
string_value:
{"id_": "192e571a-a0bd-4be1-8de4-684686d344eb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3633, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8a0d6a0b-7fe7-4d8d-9cad-0a44c95338b9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3633, "retrieve_doc": true, "source": "transformers"}, "hash": "68e788e16b16e5bd99336fbb45baed5162caec9be33f7bc2fdbbf6082c67e10b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4a5dcc13-dc93-41d9-a01a-da33b903d942", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3633, "retrieve_doc": true, "source": "transformers"}, "hash": "743d4cdff25237f3e93a96ef463c4c13ca245d0426b95beaffb432524709eefe", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ecd36066-bf27-415e-ad1d-9c42fdd1bdbc", "node_type": "1", "metadata": {}, "hash": "113cdfb0e5a37e902b267751b2595782b1841f7982bab0bb5c4da5af328728cf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5950, "end_char_idx": 9027, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In a few lines of code, you can harness the power of an LLM.


## Common pitfalls

There are many [generation strategies](generation_strategies), and sometimes the default values may not be appropriate for your use case. If your outputs aren't aligned with what you're expecting, we've created a list of the most common pitfalls and how to avoid them.

```py
>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")
>>> tokenizer.pad_token = tokenizer.eos_token  # Most LLMs don't have a pad token by default
>>> model = AutoModelForCausalLM.from_pretrained(
...     "mistralai/Mistral-7B-v0.1", device_map="auto", load_in_4bit=True
... )
```

### Generated output is too short/long

If not specified in the [`~generation.GenerationConfig`] file, `generate` returns up to 20 tokens by default. We highly recommend manually setting `max_new_tokens` in your `generate` call to control the maximum number of new tokens it can return. Keep in mind LLMs (more precisely, [decoder-only models](https://huggingface.co/learn/nlp-course/chapter1/6?fw=pt)) also return the input prompt as part of the output.


```py
>>> model_inputs = tokenizer(["A sequence of numbers: 1, 2"], return_tensors="pt").to("cuda")

>>> # By default, the output will contain up to 20 tokens
>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'A sequence of numbers: 1, 2, 3, 4, 5'

>>> # Setting `max_new_tokens` allows you to control the maximum length
>>> generated_ids = model.generate(**model_inputs, max_new_tokens=50)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'A sequence of numbers: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,'
```

### Incorrect generation mode

By default, and unless specified in the [`~generation.GenerationConfig`] file, `generate` selects the most likely token at each iteration (greedy decoding). Depending on your task, this may be undesirable; creative tasks like chatbots or writing an essay benefit from sampling. On the other hand, input-grounded tasks like audio transcription or translation benefit from greedy decoding. Enable sampling with `do_sample=True`, and you can learn more about this topic in this [blog post](https://huggingface.co/blog/how-to-generate).

```py
>>> # Set seed or reproducibility -- you don't need this unless you want full reproducibility
>>> from transformers import set_seed
>>> set_seed(42)

>>> model_inputs = tokenizer(["I am a cat."], return_tensors="pt").to("cuda")

>>> # LLM + greedy decoding = repetitive, boring output
>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'I am a cat. I am a cat. I am a cat. I am a cat'

>>> # With sampling, the output becomes more creative!
>>> generated_ids = model.generate(**model_inputs, do_sample=True)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'I am a cat.  Specifically, I am an indoor-only cat.


key: _node_content
string_value:
{"id_": "ecd36066-bf27-415e-ad1d-9c42fdd1bdbc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3633, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8a0d6a0b-7fe7-4d8d-9cad-0a44c95338b9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3633, "retrieve_doc": true, "source": "transformers"}, "hash": "68e788e16b16e5bd99336fbb45baed5162caec9be33f7bc2fdbbf6082c67e10b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "192e571a-a0bd-4be1-8de4-684686d344eb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3633, "retrieve_doc": true, "source": "transformers"}, "hash": "427de0209d3e5741d05d43abb6cac9ac4383104cd63355bc4ba8c627ea64a615", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ff269aa2-e0a7-4496-8893-f2389fcb5ba9", "node_type": "1", "metadata": {}, "hash": "c4ae41c5777a36ba310ec9edb457541ab67e3b5379e0b262c94d74b7cd631834", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7942, "end_char_idx": 10796, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Depending on your task, this may be undesirable; creative tasks like chatbots or writing an essay benefit from sampling. On the other hand, input-grounded tasks like audio transcription or translation benefit from greedy decoding. Enable sampling with `do_sample=True`, and you can learn more about this topic in this [blog post](https://huggingface.co/blog/how-to-generate).

```py
>>> # Set seed or reproducibility -- you don't need this unless you want full reproducibility
>>> from transformers import set_seed
>>> set_seed(42)

>>> model_inputs = tokenizer(["I am a cat."], return_tensors="pt").to("cuda")

>>> # LLM + greedy decoding = repetitive, boring output
>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'I am a cat. I am a cat. I am a cat. I am a cat'

>>> # With sampling, the output becomes more creative!
>>> generated_ids = model.generate(**model_inputs, do_sample=True)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'I am a cat.  Specifically, I am an indoor-only cat.  I'
```

### Wrong padding side

LLMs are [decoder-only](https://huggingface.co/learn/nlp-course/chapter1/6?fw=pt) architectures, meaning they continue to iterate on your input prompt. If your inputs do not have the same length, they need to be padded. Since LLMs are not trained to continue from pad tokens, your input needs to be left-padded. Make sure you also don't forget to pass the attention mask to generate!

```py
>>> # The tokenizer initialized above has right-padding active by default: the 1st sequence,
>>> # which is shorter, has padding on the right side. Generation fails to capture the logic.
>>> model_inputs = tokenizer(
...     ["1, 2, 3", "A, B, C, D, E"], padding=True, return_tensors="pt"
... ).to("cuda")
>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'1, 2, 33333333333'

>>> # With left-padding, it works as expected!
>>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1", padding_side="left")
>>> tokenizer.pad_token = tokenizer.eos_token  # Most LLMs don't have a pad token by default
>>> model_inputs = tokenizer(
...     ["1, 2, 3", "A, B, C, D, E"], padding=True, return_tensors="pt"
... ).to("cuda")
>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'1, 2, 3, 4, 5, 6,'
```

### Wrong prompt

Some models and tasks expect a certain input prompt format to work properly. When this format is not applied, you will get a silent performance degradation: the model kinda works, but not as well as if you were following the expected prompt. More information about prompting, including which models and tasks need to be careful, is available in this [guide](tasks/prompting).


key: _node_content
string_value:
{"id_": "ff269aa2-e0a7-4496-8893-f2389fcb5ba9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3633, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8a0d6a0b-7fe7-4d8d-9cad-0a44c95338b9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3633, "retrieve_doc": true, "source": "transformers"}, "hash": "68e788e16b16e5bd99336fbb45baed5162caec9be33f7bc2fdbbf6082c67e10b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ecd36066-bf27-415e-ad1d-9c42fdd1bdbc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3633, "retrieve_doc": true, "source": "transformers"}, "hash": "cb9f1a27b6fbe2691b3f2967a1c98ccb79498b5fbba00e6a0098d82d7541a27b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "72b80d3b-8578-4e65-9fa1-2bc61f1f4cec", "node_type": "1", "metadata": {}, "hash": "a89520078fde28d8d36aadfd5688f9bed45706a5660444e89f25497e5b21f980", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9281, "end_char_idx": 12048, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Since LLMs are not trained to continue from pad tokens, your input needs to be left-padded. Make sure you also don't forget to pass the attention mask to generate!

```py
>>> # The tokenizer initialized above has right-padding active by default: the 1st sequence,
>>> # which is shorter, has padding on the right side. Generation fails to capture the logic.
>>> model_inputs = tokenizer(
...     ["1, 2, 3", "A, B, C, D, E"], padding=True, return_tensors="pt"
... ).to("cuda")
>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'1, 2, 33333333333'

>>> # With left-padding, it works as expected!
>>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1", padding_side="left")
>>> tokenizer.pad_token = tokenizer.eos_token  # Most LLMs don't have a pad token by default
>>> model_inputs = tokenizer(
...     ["1, 2, 3", "A, B, C, D, E"], padding=True, return_tensors="pt"
... ).to("cuda")
>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'1, 2, 3, 4, 5, 6,'
```

### Wrong prompt

Some models and tasks expect a certain input prompt format to work properly. When this format is not applied, you will get a silent performance degradation: the model kinda works, but not as well as if you were following the expected prompt. More information about prompting, including which models and tasks need to be careful, is available in this [guide](tasks/prompting). Let's see an example with a chat LLM, which makes use of [chat templating](chat_templating):

```python
>>> tokenizer = AutoTokenizer.from_pretrained("HuggingFaceH4/zephyr-7b-alpha")
>>> model = AutoModelForCausalLM.from_pretrained(
...     "HuggingFaceH4/zephyr-7b-alpha", device_map="auto", load_in_4bit=True
... )
>>> set_seed(0)
>>> prompt = """How many helicopters can a human eat in one sitting? Reply as a thug."""
>>> model_inputs = tokenizer([prompt], return_tensors="pt").to("cuda")
>>> input_length = model_inputs.input_ids.shape[1]
>>> generated_ids = model.generate(**model_inputs, max_new_tokens=20)
>>> print(tokenizer.batch_decode(generated_ids[:, input_length:], skip_special_tokens=True)[0])
"I'm not a thug, but i can tell you that a human cannot eat"
>>> # Oh no, it did not follow our instruction to reply as a thug! Let's see what happens when we write
>>> # a better prompt and use the right template for this model (through `tokenizer.apply_chat_template`)

>>> set_seed(0)
>>> messages = [
...     {
...         "role": "system",
...         "content": "You are a friendly chatbot who always responds in the style of a thug",
...     },
...     {"role": "user", "content": "How many helicopters can a human eat in one sitting?


key: _node_content
string_value:
{"id_": "72b80d3b-8578-4e65-9fa1-2bc61f1f4cec", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3633, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8a0d6a0b-7fe7-4d8d-9cad-0a44c95338b9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3633, "retrieve_doc": true, "source": "transformers"}, "hash": "68e788e16b16e5bd99336fbb45baed5162caec9be33f7bc2fdbbf6082c67e10b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ff269aa2-e0a7-4496-8893-f2389fcb5ba9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3633, "retrieve_doc": true, "source": "transformers"}, "hash": "24b86945479527560de7bb53fc74d0e6831dc47874f430d8d397983b78e0ca38", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0a80d378-629b-4006-8323-c2cf4700c24b", "node_type": "1", "metadata": {}, "hash": "219ecfe18a4e199d6f3cbba60e4f0e289903260ad411dc61b7f8db47a6af37d7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10498, "end_char_idx": 13715, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
When this format is not applied, you will get a silent performance degradation: the model kinda works, but not as well as if you were following the expected prompt. More information about prompting, including which models and tasks need to be careful, is available in this [guide](tasks/prompting). Let's see an example with a chat LLM, which makes use of [chat templating](chat_templating):

```python
>>> tokenizer = AutoTokenizer.from_pretrained("HuggingFaceH4/zephyr-7b-alpha")
>>> model = AutoModelForCausalLM.from_pretrained(
...     "HuggingFaceH4/zephyr-7b-alpha", device_map="auto", load_in_4bit=True
... )
>>> set_seed(0)
>>> prompt = """How many helicopters can a human eat in one sitting? Reply as a thug."""
>>> model_inputs = tokenizer([prompt], return_tensors="pt").to("cuda")
>>> input_length = model_inputs.input_ids.shape[1]
>>> generated_ids = model.generate(**model_inputs, max_new_tokens=20)
>>> print(tokenizer.batch_decode(generated_ids[:, input_length:], skip_special_tokens=True)[0])
"I'm not a thug, but i can tell you that a human cannot eat"
>>> # Oh no, it did not follow our instruction to reply as a thug! Let's see what happens when we write
>>> # a better prompt and use the right template for this model (through `tokenizer.apply_chat_template`)

>>> set_seed(0)
>>> messages = [
...     {
...         "role": "system",
...         "content": "You are a friendly chatbot who always responds in the style of a thug",
...     },
...     {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
... ]
>>> model_inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to("cuda")
>>> input_length = model_inputs.shape[1]
>>> generated_ids = model.generate(model_inputs, do_sample=True, max_new_tokens=20)
>>> print(tokenizer.batch_decode(generated_ids[:, input_length:], skip_special_tokens=True)[0])
'None, you thug. How bout you try to focus on more useful questions?'
>>> # As we can see, it followed a proper thug style 😎
```

## Further resources

While the autoregressive generation process is relatively straightforward, making the most out of your LLM can be a challenging endeavor because there are many moving parts. For your next steps to help you dive deeper into LLM usage and understanding:

### Advanced generate usage

1. Guide on how to [control different generation methods](generation_strategies), how to set up the generation configuration file, and how to stream the output;
2. [Accelerating text generation](llm_optims);
3. [Prompt templates for chat LLMs](chat_templating);
4. [Prompt design guide](tasks/prompting);
5. API reference on [`~generation.GenerationConfig`], [`~generation.GenerationMixin.generate`], and [generate-related classes](internal/generation_utils). Most of the classes, including the logits processors, have usage examples!

### LLM leaderboards

1. [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard), which focuses on the quality of the open-source models;
2. [Open LLM-Perf Leaderboard](https://huggingface.co/spaces/optimum/llm-perf-leaderboard), which focuses on LLM throughput.

### Latency, throughput and memory utilization

1.


key: _node_content
string_value:
{"id_": "0a80d378-629b-4006-8323-c2cf4700c24b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3633, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8a0d6a0b-7fe7-4d8d-9cad-0a44c95338b9", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3633, "retrieve_doc": true, "source": "transformers"}, "hash": "68e788e16b16e5bd99336fbb45baed5162caec9be33f7bc2fdbbf6082c67e10b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "72b80d3b-8578-4e65-9fa1-2bc61f1f4cec", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3633, "retrieve_doc": true, "source": "transformers"}, "hash": "1c45e0213ae293de2ade271cb5bb9fe35a8cd3d5e23b3d8dda7910d963fa4259", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12048, "end_char_idx": 14624, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"},
... ]
>>> model_inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to("cuda")
>>> input_length = model_inputs.shape[1]
>>> generated_ids = model.generate(model_inputs, do_sample=True, max_new_tokens=20)
>>> print(tokenizer.batch_decode(generated_ids[:, input_length:], skip_special_tokens=True)[0])
'None, you thug. How bout you try to focus on more useful questions?'
>>> # As we can see, it followed a proper thug style 😎
```

## Further resources

While the autoregressive generation process is relatively straightforward, making the most out of your LLM can be a challenging endeavor because there are many moving parts. For your next steps to help you dive deeper into LLM usage and understanding:

### Advanced generate usage

1. Guide on how to [control different generation methods](generation_strategies), how to set up the generation configuration file, and how to stream the output;
2. [Accelerating text generation](llm_optims);
3. [Prompt templates for chat LLMs](chat_templating);
4. [Prompt design guide](tasks/prompting);
5. API reference on [`~generation.GenerationConfig`], [`~generation.GenerationMixin.generate`], and [generate-related classes](internal/generation_utils). Most of the classes, including the logits processors, have usage examples!

### LLM leaderboards

1. [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard), which focuses on the quality of the open-source models;
2. [Open LLM-Perf Leaderboard](https://huggingface.co/spaces/optimum/llm-perf-leaderboard), which focuses on LLM throughput.

### Latency, throughput and memory utilization

1. Guide on how to [optimize LLMs for speed and memory](llm_tutorial_optimization);
2. Guide on [quantization](main_classes/quantization) such as bitsandbytes and autogptq, which shows you how to drastically reduce your memory requirements.

### Related libraries

1. [`optimum`](https://github.com/huggingface/optimum), an extension of 🤗 Transformers that optimizes for specific hardware devices.
2. [`outlines`](https://github.com/outlines-dev/outlines), a library where you can constrain text generation (e.g. to generate JSON files);
3. [`SynCode`](https://github.com/uiuc-focal-lab/syncode), a library for context-free grammar guided generation. (e.g. JSON, SQL, Python)
4. [`text-generation-inference`](https://github.com/huggingface/text-generation-inference), a production-ready server for LLMs;
5. [`text-generation-webui`](https://github.com/oobabooga/text-generation-webui), a UI for text generation;


key: _node_content
string_value:
{"id_": "0c73be4e-0b70-4635-8571-ac3c97501dc7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/torchscript", "title": "Export to TorchScript", "tokens": 2219, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e2044816-f84e-46f2-99ea-eab21a197414", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/torchscript", "title": "Export to TorchScript", "tokens": 2219, "retrieve_doc": true, "source": "transformers"}, "hash": "43ce469e3ac7a2d78ae89c22727d8fa0878a46070d605e16e52e5854e762250b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b6524db2-8850-4c7f-a6d0-fc1775bae675", "node_type": "1", "metadata": {}, "hash": "52e5ba33aed87ce49846f926e41404dc4fc7d133f0b6f5e5ad56f603cf8d21ef", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3273, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Export to TorchScript

<Tip>

This is the very beginning of our experiments with TorchScript and we are still
exploring its capabilities with variable-input-size models. It is a focus of interest to
us and we will deepen our analysis in upcoming releases, with more code examples, a more
flexible implementation, and benchmarks comparing Python-based codes with compiled
TorchScript.

</Tip>

According to the [TorchScript documentation](https://pytorch.org/docs/stable/jit.html):

> TorchScript is a way to create serializable and optimizable models from PyTorch code.

There are two PyTorch modules, [JIT and
TRACE](https://pytorch.org/docs/stable/jit.html), that allow developers to export their
models to be reused in other programs like efficiency-oriented C++ programs.

We provide an interface that allows you to export 🤗 Transformers models to TorchScript
so they can be reused in a different environment than PyTorch-based Python programs.
Here, we explain how to export and use our models using TorchScript.

Exporting a model requires two things:

- model instantiation with the `torchscript` flag
- a forward pass with dummy inputs

These necessities imply several things developers should be careful about as detailed
below.

## TorchScript flag and tied weights

The `torchscript` flag is necessary because most of the 🤗 Transformers language models
have tied weights between their `Embedding` layer and their `Decoding` layer.
TorchScript does not allow you to export models that have tied weights, so it is
necessary to untie and clone the weights beforehand.

Models instantiated with the `torchscript` flag have their `Embedding` layer and
`Decoding` layer separated, which means that they should not be trained down the line.
Training would desynchronize the two layers, leading to unexpected results.

This is not the case for models that do not have a language model head, as those do not
have tied weights. These models can be safely exported without the `torchscript` flag.

## Dummy inputs and standard lengths

The dummy inputs are used for a models forward pass. While the inputs' values are
propagated through the layers, PyTorch keeps track of the different operations executed
on each tensor. These recorded operations are then used to create the *trace* of the
model.

The trace is created relative to the inputs' dimensions. It is therefore constrained by
the dimensions of the dummy input, and will not work for any other sequence length or
batch size. When trying with a different size, the following error is raised:

```
`The expanded size of the tensor (3) must match the existing size (7) at non-singleton dimension 2`
```

We recommended you trace the model with a dummy input size at least as large as the
largest input that will be fed to the model during inference. Padding can help fill the
missing values. However, since the model is traced with a larger input size, the
dimensions of the matrix will also be large, resulting in more calculations.

Be careful of the total number of operations done on each input and follow the
performance closely when exporting varying sequence-length models.

## Using TorchScript in Python

This section demonstrates how to save and load models as well as how to use the trace
for inference.


key: _node_content
string_value:
{"id_": "b6524db2-8850-4c7f-a6d0-fc1775bae675", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/torchscript", "title": "Export to TorchScript", "tokens": 2219, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e2044816-f84e-46f2-99ea-eab21a197414", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/torchscript", "title": "Export to TorchScript", "tokens": 2219, "retrieve_doc": true, "source": "transformers"}, "hash": "43ce469e3ac7a2d78ae89c22727d8fa0878a46070d605e16e52e5854e762250b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0c73be4e-0b70-4635-8571-ac3c97501dc7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/torchscript", "title": "Export to TorchScript", "tokens": 2219, "retrieve_doc": true, "source": "transformers"}, "hash": "dee6b8f433a21aad177805dc24af7d77bd8163bfada5ddde0bec213202d7429c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d561cb00-6f05-4721-a9d2-9648206d654b", "node_type": "1", "metadata": {}, "hash": "3e01464f563ce2f71b32676684c7482e3face88131c2997f8c07b500a5c2834d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1444, "end_char_idx": 4263, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
TorchScript does not allow you to export models that have tied weights, so it is
necessary to untie and clone the weights beforehand.

Models instantiated with the `torchscript` flag have their `Embedding` layer and
`Decoding` layer separated, which means that they should not be trained down the line.
Training would desynchronize the two layers, leading to unexpected results.

This is not the case for models that do not have a language model head, as those do not
have tied weights. These models can be safely exported without the `torchscript` flag.

## Dummy inputs and standard lengths

The dummy inputs are used for a models forward pass. While the inputs' values are
propagated through the layers, PyTorch keeps track of the different operations executed
on each tensor. These recorded operations are then used to create the *trace* of the
model.

The trace is created relative to the inputs' dimensions. It is therefore constrained by
the dimensions of the dummy input, and will not work for any other sequence length or
batch size. When trying with a different size, the following error is raised:

```
`The expanded size of the tensor (3) must match the existing size (7) at non-singleton dimension 2`
```

We recommended you trace the model with a dummy input size at least as large as the
largest input that will be fed to the model during inference. Padding can help fill the
missing values. However, since the model is traced with a larger input size, the
dimensions of the matrix will also be large, resulting in more calculations.

Be careful of the total number of operations done on each input and follow the
performance closely when exporting varying sequence-length models.

## Using TorchScript in Python

This section demonstrates how to save and load models as well as how to use the trace
for inference.

### Saving a model

To export a `BertModel` with TorchScript, instantiate `BertModel` from the `BertConfig`
class and then save it to disk under the filename `traced_bert.pt`:

```python
from transformers import BertModel, BertTokenizer, BertConfig
import torch

enc = BertTokenizer.from_pretrained("google-bert/bert-base-uncased")

# Tokenizing input text
text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
tokenized_text = enc.tokenize(text)

# Masking one of the input tokens
masked_index = 8
tokenized_text[masked_index] = "[MASK]"
indexed_tokens = enc.convert_tokens_to_ids(tokenized_text)
segments_ids = [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]

# Creating a dummy input
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])
dummy_input = [tokens_tensor, segments_tensors]

# Initializing the model with the torchscript flag
# Flag set to True even though it is not necessary as this model does not have an LM Head.


key: _node_content
string_value:
{"id_": "d561cb00-6f05-4721-a9d2-9648206d654b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/torchscript", "title": "Export to TorchScript", "tokens": 2219, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e2044816-f84e-46f2-99ea-eab21a197414", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/torchscript", "title": "Export to TorchScript", "tokens": 2219, "retrieve_doc": true, "source": "transformers"}, "hash": "43ce469e3ac7a2d78ae89c22727d8fa0878a46070d605e16e52e5854e762250b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b6524db2-8850-4c7f-a6d0-fc1775bae675", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/torchscript", "title": "Export to TorchScript", "tokens": 2219, "retrieve_doc": true, "source": "transformers"}, "hash": "cafd6bc8d33a1dde6a755aed2ccec2daea33b10ed509e2449caaf91ab23651f2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8c7cb2bd-220f-4562-8c1a-bdc27869d6b6", "node_type": "1", "metadata": {}, "hash": "903ed2d58ac048bf9b1a1582a8992dc03fa1b128394875b48eb5f15d182779d9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2809, "end_char_idx": 5978, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Padding can help fill the
missing values. However, since the model is traced with a larger input size, the
dimensions of the matrix will also be large, resulting in more calculations.

Be careful of the total number of operations done on each input and follow the
performance closely when exporting varying sequence-length models.

## Using TorchScript in Python

This section demonstrates how to save and load models as well as how to use the trace
for inference.

### Saving a model

To export a `BertModel` with TorchScript, instantiate `BertModel` from the `BertConfig`
class and then save it to disk under the filename `traced_bert.pt`:

```python
from transformers import BertModel, BertTokenizer, BertConfig
import torch

enc = BertTokenizer.from_pretrained("google-bert/bert-base-uncased")

# Tokenizing input text
text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
tokenized_text = enc.tokenize(text)

# Masking one of the input tokens
masked_index = 8
tokenized_text[masked_index] = "[MASK]"
indexed_tokens = enc.convert_tokens_to_ids(tokenized_text)
segments_ids = [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]

# Creating a dummy input
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])
dummy_input = [tokens_tensor, segments_tensors]

# Initializing the model with the torchscript flag
# Flag set to True even though it is not necessary as this model does not have an LM Head.
config = BertConfig(
    vocab_size_or_config_json_file=32000,
    hidden_size=768,
    num_hidden_layers=12,
    num_attention_heads=12,
    intermediate_size=3072,
    torchscript=True,
)

# Instantiating the model
model = BertModel(config)

# The model needs to be in evaluation mode
model.eval()

# If you are instantiating the model with *from_pretrained* you can also easily set the TorchScript flag
model = BertModel.from_pretrained("google-bert/bert-base-uncased", torchscript=True)

# Creating the trace
traced_model = torch.jit.trace(model, [tokens_tensor, segments_tensors])
torch.jit.save(traced_model, "traced_bert.pt")
```

### Loading a model

Now you can load the previously saved `BertModel`, `traced_bert.pt`, from disk and use
it on the previously initialised `dummy_input`:

```python
loaded_model = torch.jit.load("traced_bert.pt")
loaded_model.eval()

all_encoder_layers, pooled_output = loaded_model(*dummy_input)
```

### Using a traced model for inference

Use the traced model for inference by using its `__call__` dunder method:

```python
traced_model(tokens_tensor, segments_tensors)
```

## Deploy Hugging Face TorchScript models to AWS with the Neuron SDK

AWS introduced the [Amazon EC2 Inf1](https://aws.amazon.com/ec2/instance-types/inf1/)
instance family for low cost, high performance machine learning inference in the cloud.
The Inf1 instances are powered by the AWS Inferentia chip, a custom-built hardware
accelerator, specializing in deep learning inferencing workloads. [AWS
Neuron](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/#) is the SDK for
Inferentia that supports tracing and optimizing transformers models for deployment on
Inf1. The Neuron SDK provides:


key: _node_content
string_value:
{"id_": "8c7cb2bd-220f-4562-8c1a-bdc27869d6b6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/torchscript", "title": "Export to TorchScript", "tokens": 2219, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e2044816-f84e-46f2-99ea-eab21a197414", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/torchscript", "title": "Export to TorchScript", "tokens": 2219, "retrieve_doc": true, "source": "transformers"}, "hash": "43ce469e3ac7a2d78ae89c22727d8fa0878a46070d605e16e52e5854e762250b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d561cb00-6f05-4721-a9d2-9648206d654b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/torchscript", "title": "Export to TorchScript", "tokens": 2219, "retrieve_doc": true, "source": "transformers"}, "hash": "ac6483b16b1ad4de82f01aa9d4e7d686573217114c18fef9eab5874f13b7355d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5626, "end_char_idx": 8863, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The Inf1 instances are powered by the AWS Inferentia chip, a custom-built hardware
accelerator, specializing in deep learning inferencing workloads. [AWS
Neuron](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/#) is the SDK for
Inferentia that supports tracing and optimizing transformers models for deployment on
Inf1. The Neuron SDK provides:


1. Easy-to-use API with one line of code change to trace and optimize a TorchScript
   model for inference in the cloud.
2. Out of the box performance optimizations for [improved
   cost-performance](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-guide/benchmark/>).
3. Support for Hugging Face transformers models built with either
   [PyTorch](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/src/examples/pytorch/bert_tutorial/tutorial_pretrained_bert.html)
   or
   [TensorFlow](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/src/examples/tensorflow/huggingface_bert/huggingface_bert.html).

### Implications

Transformers models based on the [BERT (Bidirectional Encoder Representations from
Transformers)](https://huggingface.co/docs/transformers/main/model_doc/bert)
architecture, or its variants such as
[distilBERT](https://huggingface.co/docs/transformers/main/model_doc/distilbert) and
[roBERTa](https://huggingface.co/docs/transformers/main/model_doc/roberta) run best on
Inf1 for non-generative tasks such as extractive question answering, sequence
classification, and token classification. However, text generation tasks can still be
adapted to run on Inf1 according to this [AWS Neuron MarianMT
tutorial](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/src/examples/pytorch/transformers-marianmt.html).
More information about models that can be converted out of the box on Inferentia can be
found in the [Model Architecture
Fit](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-guide/models/models-inferentia.html#models-inferentia)
section of the Neuron documentation.

### Dependencies

Using AWS Neuron to convert models requires a [Neuron SDK
environment](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-guide/neuron-frameworks/pytorch-neuron/index.html#installation-guide)
which comes preconfigured on [AWS Deep Learning
AMI](https://docs.aws.amazon.com/dlami/latest/devguide/tutorial-inferentia-launching.html).

### Converting a model for AWS Neuron

Convert a model for AWS NEURON using the same code from [Using TorchScript in
Python](torchscript#using-torchscript-in-python) to trace a `BertModel`. Import the
`torch.neuron` framework extension to access the components of the Neuron SDK through a
Python API:

```python
from transformers import BertModel, BertTokenizer, BertConfig
import torch
import torch.neuron
```

You only need to modify the following line:

```diff
- torch.jit.trace(model, [tokens_tensor, segments_tensors])
+ torch.neuron.trace(model, [token_tensor, segments_tensors])
```

This enables the Neuron SDK to trace the model and optimize it for Inf1 instances.

To learn more about AWS Neuron SDK features, tools, example tutorials and latest
updates, please see the [AWS NeuronSDK
documentation](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/index.html).


key: _node_content
string_value:
{"id_": "06e3027d-f80a-4743-b336-163c46ed17e1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ed8d95d-bd8d-4f44-8ade-de93a4116f9c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "transformers"}, "hash": "4af0c723989f11f960cd87b3d575597837522c8fac807ee591f090bba140f09e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "455c36bc-ff3e-49b1-90af-a416344300bf", "node_type": "1", "metadata": {}, "hash": "d2d5e0eddc9d1b1deab50d5955d2a86fa61a989c75273fe3b9038b327fa732ab", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3323, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Training on TPU with TensorFlow

<Tip>

If you don't need long explanations and just want TPU code samples to get started with, check out [our TPU example notebook!](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb)

</Tip>

### What is a TPU?

A TPU is a **Tensor Processing Unit.** They are hardware designed by Google, which are used to greatly speed up the tensor computations within neural networks, much like GPUs. They can be used for both network training and inference. They are generally accessed through Google’s cloud services, but small TPUs can also be accessed directly for free through Google Colab and Kaggle Kernels.

Because [all TensorFlow models in 🤗 Transformers are Keras models](https://huggingface.co/blog/tensorflow-philosophy), most of the methods in this document are generally applicable to TPU training for any Keras model! However, there are a few points that are specific to the HuggingFace ecosystem (hug-o-system?) of Transformers and Datasets, and we’ll make sure to flag them up when we get to them.

### What kinds of TPU are available?

New users are often very confused by the range of TPUs, and the different ways to access them. The first key distinction to understand is the difference between **TPU Nodes** and **TPU VMs.**

When you use a **TPU Node**, you are effectively indirectly accessing a remote TPU. You will need a separate VM, which will initialize your network and data pipeline and then forward them to the remote node. When you use a TPU on Google Colab, you are accessing it in the **TPU Node** style.

Using TPU Nodes can have some quite unexpected behaviour for people who aren’t used to them! In particular, because the TPU is located on a physically different system to the machine you’re running your Python code on, your data cannot be local to your machine - any data pipeline that loads from your machine’s internal storage will totally fail! Instead, data must be stored in Google Cloud Storage where your data pipeline can still access it, even when the pipeline is running on the remote TPU node.

<Tip>

If you can fit all your data in memory as `np.ndarray` or `tf.Tensor`, then you can `fit()` on that data even when using Colab or a TPU Node, without needing to upload it to Google Cloud Storage.

</Tip>

<Tip>

**🤗Specific Hugging Face Tip🤗:** The methods `Dataset.to_tf_dataset()` and its higher-level wrapper `model.prepare_tf_dataset()` , which you will see throughout our TF code examples, will both fail on a TPU Node. The reason for this is that even though they create a `tf.data.Dataset` it is not a “pure” `tf.data` pipeline and uses `tf.numpy_function` or `Dataset.from_generator()` to stream data from the underlying HuggingFace `Dataset`. This HuggingFace `Dataset` is backed by data that is on a local disc and which the remote TPU Node will not be able to read.

</Tip>

The second way to access a TPU is via a **TPU VM.** When using a TPU VM, you connect directly to the machine that the TPU is attached to, much like training on a GPU VM. TPU VMs are generally easier to work with, particularly when it comes to your data pipeline. All of the above warnings do not apply to TPU VMs!

This is an opinionated document, so here’s our opinion: **Avoid using TPU Node if possible.


key: _node_content
string_value:
{"id_": "455c36bc-ff3e-49b1-90af-a416344300bf", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ed8d95d-bd8d-4f44-8ade-de93a4116f9c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "transformers"}, "hash": "4af0c723989f11f960cd87b3d575597837522c8fac807ee591f090bba140f09e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "06e3027d-f80a-4743-b336-163c46ed17e1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "transformers"}, "hash": "f305e972c2eb80aed8cb6c704868e2d5fd13339d60c0975f6fdc44c3a0074225", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0359b304-46e0-48cc-94e9-c1be8e9b9c20", "node_type": "1", "metadata": {}, "hash": "e50e8902689a2231c6ea9c642a024754c791fa7623a555fa69bef0f87a71637b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1711, "end_char_idx": 4800, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In particular, because the TPU is located on a physically different system to the machine you’re running your Python code on, your data cannot be local to your machine - any data pipeline that loads from your machine’s internal storage will totally fail! Instead, data must be stored in Google Cloud Storage where your data pipeline can still access it, even when the pipeline is running on the remote TPU node.

<Tip>

If you can fit all your data in memory as `np.ndarray` or `tf.Tensor`, then you can `fit()` on that data even when using Colab or a TPU Node, without needing to upload it to Google Cloud Storage.

</Tip>

<Tip>

**🤗Specific Hugging Face Tip🤗:** The methods `Dataset.to_tf_dataset()` and its higher-level wrapper `model.prepare_tf_dataset()` , which you will see throughout our TF code examples, will both fail on a TPU Node. The reason for this is that even though they create a `tf.data.Dataset` it is not a “pure” `tf.data` pipeline and uses `tf.numpy_function` or `Dataset.from_generator()` to stream data from the underlying HuggingFace `Dataset`. This HuggingFace `Dataset` is backed by data that is on a local disc and which the remote TPU Node will not be able to read.

</Tip>

The second way to access a TPU is via a **TPU VM.** When using a TPU VM, you connect directly to the machine that the TPU is attached to, much like training on a GPU VM. TPU VMs are generally easier to work with, particularly when it comes to your data pipeline. All of the above warnings do not apply to TPU VMs!

This is an opinionated document, so here’s our opinion: **Avoid using TPU Node if possible.** It is more confusing and more difficult to debug than TPU VMs. It is also likely to be unsupported in future - Google’s latest TPU, TPUv4, can only be accessed as a TPU VM, which suggests that TPU Nodes are increasingly going to become a “legacy” access method. However, we understand that the only free TPU access is on Colab and Kaggle Kernels, which uses TPU Node - so we’ll try to explain how to handle it if you have to! Check the [TPU example notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb) for code samples that explain this in more detail.

### What sizes of TPU are available?

A single TPU (a v2-8/v3-8/v4-8) runs 8 replicas. TPUs exist in **pods** that can run hundreds or thousands of replicas simultaneously. When you use more than a single TPU but less than a whole pod (for example, a v3-32), your TPU fleet is referred to as a **pod slice.**

When you access a free TPU via Colab, you generally get a single v2-8 TPU.

### I keep hearing about this XLA thing. What’s XLA, and how does it relate to TPUs?

XLA is an optimizing compiler, used by both TensorFlow and JAX. In JAX it is the only compiler, whereas in TensorFlow it is optional (but mandatory on TPU!). The easiest way to enable it when training a Keras model is to pass the argument `jit_compile=True` to `model.compile()`. If you don’t get any errors and performance is good, that’s a great sign that you’re ready to move to TPU!


key: _node_content
string_value:
{"id_": "0359b304-46e0-48cc-94e9-c1be8e9b9c20", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ed8d95d-bd8d-4f44-8ade-de93a4116f9c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "transformers"}, "hash": "4af0c723989f11f960cd87b3d575597837522c8fac807ee591f090bba140f09e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "455c36bc-ff3e-49b1-90af-a416344300bf", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "transformers"}, "hash": "104787d9042a685da93373c43d8f693b622e23597a5f3f27cdfc752b52da123f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f7c72463-7380-42fe-ab27-f47a02f53040", "node_type": "1", "metadata": {}, "hash": "e50200852a9698d2013f0bc889f5640a93e082097c802ae0c7a2585b6f2f38a7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3323, "end_char_idx": 6338, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
** It is more confusing and more difficult to debug than TPU VMs. It is also likely to be unsupported in future - Google’s latest TPU, TPUv4, can only be accessed as a TPU VM, which suggests that TPU Nodes are increasingly going to become a “legacy” access method. However, we understand that the only free TPU access is on Colab and Kaggle Kernels, which uses TPU Node - so we’ll try to explain how to handle it if you have to! Check the [TPU example notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb) for code samples that explain this in more detail.

### What sizes of TPU are available?

A single TPU (a v2-8/v3-8/v4-8) runs 8 replicas. TPUs exist in **pods** that can run hundreds or thousands of replicas simultaneously. When you use more than a single TPU but less than a whole pod (for example, a v3-32), your TPU fleet is referred to as a **pod slice.**

When you access a free TPU via Colab, you generally get a single v2-8 TPU.

### I keep hearing about this XLA thing. What’s XLA, and how does it relate to TPUs?

XLA is an optimizing compiler, used by both TensorFlow and JAX. In JAX it is the only compiler, whereas in TensorFlow it is optional (but mandatory on TPU!). The easiest way to enable it when training a Keras model is to pass the argument `jit_compile=True` to `model.compile()`. If you don’t get any errors and performance is good, that’s a great sign that you’re ready to move to TPU!

Debugging on TPU is generally a bit harder than on CPU/GPU, so we recommend getting your code running on CPU/GPU with XLA first before trying it on TPU. You don’t have to train for long, of course - just for a few steps to make sure that your model and data pipeline are working like you expect them to.

<Tip>

XLA compiled code is usually faster - so even if you’re not planning to run on TPU, adding `jit_compile=True` can improve your performance. Be sure to note the caveats below about XLA compatibility, though!

</Tip>

<Tip warning={true}>

**Tip born of painful experience:** Although using `jit_compile=True` is a good way to get a speed boost and test if your CPU/GPU code is XLA-compatible, it can actually cause a lot of problems if you leave it in when actually training on TPU. XLA compilation will happen implicitly on TPU, so remember to remove that line before actually running your code on a TPU!

</Tip>

### How do I make my model XLA compatible?

In many cases, your code is probably XLA-compatible already! However, there are a few things that work in normal TensorFlow that don’t work in XLA. We’ve distilled them into three core rules below:

<Tip>

**🤗Specific HuggingFace Tip🤗:** We’ve put a lot of effort into rewriting our TensorFlow models and loss functions to be XLA-compatible. Our models and loss functions generally obey rule #1 and #2 by default, so you can skip over them if you’re using `transformers` models. Don’t forget about these rules when writing your own models and loss functions, though!


key: _node_content
string_value:
{"id_": "f7c72463-7380-42fe-ab27-f47a02f53040", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ed8d95d-bd8d-4f44-8ade-de93a4116f9c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "transformers"}, "hash": "4af0c723989f11f960cd87b3d575597837522c8fac807ee591f090bba140f09e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0359b304-46e0-48cc-94e9-c1be8e9b9c20", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "transformers"}, "hash": "cbb35b187a341f5c9c0f457cf65b5fe65e348fe48e023124fbfdcc5ee2bc2cf1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "efa3005b-711c-4445-ba9b-06f59f04fa44", "node_type": "1", "metadata": {}, "hash": "a0feea18a167b7f75c395c148ecc03797172c6631ad0becf80ff8dede2f58e17", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4694, "end_char_idx": 8011, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If you don’t get any errors and performance is good, that’s a great sign that you’re ready to move to TPU!

Debugging on TPU is generally a bit harder than on CPU/GPU, so we recommend getting your code running on CPU/GPU with XLA first before trying it on TPU. You don’t have to train for long, of course - just for a few steps to make sure that your model and data pipeline are working like you expect them to.

<Tip>

XLA compiled code is usually faster - so even if you’re not planning to run on TPU, adding `jit_compile=True` can improve your performance. Be sure to note the caveats below about XLA compatibility, though!

</Tip>

<Tip warning={true}>

**Tip born of painful experience:** Although using `jit_compile=True` is a good way to get a speed boost and test if your CPU/GPU code is XLA-compatible, it can actually cause a lot of problems if you leave it in when actually training on TPU. XLA compilation will happen implicitly on TPU, so remember to remove that line before actually running your code on a TPU!

</Tip>

### How do I make my model XLA compatible?

In many cases, your code is probably XLA-compatible already! However, there are a few things that work in normal TensorFlow that don’t work in XLA. We’ve distilled them into three core rules below:

<Tip>

**🤗Specific HuggingFace Tip🤗:** We’ve put a lot of effort into rewriting our TensorFlow models and loss functions to be XLA-compatible. Our models and loss functions generally obey rule #1 and #2 by default, so you can skip over them if you’re using `transformers` models. Don’t forget about these rules when writing your own models and loss functions, though!

</Tip>

#### XLA Rule #1: Your code cannot have “data-dependent conditionals”

What that means is that any `if` statement cannot depend on values inside a `tf.Tensor`. For example, this code block cannot be compiled with XLA!

```python
if tf.reduce_sum(tensor) > 10:
    tensor = tensor / 2.0
```

This might seem very restrictive at first, but most neural net code doesn’t need to do this. You can often get around this restriction by using `tf.cond` (see the documentation [here](https://www.tensorflow.org/api_docs/python/tf/cond)) or by removing the conditional and finding a clever math trick with indicator variables instead, like so:

```python
sum_over_10 = tf.cast(tf.reduce_sum(tensor) > 10, tf.float32)
tensor = tensor / (1.0 + sum_over_10)
```

This code has exactly the same effect as the code above, but by avoiding a conditional, we ensure it will compile with XLA without problems!

#### XLA Rule #2: Your code cannot have “data-dependent shapes”

What this means is that the shape of all of the `tf.Tensor` objects in your code cannot depend on their values. For example, the function `tf.unique` cannot be compiled with XLA, because it returns a `tensor` containing one instance of each unique value in the input. The shape of this output will obviously be different depending on how repetitive the input `Tensor` was, and so XLA refuses to handle it!

In general, most neural network code obeys rule #2 by default. However, there are a few common cases where it becomes a problem. One very common one is when you use **label masking**, setting your labels to a negative value to indicate that those positions should be ignored when computing the loss.


key: _node_content
string_value:
{"id_": "efa3005b-711c-4445-ba9b-06f59f04fa44", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ed8d95d-bd8d-4f44-8ade-de93a4116f9c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "transformers"}, "hash": "4af0c723989f11f960cd87b3d575597837522c8fac807ee591f090bba140f09e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f7c72463-7380-42fe-ab27-f47a02f53040", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "transformers"}, "hash": "b059d9899140b11a2cc0a8e3a950b365e0a158bba01c1df5d075947da70e84ad", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "37c56fb1-6fd1-4daa-bd9f-185f1b6546ad", "node_type": "1", "metadata": {}, "hash": "58230d26aa0812f7a2d4e8e8ad5753b88a7bca534ec119532e0f8f34be89ea9e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6340, "end_char_idx": 9602, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

#### XLA Rule #1: Your code cannot have “data-dependent conditionals”

What that means is that any `if` statement cannot depend on values inside a `tf.Tensor`. For example, this code block cannot be compiled with XLA!

```python
if tf.reduce_sum(tensor) > 10:
    tensor = tensor / 2.0
```

This might seem very restrictive at first, but most neural net code doesn’t need to do this. You can often get around this restriction by using `tf.cond` (see the documentation [here](https://www.tensorflow.org/api_docs/python/tf/cond)) or by removing the conditional and finding a clever math trick with indicator variables instead, like so:

```python
sum_over_10 = tf.cast(tf.reduce_sum(tensor) > 10, tf.float32)
tensor = tensor / (1.0 + sum_over_10)
```

This code has exactly the same effect as the code above, but by avoiding a conditional, we ensure it will compile with XLA without problems!

#### XLA Rule #2: Your code cannot have “data-dependent shapes”

What this means is that the shape of all of the `tf.Tensor` objects in your code cannot depend on their values. For example, the function `tf.unique` cannot be compiled with XLA, because it returns a `tensor` containing one instance of each unique value in the input. The shape of this output will obviously be different depending on how repetitive the input `Tensor` was, and so XLA refuses to handle it!

In general, most neural network code obeys rule #2 by default. However, there are a few common cases where it becomes a problem. One very common one is when you use **label masking**, setting your labels to a negative value to indicate that those positions should be ignored when computing the loss. If you look at NumPy or PyTorch loss functions that support label masking, you will often see code like this that uses [boolean indexing](https://numpy.org/doc/stable/user/basics.indexing.html#boolean-array-indexing):

```python
label_mask = labels >= 0
masked_outputs = outputs[label_mask]
masked_labels = labels[label_mask]
loss = compute_loss(masked_outputs, masked_labels)
mean_loss = torch.mean(loss)
```

This code is totally fine in NumPy or PyTorch, but it breaks in XLA! Why? Because the shape of `masked_outputs` and `masked_labels` depends on how many positions are masked - that makes it a **data-dependent shape.** However, just like for rule #1, we can often rewrite this code to yield exactly the same output without any data-dependent shapes.

```python
label_mask = tf.cast(labels >= 0, tf.float32)
loss = compute_loss(outputs, labels)
loss = loss * label_mask  # Set negative label positions to 0
mean_loss = tf.reduce_sum(loss) / tf.reduce_sum(label_mask)
```

Here, we avoid data-dependent shapes by computing the loss for every position, but zeroing out the masked positions in both the numerator and denominator when we calculate the mean, which yields exactly the same result as the first block while maintaining XLA compatibility. Note that we use the same trick as in rule #1 - converting a `tf.bool` to `tf.float32` and using it as an indicator variable. This is a really useful trick, so remember it if you need to convert your own code to XLA!

#### XLA Rule #3: XLA will need to recompile your model for every different input shape it sees

This is the big one.


key: _node_content
string_value:
{"id_": "37c56fb1-6fd1-4daa-bd9f-185f1b6546ad", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ed8d95d-bd8d-4f44-8ade-de93a4116f9c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "transformers"}, "hash": "4af0c723989f11f960cd87b3d575597837522c8fac807ee591f090bba140f09e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "efa3005b-711c-4445-ba9b-06f59f04fa44", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "transformers"}, "hash": "1e4d5c7d5054178bc5a480e0d1537dfeffcfd9bf37b383bece96c2418db28f89", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "435d7987-4ba6-4750-82fd-1cd6bc5b4a15", "node_type": "1", "metadata": {}, "hash": "bceda79f31cca7fc833bfbee6c3cd792c41f7c236242780e746b2fac36212354", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8012, "end_char_idx": 11285, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If you look at NumPy or PyTorch loss functions that support label masking, you will often see code like this that uses [boolean indexing](https://numpy.org/doc/stable/user/basics.indexing.html#boolean-array-indexing):

```python
label_mask = labels >= 0
masked_outputs = outputs[label_mask]
masked_labels = labels[label_mask]
loss = compute_loss(masked_outputs, masked_labels)
mean_loss = torch.mean(loss)
```

This code is totally fine in NumPy or PyTorch, but it breaks in XLA! Why? Because the shape of `masked_outputs` and `masked_labels` depends on how many positions are masked - that makes it a **data-dependent shape.** However, just like for rule #1, we can often rewrite this code to yield exactly the same output without any data-dependent shapes.

```python
label_mask = tf.cast(labels >= 0, tf.float32)
loss = compute_loss(outputs, labels)
loss = loss * label_mask  # Set negative label positions to 0
mean_loss = tf.reduce_sum(loss) / tf.reduce_sum(label_mask)
```

Here, we avoid data-dependent shapes by computing the loss for every position, but zeroing out the masked positions in both the numerator and denominator when we calculate the mean, which yields exactly the same result as the first block while maintaining XLA compatibility. Note that we use the same trick as in rule #1 - converting a `tf.bool` to `tf.float32` and using it as an indicator variable. This is a really useful trick, so remember it if you need to convert your own code to XLA!

#### XLA Rule #3: XLA will need to recompile your model for every different input shape it sees

This is the big one. What this means is that if your input shapes are very variable, XLA will have to recompile your model over and over, which will create huge performance problems. This commonly arises in NLP models, where input texts have variable lengths after tokenization. In other modalities, static shapes are more common and this rule is much less of a problem.

How can you get around rule #3? The key is **padding** - if you pad all your inputs to the same length, and then use an `attention_mask`, you can get the same results as you’d get from variable shapes, but without any XLA issues. However, excessive padding can cause severe slowdown too - if you pad all your samples to the maximum length in the whole dataset, you might end up with batches consisting endless padding tokens, which will waste a lot of compute and memory!

There isn’t a perfect solution to this problem. However, you can try some tricks. One very useful trick is to **pad batches of samples up to a multiple of a number like 32 or 64 tokens.** This often only increases the number of tokens by a small amount, but it hugely reduces the number of unique input shapes, because every input shape now has to be a multiple of 32 or 64. Fewer unique input shapes means fewer XLA compilations!

<Tip>

**🤗Specific HuggingFace Tip🤗:** Our tokenizers and data collators have methods that can help you here. You can use `padding="max_length"` or `padding="longest"` when calling tokenizers to get them to output padded data. Our tokenizers and data collators also have a `pad_to_multiple_of` argument that you can use to reduce the number of unique input shapes you see!

</Tip>

### How do I actually train my model on TPU?


key: _node_content
string_value:
{"id_": "435d7987-4ba6-4750-82fd-1cd6bc5b4a15", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ed8d95d-bd8d-4f44-8ade-de93a4116f9c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "transformers"}, "hash": "4af0c723989f11f960cd87b3d575597837522c8fac807ee591f090bba140f09e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "37c56fb1-6fd1-4daa-bd9f-185f1b6546ad", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "transformers"}, "hash": "9c92c609ebe8bc911ce59f8ef29aa6074ec2736148408d32f94a5055c14a231d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5ac3d452-8944-430f-9f0b-7c7401586614", "node_type": "1", "metadata": {}, "hash": "f8eba4a6ba0f7f20c573ecc5e00d4e1fc61d30b1e627ac6a81006893a89e2db5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9603, "end_char_idx": 12855, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
What this means is that if your input shapes are very variable, XLA will have to recompile your model over and over, which will create huge performance problems. This commonly arises in NLP models, where input texts have variable lengths after tokenization. In other modalities, static shapes are more common and this rule is much less of a problem.

How can you get around rule #3? The key is **padding** - if you pad all your inputs to the same length, and then use an `attention_mask`, you can get the same results as you’d get from variable shapes, but without any XLA issues. However, excessive padding can cause severe slowdown too - if you pad all your samples to the maximum length in the whole dataset, you might end up with batches consisting endless padding tokens, which will waste a lot of compute and memory!

There isn’t a perfect solution to this problem. However, you can try some tricks. One very useful trick is to **pad batches of samples up to a multiple of a number like 32 or 64 tokens.** This often only increases the number of tokens by a small amount, but it hugely reduces the number of unique input shapes, because every input shape now has to be a multiple of 32 or 64. Fewer unique input shapes means fewer XLA compilations!

<Tip>

**🤗Specific HuggingFace Tip🤗:** Our tokenizers and data collators have methods that can help you here. You can use `padding="max_length"` or `padding="longest"` when calling tokenizers to get them to output padded data. Our tokenizers and data collators also have a `pad_to_multiple_of` argument that you can use to reduce the number of unique input shapes you see!

</Tip>

### How do I actually train my model on TPU?

Once your training is XLA-compatible and (if you’re using TPU Node / Colab) your dataset has been prepared appropriately, running on TPU is surprisingly easy! All you really need to change in your code is to add a few lines to initialize your TPU, and to ensure that your model and dataset are created inside a `TPUStrategy` scope. Take a look at [our TPU example notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb) to see this in action!

### Summary

There was a lot in here, so let’s summarize with a quick checklist you can follow when you want to get your model ready for TPU training:

- Make sure your code follows the three rules of XLA
- Compile your model with `jit_compile=True` on CPU/GPU and confirm that you can train it with XLA
- Either load your dataset into memory or use a TPU-compatible dataset loading approach (see [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb))
- Migrate your code either to Colab (with accelerator set to “TPU”) or a TPU VM on Google Cloud
- Add TPU initializer code (see [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb))
- Create your `TPUStrategy` and make sure dataset loading and model creation are inside the `strategy.scope()` (see [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb))
- Don’t forget to take `jit_compile=True` out again when you move to TPU!


key: _node_content
string_value:
{"id_": "5ac3d452-8944-430f-9f0b-7c7401586614", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6ed8d95d-bd8d-4f44-8ade-de93a4116f9c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "transformers"}, "hash": "4af0c723989f11f960cd87b3d575597837522c8fac807ee591f090bba140f09e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "435d7987-4ba6-4750-82fd-1cd6bc5b4a15", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/perf_train_tpu_tf", "title": "Training on TPU with TensorFlow", "tokens": 3205, "retrieve_doc": true, "source": "transformers"}, "hash": "82e384348fd6243a682c0e2a280ad367515e95bbe0a68f742f27475e9a188602", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11287, "end_char_idx": 12897, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Once your training is XLA-compatible and (if you’re using TPU Node / Colab) your dataset has been prepared appropriately, running on TPU is surprisingly easy! All you really need to change in your code is to add a few lines to initialize your TPU, and to ensure that your model and dataset are created inside a `TPUStrategy` scope. Take a look at [our TPU example notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb) to see this in action!

### Summary

There was a lot in here, so let’s summarize with a quick checklist you can follow when you want to get your model ready for TPU training:

- Make sure your code follows the three rules of XLA
- Compile your model with `jit_compile=True` on CPU/GPU and confirm that you can train it with XLA
- Either load your dataset into memory or use a TPU-compatible dataset loading approach (see [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb))
- Migrate your code either to Colab (with accelerator set to “TPU”) or a TPU VM on Google Cloud
- Add TPU initializer code (see [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb))
- Create your `TPUStrategy` and make sure dataset loading and model creation are inside the `strategy.scope()` (see [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/tpu_training-tf.ipynb))
- Don’t forget to take `jit_compile=True` out again when you move to TPU!
- 🙏🙏🙏🥺🥺🥺
- Call model.fit()
- You did it!


key: _node_content
string_value:
{"id_": "a2b43a1d-ea88-4bb8-83a7-a658af3a347a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44f7a3a9-fe70-4a3f-a978-6ccea8ad5502", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "hash": "2eb2ac24706ad47f1e22f4a3d3b118679148add84b768c97708831f68f226fe0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "eaf1ecc7-747c-4e62-b926-3ee817e6c434", "node_type": "1", "metadata": {}, "hash": "2cbdb6bc6412b2495a3fc1c0fb67e5ec6e58871c579962970216247ae10cc41d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 4461, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Quick tour

[[open-in-colab]]

Get up and running with 🤗 Transformers! Whether you're a developer or an everyday user, this quick tour will help you get started and show you how to use the [`pipeline`] for inference, load a pretrained model and preprocessor with an [AutoClass](./model_doc/auto), and quickly train a model with PyTorch or TensorFlow. If you're a beginner, we recommend checking out our tutorials or [course](https://huggingface.co/course/chapter1/1) next for more in-depth explanations of the concepts introduced here.

Before you begin, make sure you have all the necessary libraries installed:

```bash
!pip install transformers datasets evaluate accelerate
```

You'll also need to install your preferred machine learning framework:

<frameworkcontent>
<pt>

```bash
pip install torch
```
</pt>
<tf>

```bash
pip install tensorflow
```
</tf>
</frameworkcontent>

## Pipeline

<Youtube id="tiZFewofSLM"/>

The [`pipeline`] is the easiest and fastest way to use a pretrained model for inference. You can use the [`pipeline`] out-of-the-box for many tasks across different modalities, some of which are shown in the table below:

<Tip>

For a complete list of available tasks, check out the [pipeline API reference](./main_classes/pipelines).

</Tip>

| **Task**                     | **Description**                                                                                              | **Modality**    | **Pipeline identifier**                       |
|------------------------------|--------------------------------------------------------------------------------------------------------------|-----------------|-----------------------------------------------|
| Text classification          | assign a label to a given sequence of text                                                                   | NLP             | pipeline(task=“sentiment-analysis”)           |
| Text generation              | generate text given a prompt                                                                                 | NLP             | pipeline(task=“text-generation”)              |
| Summarization                | generate a summary of a sequence of text or document                                                         | NLP             | pipeline(task=“summarization”)                |
| Image classification         | assign a label to an image                                                                                   | Computer vision | pipeline(task=“image-classification”)         |
| Image segmentation           | assign a label to each individual pixel of an image (supports semantic, panoptic, and instance segmentation) | Computer vision | pipeline(task=“image-segmentation”)           |
| Object detection             | predict the bounding boxes and classes of objects in an image                                                | Computer vision | pipeline(task=“object-detection”)             |
| Audio classification         | assign a label to some audio data                                                                            | Audio           | pipeline(task=“audio-classification”)         |
| Automatic speech recognition | transcribe speech into text                                                                                  | Audio           | pipeline(task=“automatic-speech-recognition”) |
| Visual question answering    | answer a question about the image, given an image and a question                                             | Multimodal      | pipeline(task=“vqa”)                          |
| Document question answering  | answer a question about the document, given a document and a question                                        | Multimodal      | pipeline(task="document-question-answering")  |
| Image captioning             | generate a caption for a given image                                                                         | Multimodal      | pipeline(task="image-to-text")                |

Start by creating an instance of [`pipeline`] and specifying a task you want to use it for. In this guide, you'll use the [`pipeline`] for sentiment analysis as an example:

```py
>>> from transformers import pipeline

>>> classifier = pipeline("sentiment-analysis")
```

The [`pipeline`] downloads and caches a default [pretrained model](https://huggingface.co/distilbert/distilbert-base-uncased-finetuned-sst-2-english) and tokenizer for sentiment analysis.


key: _node_content
string_value:
{"id_": "eaf1ecc7-747c-4e62-b926-3ee817e6c434", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44f7a3a9-fe70-4a3f-a978-6ccea8ad5502", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "hash": "2eb2ac24706ad47f1e22f4a3d3b118679148add84b768c97708831f68f226fe0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a2b43a1d-ea88-4bb8-83a7-a658af3a347a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "hash": "9c970a7792b587303c8623712377b6a7e1082bb2a51557de26e64b6e5bb03a3f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c97523bf-2d4b-42f5-8ef7-8a480abb937a", "node_type": "1", "metadata": {}, "hash": "edf7908e6722d5c497727af26f915dbb5a8f1176cf8766e11f4db6119f4a7f50", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4094, "end_char_idx": 7088, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In this guide, you'll use the [`pipeline`] for sentiment analysis as an example:

```py
>>> from transformers import pipeline

>>> classifier = pipeline("sentiment-analysis")
```

The [`pipeline`] downloads and caches a default [pretrained model](https://huggingface.co/distilbert/distilbert-base-uncased-finetuned-sst-2-english) and tokenizer for sentiment analysis. Now you can use the `classifier` on your target text:

```py
>>> classifier("We are very happy to show you the 🤗 Transformers library.")
[{'label': 'POSITIVE', 'score': 0.9998}]
```

If you have more than one input, pass your inputs as a list to the [`pipeline`] to return a list of dictionaries:

```py
>>> results = classifier(["We are very happy to show you the 🤗 Transformers library.", "We hope you don't hate it."])
>>> for result in results:
...     print(f"label: {result['label']}, with score: {round(result['score'], 4)}")
label: POSITIVE, with score: 0.9998
label: NEGATIVE, with score: 0.5309
```

The [`pipeline`] can also iterate over an entire dataset for any task you like. For this example, let's choose automatic speech recognition as our task:

```py
>>> import torch
>>> from transformers import pipeline

>>> speech_recognizer = pipeline("automatic-speech-recognition", model="facebook/wav2vec2-base-960h")
```

Load an audio dataset (see the 🤗 Datasets [Quick Start](https://huggingface.co/docs/datasets/quickstart#audio) for more details) you'd like to iterate over. For example, load the [MInDS-14](https://huggingface.co/datasets/PolyAI/minds14) dataset:

```py
>>> from datasets import load_dataset, Audio

>>> dataset = load_dataset("PolyAI/minds14", name="en-US", split="train")  # doctest: +IGNORE_RESULT
```

You need to make sure the sampling rate of the dataset matches the sampling 
rate [`facebook/wav2vec2-base-960h`](https://huggingface.co/facebook/wav2vec2-base-960h) was trained on:

```py
>>> dataset = dataset.cast_column("audio", Audio(sampling_rate=speech_recognizer.feature_extractor.sampling_rate))
```

The audio files are automatically loaded and resampled when calling the `"audio"` column.
Extract the raw waveform arrays from the first 4 samples and pass it as a list to the pipeline:

```py
>>> result = speech_recognizer(dataset[:4]["audio"])
>>> print([d["text"] for d in result])
['I WOULD LIKE TO SET UP A JOINT ACCOUNT WITH MY PARTNER HOW DO I PROCEED WITH DOING THAT', "FONDERING HOW I'D SET UP A JOIN TO HELL T WITH MY WIFE AND WHERE THE AP MIGHT BE", "I I'D LIKE TOY SET UP A JOINT ACCOUNT WITH MY PARTNER I'M NOT SEEING THE OPTION TO DO IT ON THE APSO I CALLED IN TO GET SOME HELP CAN I JUST DO IT OVER THE PHONE WITH YOU AND GIVE YOU THE INFORMATION OR SHOULD I DO IT IN THE AP AN I'M MISSING SOMETHING UQUETTE HAD PREFERRED TO JUST DO IT OVER THE PHONE OF POSSIBLE THINGS", 'HOW DO I FURN A JOINA COUT']
```

For larger datasets where the inputs are big (like in speech or vision), you'll want to pass a generator instead of a list to load all the inputs in memory.


key: _node_content
string_value:
{"id_": "c97523bf-2d4b-42f5-8ef7-8a480abb937a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44f7a3a9-fe70-4a3f-a978-6ccea8ad5502", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "hash": "2eb2ac24706ad47f1e22f4a3d3b118679148add84b768c97708831f68f226fe0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "eaf1ecc7-747c-4e62-b926-3ee817e6c434", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "hash": "4905516afe2deab06cfc4b20b1a1974cffdaf73db137b27e6c3f2002bbd60de5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ab19b5b8-4d91-4578-b2c3-8bc546fd2e00", "node_type": "1", "metadata": {}, "hash": "6d2f53bb2cf2f564603ea8e51ba990c158529239531bd87c96707ac5af69e07c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6200, "end_char_idx": 9335, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Extract the raw waveform arrays from the first 4 samples and pass it as a list to the pipeline:

```py
>>> result = speech_recognizer(dataset[:4]["audio"])
>>> print([d["text"] for d in result])
['I WOULD LIKE TO SET UP A JOINT ACCOUNT WITH MY PARTNER HOW DO I PROCEED WITH DOING THAT', "FONDERING HOW I'D SET UP A JOIN TO HELL T WITH MY WIFE AND WHERE THE AP MIGHT BE", "I I'D LIKE TOY SET UP A JOINT ACCOUNT WITH MY PARTNER I'M NOT SEEING THE OPTION TO DO IT ON THE APSO I CALLED IN TO GET SOME HELP CAN I JUST DO IT OVER THE PHONE WITH YOU AND GIVE YOU THE INFORMATION OR SHOULD I DO IT IN THE AP AN I'M MISSING SOMETHING UQUETTE HAD PREFERRED TO JUST DO IT OVER THE PHONE OF POSSIBLE THINGS", 'HOW DO I FURN A JOINA COUT']
```

For larger datasets where the inputs are big (like in speech or vision), you'll want to pass a generator instead of a list to load all the inputs in memory. Take a look at the [pipeline API reference](./main_classes/pipelines) for more information.

### Use another model and tokenizer in the pipeline

The [`pipeline`] can accommodate any model from the [Hub](https://huggingface.co/models), making it easy to adapt the [`pipeline`] for other use-cases. For example, if you'd like a model capable of handling French text, use the tags on the Hub to filter for an appropriate model. The top filtered result returns a multilingual [BERT model](https://huggingface.co/nlptown/bert-base-multilingual-uncased-sentiment) finetuned for sentiment analysis you can use for French text:

```py
>>> model_name = "nlptown/bert-base-multilingual-uncased-sentiment"
```

<frameworkcontent>
<pt>
Use [`AutoModelForSequenceClassification`] and [`AutoTokenizer`] to load the pretrained model and it's associated tokenizer (more on an `AutoClass` in the next section):

```py
>>> from transformers import AutoTokenizer, AutoModelForSequenceClassification

>>> model = AutoModelForSequenceClassification.from_pretrained(model_name)
>>> tokenizer = AutoTokenizer.from_pretrained(model_name)
```
</pt>
<tf>
Use [`TFAutoModelForSequenceClassification`] and [`AutoTokenizer`] to load the pretrained model and it's associated tokenizer (more on an `TFAutoClass` in the next section):

```py
>>> from transformers import AutoTokenizer, TFAutoModelForSequenceClassification

>>> model = TFAutoModelForSequenceClassification.from_pretrained(model_name)
>>> tokenizer = AutoTokenizer.from_pretrained(model_name)
```
</tf>
</frameworkcontent>

Specify the model and tokenizer in the [`pipeline`], and now you can apply the `classifier` on French text:

```py
>>> classifier = pipeline("sentiment-analysis", model=model, tokenizer=tokenizer)
>>> classifier("Nous sommes très heureux de vous présenter la bibliothèque 🤗 Transformers.")
[{'label': '5 stars', 'score': 0.7273}]
```

If you can't find a model for your use-case, you'll need to finetune a pretrained model on your data. Take a look at our [finetuning tutorial](./training) to learn how. Finally, after you've finetuned your pretrained model, please consider [sharing](./model_sharing) the model with the community on the Hub to democratize machine learning for everyone!


key: _node_content
string_value:
{"id_": "ab19b5b8-4d91-4578-b2c3-8bc546fd2e00", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44f7a3a9-fe70-4a3f-a978-6ccea8ad5502", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "hash": "2eb2ac24706ad47f1e22f4a3d3b118679148add84b768c97708831f68f226fe0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c97523bf-2d4b-42f5-8ef7-8a480abb937a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "hash": "0c9cc573a46166e958fbca5983ecb979b64c3a5feb51fc32e08fc97587f9124d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a8e5b8c3-f054-4490-a6ce-4d862b764547", "node_type": "1", "metadata": {}, "hash": "72d817d147d8d680bdfe0ddc949cc311d091d9696f1860abdac2c976400ec12a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8938, "end_char_idx": 11779, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[{'label': '5 stars', 'score': 0.7273}]
```

If you can't find a model for your use-case, you'll need to finetune a pretrained model on your data. Take a look at our [finetuning tutorial](./training) to learn how. Finally, after you've finetuned your pretrained model, please consider [sharing](./model_sharing) the model with the community on the Hub to democratize machine learning for everyone! 🤗

## AutoClass

<Youtube id="AhChOFRegn4"/>

Under the hood, the [`AutoModelForSequenceClassification`] and [`AutoTokenizer`] classes work together to power the [`pipeline`] you used above. An [AutoClass](./model_doc/auto) is a shortcut that automatically retrieves the architecture of a pretrained model from its name or path. You only need to select the appropriate `AutoClass` for your task and it's associated preprocessing class. 

Let's return to the example from the previous section and see how you can use the `AutoClass` to replicate the results of the [`pipeline`].

### AutoTokenizer

A tokenizer is responsible for preprocessing text into an array of numbers as inputs to a model. There are multiple rules that govern the tokenization process, including how to split a word and at what level words should be split (learn more about tokenization in the [tokenizer summary](./tokenizer_summary)). The most important thing to remember is you need to instantiate a tokenizer with the same model name to ensure you're using the same tokenization rules a model was pretrained with.

Load a tokenizer with [`AutoTokenizer`]:

```py
>>> from transformers import AutoTokenizer

>>> model_name = "nlptown/bert-base-multilingual-uncased-sentiment"
>>> tokenizer = AutoTokenizer.from_pretrained(model_name)
```

Pass your text to the tokenizer:

```py
>>> encoding = tokenizer("We are very happy to show you the 🤗 Transformers library.")
>>> print(encoding)
{'input_ids': [101, 11312, 10320, 12495, 19308, 10114, 11391, 10855, 10103, 100, 58263, 13299, 119, 102],
 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
```

The tokenizer returns a dictionary containing:

* [input_ids](./glossary#input-ids): numerical representations of your tokens.
* [attention_mask](./glossary#attention-mask): indicates which tokens should be attended to.

A tokenizer can also accept a list of inputs, and pad and truncate the text to return a batch with uniform length:

<frameworkcontent>
<pt>

```py
>>> pt_batch = tokenizer(
...     ["We are very happy to show you the 🤗 Transformers library.", "We hope you don't hate it."],
...     padding=True,
...     truncation=True,
...     max_length=512,
...     return_tensors="pt",
... )
```
</pt>
<tf>

```py
>>> tf_batch = tokenizer(
...     ["We are very happy to show you the 🤗 Transformers library.", "We hope you don't hate it.


key: _node_content
string_value:
{"id_": "a8e5b8c3-f054-4490-a6ce-4d862b764547", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44f7a3a9-fe70-4a3f-a978-6ccea8ad5502", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "hash": "2eb2ac24706ad47f1e22f4a3d3b118679148add84b768c97708831f68f226fe0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ab19b5b8-4d91-4578-b2c3-8bc546fd2e00", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "hash": "5f3b9b0349e5656b0664c1b17905298ce2e168f2f66677c3dd7ff1ee5a47e066", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a7c85d4e-4f7d-422c-89c5-be00469d75d1", "node_type": "1", "metadata": {}, "hash": "4bda205eb754f0a8adfb712a013aba8e32a115c1437d23bfd32022cf56b9aed6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10776, "end_char_idx": 13097, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> print(encoding)
{'input_ids': [101, 11312, 10320, 12495, 19308, 10114, 11391, 10855, 10103, 100, 58263, 13299, 119, 102],
 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
```

The tokenizer returns a dictionary containing:

* [input_ids](./glossary#input-ids): numerical representations of your tokens.
* [attention_mask](./glossary#attention-mask): indicates which tokens should be attended to.

A tokenizer can also accept a list of inputs, and pad and truncate the text to return a batch with uniform length:

<frameworkcontent>
<pt>

```py
>>> pt_batch = tokenizer(
...     ["We are very happy to show you the 🤗 Transformers library.", "We hope you don't hate it."],
...     padding=True,
...     truncation=True,
...     max_length=512,
...     return_tensors="pt",
... )
```
</pt>
<tf>

```py
>>> tf_batch = tokenizer(
...     ["We are very happy to show you the 🤗 Transformers library.", "We hope you don't hate it."],
...     padding=True,
...     truncation=True,
...     max_length=512,
...     return_tensors="tf",
... )
```
</tf>
</frameworkcontent>

<Tip>

Check out the [preprocess](./preprocessing) tutorial for more details about tokenization, and how to use an [`AutoImageProcessor`], [`AutoFeatureExtractor`] and [`AutoProcessor`] to preprocess image, audio, and multimodal inputs.

</Tip>

### AutoModel

<frameworkcontent>
<pt>
🤗 Transformers provides a simple and unified way to load pretrained instances. This means you can load an [`AutoModel`] like you would load an [`AutoTokenizer`]. The only difference is selecting the correct [`AutoModel`] for the task. For text (or sequence) classification, you should load [`AutoModelForSequenceClassification`]:

```py
>>> from transformers import AutoModelForSequenceClassification

>>> model_name = "nlptown/bert-base-multilingual-uncased-sentiment"
>>> pt_model = AutoModelForSequenceClassification.from_pretrained(model_name)
```

<Tip>

See the [task summary](./task_summary) for tasks supported by an [`AutoModel`] class.

</Tip>

Now pass your preprocessed batch of inputs directly to the model. You just have to unpack the dictionary by adding `**`:

```py
>>> pt_outputs = pt_model(**pt_batch)
```

The model outputs the final activations in the `logits` attribute.


key: _node_content
string_value:
{"id_": "a7c85d4e-4f7d-422c-89c5-be00469d75d1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44f7a3a9-fe70-4a3f-a978-6ccea8ad5502", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "hash": "2eb2ac24706ad47f1e22f4a3d3b118679148add84b768c97708831f68f226fe0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a8e5b8c3-f054-4490-a6ce-4d862b764547", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "hash": "d7d4fccb24190811e1d2f3bb997496ac896b87b87c8da0053c7b48d4d8fe0c8a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bdb430dc-55d2-482b-971c-37d595e33714", "node_type": "1", "metadata": {}, "hash": "fd600be74cf70414fd62fd10e349dd281905975cbed45424c415d31b56104e78", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11494, "end_char_idx": 14322, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
", "We hope you don't hate it."],
...     padding=True,
...     truncation=True,
...     max_length=512,
...     return_tensors="pt",
... )
```
</pt>
<tf>

```py
>>> tf_batch = tokenizer(
...     ["We are very happy to show you the 🤗 Transformers library.", "We hope you don't hate it."],
...     padding=True,
...     truncation=True,
...     max_length=512,
...     return_tensors="tf",
... )
```
</tf>
</frameworkcontent>

<Tip>

Check out the [preprocess](./preprocessing) tutorial for more details about tokenization, and how to use an [`AutoImageProcessor`], [`AutoFeatureExtractor`] and [`AutoProcessor`] to preprocess image, audio, and multimodal inputs.

</Tip>

### AutoModel

<frameworkcontent>
<pt>
🤗 Transformers provides a simple and unified way to load pretrained instances. This means you can load an [`AutoModel`] like you would load an [`AutoTokenizer`]. The only difference is selecting the correct [`AutoModel`] for the task. For text (or sequence) classification, you should load [`AutoModelForSequenceClassification`]:

```py
>>> from transformers import AutoModelForSequenceClassification

>>> model_name = "nlptown/bert-base-multilingual-uncased-sentiment"
>>> pt_model = AutoModelForSequenceClassification.from_pretrained(model_name)
```

<Tip>

See the [task summary](./task_summary) for tasks supported by an [`AutoModel`] class.

</Tip>

Now pass your preprocessed batch of inputs directly to the model. You just have to unpack the dictionary by adding `**`:

```py
>>> pt_outputs = pt_model(**pt_batch)
```

The model outputs the final activations in the `logits` attribute. Apply the softmax function to the `logits` to retrieve the probabilities:

```py
>>> from torch import nn

>>> pt_predictions = nn.functional.softmax(pt_outputs.logits, dim=-1)
>>> print(pt_predictions)
tensor([[0.0021, 0.0018, 0.0115, 0.2121, 0.7725],
        [0.2084, 0.1826, 0.1969, 0.1755, 0.2365]], grad_fn=<SoftmaxBackward0>)
```
</pt>
<tf>
🤗 Transformers provides a simple and unified way to load pretrained instances. This means you can load an [`TFAutoModel`] like you would load an [`AutoTokenizer`]. The only difference is selecting the correct [`TFAutoModel`] for the task. For text (or sequence) classification, you should load [`TFAutoModelForSequenceClassification`]:

```py
>>> from transformers import TFAutoModelForSequenceClassification

>>> model_name = "nlptown/bert-base-multilingual-uncased-sentiment"
>>> tf_model = TFAutoModelForSequenceClassification.from_pretrained(model_name)
```

<Tip>

See the [task summary](./task_summary) for tasks supported by an [`AutoModel`] class.

</Tip>

Now pass your preprocessed batch of inputs directly to the model. You can pass the tensors as-is:

```py
>>> tf_outputs = tf_model(tf_batch)
```

The model outputs the final activations in the `logits` attribute.


key: _node_content
string_value:
{"id_": "bdb430dc-55d2-482b-971c-37d595e33714", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44f7a3a9-fe70-4a3f-a978-6ccea8ad5502", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "hash": "2eb2ac24706ad47f1e22f4a3d3b118679148add84b768c97708831f68f226fe0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a7c85d4e-4f7d-422c-89c5-be00469d75d1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "hash": "c96528b83dd9cc29b9cf1641d769e20c4b5b32ede3005cd1b1031b287a20380e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0a852a0b-7235-4125-b187-0baf99024f00", "node_type": "1", "metadata": {}, "hash": "994a8fa4f2c43b35b4b170b5cfd214a12b0b64a18abe435becdaea554f2c7cc5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12852, "end_char_idx": 15017, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

Now pass your preprocessed batch of inputs directly to the model. You just have to unpack the dictionary by adding `**`:

```py
>>> pt_outputs = pt_model(**pt_batch)
```

The model outputs the final activations in the `logits` attribute. Apply the softmax function to the `logits` to retrieve the probabilities:

```py
>>> from torch import nn

>>> pt_predictions = nn.functional.softmax(pt_outputs.logits, dim=-1)
>>> print(pt_predictions)
tensor([[0.0021, 0.0018, 0.0115, 0.2121, 0.7725],
        [0.2084, 0.1826, 0.1969, 0.1755, 0.2365]], grad_fn=<SoftmaxBackward0>)
```
</pt>
<tf>
🤗 Transformers provides a simple and unified way to load pretrained instances. This means you can load an [`TFAutoModel`] like you would load an [`AutoTokenizer`]. The only difference is selecting the correct [`TFAutoModel`] for the task. For text (or sequence) classification, you should load [`TFAutoModelForSequenceClassification`]:

```py
>>> from transformers import TFAutoModelForSequenceClassification

>>> model_name = "nlptown/bert-base-multilingual-uncased-sentiment"
>>> tf_model = TFAutoModelForSequenceClassification.from_pretrained(model_name)
```

<Tip>

See the [task summary](./task_summary) for tasks supported by an [`AutoModel`] class.

</Tip>

Now pass your preprocessed batch of inputs directly to the model. You can pass the tensors as-is:

```py
>>> tf_outputs = tf_model(tf_batch)
```

The model outputs the final activations in the `logits` attribute. Apply the softmax function to the `logits` to retrieve the probabilities:

```py
>>> import tensorflow as tf

>>> tf_predictions = tf.nn.softmax(tf_outputs.logits, axis=-1)
>>> tf_predictions  # doctest: +IGNORE_RESULT
```
</tf>
</frameworkcontent>

<Tip>

All 🤗 Transformers models (PyTorch or TensorFlow) output the tensors *before* the final activation
function (like softmax) because the final activation function is often fused with the loss. Model outputs are special dataclasses so their attributes are autocompleted in an IDE. The model outputs behave like a tuple or a dictionary (you can index with an integer, a slice or a string) in which case, attributes that are None are ignored.


key: _node_content
string_value:
{"id_": "0a852a0b-7235-4125-b187-0baf99024f00", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44f7a3a9-fe70-4a3f-a978-6ccea8ad5502", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "hash": "2eb2ac24706ad47f1e22f4a3d3b118679148add84b768c97708831f68f226fe0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bdb430dc-55d2-482b-971c-37d595e33714", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "hash": "9a6b6b0307a88bb71237d27ab252eaa3baff48514ed49e3ca15124af07a8c255", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3767fb7b-c6f5-406f-8fc7-f43d74845ba3", "node_type": "1", "metadata": {}, "hash": "2baf2fec40273bdf1b3868a72f659f29affd0f09d5161aaa179d466f2c429ccb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13524, "end_char_idx": 16219, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This means you can load an [`TFAutoModel`] like you would load an [`AutoTokenizer`]. The only difference is selecting the correct [`TFAutoModel`] for the task. For text (or sequence) classification, you should load [`TFAutoModelForSequenceClassification`]:

```py
>>> from transformers import TFAutoModelForSequenceClassification

>>> model_name = "nlptown/bert-base-multilingual-uncased-sentiment"
>>> tf_model = TFAutoModelForSequenceClassification.from_pretrained(model_name)
```

<Tip>

See the [task summary](./task_summary) for tasks supported by an [`AutoModel`] class.

</Tip>

Now pass your preprocessed batch of inputs directly to the model. You can pass the tensors as-is:

```py
>>> tf_outputs = tf_model(tf_batch)
```

The model outputs the final activations in the `logits` attribute. Apply the softmax function to the `logits` to retrieve the probabilities:

```py
>>> import tensorflow as tf

>>> tf_predictions = tf.nn.softmax(tf_outputs.logits, axis=-1)
>>> tf_predictions  # doctest: +IGNORE_RESULT
```
</tf>
</frameworkcontent>

<Tip>

All 🤗 Transformers models (PyTorch or TensorFlow) output the tensors *before* the final activation
function (like softmax) because the final activation function is often fused with the loss. Model outputs are special dataclasses so their attributes are autocompleted in an IDE. The model outputs behave like a tuple or a dictionary (you can index with an integer, a slice or a string) in which case, attributes that are None are ignored.

</Tip>

### Save a model

<frameworkcontent>
<pt>
Once your model is fine-tuned, you can save it with its tokenizer using [`PreTrainedModel.save_pretrained`]:

```py
>>> pt_save_directory = "./pt_save_pretrained"
>>> tokenizer.save_pretrained(pt_save_directory)  # doctest: +IGNORE_RESULT
>>> pt_model.save_pretrained(pt_save_directory)
```

When you are ready to use the model again, reload it with [`PreTrainedModel.from_pretrained`]:

```py
>>> pt_model = AutoModelForSequenceClassification.from_pretrained("./pt_save_pretrained")
```
</pt>
<tf>
Once your model is fine-tuned, you can save it with its tokenizer using [`TFPreTrainedModel.save_pretrained`]:

```py
>>> tf_save_directory = "./tf_save_pretrained"
>>> tokenizer.save_pretrained(tf_save_directory)  # doctest: +IGNORE_RESULT
>>> tf_model.save_pretrained(tf_save_directory)
```

When you are ready to use the model again, reload it with [`TFPreTrainedModel.from_pretrained`]:

```py
>>> tf_model = TFAutoModelForSequenceClassification.from_pretrained("./tf_save_pretrained")
```
</tf>
</frameworkcontent>

One particularly cool 🤗 Transformers feature is the ability to save a model and reload it as either a PyTorch or TensorFlow model.


key: _node_content
string_value:
{"id_": "3767fb7b-c6f5-406f-8fc7-f43d74845ba3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44f7a3a9-fe70-4a3f-a978-6ccea8ad5502", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "hash": "2eb2ac24706ad47f1e22f4a3d3b118679148add84b768c97708831f68f226fe0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0a852a0b-7235-4125-b187-0baf99024f00", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "hash": "8e41a1d29ec8710a399e13b4b5a8c07d310b61418ebade44bf667558ce6e589f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "eb8a86e6-d62b-4d69-ad9f-feffecd39ce9", "node_type": "1", "metadata": {}, "hash": "b428c2405a58a5cbe5d38fd98ca9ac2d25e96810c349ffee301f0b22621ed4c4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14771, "end_char_idx": 18154, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Model outputs are special dataclasses so their attributes are autocompleted in an IDE. The model outputs behave like a tuple or a dictionary (you can index with an integer, a slice or a string) in which case, attributes that are None are ignored.

</Tip>

### Save a model

<frameworkcontent>
<pt>
Once your model is fine-tuned, you can save it with its tokenizer using [`PreTrainedModel.save_pretrained`]:

```py
>>> pt_save_directory = "./pt_save_pretrained"
>>> tokenizer.save_pretrained(pt_save_directory)  # doctest: +IGNORE_RESULT
>>> pt_model.save_pretrained(pt_save_directory)
```

When you are ready to use the model again, reload it with [`PreTrainedModel.from_pretrained`]:

```py
>>> pt_model = AutoModelForSequenceClassification.from_pretrained("./pt_save_pretrained")
```
</pt>
<tf>
Once your model is fine-tuned, you can save it with its tokenizer using [`TFPreTrainedModel.save_pretrained`]:

```py
>>> tf_save_directory = "./tf_save_pretrained"
>>> tokenizer.save_pretrained(tf_save_directory)  # doctest: +IGNORE_RESULT
>>> tf_model.save_pretrained(tf_save_directory)
```

When you are ready to use the model again, reload it with [`TFPreTrainedModel.from_pretrained`]:

```py
>>> tf_model = TFAutoModelForSequenceClassification.from_pretrained("./tf_save_pretrained")
```
</tf>
</frameworkcontent>

One particularly cool 🤗 Transformers feature is the ability to save a model and reload it as either a PyTorch or TensorFlow model. The `from_pt` or `from_tf` parameter can convert the model from one framework to the other:

<frameworkcontent>
<pt>

```py
>>> from transformers import AutoModel

>>> tokenizer = AutoTokenizer.from_pretrained(tf_save_directory)
>>> pt_model = AutoModelForSequenceClassification.from_pretrained(tf_save_directory, from_tf=True)
```
</pt>
<tf>

```py
>>> from transformers import TFAutoModel

>>> tokenizer = AutoTokenizer.from_pretrained(pt_save_directory)
>>> tf_model = TFAutoModelForSequenceClassification.from_pretrained(pt_save_directory, from_pt=True)
```
</tf>
</frameworkcontent>

## Custom model builds

You can modify the model's configuration class to change how a model is built. The configuration specifies a model's attributes, such as the number of hidden layers or attention heads. You start from scratch when you initialize a model from a custom configuration class. The model attributes are randomly initialized, and you'll need to train the model before you can use it to get meaningful results.

Start by importing [`AutoConfig`], and then load the pretrained model you want to modify. Within [`AutoConfig.from_pretrained`], you can specify the attribute you want to change, such as the number of attention heads:

```py
>>> from transformers import AutoConfig

>>> my_config = AutoConfig.from_pretrained("distilbert/distilbert-base-uncased", n_heads=12)
```

<frameworkcontent>
<pt>
Create a model from your custom configuration with [`AutoModel.from_config`]:

```py
>>> from transformers import AutoModel

>>> my_model = AutoModel.from_config(my_config)
```
</pt>
<tf>
Create a model from your custom configuration with [`TFAutoModel.from_config`]:

```py
>>> from transformers import TFAutoModel

>>> my_model = TFAutoModel.from_config(my_config)
```
</tf>
</frameworkcontent>

Take a look at the [Create a custom architecture](./create_a_model) guide for more information about building custom configurations.


key: _node_content
string_value:
{"id_": "eb8a86e6-d62b-4d69-ad9f-feffecd39ce9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44f7a3a9-fe70-4a3f-a978-6ccea8ad5502", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "hash": "2eb2ac24706ad47f1e22f4a3d3b118679148add84b768c97708831f68f226fe0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3767fb7b-c6f5-406f-8fc7-f43d74845ba3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "hash": "9091881359abca90784cba18befe26360370dc70a1fa022467ab3b95732dfeaf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5da5f980-f029-4378-9e35-c65d378345e5", "node_type": "1", "metadata": {}, "hash": "75e61de65b9b3126c2f3b0f785aeae98034f23cbc1c8198e6faa886ec8b6c9a3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16912, "end_char_idx": 20239, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The configuration specifies a model's attributes, such as the number of hidden layers or attention heads. You start from scratch when you initialize a model from a custom configuration class. The model attributes are randomly initialized, and you'll need to train the model before you can use it to get meaningful results.

Start by importing [`AutoConfig`], and then load the pretrained model you want to modify. Within [`AutoConfig.from_pretrained`], you can specify the attribute you want to change, such as the number of attention heads:

```py
>>> from transformers import AutoConfig

>>> my_config = AutoConfig.from_pretrained("distilbert/distilbert-base-uncased", n_heads=12)
```

<frameworkcontent>
<pt>
Create a model from your custom configuration with [`AutoModel.from_config`]:

```py
>>> from transformers import AutoModel

>>> my_model = AutoModel.from_config(my_config)
```
</pt>
<tf>
Create a model from your custom configuration with [`TFAutoModel.from_config`]:

```py
>>> from transformers import TFAutoModel

>>> my_model = TFAutoModel.from_config(my_config)
```
</tf>
</frameworkcontent>

Take a look at the [Create a custom architecture](./create_a_model) guide for more information about building custom configurations.

## Trainer - a PyTorch optimized training loop

All models are a standard [`torch.nn.Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) so you can use them in any typical training loop. While you can write your own training loop, 🤗 Transformers provides a [`Trainer`] class for PyTorch, which contains the basic training loop and adds additional functionality for features like distributed training, mixed precision, and more.

Depending on your task, you'll typically pass the following parameters to [`Trainer`]:

1. You'll start with a [`PreTrainedModel`] or a [`torch.nn.Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module):

   ```py
   >>> from transformers import AutoModelForSequenceClassification

   >>> model = AutoModelForSequenceClassification.from_pretrained("distilbert/distilbert-base-uncased")
   ```

2. [`TrainingArguments`] contains the model hyperparameters you can change like learning rate, batch size, and the number of epochs to train for. The default values are used if you don't specify any training arguments:

   ```py
   >>> from transformers import TrainingArguments

   >>> training_args = TrainingArguments(
   ...     output_dir="path/to/save/folder/",
   ...     learning_rate=2e-5,
   ...     per_device_train_batch_size=8,
   ...     per_device_eval_batch_size=8,
   ...     num_train_epochs=2,
   ... )
   ```

3. Load a preprocessing class like a tokenizer, image processor, feature extractor, or processor:

   ```py
   >>> from transformers import AutoTokenizer

   >>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
   ```

4. Load a dataset:

   ```py
   >>> from datasets import load_dataset

   >>> dataset = load_dataset("rotten_tomatoes")  # doctest: +IGNORE_RESULT
   ```

5. Create a function to tokenize the dataset:

   ```py
   >>> def tokenize_dataset(dataset):
   ...     return tokenizer(dataset["text"])
   ```

   Then apply it over the entire dataset with [`~datasets.Dataset.map`]:

   ```py
   >>> dataset = dataset.map(tokenize_dataset, batched=True)
   ```

6.


key: _node_content
string_value:
{"id_": "5da5f980-f029-4378-9e35-c65d378345e5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44f7a3a9-fe70-4a3f-a978-6ccea8ad5502", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "hash": "2eb2ac24706ad47f1e22f4a3d3b118679148add84b768c97708831f68f226fe0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "eb8a86e6-d62b-4d69-ad9f-feffecd39ce9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "hash": "1de8829a40d324785acb95b7fb8553e5c4a88667bd695f4d8c116e1eea5b3924", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9eeca6b5-cb69-431a-80fd-9eb6789e9d08", "node_type": "1", "metadata": {}, "hash": "61a8a9c1a5e20ea7b470219acc1c9008b652fbc04245b056eb7b91ddb9f590d6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18599, "end_char_idx": 21737, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Depending on your task, you'll typically pass the following parameters to [`Trainer`]:

1. You'll start with a [`PreTrainedModel`] or a [`torch.nn.Module`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module):

   ```py
   >>> from transformers import AutoModelForSequenceClassification

   >>> model = AutoModelForSequenceClassification.from_pretrained("distilbert/distilbert-base-uncased")
   ```

2. [`TrainingArguments`] contains the model hyperparameters you can change like learning rate, batch size, and the number of epochs to train for. The default values are used if you don't specify any training arguments:

   ```py
   >>> from transformers import TrainingArguments

   >>> training_args = TrainingArguments(
   ...     output_dir="path/to/save/folder/",
   ...     learning_rate=2e-5,
   ...     per_device_train_batch_size=8,
   ...     per_device_eval_batch_size=8,
   ...     num_train_epochs=2,
   ... )
   ```

3. Load a preprocessing class like a tokenizer, image processor, feature extractor, or processor:

   ```py
   >>> from transformers import AutoTokenizer

   >>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
   ```

4. Load a dataset:

   ```py
   >>> from datasets import load_dataset

   >>> dataset = load_dataset("rotten_tomatoes")  # doctest: +IGNORE_RESULT
   ```

5. Create a function to tokenize the dataset:

   ```py
   >>> def tokenize_dataset(dataset):
   ...     return tokenizer(dataset["text"])
   ```

   Then apply it over the entire dataset with [`~datasets.Dataset.map`]:

   ```py
   >>> dataset = dataset.map(tokenize_dataset, batched=True)
   ```

6. A [`DataCollatorWithPadding`] to create a batch of examples from your dataset:

   ```py
   >>> from transformers import DataCollatorWithPadding

   >>> data_collator = DataCollatorWithPadding(tokenizer=tokenizer)
   ```

Now gather all these classes in [`Trainer`]:

```py
>>> from transformers import Trainer

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=dataset["train"],
...     eval_dataset=dataset["test"],
...     tokenizer=tokenizer,
...     data_collator=data_collator,
... )  # doctest: +SKIP
```

When you're ready, call [`~Trainer.train`] to start training:

```py
>>> trainer.train()  # doctest: +SKIP
```

<Tip>

For tasks - like translation or summarization - that use a sequence-to-sequence model, use the [`Seq2SeqTrainer`] and [`Seq2SeqTrainingArguments`] classes instead.

</Tip>

You can customize the training loop behavior by subclassing the methods inside [`Trainer`]. This allows you to customize features such as the loss function, optimizer, and scheduler. Take a look at the [`Trainer`] reference for which methods can be subclassed. 

The other way to customize the training loop is by using [Callbacks](./main_classes/callback). You can use callbacks to integrate with other libraries and inspect the training loop to report on progress or stop the training early. Callbacks do not modify anything in the training loop itself. To customize something like the loss function, you need to subclass the [`Trainer`] instead.


key: _node_content
string_value:
{"id_": "9eeca6b5-cb69-431a-80fd-9eb6789e9d08", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44f7a3a9-fe70-4a3f-a978-6ccea8ad5502", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "hash": "2eb2ac24706ad47f1e22f4a3d3b118679148add84b768c97708831f68f226fe0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5da5f980-f029-4378-9e35-c65d378345e5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "hash": "51d1a2cb568a6795fb423312fc2dfc18ec440145e88736eb21c5f168278f866a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "13b61fbe-8509-4bf7-9d59-c3c93d108373", "node_type": "1", "metadata": {}, "hash": "e44743883676472df595ad995b2d82e1f791f6645c551eaf22ae8065266fee36", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20240, "end_char_idx": 23501, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
A [`DataCollatorWithPadding`] to create a batch of examples from your dataset:

   ```py
   >>> from transformers import DataCollatorWithPadding

   >>> data_collator = DataCollatorWithPadding(tokenizer=tokenizer)
   ```

Now gather all these classes in [`Trainer`]:

```py
>>> from transformers import Trainer

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=dataset["train"],
...     eval_dataset=dataset["test"],
...     tokenizer=tokenizer,
...     data_collator=data_collator,
... )  # doctest: +SKIP
```

When you're ready, call [`~Trainer.train`] to start training:

```py
>>> trainer.train()  # doctest: +SKIP
```

<Tip>

For tasks - like translation or summarization - that use a sequence-to-sequence model, use the [`Seq2SeqTrainer`] and [`Seq2SeqTrainingArguments`] classes instead.

</Tip>

You can customize the training loop behavior by subclassing the methods inside [`Trainer`]. This allows you to customize features such as the loss function, optimizer, and scheduler. Take a look at the [`Trainer`] reference for which methods can be subclassed. 

The other way to customize the training loop is by using [Callbacks](./main_classes/callback). You can use callbacks to integrate with other libraries and inspect the training loop to report on progress or stop the training early. Callbacks do not modify anything in the training loop itself. To customize something like the loss function, you need to subclass the [`Trainer`] instead.

## Train with TensorFlow

All models are a standard [`tf.keras.Model`](https://www.tensorflow.org/api_docs/python/tf/keras/Model) so they can be trained in TensorFlow with the [Keras](https://keras.io/) API. 🤗 Transformers provides the [`~TFPreTrainedModel.prepare_tf_dataset`] method to easily load your dataset as a `tf.data.Dataset` so you can start training right away with Keras' [`compile`](https://keras.io/api/models/model_training_apis/#compile-method) and [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) methods.

1. You'll start with a [`TFPreTrainedModel`] or a [`tf.keras.Model`](https://www.tensorflow.org/api_docs/python/tf/keras/Model):

   ```py
   >>> from transformers import TFAutoModelForSequenceClassification

   >>> model = TFAutoModelForSequenceClassification.from_pretrained("distilbert/distilbert-base-uncased")
   ```

2. Load a preprocessing class like a tokenizer, image processor, feature extractor, or processor:

   ```py
   >>> from transformers import AutoTokenizer

   >>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
   ```

3. Create a function to tokenize the dataset:

   ```py
   >>> def tokenize_dataset(dataset):
   ...     return tokenizer(dataset["text"])  # doctest: +SKIP
   ```

4. Apply the tokenizer over the entire dataset with [`~datasets.Dataset.map`] and then pass the dataset and tokenizer to [`~TFPreTrainedModel.prepare_tf_dataset`]. You can also change the batch size and shuffle the dataset here if you'd like:

   ```py
   >>> dataset = dataset.map(tokenize_dataset)  # doctest: +SKIP
   >>> tf_dataset = model.prepare_tf_dataset(
   ...     dataset["train"], batch_size=16, shuffle=True, tokenizer=tokenizer
   ... )  # doctest: +SKIP
   ```

5.


key: _node_content
string_value:
{"id_": "13b61fbe-8509-4bf7-9d59-c3c93d108373", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44f7a3a9-fe70-4a3f-a978-6ccea8ad5502", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "hash": "2eb2ac24706ad47f1e22f4a3d3b118679148add84b768c97708831f68f226fe0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9eeca6b5-cb69-431a-80fd-9eb6789e9d08", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/quicktour", "title": "Quick tour", "tokens": 5735, "retrieve_doc": true, "source": "transformers"}, "hash": "23361400bf874404ef9d8924b14dd1c28533f990a238826d12a9f0d586b94dab", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21947, "end_char_idx": 24255, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
🤗 Transformers provides the [`~TFPreTrainedModel.prepare_tf_dataset`] method to easily load your dataset as a `tf.data.Dataset` so you can start training right away with Keras' [`compile`](https://keras.io/api/models/model_training_apis/#compile-method) and [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) methods.

1. You'll start with a [`TFPreTrainedModel`] or a [`tf.keras.Model`](https://www.tensorflow.org/api_docs/python/tf/keras/Model):

   ```py
   >>> from transformers import TFAutoModelForSequenceClassification

   >>> model = TFAutoModelForSequenceClassification.from_pretrained("distilbert/distilbert-base-uncased")
   ```

2. Load a preprocessing class like a tokenizer, image processor, feature extractor, or processor:

   ```py
   >>> from transformers import AutoTokenizer

   >>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
   ```

3. Create a function to tokenize the dataset:

   ```py
   >>> def tokenize_dataset(dataset):
   ...     return tokenizer(dataset["text"])  # doctest: +SKIP
   ```

4. Apply the tokenizer over the entire dataset with [`~datasets.Dataset.map`] and then pass the dataset and tokenizer to [`~TFPreTrainedModel.prepare_tf_dataset`]. You can also change the batch size and shuffle the dataset here if you'd like:

   ```py
   >>> dataset = dataset.map(tokenize_dataset)  # doctest: +SKIP
   >>> tf_dataset = model.prepare_tf_dataset(
   ...     dataset["train"], batch_size=16, shuffle=True, tokenizer=tokenizer
   ... )  # doctest: +SKIP
   ```

5. When you're ready, you can call `compile` and `fit` to start training. Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

   ```py
   >>> from tensorflow.keras.optimizers import Adam

   >>> model.compile(optimizer='adam')  # No loss argument!
   >>> model.fit(tf_dataset)  # doctest: +SKIP
   ```

## What's next?

Now that you've completed the 🤗 Transformers quick tour, check out our guides and learn how to do more specific things like writing a custom model, fine-tuning a model for a task, and how to train a model with a script. If you're interested in learning more about 🤗 Transformers core concepts, grab a cup of coffee and take a look at our Conceptual Guides!


key: _node_content
string_value:
{"id_": "4594d633-a328-4441-a476-4588301d24de", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1e982a6-1f67-4a2e-a6ce-c33e6127e75e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "hash": "27d2530e4fe585d3fe3385ce93b69adf7f2f8d64bd0acaaa01d1ae6ece2fcb7c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9d6a59fb-292c-4dce-9534-8288ed2728fb", "node_type": "1", "metadata": {}, "hash": "c73989040327e25a5f79f2fa66909924a0b5781cbb3e570b886f45e06b1107f1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3469, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Trainer

The [`Trainer`] is a complete training and evaluation loop for PyTorch models implemented in the Transformers library. You only need to pass it the necessary pieces for training (model, tokenizer, dataset, evaluation function, training hyperparameters, etc.), and the [`Trainer`] class takes care of the rest. This makes it easier to start training faster without manually writing your own training loop. But at the same time, [`Trainer`] is very customizable and offers a ton of training options so you can tailor it to your exact training needs.

<Tip>

In addition to the [`Trainer`] class, Transformers also provides a [`Seq2SeqTrainer`] class for sequence-to-sequence tasks like translation or summarization. There is also the [`~trl.SFTTrainer`] class from the [TRL](https://hf.co/docs/trl) library which wraps the [`Trainer`] class and is optimized for training language models like Llama-2 and Mistral with autoregressive techniques. [`~trl.SFTTrainer`] also supports features like sequence packing, LoRA, quantization, and DeepSpeed for efficiently scaling to any model size.

<br>

Feel free to check out the [API reference](./main_classes/trainer) for these other [`Trainer`]-type classes to learn more about when to use which one. In general, [`Trainer`] is the most versatile option and is appropriate for a broad spectrum of tasks. [`Seq2SeqTrainer`] is designed for sequence-to-sequence tasks and [`~trl.SFTTrainer`] is designed for training language models.

</Tip>

Before you start, make sure [Accelerate](https://hf.co/docs/accelerate) - a library for enabling and running PyTorch training across distributed environments - is installed.

```bash
pip install accelerate

# upgrade
pip install accelerate --upgrade
```

This guide provides an overview of the [`Trainer`] class.

## Basic usage

[`Trainer`] includes all the code you'll find in a basic training loop:

1. perform a training step to calculate the loss
2. calculate the gradients with the [`~accelerate.Accelerator.backward`] method
3. update the weights based on the gradients
4. repeat this process until you've reached a predetermined number of epochs

The [`Trainer`] class abstracts all of this code away so you don't have to worry about manually writing a training loop every time or if you're just getting started with PyTorch and training. You only need to provide the essential components required for training, such as a model and a dataset, and the [`Trainer`] class handles everything else.

If you want to specify any training options or hyperparameters, you can find them in the [`TrainingArguments`] class. For example, let's define where to save the model in `output_dir` and push the model to the Hub after training with `push_to_hub=True`.

```py
from transformers import TrainingArguments

training_args = TrainingArguments(
    output_dir="your-model",
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    num_train_epochs=2,
    weight_decay=0.01,
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    push_to_hub=True,
)
```

Pass `training_args` to the [`Trainer`] along with a model, dataset, something to preprocess the dataset with (depending on your data type it could be a tokenizer, feature extractor or image processor), a data collator, and a function to compute the metrics you want to track during training.

Finally, call [`~Trainer.train`] to start training!


key: _node_content
string_value:
{"id_": "9d6a59fb-292c-4dce-9534-8288ed2728fb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1e982a6-1f67-4a2e-a6ce-c33e6127e75e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "hash": "27d2530e4fe585d3fe3385ce93b69adf7f2f8d64bd0acaaa01d1ae6ece2fcb7c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4594d633-a328-4441-a476-4588301d24de", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "hash": "52d3fab8182c3315cac03bfa4fe3732509c2389d9f464e23b9f7f32bf2affe0f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c2d4acf6-22f2-443a-957e-eba4c6f7ce1d", "node_type": "1", "metadata": {}, "hash": "b83cef4bfd4a7f7a31fdd599e3d7927cf609105e8ab369dbf3436366f21f9a0a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1808, "end_char_idx": 5274, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Basic usage

[`Trainer`] includes all the code you'll find in a basic training loop:

1. perform a training step to calculate the loss
2. calculate the gradients with the [`~accelerate.Accelerator.backward`] method
3. update the weights based on the gradients
4. repeat this process until you've reached a predetermined number of epochs

The [`Trainer`] class abstracts all of this code away so you don't have to worry about manually writing a training loop every time or if you're just getting started with PyTorch and training. You only need to provide the essential components required for training, such as a model and a dataset, and the [`Trainer`] class handles everything else.

If you want to specify any training options or hyperparameters, you can find them in the [`TrainingArguments`] class. For example, let's define where to save the model in `output_dir` and push the model to the Hub after training with `push_to_hub=True`.

```py
from transformers import TrainingArguments

training_args = TrainingArguments(
    output_dir="your-model",
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    num_train_epochs=2,
    weight_decay=0.01,
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    push_to_hub=True,
)
```

Pass `training_args` to the [`Trainer`] along with a model, dataset, something to preprocess the dataset with (depending on your data type it could be a tokenizer, feature extractor or image processor), a data collator, and a function to compute the metrics you want to track during training.

Finally, call [`~Trainer.train`] to start training!

```py
from transformers import Trainer

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["test"],
    tokenizer=tokenizer,
    data_collator=data_collator,
    compute_metrics=compute_metrics,
)

trainer.train()
```

### Checkpoints

The [`Trainer`] class saves your model checkpoints to the directory specified in the `output_dir` parameter of [`TrainingArguments`]. You'll find the checkpoints saved in a `checkpoint-000` subfolder where the numbers at the end correspond to the training step. Saving checkpoints are useful for resuming training later.

```py
# resume from latest checkpoint
trainer.train(resume_from_checkpoint=True)

# resume from specific checkpoint saved in output directory
trainer.train(resume_from_checkpoint="your-model/checkpoint-1000")
```

You can save your checkpoints (the optimizer state is not saved by default) to the Hub by setting `push_to_hub=True` in [`TrainingArguments`] to commit and push them. Other options for deciding how your checkpoints are saved are set up in the [`hub_strategy`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.hub_strategy) parameter:

* `hub_strategy="checkpoint"` pushes the latest checkpoint to a subfolder named "last-checkpoint" from which you can resume training
* `hub_strategy="all_checkpoints"` pushes all checkpoints to the directory defined in `output_dir` (you'll see one checkpoint per folder in your model repository)

When you resume training from a checkpoint, the [`Trainer`] tries to keep the Python, NumPy, and PyTorch RNG states the same as they were when the checkpoint was saved. But because PyTorch has various non-deterministic default settings, the RNG states aren't guaranteed to be the same.


key: _node_content
string_value:
{"id_": "c2d4acf6-22f2-443a-957e-eba4c6f7ce1d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1e982a6-1f67-4a2e-a6ce-c33e6127e75e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "hash": "27d2530e4fe585d3fe3385ce93b69adf7f2f8d64bd0acaaa01d1ae6ece2fcb7c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9d6a59fb-292c-4dce-9534-8288ed2728fb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "hash": "1c77675e1a16080d77f291e0606cfb759e6cadb2bf1e5201c4c05c79d3b51b81", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "17c79740-b64b-408a-8e62-3dd5b6a2103c", "node_type": "1", "metadata": {}, "hash": "3ac69d12e3599402cd8789085c6f2e23b4d38232fc5618ec5dd5fbd556e8bc2b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3471, "end_char_idx": 7020, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
from transformers import Trainer

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["test"],
    tokenizer=tokenizer,
    data_collator=data_collator,
    compute_metrics=compute_metrics,
)

trainer.train()
```

### Checkpoints

The [`Trainer`] class saves your model checkpoints to the directory specified in the `output_dir` parameter of [`TrainingArguments`]. You'll find the checkpoints saved in a `checkpoint-000` subfolder where the numbers at the end correspond to the training step. Saving checkpoints are useful for resuming training later.

```py
# resume from latest checkpoint
trainer.train(resume_from_checkpoint=True)

# resume from specific checkpoint saved in output directory
trainer.train(resume_from_checkpoint="your-model/checkpoint-1000")
```

You can save your checkpoints (the optimizer state is not saved by default) to the Hub by setting `push_to_hub=True` in [`TrainingArguments`] to commit and push them. Other options for deciding how your checkpoints are saved are set up in the [`hub_strategy`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.hub_strategy) parameter:

* `hub_strategy="checkpoint"` pushes the latest checkpoint to a subfolder named "last-checkpoint" from which you can resume training
* `hub_strategy="all_checkpoints"` pushes all checkpoints to the directory defined in `output_dir` (you'll see one checkpoint per folder in your model repository)

When you resume training from a checkpoint, the [`Trainer`] tries to keep the Python, NumPy, and PyTorch RNG states the same as they were when the checkpoint was saved. But because PyTorch has various non-deterministic default settings, the RNG states aren't guaranteed to be the same. If you want to enable full determinism, take a look at the [Controlling sources of randomness](https://pytorch.org/docs/stable/notes/randomness#controlling-sources-of-randomness) guide to learn what you can enable to make your training fully deterministic. Keep in mind though that by making certain settings deterministic, training may be slower.

## Customize the Trainer

While the [`Trainer`] class is designed to be accessible and easy-to-use, it also offers a lot of customizability for more adventurous users. Many of the [`Trainer`]'s method can be subclassed and overridden to support the functionality you want, without having to rewrite the entire training loop from scratch to accommodate it. These methods include:

* [`~Trainer.get_train_dataloader`] creates a training DataLoader
* [`~Trainer.get_eval_dataloader`] creates an evaluation DataLoader
* [`~Trainer.get_test_dataloader`] creates a test DataLoader
* [`~Trainer.log`] logs information on the various objects that watch training
* [`~Trainer.create_optimizer_and_scheduler`] creates an optimizer and learning rate scheduler if they weren't passed in the `__init__`; these can also be separately customized with [`~Trainer.create_optimizer`] and [`~Trainer.create_scheduler`] respectively
* [`~Trainer.compute_loss`] computes the loss on a batch of training inputs
* [`~Trainer.training_step`] performs the training step
* [`~Trainer.prediction_step`] performs the prediction and test step
* [`~Trainer.evaluate`] evaluates the model and returns the evaluation metrics
* [`~Trainer.predict`] makes predictions (with metrics if labels are available) on the test set

For example, if you want to customize the [`~Trainer.compute_loss`] method to use a weighted loss instead.


key: _node_content
string_value:
{"id_": "17c79740-b64b-408a-8e62-3dd5b6a2103c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1e982a6-1f67-4a2e-a6ce-c33e6127e75e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "hash": "27d2530e4fe585d3fe3385ce93b69adf7f2f8d64bd0acaaa01d1ae6ece2fcb7c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c2d4acf6-22f2-443a-957e-eba4c6f7ce1d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "hash": "1daed3953d8c889367ba029d96704865bfa0a67ecb5bc1e247450eadd9ff6341", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0ccd0d0b-acce-4ffb-b59e-6d6e06db23d9", "node_type": "1", "metadata": {}, "hash": "8096f1b996cc2c648faada935544920a581958d5fb48441d5ee0bb077dd6baaa", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5275, "end_char_idx": 8613, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If you want to enable full determinism, take a look at the [Controlling sources of randomness](https://pytorch.org/docs/stable/notes/randomness#controlling-sources-of-randomness) guide to learn what you can enable to make your training fully deterministic. Keep in mind though that by making certain settings deterministic, training may be slower.

## Customize the Trainer

While the [`Trainer`] class is designed to be accessible and easy-to-use, it also offers a lot of customizability for more adventurous users. Many of the [`Trainer`]'s method can be subclassed and overridden to support the functionality you want, without having to rewrite the entire training loop from scratch to accommodate it. These methods include:

* [`~Trainer.get_train_dataloader`] creates a training DataLoader
* [`~Trainer.get_eval_dataloader`] creates an evaluation DataLoader
* [`~Trainer.get_test_dataloader`] creates a test DataLoader
* [`~Trainer.log`] logs information on the various objects that watch training
* [`~Trainer.create_optimizer_and_scheduler`] creates an optimizer and learning rate scheduler if they weren't passed in the `__init__`; these can also be separately customized with [`~Trainer.create_optimizer`] and [`~Trainer.create_scheduler`] respectively
* [`~Trainer.compute_loss`] computes the loss on a batch of training inputs
* [`~Trainer.training_step`] performs the training step
* [`~Trainer.prediction_step`] performs the prediction and test step
* [`~Trainer.evaluate`] evaluates the model and returns the evaluation metrics
* [`~Trainer.predict`] makes predictions (with metrics if labels are available) on the test set

For example, if you want to customize the [`~Trainer.compute_loss`] method to use a weighted loss instead.

```py
from torch import nn
from transformers import Trainer

class CustomTrainer(Trainer):
    def compute_loss(self, model, inputs, return_outputs=False):
        labels = inputs.pop("labels")
        # forward pass
        outputs = model(**inputs)
        logits = outputs.get("logits")
        # compute custom loss for 3 labels with different weights
        loss_fct = nn.CrossEntropyLoss(weight=torch.tensor([1.0, 2.0, 3.0], device=model.device))
        loss = loss_fct(logits.view(-1, self.model.config.num_labels), labels.view(-1))
        return (loss, outputs) if return_outputs else loss
```

### Callbacks

Another option for customizing the [`Trainer`] is to use [callbacks](callbacks). Callbacks *don't change* anything in the training loop. They inspect the training loop state and then execute some action (early stopping, logging results, etc.) depending on the state. In other words, a callback can't be used to implement something like a custom loss function and you'll need to subclass and override the [`~Trainer.compute_loss`] method for that.

For example, if you want to add an early stopping callback to the training loop after 10 steps.

```py
from transformers import TrainerCallback

class EarlyStoppingCallback(TrainerCallback):
    def __init__(self, num_steps=10):
        self.num_steps = num_steps
    
    def on_step_end(self, args, state, control, **kwargs):
        if state.global_step >= self.num_steps:
            return {"should_training_stop": True}
        else:
            return {}
```

Then pass it to the [`Trainer`]'s `callback` parameter.


key: _node_content
string_value:
{"id_": "0ccd0d0b-acce-4ffb-b59e-6d6e06db23d9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1e982a6-1f67-4a2e-a6ce-c33e6127e75e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "hash": "27d2530e4fe585d3fe3385ce93b69adf7f2f8d64bd0acaaa01d1ae6ece2fcb7c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "17c79740-b64b-408a-8e62-3dd5b6a2103c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "hash": "7f475a5f0cdce0113d2dd1f8ddbf36ba35a275a7fd3c926a5b74e920da5c60ab", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "063df60c-daa2-4427-8c73-bfc454ee241f", "node_type": "1", "metadata": {}, "hash": "e4a692305e1d9e04ce8d5de4ed4a2d52c947f5bb02a47025960726f306177531", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7022, "end_char_idx": 10172, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
from torch import nn
from transformers import Trainer

class CustomTrainer(Trainer):
    def compute_loss(self, model, inputs, return_outputs=False):
        labels = inputs.pop("labels")
        # forward pass
        outputs = model(**inputs)
        logits = outputs.get("logits")
        # compute custom loss for 3 labels with different weights
        loss_fct = nn.CrossEntropyLoss(weight=torch.tensor([1.0, 2.0, 3.0], device=model.device))
        loss = loss_fct(logits.view(-1, self.model.config.num_labels), labels.view(-1))
        return (loss, outputs) if return_outputs else loss
```

### Callbacks

Another option for customizing the [`Trainer`] is to use [callbacks](callbacks). Callbacks *don't change* anything in the training loop. They inspect the training loop state and then execute some action (early stopping, logging results, etc.) depending on the state. In other words, a callback can't be used to implement something like a custom loss function and you'll need to subclass and override the [`~Trainer.compute_loss`] method for that.

For example, if you want to add an early stopping callback to the training loop after 10 steps.

```py
from transformers import TrainerCallback

class EarlyStoppingCallback(TrainerCallback):
    def __init__(self, num_steps=10):
        self.num_steps = num_steps
    
    def on_step_end(self, args, state, control, **kwargs):
        if state.global_step >= self.num_steps:
            return {"should_training_stop": True}
        else:
            return {}
```

Then pass it to the [`Trainer`]'s `callback` parameter.

```py
from transformers import Trainer

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["test"],
    tokenizer=tokenizer,
    data_collator=data_collator,
    compute_metrics=compute_metrics,
    callback=[EarlyStoppingCallback()],
)
```

## Logging

<Tip>

Check out the [logging](./main_classes/logging) API reference for more information about the different logging levels.

</Tip>

The [`Trainer`] is set to `logging.INFO` by default which reports errors, warnings, and other basic information. A [`Trainer`] replica - in distributed environments - is set to `logging.WARNING` which only reports errors and warnings. You can change the logging level with the [`log_level`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.log_level) and [`log_level_replica`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.log_level_replica) parameters in [`TrainingArguments`].

To configure the log level setting for each node, use the [`log_on_each_node`](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.TrainingArguments.log_on_each_node) parameter to determine whether to use the log level on each node or only on the main node.

<Tip>

[`Trainer`] sets the log level separately for each node in the [`Trainer.__init__`] method, so you may want to consider setting this sooner if you're using other Transformers functionalities before creating the [`Trainer`] object.


key: _node_content
string_value:
{"id_": "063df60c-daa2-4427-8c73-bfc454ee241f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1e982a6-1f67-4a2e-a6ce-c33e6127e75e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "hash": "27d2530e4fe585d3fe3385ce93b69adf7f2f8d64bd0acaaa01d1ae6ece2fcb7c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0ccd0d0b-acce-4ffb-b59e-6d6e06db23d9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "hash": "ff3daf0f15d344022233d661393eff473b04bb91175a1896916524d4afcc6e2e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "61f340ae-a978-4795-bba3-5a0190326a65", "node_type": "1", "metadata": {}, "hash": "4c2014ce0c4bc6aa9a80f5fcd23c7ad9c5139b6abfc29ff4769fea5a0c8014cb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8615, "end_char_idx": 11854, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
from transformers import Trainer

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["test"],
    tokenizer=tokenizer,
    data_collator=data_collator,
    compute_metrics=compute_metrics,
    callback=[EarlyStoppingCallback()],
)
```

## Logging

<Tip>

Check out the [logging](./main_classes/logging) API reference for more information about the different logging levels.

</Tip>

The [`Trainer`] is set to `logging.INFO` by default which reports errors, warnings, and other basic information. A [`Trainer`] replica - in distributed environments - is set to `logging.WARNING` which only reports errors and warnings. You can change the logging level with the [`log_level`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.log_level) and [`log_level_replica`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.log_level_replica) parameters in [`TrainingArguments`].

To configure the log level setting for each node, use the [`log_on_each_node`](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.TrainingArguments.log_on_each_node) parameter to determine whether to use the log level on each node or only on the main node.

<Tip>

[`Trainer`] sets the log level separately for each node in the [`Trainer.__init__`] method, so you may want to consider setting this sooner if you're using other Transformers functionalities before creating the [`Trainer`] object.

</Tip>

For example, to set your main code and modules to use the same log level according to each node:

```py
logger = logging.getLogger(__name__)

logging.basicConfig(
    format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
    datefmt="%m/%d/%Y %H:%M:%S",
    handlers=[logging.StreamHandler(sys.stdout)],
)

log_level = training_args.get_process_log_level()
logger.setLevel(log_level)
datasets.utils.logging.set_verbosity(log_level)
transformers.utils.logging.set_verbosity(log_level)

trainer = Trainer(...)
```

Use different combinations of `log_level` and `log_level_replica` to configure what gets logged on each of the nodes.

<hfoptions id="logging">
<hfoption id="single node">

```bash
my_app.py ... --log_level warning --log_level_replica error
```

</hfoption>
<hfoption id="multi-node">

Add the `log_on_each_node 0` parameter for multi-node environments.

```bash
my_app.py ... --log_level warning --log_level_replica error --log_on_each_node 0

# set to only report errors
my_app.py ... --log_level error --log_level_replica error --log_on_each_node 0
```

</hfoption>
</hfoptions>

## NEFTune

[NEFTune](https://hf.co/papers/2310.05914) is a technique that can improve performance by adding noise to the embedding vectors during training. To enable it in [`Trainer`], set the `neftune_noise_alpha` parameter in [`TrainingArguments`] to control how much noise is added.

```py
from transformers import TrainingArguments, Trainer

training_args = TrainingArguments(..., neftune_noise_alpha=0.1)
trainer = Trainer(..., args=training_args)
```

NEFTune is disabled after training to restore the original embedding layer to avoid any unexpected behavior.


key: _node_content
string_value:
{"id_": "61f340ae-a978-4795-bba3-5a0190326a65", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1e982a6-1f67-4a2e-a6ce-c33e6127e75e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "hash": "27d2530e4fe585d3fe3385ce93b69adf7f2f8d64bd0acaaa01d1ae6ece2fcb7c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "063df60c-daa2-4427-8c73-bfc454ee241f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "hash": "81aee753783385acd62f363a19493531c5ed7dc02aa4ae8bf9169fe860d91191", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "61e3f54c-9da7-42a1-9caa-2d6454e19b37", "node_type": "1", "metadata": {}, "hash": "93e7e11f32cb9b40ad8f5d7158e19a1d5924c07148ef698791d4005dd41c405e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10823, "end_char_idx": 13741, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<hfoptions id="logging">
<hfoption id="single node">

```bash
my_app.py ... --log_level warning --log_level_replica error
```

</hfoption>
<hfoption id="multi-node">

Add the `log_on_each_node 0` parameter for multi-node environments.

```bash
my_app.py ... --log_level warning --log_level_replica error --log_on_each_node 0

# set to only report errors
my_app.py ... --log_level error --log_level_replica error --log_on_each_node 0
```

</hfoption>
</hfoptions>

## NEFTune

[NEFTune](https://hf.co/papers/2310.05914) is a technique that can improve performance by adding noise to the embedding vectors during training. To enable it in [`Trainer`], set the `neftune_noise_alpha` parameter in [`TrainingArguments`] to control how much noise is added.

```py
from transformers import TrainingArguments, Trainer

training_args = TrainingArguments(..., neftune_noise_alpha=0.1)
trainer = Trainer(..., args=training_args)
```

NEFTune is disabled after training to restore the original embedding layer to avoid any unexpected behavior.

## GaLore

Gradient Low-Rank Projection (GaLore) is a memory-efficient low-rank training strategy that allows full-parameter learning but is more memory-efficient than common low-rank adaptation methods, such as LoRA.

First make sure to install GaLore official repository:

```bash
pip install galore-torch
```

Then simply add one of `["galore_adamw", "galore_adafactor", "galore_adamw_8bit"]` in `optim` together with `optim_target_modules`, which can be a list of strings, regex or full path corresponding to the target module names you want to adapt. Below is an end-to-end example script (make sure to `pip install trl datasets`):

```python
import torch
import datasets
import trl

from transformers import TrainingArguments, AutoConfig, AutoTokenizer, AutoModelForCausalLM

train_dataset = datasets.load_dataset('imdb', split='train')

args = TrainingArguments(
    output_dir="./test-galore",
    max_steps=100,
    per_device_train_batch_size=2,
    optim="galore_adamw",
    optim_target_modules=[r".*.attn.*", r".*.mlp.*"]
)

model_id = "google/gemma-2b"

config = AutoConfig.from_pretrained(model_id)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_config(config).to(0)

trainer = trl.SFTTrainer(
    model=model, 
    args=args,
    train_dataset=train_dataset,
    dataset_text_field='text',
    max_seq_length=512,
)

trainer.train()
```

To pass extra arguments supports by GaLore, you should pass correctly `optim_args`, for example:

```python
import torch
import datasets
import trl

from transformers import TrainingArguments, AutoConfig, AutoTokenizer, AutoModelForCausalLM

train_dataset = datasets.load_dataset('imdb', split='train')

args = TrainingArguments(
    output_dir="./test-galore",
    max_steps=100,
    per_device_train_batch_size=2,
    optim="galore_adamw",
    optim_target_modules=[r".*.attn.*", r".*.mlp.


key: _node_content
string_value:
{"id_": "61e3f54c-9da7-42a1-9caa-2d6454e19b37", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1e982a6-1f67-4a2e-a6ce-c33e6127e75e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "hash": "27d2530e4fe585d3fe3385ce93b69adf7f2f8d64bd0acaaa01d1ae6ece2fcb7c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "61f340ae-a978-4795-bba3-5a0190326a65", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "hash": "0a8641f6eb0e6ad3c6dc5ff5f0ffe2dd3a62685f884a8cec7ffa228663315305", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "94c27b37-427e-48b6-8613-6a94f8f60662", "node_type": "1", "metadata": {}, "hash": "9efbfd82785d2634ae939489bd21e8d8692cd9fe98ac6b7501813f9896c7a271", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12412, "end_char_idx": 15183, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Below is an end-to-end example script (make sure to `pip install trl datasets`):

```python
import torch
import datasets
import trl

from transformers import TrainingArguments, AutoConfig, AutoTokenizer, AutoModelForCausalLM

train_dataset = datasets.load_dataset('imdb', split='train')

args = TrainingArguments(
    output_dir="./test-galore",
    max_steps=100,
    per_device_train_batch_size=2,
    optim="galore_adamw",
    optim_target_modules=[r".*.attn.*", r".*.mlp.*"]
)

model_id = "google/gemma-2b"

config = AutoConfig.from_pretrained(model_id)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_config(config).to(0)

trainer = trl.SFTTrainer(
    model=model, 
    args=args,
    train_dataset=train_dataset,
    dataset_text_field='text',
    max_seq_length=512,
)

trainer.train()
```

To pass extra arguments supports by GaLore, you should pass correctly `optim_args`, for example:

```python
import torch
import datasets
import trl

from transformers import TrainingArguments, AutoConfig, AutoTokenizer, AutoModelForCausalLM

train_dataset = datasets.load_dataset('imdb', split='train')

args = TrainingArguments(
    output_dir="./test-galore",
    max_steps=100,
    per_device_train_batch_size=2,
    optim="galore_adamw",
    optim_target_modules=[r".*.attn.*", r".*.mlp.*"],
    optim_args="rank=64, update_proj_gap=100, scale=0.10",
)

model_id = "google/gemma-2b"

config = AutoConfig.from_pretrained(model_id)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_config(config).to(0)

trainer = trl.SFTTrainer(
    model=model, 
    args=args,
    train_dataset=train_dataset,
    dataset_text_field='text',
    max_seq_length=512,
)

trainer.train()
```

You can read more about the method in the [original repository](https://github.com/jiaweizzhao/GaLore) or the [paper](https://arxiv.org/abs/2403.03507).

Currently you can only train Linear layers that are considered as GaLore layers and will use low-rank decomposition to be trained while remaining layers will be optimized in the conventional manner.

Note it will take a bit of time before starting the training (~3 minutes for a 2B model on a NVIDIA A100), but training should go smoothly afterwards.

You can also perform layer-wise optimization by post-pending the optimizer name with `layerwise` like below:

```python
import torch
import datasets
import trl

from transformers import TrainingArguments, AutoConfig, AutoTokenizer, AutoModelForCausalLM

train_dataset = datasets.load_dataset('imdb', split='train')

args = TrainingArguments(
    output_dir="./test-galore",
    max_steps=100,
    per_device_train_batch_size=2,
    optim="galore_adamw_layerwise",
    optim_target_modules=[r".*.attn.*", r".*.mlp.


key: _node_content
string_value:
{"id_": "94c27b37-427e-48b6-8613-6a94f8f60662", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1e982a6-1f67-4a2e-a6ce-c33e6127e75e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "hash": "27d2530e4fe585d3fe3385ce93b69adf7f2f8d64bd0acaaa01d1ae6ece2fcb7c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "61e3f54c-9da7-42a1-9caa-2d6454e19b37", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "hash": "08bc19bd724161f11429f56920e5e079b12b3720be064d1681471f8c67e0ef35", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bc4cdd72-cd02-4238-8ff4-a9483bee72a4", "node_type": "1", "metadata": {}, "hash": "55f6c0e91b5ffdb1eff341c67ade304597c4cc9cac2b8a0bc8a61c53101f2f63", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13728, "end_char_idx": 16869, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
*", r".*.mlp.*"],
    optim_args="rank=64, update_proj_gap=100, scale=0.10",
)

model_id = "google/gemma-2b"

config = AutoConfig.from_pretrained(model_id)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_config(config).to(0)

trainer = trl.SFTTrainer(
    model=model, 
    args=args,
    train_dataset=train_dataset,
    dataset_text_field='text',
    max_seq_length=512,
)

trainer.train()
```

You can read more about the method in the [original repository](https://github.com/jiaweizzhao/GaLore) or the [paper](https://arxiv.org/abs/2403.03507).

Currently you can only train Linear layers that are considered as GaLore layers and will use low-rank decomposition to be trained while remaining layers will be optimized in the conventional manner.

Note it will take a bit of time before starting the training (~3 minutes for a 2B model on a NVIDIA A100), but training should go smoothly afterwards.

You can also perform layer-wise optimization by post-pending the optimizer name with `layerwise` like below:

```python
import torch
import datasets
import trl

from transformers import TrainingArguments, AutoConfig, AutoTokenizer, AutoModelForCausalLM

train_dataset = datasets.load_dataset('imdb', split='train')

args = TrainingArguments(
    output_dir="./test-galore",
    max_steps=100,
    per_device_train_batch_size=2,
    optim="galore_adamw_layerwise",
    optim_target_modules=[r".*.attn.*", r".*.mlp.*"]
)

model_id = "google/gemma-2b"

config = AutoConfig.from_pretrained(model_id)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_config(config).to(0)

trainer = trl.SFTTrainer(
    model=model, 
    args=args,
    train_dataset=train_dataset,
    dataset_text_field='text',
    max_seq_length=512,
)

trainer.train()
```

Note layerwise optimization is a bit experimental and does not support DDP (Distributed Data Parallel), thus you can run the training script only on a single GPU. Please see [this appropriate section](https://github.com/jiaweizzhao/GaLore?tab=readme-ov-file#train-7b-model-with-a-single-gpu-with-24gb-memory) for more details. Other features such as gradient clipping, DeepSpeed, etc might not be supported out of the box. Please [raise an issue on GitHub](https://github.com/huggingface/transformers/issues) if you encounter such issue.

## Liger Kernel

[Liger-Kernel](https://github.com/linkedin/Liger-Kernel) Kernel is a collection of Triton kernels developed by Linkedin designed specifically for LLM training. We have implemented Hugging Face Compatible RMSNorm, RoPE, SwiGLU, CrossEntropy, FusedLinearCrossEntropy, and more to come. It can effectively increase multi-GPU training throughput by 20% and reduces memory usage by 60%. The kernel works out of the box with flash attention, PyTorch FSDP, and Microsoft DeepSpeed.

<Tip>
Gain +20% throughput and reduce memory usage by 60% on LLaMA 3-8B model training. Achieve longer context lengths and larger batch sizes. It’s also useful if you want to scale up your model to multi-head training or large vocabulary sizes. Unleash multi-head training (medusa) and more.


key: _node_content
string_value:
{"id_": "bc4cdd72-cd02-4238-8ff4-a9483bee72a4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1e982a6-1f67-4a2e-a6ce-c33e6127e75e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "hash": "27d2530e4fe585d3fe3385ce93b69adf7f2f8d64bd0acaaa01d1ae6ece2fcb7c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "94c27b37-427e-48b6-8613-6a94f8f60662", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "hash": "7f381950eea268712579fb1bb51891c385d42380ccec542c3bfebb53f6dcd32a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "74582ca9-f8f1-4723-ab4e-fa77d41dd261", "node_type": "1", "metadata": {}, "hash": "0c52ced0e1f194ffee9145fb8d63f82f32b060a8cafeed840254f594e4b48846", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15709, "end_char_idx": 18767, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Please see [this appropriate section](https://github.com/jiaweizzhao/GaLore?tab=readme-ov-file#train-7b-model-with-a-single-gpu-with-24gb-memory) for more details. Other features such as gradient clipping, DeepSpeed, etc might not be supported out of the box. Please [raise an issue on GitHub](https://github.com/huggingface/transformers/issues) if you encounter such issue.

## Liger Kernel

[Liger-Kernel](https://github.com/linkedin/Liger-Kernel) Kernel is a collection of Triton kernels developed by Linkedin designed specifically for LLM training. We have implemented Hugging Face Compatible RMSNorm, RoPE, SwiGLU, CrossEntropy, FusedLinearCrossEntropy, and more to come. It can effectively increase multi-GPU training throughput by 20% and reduces memory usage by 60%. The kernel works out of the box with flash attention, PyTorch FSDP, and Microsoft DeepSpeed.

<Tip>
Gain +20% throughput and reduce memory usage by 60% on LLaMA 3-8B model training. Achieve longer context lengths and larger batch sizes. It’s also useful if you want to scale up your model to multi-head training or large vocabulary sizes. Unleash multi-head training (medusa) and more. See details and examples in [Liger](https://github.com/linkedin/Liger-Kernel/tree/main/examples)
</Tip>

First make sure to install Liger official repository:
```bash
pip install liger-kernel
```

You should pass `use_liger_kernel=True` to apply liger kernel on your model, for example:

```py
from transformers import TrainingArguments

training_args = TrainingArguments(
    output_dir="your-model",
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    num_train_epochs=2,
    weight_decay=0.01,
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    push_to_hub=True,
    use_liger_kernel=True
)
```

The kernel supports the Llama, Gemma, Mistral, and Mixtral model architectures. The most up-to-date list of supported models can be found [here](https://github.com/linkedin/Liger-Kernel). When `use_liger_kernel` is set to `True`, the corresponding layers in the original model will be patched with Liger's efficient implementation, so you don't need to do anything extra other than setting the argument value.

## LOMO optimizer

The LOMO optimizers have been introduced in [Full Parameter Fine-Tuning for Large Language Models with Limited Resources](https://hf.co/papers/2306.09782) and [AdaLomo: Low-memory Optimization with Adaptive Learning Rate](https://hf.co/papers/2310.10195). 
They both consist of an efficient full-parameter fine-tuning method. These optimizers fuse the gradient computation and the parameter update in one step to reduce memory usage. Supported optimizers for LOMO are `"lomo"` and `"adalomo"`. First either install LOMO from pypi `pip install lomo-optim` or install it from source with `pip install git+https://github.com/OpenLMLab/LOMO.git`. 

<Tip>

According to the authors, it is recommended to use `AdaLomo` without `grad_norm` to get better performance and higher throughput.


key: _node_content
string_value:
{"id_": "74582ca9-f8f1-4723-ab4e-fa77d41dd261", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1e982a6-1f67-4a2e-a6ce-c33e6127e75e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "hash": "27d2530e4fe585d3fe3385ce93b69adf7f2f8d64bd0acaaa01d1ae6ece2fcb7c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bc4cdd72-cd02-4238-8ff4-a9483bee72a4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "hash": "8d190628fd381018a5f28702e8b3b83eb3c75e8dc5a7695ff9bc4f11ff1fa6b1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b97fb182-327f-4f54-b352-dad4c9987f82", "node_type": "1", "metadata": {}, "hash": "55309d51240a7c8e9793ebd8b9409ed9bad9cba8a64f7cd12c3ee9b6c37d1440", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17636, "end_char_idx": 20156, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The most up-to-date list of supported models can be found [here](https://github.com/linkedin/Liger-Kernel). When `use_liger_kernel` is set to `True`, the corresponding layers in the original model will be patched with Liger's efficient implementation, so you don't need to do anything extra other than setting the argument value.

## LOMO optimizer

The LOMO optimizers have been introduced in [Full Parameter Fine-Tuning for Large Language Models with Limited Resources](https://hf.co/papers/2306.09782) and [AdaLomo: Low-memory Optimization with Adaptive Learning Rate](https://hf.co/papers/2310.10195). 
They both consist of an efficient full-parameter fine-tuning method. These optimizers fuse the gradient computation and the parameter update in one step to reduce memory usage. Supported optimizers for LOMO are `"lomo"` and `"adalomo"`. First either install LOMO from pypi `pip install lomo-optim` or install it from source with `pip install git+https://github.com/OpenLMLab/LOMO.git`. 

<Tip>

According to the authors, it is recommended to use `AdaLomo` without `grad_norm` to get better performance and higher throughput.

</Tip>

Below is a simple script to demonstrate how to fine-tune [google/gemma-2b](https://huggingface.co/google/gemma-2b) on IMDB dataset in full precision:

```python
import torch
import datasets
from transformers import TrainingArguments, AutoTokenizer, AutoModelForCausalLM
import trl

train_dataset = datasets.load_dataset('imdb', split='train')

args = TrainingArguments(
    output_dir="./test-lomo",
    max_steps=1000,
    per_device_train_batch_size=4,
    optim="adalomo",
    gradient_checkpointing=True,
    logging_strategy="steps",
    logging_steps=1,
    learning_rate=2e-6,
    save_strategy="no",
    run_name="lomo-imdb",
)

model_id = "google/gemma-2b"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, low_cpu_mem_usage=True).to(0)

trainer = trl.SFTTrainer(
    model=model, 
    args=args,
    train_dataset=train_dataset,
    dataset_text_field='text',
    max_seq_length=1024,
)

trainer.train()
```

## GrokAdamW optimizer

The GrokAdamW optimizer is designed to enhance training performance and stability, particularly for models that benefit from grokking signal functions. To use GrokAdamW, first install the optimizer package with `pip install grokadamw`.

<Tip>

GrokAdamW is particularly useful for models that require advanced optimization techniques to achieve better performance and stability.


key: _node_content
string_value:
{"id_": "b97fb182-327f-4f54-b352-dad4c9987f82", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1e982a6-1f67-4a2e-a6ce-c33e6127e75e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "hash": "27d2530e4fe585d3fe3385ce93b69adf7f2f8d64bd0acaaa01d1ae6ece2fcb7c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "74582ca9-f8f1-4723-ab4e-fa77d41dd261", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "hash": "acd1afa78721100e0f2913d0c129fb81ed3b858af24ade0301d190a8db331699", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c7eece16-1631-4cfd-b2d0-07c0f4990f83", "node_type": "1", "metadata": {}, "hash": "0de290bac1c5238691179d8393f820249839aaf8e4034be117621a5a47d3c0f5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18631, "end_char_idx": 21812, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<Tip>

According to the authors, it is recommended to use `AdaLomo` without `grad_norm` to get better performance and higher throughput.

</Tip>

Below is a simple script to demonstrate how to fine-tune [google/gemma-2b](https://huggingface.co/google/gemma-2b) on IMDB dataset in full precision:

```python
import torch
import datasets
from transformers import TrainingArguments, AutoTokenizer, AutoModelForCausalLM
import trl

train_dataset = datasets.load_dataset('imdb', split='train')

args = TrainingArguments(
    output_dir="./test-lomo",
    max_steps=1000,
    per_device_train_batch_size=4,
    optim="adalomo",
    gradient_checkpointing=True,
    logging_strategy="steps",
    logging_steps=1,
    learning_rate=2e-6,
    save_strategy="no",
    run_name="lomo-imdb",
)

model_id = "google/gemma-2b"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, low_cpu_mem_usage=True).to(0)

trainer = trl.SFTTrainer(
    model=model, 
    args=args,
    train_dataset=train_dataset,
    dataset_text_field='text',
    max_seq_length=1024,
)

trainer.train()
```

## GrokAdamW optimizer

The GrokAdamW optimizer is designed to enhance training performance and stability, particularly for models that benefit from grokking signal functions. To use GrokAdamW, first install the optimizer package with `pip install grokadamw`.

<Tip>

GrokAdamW is particularly useful for models that require advanced optimization techniques to achieve better performance and stability.

</Tip>

Below is a simple script to demonstrate how to fine-tune [google/gemma-2b](https://huggingface.co/google/gemma-2b) on the IMDB dataset using the GrokAdamW optimizer:

```python
import torch
import datasets
from transformers import TrainingArguments, AutoTokenizer, AutoModelForCausalLM, Trainer

# Load the IMDB dataset
train_dataset = datasets.load_dataset('imdb', split='train')

# Define the training arguments
args = TrainingArguments(
    output_dir="./test-grokadamw",
    max_steps=1000,
    per_device_train_batch_size=4,
    optim="grokadamw",
    logging_strategy="steps",
    logging_steps=1,
    learning_rate=2e-5,
    save_strategy="no",
    run_name="grokadamw-imdb",
)

# Load the model and tokenizer
model_id = "google/gemma-2b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, low_cpu_mem_usage=True).to(0)

# Initialize the Trainer
trainer = Trainer(
    model=model,
    args=args,
    train_dataset=train_dataset,
)

# Train the model
trainer.train()
```

This script demonstrates how to fine-tune the `google/gemma-2b` model on the IMDB dataset using the GrokAdamW optimizer. The `TrainingArguments` are configured to use GrokAdamW, and the dataset is passed to the `Trainer` for training.

## Accelerate and Trainer

The [`Trainer`] class is powered by [Accelerate](https://hf.co/docs/accelerate), a library for easily training PyTorch models in distributed environments with support for integrations such as [FullyShardedDataParallel (FSDP)](https://pytorch.org/blog/introducing-pytorch-fully-sharded-data-parallel-api/) and [DeepSpeed](https://www.deepspeed.ai/).


key: _node_content
string_value:
{"id_": "c7eece16-1631-4cfd-b2d0-07c0f4990f83", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1e982a6-1f67-4a2e-a6ce-c33e6127e75e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "hash": "27d2530e4fe585d3fe3385ce93b69adf7f2f8d64bd0acaaa01d1ae6ece2fcb7c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b97fb182-327f-4f54-b352-dad4c9987f82", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "hash": "05e78b1535be85bc24649b390571c4540803f196f72ed2ede025de45422bc603", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b0c4badb-e5ab-4394-b8cd-7d216b2e8499", "node_type": "1", "metadata": {}, "hash": "1b3b0d4b9b3751e17114afe25fb3859b942fe1059164a71fa2c46d591dcd62ba", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21322, "end_char_idx": 22268, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The `TrainingArguments` are configured to use GrokAdamW, and the dataset is passed to the `Trainer` for training.

## Accelerate and Trainer

The [`Trainer`] class is powered by [Accelerate](https://hf.co/docs/accelerate), a library for easily training PyTorch models in distributed environments with support for integrations such as [FullyShardedDataParallel (FSDP)](https://pytorch.org/blog/introducing-pytorch-fully-sharded-data-parallel-api/) and [DeepSpeed](https://www.deepspeed.ai/).

<Tip>

Learn more about FSDP sharding strategies, CPU offloading, and more with the [`Trainer`] in the [Fully Sharded Data Parallel](fsdp) guide.

</Tip>

To use Accelerate with [`Trainer`], run the [`accelerate.config`](https://huggingface.co/docs/accelerate/package_reference/cli#accelerate-config) command to set up training for your training environment. This command creates a `config_file.yaml` that'll be used when you launch your training script.


key: _node_content
string_value:
{"id_": "b0c4badb-e5ab-4394-b8cd-7d216b2e8499", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1e982a6-1f67-4a2e-a6ce-c33e6127e75e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "hash": "27d2530e4fe585d3fe3385ce93b69adf7f2f8d64bd0acaaa01d1ae6ece2fcb7c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c7eece16-1631-4cfd-b2d0-07c0f4990f83", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "hash": "f6ef11ae3776310a37443a0f2feccb49ea42cedf5fac7eccc3c079dc53c5a49e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e9c489c9-28cc-4f58-8bbe-d9f99703e0fc", "node_type": "1", "metadata": {}, "hash": "b86580875dbfe86cc0625fe7769988277b2a76725355a704803eb4d60bf484cc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21322, "end_char_idx": 25143, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The `TrainingArguments` are configured to use GrokAdamW, and the dataset is passed to the `Trainer` for training.

## Accelerate and Trainer

The [`Trainer`] class is powered by [Accelerate](https://hf.co/docs/accelerate), a library for easily training PyTorch models in distributed environments with support for integrations such as [FullyShardedDataParallel (FSDP)](https://pytorch.org/blog/introducing-pytorch-fully-sharded-data-parallel-api/) and [DeepSpeed](https://www.deepspeed.ai/).

<Tip>

Learn more about FSDP sharding strategies, CPU offloading, and more with the [`Trainer`] in the [Fully Sharded Data Parallel](fsdp) guide.

</Tip>

To use Accelerate with [`Trainer`], run the [`accelerate.config`](https://huggingface.co/docs/accelerate/package_reference/cli#accelerate-config) command to set up training for your training environment. This command creates a `config_file.yaml` that'll be used when you launch your training script. For example, some example configurations you can setup are:

<hfoptions id="config">
<hfoption id="DistributedDataParallel">

```yml
compute_environment: LOCAL_MACHINE                                                                                             
distributed_type: MULTI_GPU                                                                                                    
downcast_bf16: 'no'
gpu_ids: all
machine_rank: 0 #change rank as per the node
main_process_ip: 192.168.20.1
main_process_port: 9898
main_training_function: main
mixed_precision: fp16
num_machines: 2
num_processes: 8
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
```

</hfoption>
<hfoption id="FSDP">

```yml
compute_environment: LOCAL_MACHINE
distributed_type: FSDP
downcast_bf16: 'no'
fsdp_config:
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_backward_prefetch_policy: BACKWARD_PRE
  fsdp_forward_prefetch: true
  fsdp_offload_params: false
  fsdp_sharding_strategy: 1
  fsdp_state_dict_type: FULL_STATE_DICT
  fsdp_sync_module_states: true
  fsdp_transformer_layer_cls_to_wrap: BertLayer
  fsdp_use_orig_params: true
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 2
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
```

</hfoption>
<hfoption id="DeepSpeed">

```yml
compute_environment: LOCAL_MACHINE
deepspeed_config:
  deepspeed_config_file: /home/user/configs/ds_zero3_config.json
  zero3_init_flag: true
distributed_type: DEEPSPEED
downcast_bf16: 'no'
machine_rank: 0
main_training_function: main
num_machines: 1
num_processes: 4
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
```

</hfoption>
<hfoption id="DeepSpeed with Accelerate plugin">

```yml
compute_environment: LOCAL_MACHINE                                                                                             
deepspeed_config:                                                                                                              
  gradient_accumulation_steps: 1
  gradient_clipping: 0.7
  offload_optimizer_device: cpu
  offload_param_device: cpu
  zero3_init_flag: true
  zero_stage: 2
distributed_type: DEEPSPEED
downcast_bf16: 'no'
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 4
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
```

</hfoption>
</hfoptions>

The [`accelerate_launch`](https://huggingface.co/docs/accelerate/package_reference/cli#accelerate-launch) command is the recommended way to launch your training script on a distributed system with Accelerate and [`Trainer`] with the parameters specified in `config_file.yaml`.


key: _node_content
string_value:
{"id_": "e9c489c9-28cc-4f58-8bbe-d9f99703e0fc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e1e982a6-1f67-4a2e-a6ce-c33e6127e75e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "hash": "27d2530e4fe585d3fe3385ce93b69adf7f2f8d64bd0acaaa01d1ae6ece2fcb7c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b0c4badb-e5ab-4394-b8cd-7d216b2e8499", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/trainer", "title": "Trainer", "tokens": 6527, "retrieve_doc": true, "source": "transformers"}, "hash": "adbfe856b65e6dcece910efca2bb9905ac944938cbdc60fdd49e054a41d943f9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 25144, "end_char_idx": 26800, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This file is saved to the Accelerate cache folder and automatically loaded when you run `accelerate_launch`.

For example, to run the [run_glue.py](https://github.com/huggingface/transformers/blob/f4db565b695582891e43a5e042e5d318e28f20b8/examples/pytorch/text-classification/run_glue.py#L4) training script with the FSDP configuration:

```bash
accelerate launch \
    ./examples/pytorch/text-classification/run_glue.py \
    --model_name_or_path google-bert/bert-base-cased \
    --task_name $TASK_NAME \
    --do_train \
    --do_eval \
    --max_seq_length 128 \
    --per_device_train_batch_size 16 \
    --learning_rate 5e-5 \
    --num_train_epochs 3 \
    --output_dir /tmp/$TASK_NAME/ \
    --overwrite_output_dir
```

You could also specify the parameters from the `config_file.yaml` file directly in the command line:

```bash
accelerate launch --num_processes=2 \
    --use_fsdp \
    --mixed_precision=bf16 \
    --fsdp_auto_wrap_policy=TRANSFORMER_BASED_WRAP  \
    --fsdp_transformer_layer_cls_to_wrap="BertLayer" \
    --fsdp_sharding_strategy=1 \
    --fsdp_state_dict_type=FULL_STATE_DICT \
    ./examples/pytorch/text-classification/run_glue.py
    --model_name_or_path google-bert/bert-base-cased \
    --task_name $TASK_NAME \
    --do_train \
    --do_eval \
    --max_seq_length 128 \
    --per_device_train_batch_size 16 \
    --learning_rate 5e-5 \
    --num_train_epochs 3 \
    --output_dir /tmp/$TASK_NAME/ \
    --overwrite_output_dir
```

Check out the [Launching your Accelerate scripts](https://huggingface.co/docs/accelerate/basic_tutorials/launch) tutorial to learn more about `accelerate_launch` and custom configurations.


key: _node_content
string_value:
{"id_": "5ed29efa-bf46-44b1-b84b-ab2564d6f69b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0d88f10e-52bf-45ee-ab95-c9bd80aa550a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "transformers"}, "hash": "a4b63f4ecac3816553904c23884eabea9b42923c6eedc0bf4c6a97ad144f0f97", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4e691712-7eef-472c-90f6-b1d4b9d6e66e", "node_type": "1", "metadata": {}, "hash": "69d81e0588f6a06e36c9b5d8e4f960e87c81dbbb4bee32daa3509f7272d6abc0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3004, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Train with a script

Along with the 🤗 Transformers [notebooks](./notebooks), there are also example scripts demonstrating how to train a model for a task with [PyTorch](https://github.com/huggingface/transformers/tree/main/examples/pytorch), [TensorFlow](https://github.com/huggingface/transformers/tree/main/examples/tensorflow), or [JAX/Flax](https://github.com/huggingface/transformers/tree/main/examples/flax).

You will also find scripts we've used in our [research projects](https://github.com/huggingface/transformers/tree/main/examples/research_projects) and [legacy examples](https://github.com/huggingface/transformers/tree/main/examples/legacy) which are mostly community contributed. These scripts are not actively maintained and require a specific version of 🤗 Transformers that will most likely be incompatible with the latest version of the library.

The example scripts are not expected to work out-of-the-box on every problem, and you may need to adapt the script to the problem you're trying to solve. To help you with this, most of the scripts fully expose how data is preprocessed, allowing you to edit it as necessary for your use case.

For any feature you'd like to implement in an example script, please discuss it on the [forum](https://discuss.huggingface.co/) or in an [issue](https://github.com/huggingface/transformers/issues) before submitting a Pull Request. While we welcome bug fixes, it is unlikely we will merge a Pull Request that adds more functionality at the cost of readability.

This guide will show you how to run an example summarization training script in [PyTorch](https://github.com/huggingface/transformers/tree/main/examples/pytorch/summarization) and [TensorFlow](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/summarization). All examples are expected to work with both frameworks unless otherwise specified.

## Setup

To successfully run the latest version of the example scripts, you have to **install 🤗 Transformers from source** in a new virtual environment:

```bash
git clone https://github.com/huggingface/transformers
cd transformers
pip install .
```

For older versions of the example scripts, click on the toggle below:

<details>
  <summary>Examples for older versions of 🤗 Transformers</summary>
	<ul>
		<li><a href="https://github.com/huggingface/transformers/tree/v4.5.1/examples">v4.5.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v4.4.2/examples">v4.4.2</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v4.3.3/examples">v4.3.3</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v4.2.2/examples">v4.2.2</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v4.1.1/examples">v4.1.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v4.0.1/examples">v4.0.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v3.5.1/examples">v3.5.1</a></li>
		<li><a href="https://github.


key: _node_content
string_value:
{"id_": "4e691712-7eef-472c-90f6-b1d4b9d6e66e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0d88f10e-52bf-45ee-ab95-c9bd80aa550a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "transformers"}, "hash": "a4b63f4ecac3816553904c23884eabea9b42923c6eedc0bf4c6a97ad144f0f97", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5ed29efa-bf46-44b1-b84b-ab2564d6f69b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "transformers"}, "hash": "ea88433190fa073cd99555105329d30ee3b4ba0224d6ea9df5ae42be0b9d1565", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "090c97c6-5e83-4011-b620-c8db7549dd9c", "node_type": "1", "metadata": {}, "hash": "458b5668fdbef749238e11a3f8337f43be0500b308455f510069d1633a212001", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1805, "end_char_idx": 4034, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
All examples are expected to work with both frameworks unless otherwise specified.

## Setup

To successfully run the latest version of the example scripts, you have to **install 🤗 Transformers from source** in a new virtual environment:

```bash
git clone https://github.com/huggingface/transformers
cd transformers
pip install .
```

For older versions of the example scripts, click on the toggle below:

<details>
  <summary>Examples for older versions of 🤗 Transformers</summary>
	<ul>
		<li><a href="https://github.com/huggingface/transformers/tree/v4.5.1/examples">v4.5.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v4.4.2/examples">v4.4.2</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v4.3.3/examples">v4.3.3</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v4.2.2/examples">v4.2.2</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v4.1.1/examples">v4.1.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v4.0.1/examples">v4.0.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v3.5.1/examples">v3.5.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v3.4.0/examples">v3.4.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v3.3.1/examples">v3.3.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v3.2.0/examples">v3.2.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v3.1.0/examples">v3.1.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v3.0.2/examples">v3.0.2</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.11.0/examples">v2.11.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.10.0/examples">v2.10.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.9.1/examples">v2.9.1</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.8.0/examples">v2.8.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.7.0/examples">v2.7.0</a></li>
		<li><a href="https://github.com/huggingface/transformers/tree/v2.6.0/examples">v2.6.


key: _node_content
string_value:
{"id_": "090c97c6-5e83-4011-b620-c8db7549dd9c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0d88f10e-52bf-45ee-ab95-c9bd80aa550a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "transformers"}, "hash": "a4b63f4ecac3816553904c23884eabea9b42923c6eedc0bf4c6a97ad144f0f97", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4e691712-7eef-472c-90f6-b1d4b9d6e66e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/run_scripts", "title": "Train with a script", "tokens": 4440, "retrieve_doc": true, "source": "transformers"}, "hash": "9f6d05c63e7561925124f1906415611f4abbe74b64c420fdfc61c53966bd1a47", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "309349f8-bf4a-48ad-9ed9-f4443af3efd6", "node_type": "1", "metadata": {}, "hash": "a675129f05031861d2a77a33bee7e5eb4300fa8c381eeee7ca4c27744c7713ab", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3041, "end_char_idx": 5066, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


