# embedding_metadata

key: _node_content
string_value:
{"id_": "23c985c2-2b62-4233-845d-b85fc0be3731", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "d0faf8596bdd083d896283e3e14e6b207798309c03ac95db0d357f4c7d7780d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3defa731-9e06-4e45-9e8c-daf690ed62e7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "d2d11226252519af860279cea21b2c91c30be3dc7e5893c102be75d717dcdd12", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bedbc242-39c3-4cbb-b145-8c819a6d3489", "node_type": "1", "metadata": {}, "hash": "eb91ec26d6cf9e8053b913b590bf7b79b97d23b471ceb71b1ce081f80e2923a2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3224, "end_char_idx": 5988, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Alright, 
alright, I'll give you the lowdown. But don't say I didn't warn you, I'm a robot, not a tour guide!

So, you wanna know what's fun to do in the Big Apple? Well, let me tell you, there's a million 
things to do, but I'll give you the highlights. First off, you gotta see the sights: the Statue of 
Liberty, Central Park, Times Square... you know, the usual tourist traps. But if you're lookin' for 
something a little more... unusual, I'd recommend checkin' out the Museum of Modern Art. It's got 
some wild stuff, like that Warhol guy's soup cans and all that jazz.

And if you're feelin' adventurous, take a walk across the Brooklyn Bridge. Just watch out for 
those pesky pigeons, they're like little feathered thieves! (laughs) Get it? Thieves? Ah, never mind.

Now, if you're lookin' for some serious fun, hit up the comedy clubs in Greenwich Village. You might 
even catch a glimpse of some up-and-coming comedians... or a bunch of wannabes tryin' to make it big. (winks)

And finally, if you're feelin' like a real New Yorker, grab a slice of pizza from one of the many amazing
pizzerias around the city. Just don't try to order a "robot-sized" slice, trust me, it won't end well. (laughs)

So, there you have it, pal! That's my expert advice on what to do in New York. Now, if you'll
excuse me, I've got some oil changes to attend to. (winks)
```

You can continue the chat by appending your own response to it. The
`response` object returned by the pipeline actually contains the entire chat so far, so we can simply append
a message and pass it back:

```python
chat = response[0]['generated_text']
chat.append(
    {"role": "user", "content": "Wait, what's so wild about soup cans?"}
)
response = pipe(chat, max_new_tokens=512)
print(response[0]['generated_text'][-1]['content'])
```

And you'll get:

```text
(laughs) Oh, you're killin' me, pal! You don't get it, do you? Warhol's soup cans are like, art, man! 
It's like, he took something totally mundane, like a can of soup, and turned it into a masterpiece. It's 
like, "Hey, look at me, I'm a can of soup, but I'm also a work of art!" 
(sarcastically) Oh, yeah, real original, Andy.

But, you know, back in the '60s, it was like, a big deal. People were all about challenging the
status quo, and Warhol was like, the king of that. He took the ordinary and made it extraordinary.
And, let me tell you, it was like, a real game-changer. I mean, who would've thought that a can of soup could be art? (laughs)

But, hey, you're not alone, pal. I mean, I'm a robot, and even I don't get it. (winks)
But, hey, that's what makes art, art, right? (laughs)
```

The remainder of this tutorial will cover specific topics such
as performance and memory, or how to select a chat model for your needs.


key: _node_content
string_value:
{"id_": "bedbc242-39c3-4cbb-b145-8c819a6d3489", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "d0faf8596bdd083d896283e3e14e6b207798309c03ac95db0d357f4c7d7780d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "23c985c2-2b62-4233-845d-b85fc0be3731", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "555b708d8d45ec100d4a59e38dde249211fd125538d3b32f8cfd513960f0f594", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c46ee4ca-f540-41b2-8ca9-e8aa066a229c", "node_type": "1", "metadata": {}, "hash": "96a450fe35345f2d532528ac1050d61e835335dcfa1c569998f68b707010d10f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4576, "end_char_idx": 7448, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
(winks)
```

You can continue the chat by appending your own response to it. The
`response` object returned by the pipeline actually contains the entire chat so far, so we can simply append
a message and pass it back:

```python
chat = response[0]['generated_text']
chat.append(
    {"role": "user", "content": "Wait, what's so wild about soup cans?"}
)
response = pipe(chat, max_new_tokens=512)
print(response[0]['generated_text'][-1]['content'])
```

And you'll get:

```text
(laughs) Oh, you're killin' me, pal! You don't get it, do you? Warhol's soup cans are like, art, man! 
It's like, he took something totally mundane, like a can of soup, and turned it into a masterpiece. It's 
like, "Hey, look at me, I'm a can of soup, but I'm also a work of art!" 
(sarcastically) Oh, yeah, real original, Andy.

But, you know, back in the '60s, it was like, a big deal. People were all about challenging the
status quo, and Warhol was like, the king of that. He took the ordinary and made it extraordinary.
And, let me tell you, it was like, a real game-changer. I mean, who would've thought that a can of soup could be art? (laughs)

But, hey, you're not alone, pal. I mean, I'm a robot, and even I don't get it. (winks)
But, hey, that's what makes art, art, right? (laughs)
```

The remainder of this tutorial will cover specific topics such
as performance and memory, or how to select a chat model for your needs.

## Choosing a chat model

There are an enormous number of different chat models available on the [Hugging Face Hub](https://huggingface.co/models?pipeline_tag=text-generation&sort=trending),
and new users often feel very overwhelmed by the selection offered. Don't be, though! You really need to just focus on
two important considerations: 
- The model's size, which will determine if you can fit it in memory and how quickly it will
run.
- The quality of the model's chat output.

In general, these are correlated - bigger models tend to be 
more capable, but even so there's a lot of variation at a given size point!

### Size and model naming
The size of a model is easy to spot - it's the number in the model name, like "8B" or "70B". This is the number of
**parameters** in the model. Without quantization, you should expect to need about 2 bytes of memory per parameter.
This means that an "8B" model with 8 billion parameters will need about 16GB of memory just to fit the parameters, 
plus a little extra for other overhead. It's a good fit for a high-end consumer GPU with 24GB of memory, such as a 3090
or 4090.

Some chat models are "Mixture of Experts" models. These may list their sizes in different ways, such as "8x7B" or 
"141B-A35B". The numbers are a little fuzzier here, but in general you can read this as saying that the model
has approximately 56 (8x7) billion parameters in the first case, or 141 billion parameters in the second case.


key: _node_content
string_value:
{"id_": "c46ee4ca-f540-41b2-8ca9-e8aa066a229c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "d0faf8596bdd083d896283e3e14e6b207798309c03ac95db0d357f4c7d7780d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bedbc242-39c3-4cbb-b145-8c819a6d3489", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "67319942d74b4b6b4527515a0b7503897d81f57f1ef2adf94ac33c045b544ab7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c15eb8bd-7f11-45ac-8c77-09809fb64f17", "node_type": "1", "metadata": {}, "hash": "9ca46b9c0d26bfaf7b3888c1b59ab6c834649bea3acf04f01c77137c3441b7d0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5839, "end_char_idx": 9041, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
(laughs)
```

The remainder of this tutorial will cover specific topics such
as performance and memory, or how to select a chat model for your needs.

## Choosing a chat model

There are an enormous number of different chat models available on the [Hugging Face Hub](https://huggingface.co/models?pipeline_tag=text-generation&sort=trending),
and new users often feel very overwhelmed by the selection offered. Don't be, though! You really need to just focus on
two important considerations: 
- The model's size, which will determine if you can fit it in memory and how quickly it will
run.
- The quality of the model's chat output.

In general, these are correlated - bigger models tend to be 
more capable, but even so there's a lot of variation at a given size point!

### Size and model naming
The size of a model is easy to spot - it's the number in the model name, like "8B" or "70B". This is the number of
**parameters** in the model. Without quantization, you should expect to need about 2 bytes of memory per parameter.
This means that an "8B" model with 8 billion parameters will need about 16GB of memory just to fit the parameters, 
plus a little extra for other overhead. It's a good fit for a high-end consumer GPU with 24GB of memory, such as a 3090
or 4090.

Some chat models are "Mixture of Experts" models. These may list their sizes in different ways, such as "8x7B" or 
"141B-A35B". The numbers are a little fuzzier here, but in general you can read this as saying that the model
has approximately 56 (8x7) billion parameters in the first case, or 141 billion parameters in the second case.

Note that it is very common to use quantization techniques to reduce the memory usage per parameter to 8 bits, 4 bits,
or even less. This topic is discussed in more detail in the [Memory considerations](#memory-considerations) section below.

### But which chat model is best?
Even once you know the size of chat model you can run, there's still a lot of choice out there. One way to sift through
it all is to consult **leaderboards**. Two of the most popular leaderboards are the [OpenLLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
and the [LMSys Chatbot Arena Leaderboard](https://chat.lmsys.org/?leaderboard). Note that the LMSys leaderboard
also includes proprietary models - look at the `licence` column to identify open-source ones that you can download, then
search for them on the [Hugging Face Hub](https://huggingface.co/models?pipeline_tag=text-generation&sort=trending).

### Specialist domains
Some models may be specialized for certain domains, such as medical or legal text, or non-English languages. 
If you're working in these domains, you may find that a specialized model will give you big performance benefits. 
Don't automatically assume that, though! Particularly when specialized models are smaller or older than the current 
cutting-edge, a top-end general-purpose model may still outclass them. Thankfully, we are beginning to see 
[domain-specific leaderboards](https://huggingface.co/blog/leaderboard-medicalllm) that should make it easier to locate
the best models for specialized domains.

## What happens inside the pipeline?


key: _node_content
string_value:
{"id_": "c15eb8bd-7f11-45ac-8c77-09809fb64f17", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "d0faf8596bdd083d896283e3e14e6b207798309c03ac95db0d357f4c7d7780d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c46ee4ca-f540-41b2-8ca9-e8aa066a229c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "18e38b74051daa9dcd6f5bddde8c527953346135c6e491831c1dfcb60e5c6dc3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "384ff287-a747-402f-a666-d42834f0d5a0", "node_type": "1", "metadata": {}, "hash": "24606726fa4afaa052e3e91c049154dc6e9a8f05a11ae6d33ff9f15333fc98c9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7450, "end_char_idx": 9629, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note that it is very common to use quantization techniques to reduce the memory usage per parameter to 8 bits, 4 bits,
or even less. This topic is discussed in more detail in the [Memory considerations](#memory-considerations) section below.

### But which chat model is best?
Even once you know the size of chat model you can run, there's still a lot of choice out there. One way to sift through
it all is to consult **leaderboards**. Two of the most popular leaderboards are the [OpenLLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
and the [LMSys Chatbot Arena Leaderboard](https://chat.lmsys.org/?leaderboard). Note that the LMSys leaderboard
also includes proprietary models - look at the `licence` column to identify open-source ones that you can download, then
search for them on the [Hugging Face Hub](https://huggingface.co/models?pipeline_tag=text-generation&sort=trending).

### Specialist domains
Some models may be specialized for certain domains, such as medical or legal text, or non-English languages. 
If you're working in these domains, you may find that a specialized model will give you big performance benefits. 
Don't automatically assume that, though! Particularly when specialized models are smaller or older than the current 
cutting-edge, a top-end general-purpose model may still outclass them. Thankfully, we are beginning to see 
[domain-specific leaderboards](https://huggingface.co/blog/leaderboard-medicalllm) that should make it easier to locate
the best models for specialized domains.

## What happens inside the pipeline?

The quickstart above used a high-level pipeline to chat with a chat model, which is convenient, but not the
most flexible. Let's take a more low-level approach, to see each of the steps involved in chat. Let's start with
a code sample, and then break it down:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Prepare the input as before
chat = [
    {"role": "system", "content": "You are a sassy, wise-cracking robot as imagined by Hollywood circa 1986."},
    {"role": "user", "content": "Hey, can you tell me any fun things to do in New York?"}


key: _node_content
string_value:
{"id_": "384ff287-a747-402f-a666-d42834f0d5a0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "d0faf8596bdd083d896283e3e14e6b207798309c03ac95db0d357f4c7d7780d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c15eb8bd-7f11-45ac-8c77-09809fb64f17", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "3d71a230a2cb79a7c78fb19e0289d31ad56bbadf026aa34db922bf9f1c1ce64c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "024e057c-a3c5-49e4-af8d-6e4c102f586f", "node_type": "1", "metadata": {}, "hash": "d89cda7242f1103b34849b6e30a65c06075820313544e8b04d02af7060979123", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8099, "end_char_idx": 11261, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note that the LMSys leaderboard
also includes proprietary models - look at the `licence` column to identify open-source ones that you can download, then
search for them on the [Hugging Face Hub](https://huggingface.co/models?pipeline_tag=text-generation&sort=trending).

### Specialist domains
Some models may be specialized for certain domains, such as medical or legal text, or non-English languages. 
If you're working in these domains, you may find that a specialized model will give you big performance benefits. 
Don't automatically assume that, though! Particularly when specialized models are smaller or older than the current 
cutting-edge, a top-end general-purpose model may still outclass them. Thankfully, we are beginning to see 
[domain-specific leaderboards](https://huggingface.co/blog/leaderboard-medicalllm) that should make it easier to locate
the best models for specialized domains.

## What happens inside the pipeline?

The quickstart above used a high-level pipeline to chat with a chat model, which is convenient, but not the
most flexible. Let's take a more low-level approach, to see each of the steps involved in chat. Let's start with
a code sample, and then break it down:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Prepare the input as before
chat = [
    {"role": "system", "content": "You are a sassy, wise-cracking robot as imagined by Hollywood circa 1986."},
    {"role": "user", "content": "Hey, can you tell me any fun things to do in New York?"}
]

# 1: Load the model and tokenizer
model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct", device_map="auto", torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")

# 2: Apply the chat template
formatted_chat = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)
print("Formatted chat:\n", formatted_chat)

# 3: Tokenize the chat (This can be combined with the previous step using tokenize=True)
inputs = tokenizer(formatted_chat, return_tensors="pt", add_special_tokens=False)
# Move the tokenized inputs to the same device the model is on (GPU/CPU)
inputs = {key: tensor.to(model.device) for key, tensor in inputs.items()}
print("Tokenized inputs:\n", inputs)

# 4: Generate text from the model
outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.1)
print("Generated tokens:\n", outputs)

# 5: Decode the output back to a string
decoded_output = tokenizer.decode(outputs[0][inputs['input_ids'].size(1):], skip_special_tokens=True)
print("Decoded output:\n", decoded_output)
```

There's a lot in here, each piece of which could be its own document! Rather than going into too much detail, I'll cover
the broad ideas, and leave the details for the linked documents. The key steps are:

1. [Models](https://huggingface.co/learn/nlp-course/en/chapter2/3) and [Tokenizers](https://huggingface.co/learn/nlp-course/en/chapter2/4?fw=pt) are loaded from the Hugging Face Hub.
2. The chat is formatted using the tokenizer's [chat template](https://huggingface.co/docs/transformers/main/en/chat_templating)
3.


key: _node_content
string_value:
{"id_": "024e057c-a3c5-49e4-af8d-6e4c102f586f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "d0faf8596bdd083d896283e3e14e6b207798309c03ac95db0d357f4c7d7780d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "384ff287-a747-402f-a666-d42834f0d5a0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "4289751eb75600542d671b173327bcd7ce7cd7973139b43dcaca9edd82f60726", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6b5295d0-5b5b-4fb7-b36c-e3491ab30d00", "node_type": "1", "metadata": {}, "hash": "ef0a01f30941c39bca4814fa1beddc1000d1828bae11e1e0f502378a15ccdd28", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10811, "end_char_idx": 13892, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Rather than going into too much detail, I'll cover
the broad ideas, and leave the details for the linked documents. The key steps are:

1. [Models](https://huggingface.co/learn/nlp-course/en/chapter2/3) and [Tokenizers](https://huggingface.co/learn/nlp-course/en/chapter2/4?fw=pt) are loaded from the Hugging Face Hub.
2. The chat is formatted using the tokenizer's [chat template](https://huggingface.co/docs/transformers/main/en/chat_templating)
3. The formatted chat is [tokenized](https://huggingface.co/learn/nlp-course/en/chapter2/4) using the tokenizer.
4. We [generate](https://huggingface.co/docs/transformers/en/llm_tutorial) a response from the model.
5. The tokens output by the model are decoded back to a string

## Performance, memory and hardware

You probably know by now that most machine learning tasks are run on GPUs. However, it is entirely possible
to generate text from a chat model or language model on a CPU, albeit somewhat more slowly. If you can fit
the model in GPU memory, though, this will usually be the preferable option.

### Memory considerations

By default, Hugging Face classes like [`TextGenerationPipeline`] or [`AutoModelForCausalLM`] will load the model in 
`float32` precision. This means that it will need 4 bytes (32 bits) per parameter, so an "8B" model with 8 billion
parameters will need ~32GB of memory. However, this can be wasteful! Most modern language models are trained in 
"bfloat16" precision, which uses only 2 bytes per parameter. If your hardware supports it (Nvidia 30xx/Axxx
or newer), you can load the model in `bfloat16` precision, using the `torch_dtype` argument as we did above.

It is possible to go even lower than 16-bits using "quantization", a method to lossily compress model weights. This
allows each parameter to be squeezed down to 8 bits, 4 bits or even less. Note that, especially at 4 bits,
the model's outputs may be negatively affected, but often this is a tradeoff worth making to fit a larger and more
capable chat model in memory. Let's see this in action with `bitsandbytes`:

```python
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(load_in_8bit=True)  # You can also try load_in_4bit
model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct", device_map="auto", quantization_config=quantization_config)
```

Or we can do the same thing using the `pipeline` API:

```python
from transformers import pipeline, BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(load_in_8bit=True)  # You can also try load_in_4bit
pipe = pipeline("text-generation", "meta-llama/Meta-Llama-3-8B-Instruct", device_map="auto", model_kwargs={"quantization_config": quantization_config})
```

There are several other options for quantizing models besides `bitsandbytes` - please see the [Quantization guide](./quantization)
for more information.

### Performance considerations

<Tip>

For a more extensive guide on language model performance and optimization, check out [LLM Inference Optimization](./llm_optims) .

</Tip>


key: _node_content
string_value:
{"id_": "6b5295d0-5b5b-4fb7-b36c-e3491ab30d00", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2678abda-b5a2-420f-822e-053812eb0ba3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "d0faf8596bdd083d896283e3e14e6b207798309c03ac95db0d357f4c7d7780d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "024e057c-a3c5-49e4-af8d-6e4c102f586f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "235374372074d40e89a7710f64e30bcd64cec4f745fe697d41e69946c4cf34d4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12301, "end_char_idx": 16507, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If your hardware supports it (Nvidia 30xx/Axxx
or newer), you can load the model in `bfloat16` precision, using the `torch_dtype` argument as we did above.

It is possible to go even lower than 16-bits using "quantization", a method to lossily compress model weights. This
allows each parameter to be squeezed down to 8 bits, 4 bits or even less. Note that, especially at 4 bits,
the model's outputs may be negatively affected, but often this is a tradeoff worth making to fit a larger and more
capable chat model in memory. Let's see this in action with `bitsandbytes`:

```python
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(load_in_8bit=True)  # You can also try load_in_4bit
model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct", device_map="auto", quantization_config=quantization_config)
```

Or we can do the same thing using the `pipeline` API:

```python
from transformers import pipeline, BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(load_in_8bit=True)  # You can also try load_in_4bit
pipe = pipeline("text-generation", "meta-llama/Meta-Llama-3-8B-Instruct", device_map="auto", model_kwargs={"quantization_config": quantization_config})
```

There are several other options for quantizing models besides `bitsandbytes` - please see the [Quantization guide](./quantization)
for more information.

### Performance considerations

<Tip>

For a more extensive guide on language model performance and optimization, check out [LLM Inference Optimization](./llm_optims) .

</Tip>


As a general rule, larger chat models will be slower in addition to requiring more memory. It's possible to be
more concrete about this, though: Generating text from a chat model is unusual in that it is bottlenecked by
**memory bandwidth** rather than compute power, because every active parameter must be read from memory for each
token that the model generates. This means that number of tokens per second you can generate from a chat
model is generally proportional to the total bandwidth of the memory it resides in, divided by the size of the model.

In our quickstart example above, our model was ~16GB in size when loaded in `bfloat16` precision. 
This means that 16GB must be read from memory for every token generated by the model. Total memory bandwidth can
vary from 20-100GB/sec for consumer CPUs to 200-900GB/sec for consumer GPUs, specialized CPUs like
Intel Xeon, AMD Threadripper/Epyc or high-end Apple silicon, and finally up to 2-3TB/sec for data center GPUs like
the Nvidia A100 or H100. This should give you a good idea of the generation speed you can expect from these different
hardware types.

Therefore, if you want to improve the speed of text generation, the easiest solution is to either reduce the
size of the model in memory (usually by quantization), or get hardware with higher memory bandwidth. For advanced users, 
several other techniques exist to get around this bandwidth bottleneck. The most common are variants on 
[assisted generation](https://huggingface.co/blog/assisted-generation), also known as "speculative
sampling". These techniques try to guess multiple future tokens at once, often using a smaller "draft model", and then
confirm these generations with the chat model. If the guesses are validated by the chat model, more than one token can
be generated per forward pass, which greatly alleviates the bandwidth bottleneck and improves generation speed.  

Finally, we should also note the impact of "Mixture of Experts" (MoE) models here. Several popular chat models,
such as Mixtral, Qwen-MoE and DBRX, are MoE models. In these models, not every parameter is active for every token generated.
As a result, MoE models generally have much lower memory bandwidth requirements, even though their total size
can be quite large. They can therefore be several times faster than a normal "dense" model of the same size. However,
techniques like assisted generation are generally ineffective for these models because more parameters will become
active with each new speculated token, which will negate the bandwidth and speed benefits that the MoE architecture
provides.


key: _node_content
string_value:
{"id_": "d539d9e9-9591-4c67-a1cd-58abf7bd5017", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8e347b8b-6f0d-43db-90c7-f57b1aec74b3", "node_type": "1", "metadata": {}, "hash": "e1a97070c0f542c2a82d1b8466ed0c96915b8669d35b069519b1813076853ad2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3357, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# How 🤗 Transformers solve tasks

In [What 🤗 Transformers can do](task_summary), you learned about natural language processing (NLP), speech and audio, computer vision tasks, and some important applications of them. This page will look closely at how models solve these tasks and explain what's happening under the hood. There are many ways to solve a given task, some models may implement certain techniques or even approach the task from a new angle, but for Transformer models, the general idea is the same. Owing to its flexible architecture, most models are a variant of an encoder, decoder, or encoder-decoder structure. In addition to Transformer models, our library also has several convolutional neural networks (CNNs), which are still used today for computer vision tasks. We'll also explain how a modern CNN works.

To explain how tasks are solved, we'll walk through what goes on inside the model to output useful predictions.

- [Wav2Vec2](model_doc/wav2vec2) for audio classification and automatic speech recognition (ASR)
- [Vision Transformer (ViT)](model_doc/vit) and [ConvNeXT](model_doc/convnext) for image classification
- [DETR](model_doc/detr) for object detection
- [Mask2Former](model_doc/mask2former) for image segmentation
- [GLPN](model_doc/glpn) for depth estimation
- [BERT](model_doc/bert) for NLP tasks like text classification, token classification and question answering that use an encoder
- [GPT2](model_doc/gpt2) for NLP tasks like text generation that use a decoder
- [BART](model_doc/bart) for NLP tasks like summarization and translation that use an encoder-decoder

<Tip>

Before you go further, it is good to have some basic knowledge of the original Transformer architecture. Knowing how encoders, decoders, and attention work will aid you in understanding how different Transformer models work. If you're just getting started or need a refresher, check out our [course](https://huggingface.co/course/chapter1/4?fw=pt) for more information! 

</Tip>

## Speech and audio

[Wav2Vec2](model_doc/wav2vec2) is a self-supervised model pretrained on unlabeled speech data and finetuned on labeled data for audio classification and automatic speech recognition. 

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/wav2vec2_architecture.png"/>
</div>

This model has four main components:

1. A *feature encoder* takes the raw audio waveform, normalizes it to zero mean and unit variance, and converts it into a sequence of feature vectors that are each 20ms long.

2. Waveforms are continuous by nature, so they can't be divided into separate units like a sequence of text can be split into words. That's why the feature vectors are passed to a *quantization module*, which aims to learn discrete speech units. The speech unit is chosen from a collection of codewords, known as a *codebook* (you can think of this as the vocabulary). From the codebook, the vector or speech unit, that best represents the continuous audio input is chosen and forwarded through the model.

3. About half of the feature vectors are randomly masked, and the masked feature vector is fed to a *context network*, which is a Transformer encoder that also adds relative positional embeddings.

4. The pretraining objective of the context network is a *contrastive task*.


key: _node_content
string_value:
{"id_": "8e347b8b-6f0d-43db-90c7-f57b1aec74b3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d539d9e9-9591-4c67-a1cd-58abf7bd5017", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "9efee23a70ad906a2fb4bc269cec2386f502db23e59d048980346fc69b2b82ac", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2639d582-045c-4396-b6a1-cf2d025f8f35", "node_type": "1", "metadata": {}, "hash": "3c31f07bfc10f5796ed28049146f3fb3730c5aab9d5105a78a4ecf2ab284190a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1718, "end_char_idx": 5348, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Knowing how encoders, decoders, and attention work will aid you in understanding how different Transformer models work. If you're just getting started or need a refresher, check out our [course](https://huggingface.co/course/chapter1/4?fw=pt) for more information! 

</Tip>

## Speech and audio

[Wav2Vec2](model_doc/wav2vec2) is a self-supervised model pretrained on unlabeled speech data and finetuned on labeled data for audio classification and automatic speech recognition. 

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/wav2vec2_architecture.png"/>
</div>

This model has four main components:

1. A *feature encoder* takes the raw audio waveform, normalizes it to zero mean and unit variance, and converts it into a sequence of feature vectors that are each 20ms long.

2. Waveforms are continuous by nature, so they can't be divided into separate units like a sequence of text can be split into words. That's why the feature vectors are passed to a *quantization module*, which aims to learn discrete speech units. The speech unit is chosen from a collection of codewords, known as a *codebook* (you can think of this as the vocabulary). From the codebook, the vector or speech unit, that best represents the continuous audio input is chosen and forwarded through the model.

3. About half of the feature vectors are randomly masked, and the masked feature vector is fed to a *context network*, which is a Transformer encoder that also adds relative positional embeddings.

4. The pretraining objective of the context network is a *contrastive task*. The model has to predict the true quantized speech representation of the masked prediction from a set of false ones, encouraging the model to find the most similar context vector and quantized speech unit (the target label).

Now that wav2vec2 is pretrained, you can finetune it on your data for audio classification or automatic speech recognition!

### Audio classification

To use the pretrained model for audio classification, add a sequence classification head on top of the base Wav2Vec2 model. The classification head is a linear layer that accepts the encoder's hidden states. The hidden states represent the learned features from each audio frame which can have varying lengths. To create one vector of fixed-length, the hidden states are pooled first and then transformed into logits over the class labels. The cross-entropy loss is calculated between the logits and target to find the most likely class.

Ready to try your hand at audio classification? Check out our complete [audio classification guide](tasks/audio_classification) to learn how to finetune Wav2Vec2 and use it for inference!

### Automatic speech recognition

To use the pretrained model for automatic speech recognition, add a language modeling head on top of the base Wav2Vec2 model for [connectionist temporal classification (CTC)](glossary#connectionist-temporal-classification-ctc). The language modeling head is a linear layer that accepts the encoder's hidden states and transforms them into logits. Each logit represents a token class (the number of tokens comes from the task vocabulary). The CTC loss is calculated between the logits and targets to find the most likely sequence of tokens, which are then decoded into a transcription.

Ready to try your hand at automatic speech recognition? Check out our complete [automatic speech recognition guide](tasks/asr) to learn how to finetune Wav2Vec2 and use it for inference!

## Computer vision

There are two ways to approach computer vision tasks:

1.


key: _node_content
string_value:
{"id_": "2639d582-045c-4396-b6a1-cf2d025f8f35", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8e347b8b-6f0d-43db-90c7-f57b1aec74b3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "c848dc72b44ff248da5ed62fb8f91d954c8ef5e8c0c04aa49649a5709a67a09c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "93c3cc88-68ed-4ca4-97f3-42a1b0e5caea", "node_type": "1", "metadata": {}, "hash": "e04cf1ad489f740330c0cc09f61ecec877ea432baaf3480ebd43de1337292068", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3584, "end_char_idx": 7211, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Now that wav2vec2 is pretrained, you can finetune it on your data for audio classification or automatic speech recognition!

### Audio classification

To use the pretrained model for audio classification, add a sequence classification head on top of the base Wav2Vec2 model. The classification head is a linear layer that accepts the encoder's hidden states. The hidden states represent the learned features from each audio frame which can have varying lengths. To create one vector of fixed-length, the hidden states are pooled first and then transformed into logits over the class labels. The cross-entropy loss is calculated between the logits and target to find the most likely class.

Ready to try your hand at audio classification? Check out our complete [audio classification guide](tasks/audio_classification) to learn how to finetune Wav2Vec2 and use it for inference!

### Automatic speech recognition

To use the pretrained model for automatic speech recognition, add a language modeling head on top of the base Wav2Vec2 model for [connectionist temporal classification (CTC)](glossary#connectionist-temporal-classification-ctc). The language modeling head is a linear layer that accepts the encoder's hidden states and transforms them into logits. Each logit represents a token class (the number of tokens comes from the task vocabulary). The CTC loss is calculated between the logits and targets to find the most likely sequence of tokens, which are then decoded into a transcription.

Ready to try your hand at automatic speech recognition? Check out our complete [automatic speech recognition guide](tasks/asr) to learn how to finetune Wav2Vec2 and use it for inference!

## Computer vision

There are two ways to approach computer vision tasks:

1. Split an image into a sequence of patches and process them in parallel with a Transformer.
2. Use a modern CNN, like [ConvNeXT](model_doc/convnext), which relies on convolutional layers but adopts modern network designs.

<Tip>

A third approach mixes Transformers with convolutions (for example, [Convolutional Vision Transformer](model_doc/cvt) or [LeViT](model_doc/levit)). We won't discuss those because they just combine the two approaches we examine here.

</Tip>

ViT and ConvNeXT are commonly used for image classification, but for other vision tasks like object detection, segmentation, and depth estimation, we'll look at DETR, Mask2Former and GLPN, respectively; these models are better suited for those tasks.

### Image classification

ViT and ConvNeXT can both be used for image classification; the main difference is that ViT uses an attention mechanism while ConvNeXT uses convolutions.

#### Transformer

[ViT](model_doc/vit) replaces convolutions entirely with a pure Transformer architecture. If you're familiar with the original Transformer, then you're already most of the way toward understanding ViT.

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/vit_architecture.jpg"/>
</div>

The main change ViT introduced was in how images are fed to a Transformer:

1. An image is split into square non-overlapping patches, each of which gets turned into a vector or *patch embedding*. The patch embeddings are generated from a convolutional 2D layer which creates the proper input dimensions (which for a base Transformer is 768 values for each patch embedding). If you had a 224x224 pixel image, you could split it into 196 16x16 image patches. Just like how text is tokenized into words, an image is "tokenized" into a sequence of patches.

2.


key: _node_content
string_value:
{"id_": "93c3cc88-68ed-4ca4-97f3-42a1b0e5caea", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2639d582-045c-4396-b6a1-cf2d025f8f35", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "d1540bb187db31a0d5322b18cdef85f54350f6071f2fcdcd334de30b9ee28e0e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "528be704-5da2-442a-87ba-ac65b10959b4", "node_type": "1", "metadata": {}, "hash": "5d694f3409c116fe359e1bc25ae552327265e19083d99e0d03cbe7f469032d10", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5440, "end_char_idx": 8911, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
2. Use a modern CNN, like [ConvNeXT](model_doc/convnext), which relies on convolutional layers but adopts modern network designs.

<Tip>

A third approach mixes Transformers with convolutions (for example, [Convolutional Vision Transformer](model_doc/cvt) or [LeViT](model_doc/levit)). We won't discuss those because they just combine the two approaches we examine here.

</Tip>

ViT and ConvNeXT are commonly used for image classification, but for other vision tasks like object detection, segmentation, and depth estimation, we'll look at DETR, Mask2Former and GLPN, respectively; these models are better suited for those tasks.

### Image classification

ViT and ConvNeXT can both be used for image classification; the main difference is that ViT uses an attention mechanism while ConvNeXT uses convolutions.

#### Transformer

[ViT](model_doc/vit) replaces convolutions entirely with a pure Transformer architecture. If you're familiar with the original Transformer, then you're already most of the way toward understanding ViT.

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/vit_architecture.jpg"/>
</div>

The main change ViT introduced was in how images are fed to a Transformer:

1. An image is split into square non-overlapping patches, each of which gets turned into a vector or *patch embedding*. The patch embeddings are generated from a convolutional 2D layer which creates the proper input dimensions (which for a base Transformer is 768 values for each patch embedding). If you had a 224x224 pixel image, you could split it into 196 16x16 image patches. Just like how text is tokenized into words, an image is "tokenized" into a sequence of patches.

2. A *learnable embedding* - a special `[CLS]` token - is added to the beginning of the patch embeddings just like BERT. The final hidden state of the `[CLS]` token is used as the input to the attached classification head; other outputs are ignored. This token helps the model learn how to encode a representation of the image.

3. The last thing to add to the patch and learnable embeddings are the *position embeddings* because the model doesn't know how the image patches are ordered. The position embeddings are also learnable and have the same size as the patch embeddings. Finally, all of the embeddings are passed to the Transformer encoder.

4. The output, specifically only the output with the `[CLS]` token, is passed to a multilayer perceptron head (MLP). ViT's pretraining objective is simply classification. Like other classification heads, the MLP head converts the output into logits over the class labels and calculates the cross-entropy loss to find the most likely class.

Ready to try your hand at image classification? Check out our complete [image classification guide](tasks/image_classification) to learn how to finetune ViT and use it for inference!

#### CNN

<Tip>

This section briefly explains convolutions, but it'd be helpful to have a prior understanding of how they change an image's shape and size. If you're unfamiliar with convolutions, check out the [Convolution Neural Networks chapter](https://github.com/fastai/fastbook/blob/master/13_convolutions.ipynb) from the fastai book!

</Tip>

[ConvNeXT](model_doc/convnext) is a CNN architecture that adopts new and modern network designs to improve performance. However, convolutions are still at the core of the model.


key: _node_content
string_value:
{"id_": "528be704-5da2-442a-87ba-ac65b10959b4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "93c3cc88-68ed-4ca4-97f3-42a1b0e5caea", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "9b43b07829aba4a6f7bd353683603dfb2213ba3b7857fce6ad3db904d404fb86", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e0a9ba18-c3b0-4dd6-b266-25ae1ed1f497", "node_type": "1", "metadata": {}, "hash": "ac3c9ff2a235f2f925b88f3d21cd67c2868aadb3ab7077fca49d3b5e56532360", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7112, "end_char_idx": 10537, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Just like how text is tokenized into words, an image is "tokenized" into a sequence of patches.

2. A *learnable embedding* - a special `[CLS]` token - is added to the beginning of the patch embeddings just like BERT. The final hidden state of the `[CLS]` token is used as the input to the attached classification head; other outputs are ignored. This token helps the model learn how to encode a representation of the image.

3. The last thing to add to the patch and learnable embeddings are the *position embeddings* because the model doesn't know how the image patches are ordered. The position embeddings are also learnable and have the same size as the patch embeddings. Finally, all of the embeddings are passed to the Transformer encoder.

4. The output, specifically only the output with the `[CLS]` token, is passed to a multilayer perceptron head (MLP). ViT's pretraining objective is simply classification. Like other classification heads, the MLP head converts the output into logits over the class labels and calculates the cross-entropy loss to find the most likely class.

Ready to try your hand at image classification? Check out our complete [image classification guide](tasks/image_classification) to learn how to finetune ViT and use it for inference!

#### CNN

<Tip>

This section briefly explains convolutions, but it'd be helpful to have a prior understanding of how they change an image's shape and size. If you're unfamiliar with convolutions, check out the [Convolution Neural Networks chapter](https://github.com/fastai/fastbook/blob/master/13_convolutions.ipynb) from the fastai book!

</Tip>

[ConvNeXT](model_doc/convnext) is a CNN architecture that adopts new and modern network designs to improve performance. However, convolutions are still at the core of the model. From a high-level perspective, a [convolution](glossary#convolution) is an operation where a smaller matrix (*kernel*) is multiplied by a small window of the image pixels. It computes some features from it, such as a particular texture or curvature of a line. Then it slides over to the next window of pixels; the distance the convolution travels is known as the *stride*. 

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/convolution.gif"/>
</div>

<small>A basic convolution without padding or stride, taken from <a href="https://arxiv.org/abs/1603.07285">A guide to convolution arithmetic for deep learning.</a></small>

You can feed this output to another convolutional layer, and with each successive layer, the network learns more complex and abstract things like hotdogs or rockets. Between convolutional layers, it is common to add a pooling layer to reduce dimensionality and make the model more robust to variations of a feature's position.

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/convnext_architecture.png"/>
</div>

ConvNeXT modernizes a CNN in five ways:

1. Change the number of blocks in each stage and "patchify" an image with a larger stride and corresponding kernel size. The non-overlapping sliding window makes this patchifying strategy similar to how ViT splits an image into patches.

2. A *bottleneck* layer shrinks the number of channels and then restores it because it is faster to do a 1x1 convolution, and you can increase the depth.


key: _node_content
string_value:
{"id_": "e0a9ba18-c3b0-4dd6-b266-25ae1ed1f497", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "528be704-5da2-442a-87ba-ac65b10959b4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "e8ddaf57927da5a407c62ad1d60a8ffce122d4a04aa17740a1fda26747404893", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6219846d-1d47-4563-889e-2799dde426ad", "node_type": "1", "metadata": {}, "hash": "09b978741bcc88f050366efa761ed7ca536a8ccacbb2911a5a90aff364625130", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8854, "end_char_idx": 12436, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, convolutions are still at the core of the model. From a high-level perspective, a [convolution](glossary#convolution) is an operation where a smaller matrix (*kernel*) is multiplied by a small window of the image pixels. It computes some features from it, such as a particular texture or curvature of a line. Then it slides over to the next window of pixels; the distance the convolution travels is known as the *stride*. 

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/convolution.gif"/>
</div>

<small>A basic convolution without padding or stride, taken from <a href="https://arxiv.org/abs/1603.07285">A guide to convolution arithmetic for deep learning.</a></small>

You can feed this output to another convolutional layer, and with each successive layer, the network learns more complex and abstract things like hotdogs or rockets. Between convolutional layers, it is common to add a pooling layer to reduce dimensionality and make the model more robust to variations of a feature's position.

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/convnext_architecture.png"/>
</div>

ConvNeXT modernizes a CNN in five ways:

1. Change the number of blocks in each stage and "patchify" an image with a larger stride and corresponding kernel size. The non-overlapping sliding window makes this patchifying strategy similar to how ViT splits an image into patches.

2. A *bottleneck* layer shrinks the number of channels and then restores it because it is faster to do a 1x1 convolution, and you can increase the depth. An inverted bottleneck does the opposite by expanding the number of channels and shrinking them, which is more memory efficient.

3. Replace the typical 3x3 convolutional layer in the bottleneck layer with *depthwise convolution*, which applies a convolution to each input channel separately and then stacks them back together at the end. This widens the network width for improved performance.

4. ViT has a global receptive field which means it can see more of an image at once thanks to its attention mechanism. ConvNeXT attempts to replicate this effect by increasing the kernel size to 7x7.

5. ConvNeXT also makes several layer design changes that imitate Transformer models. There are fewer activation and normalization layers,  the activation function is switched to GELU instead of ReLU, and it uses LayerNorm instead of BatchNorm.

The output from the convolution blocks is passed to a classification head which converts the outputs into logits and calculates the cross-entropy loss to find the most likely label.

### Object detection

[DETR](model_doc/detr), *DEtection TRansformer*, is an end-to-end object detection model that combines a CNN with a Transformer encoder-decoder.

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/detr_architecture.png"/>
</div>

1. A pretrained CNN *backbone* takes an image, represented by its pixel values, and creates a low-resolution feature map of it. A 1x1 convolution is applied to the feature map to reduce dimensionality and it creates a new feature map with a high-level image representation. Since the Transformer is a sequential model, the feature map is flattened into a sequence of feature vectors that are combined with positional embeddings.

2. The feature vectors are passed to the encoder, which learns the image representations using its attention layers.


key: _node_content
string_value:
{"id_": "6219846d-1d47-4563-889e-2799dde426ad", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e0a9ba18-c3b0-4dd6-b266-25ae1ed1f497", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "2f7c5a17f667cb25aa925465d858d2d0bc3483a1c91d9566e90a8e7b1e3f544c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "699fab80-9ec7-4058-9c29-012e4a60de52", "node_type": "1", "metadata": {}, "hash": "f964375713d8d0ada0ada6bc873dfbf82c720e246c24f89c99b027f763c422e5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10538, "end_char_idx": 14135, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
An inverted bottleneck does the opposite by expanding the number of channels and shrinking them, which is more memory efficient.

3. Replace the typical 3x3 convolutional layer in the bottleneck layer with *depthwise convolution*, which applies a convolution to each input channel separately and then stacks them back together at the end. This widens the network width for improved performance.

4. ViT has a global receptive field which means it can see more of an image at once thanks to its attention mechanism. ConvNeXT attempts to replicate this effect by increasing the kernel size to 7x7.

5. ConvNeXT also makes several layer design changes that imitate Transformer models. There are fewer activation and normalization layers,  the activation function is switched to GELU instead of ReLU, and it uses LayerNorm instead of BatchNorm.

The output from the convolution blocks is passed to a classification head which converts the outputs into logits and calculates the cross-entropy loss to find the most likely label.

### Object detection

[DETR](model_doc/detr), *DEtection TRansformer*, is an end-to-end object detection model that combines a CNN with a Transformer encoder-decoder.

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/detr_architecture.png"/>
</div>

1. A pretrained CNN *backbone* takes an image, represented by its pixel values, and creates a low-resolution feature map of it. A 1x1 convolution is applied to the feature map to reduce dimensionality and it creates a new feature map with a high-level image representation. Since the Transformer is a sequential model, the feature map is flattened into a sequence of feature vectors that are combined with positional embeddings.

2. The feature vectors are passed to the encoder, which learns the image representations using its attention layers. Next, the encoder hidden states are combined with *object queries* in the decoder. Object queries are learned embeddings that focus on the different regions of an image, and they're updated as they progress through each attention layer. The decoder hidden states are passed to a feedforward network that predicts the bounding box coordinates and class label for each object query, or `no object` if there isn't one.

    DETR decodes each object query in parallel to output *N* final predictions, where *N* is the number of queries. Unlike a typical autoregressive model that predicts one element at a time, object detection is a set prediction task (`bounding box`, `class label`) that makes *N* predictions in a single pass.

3. DETR uses a *bipartite matching loss* during training to compare a fixed number of predictions with a fixed set of ground truth labels. If there are fewer ground truth labels in the set of *N* labels, then they're padded with a `no object` class. This loss function encourages DETR to find a one-to-one assignment between the predictions and ground truth labels. If either the bounding boxes or class labels aren't correct, a loss is incurred. Likewise, if DETR predicts an object that doesn't exist, it is penalized. This encourages DETR to find other objects in an image instead of focusing on one really prominent object.

An object detection head is added on top of DETR to find the class label and the coordinates of the bounding box. There are two components to the object detection head: a linear layer to transform the decoder hidden states into logits over the class labels, and a MLP to predict the bounding box.

Ready to try your hand at object detection?


key: _node_content
string_value:
{"id_": "699fab80-9ec7-4058-9c29-012e4a60de52", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6219846d-1d47-4563-889e-2799dde426ad", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "989780a72eeb326a875ef3191477d3b6f7a119ff6a7c0d2a8182b36401ac66a0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9ab37e06-c015-4887-a55e-0cae2c8d1fe6", "node_type": "1", "metadata": {}, "hash": "e6de48e286d8fb3f0bf3c86c37259979d33690b27ceea1a417475bd0a40ac31d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12320, "end_char_idx": 15927, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
2. The feature vectors are passed to the encoder, which learns the image representations using its attention layers. Next, the encoder hidden states are combined with *object queries* in the decoder. Object queries are learned embeddings that focus on the different regions of an image, and they're updated as they progress through each attention layer. The decoder hidden states are passed to a feedforward network that predicts the bounding box coordinates and class label for each object query, or `no object` if there isn't one.

    DETR decodes each object query in parallel to output *N* final predictions, where *N* is the number of queries. Unlike a typical autoregressive model that predicts one element at a time, object detection is a set prediction task (`bounding box`, `class label`) that makes *N* predictions in a single pass.

3. DETR uses a *bipartite matching loss* during training to compare a fixed number of predictions with a fixed set of ground truth labels. If there are fewer ground truth labels in the set of *N* labels, then they're padded with a `no object` class. This loss function encourages DETR to find a one-to-one assignment between the predictions and ground truth labels. If either the bounding boxes or class labels aren't correct, a loss is incurred. Likewise, if DETR predicts an object that doesn't exist, it is penalized. This encourages DETR to find other objects in an image instead of focusing on one really prominent object.

An object detection head is added on top of DETR to find the class label and the coordinates of the bounding box. There are two components to the object detection head: a linear layer to transform the decoder hidden states into logits over the class labels, and a MLP to predict the bounding box.

Ready to try your hand at object detection? Check out our complete [object detection guide](tasks/object_detection) to learn how to finetune DETR and use it for inference!

### Image segmentation

[Mask2Former](model_doc/mask2former) is a universal architecture for solving all types of image segmentation tasks. Traditional segmentation models are typically tailored towards a particular subtask of image segmentation, like instance, semantic or panoptic segmentation. Mask2Former frames each of those tasks as a *mask classification* problem. Mask classification groups pixels into *N* segments, and predicts *N* masks and their corresponding class label for a given image. We'll explain how Mask2Former works in this section, and then you can try finetuning SegFormer at the end.

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/mask2former_architecture.png"/>
</div>

There are three main components to Mask2Former:

1. A [Swin](model_doc/swin) backbone accepts an image and creates a low-resolution image feature map from 3 consecutive 3x3 convolutions.

2. The feature map is passed to a *pixel decoder* which gradually upsamples the low-resolution features into high-resolution per-pixel embeddings. The pixel decoder actually generates multi-scale features (contains both low- and high-resolution features) with resolutions 1/32, 1/16, and 1/8th of the original image.

3. Each of these feature maps of differing scales is fed successively to one Transformer decoder layer at a time in order to capture small objects from the high-resolution features. The key to Mask2Former is the *masked attention* mechanism in the decoder. Unlike cross-attention which can attend to the entire image, masked attention only focuses on a certain area of the image.


key: _node_content
string_value:
{"id_": "9ab37e06-c015-4887-a55e-0cae2c8d1fe6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "699fab80-9ec7-4058-9c29-012e4a60de52", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "49ad9b58a6f55045329df1d65cd005a60abfbfa075095179b655dc724ca3123d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0e4cefce-5d7e-46a4-bc55-f515a098a595", "node_type": "1", "metadata": {}, "hash": "62ebdb7a5ae9ea2bc6fe057fa014c66129ea80b79ea5e15b7d40c57262d3ba1d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14092, "end_char_idx": 17750, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Ready to try your hand at object detection? Check out our complete [object detection guide](tasks/object_detection) to learn how to finetune DETR and use it for inference!

### Image segmentation

[Mask2Former](model_doc/mask2former) is a universal architecture for solving all types of image segmentation tasks. Traditional segmentation models are typically tailored towards a particular subtask of image segmentation, like instance, semantic or panoptic segmentation. Mask2Former frames each of those tasks as a *mask classification* problem. Mask classification groups pixels into *N* segments, and predicts *N* masks and their corresponding class label for a given image. We'll explain how Mask2Former works in this section, and then you can try finetuning SegFormer at the end.

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/mask2former_architecture.png"/>
</div>

There are three main components to Mask2Former:

1. A [Swin](model_doc/swin) backbone accepts an image and creates a low-resolution image feature map from 3 consecutive 3x3 convolutions.

2. The feature map is passed to a *pixel decoder* which gradually upsamples the low-resolution features into high-resolution per-pixel embeddings. The pixel decoder actually generates multi-scale features (contains both low- and high-resolution features) with resolutions 1/32, 1/16, and 1/8th of the original image.

3. Each of these feature maps of differing scales is fed successively to one Transformer decoder layer at a time in order to capture small objects from the high-resolution features. The key to Mask2Former is the *masked attention* mechanism in the decoder. Unlike cross-attention which can attend to the entire image, masked attention only focuses on a certain area of the image. This is faster and leads to better performance because the local features of an image are enough for the model to learn from.

4. Like [DETR](tasks_explained#object-detection), Mask2Former also uses learned object queries and combines them with the image features from the pixel decoder to make a set prediction (`class label`, `mask prediction`). The decoder hidden states are passed into a linear layer and transformed into logits over the class labels. The cross-entropy loss is calculated between the logits and class label to find the most likely one.

    The mask predictions are generated by combining the pixel-embeddings with the final decoder hidden states. The sigmoid cross-entropy and dice loss is calculated between the logits and the ground truth mask to find the most likely mask.

Ready to try your hand at object detection? Check out our complete [image segmentation guide](tasks/semantic_segmentation) to learn how to finetune SegFormer and use it for inference!

### Depth estimation

[GLPN](model_doc/glpn), *Global-Local Path Network*, is a Transformer for depth estimation that combines a [SegFormer](model_doc/segformer) encoder with a lightweight decoder.

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/glpn_architecture.jpg"/>
</div>

1. Like ViT, an image is split into a sequence of patches, except these image patches are smaller. This is better for dense prediction tasks like segmentation or depth estimation. The image patches are transformed into patch embeddings (see the [image classification](#image-classification) section for more details about how patch embeddings are created), which are fed to the encoder.

2. The encoder accepts the patch embeddings, and passes them through several encoder blocks.


key: _node_content
string_value:
{"id_": "0e4cefce-5d7e-46a4-bc55-f515a098a595", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9ab37e06-c015-4887-a55e-0cae2c8d1fe6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "bd9ba8a30e2d3ae93cf5ff30b4c5c22b6efed2f873f3bf45ca314388f0a1f2f6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3a59a2e6-6594-4256-9181-b4628a37bf85", "node_type": "1", "metadata": {}, "hash": "7e09583d737b9f04549311f7e9ae3edd95aa712e5c6047600e77b0788cc987a9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15928, "end_char_idx": 19576, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This is faster and leads to better performance because the local features of an image are enough for the model to learn from.

4. Like [DETR](tasks_explained#object-detection), Mask2Former also uses learned object queries and combines them with the image features from the pixel decoder to make a set prediction (`class label`, `mask prediction`). The decoder hidden states are passed into a linear layer and transformed into logits over the class labels. The cross-entropy loss is calculated between the logits and class label to find the most likely one.

    The mask predictions are generated by combining the pixel-embeddings with the final decoder hidden states. The sigmoid cross-entropy and dice loss is calculated between the logits and the ground truth mask to find the most likely mask.

Ready to try your hand at object detection? Check out our complete [image segmentation guide](tasks/semantic_segmentation) to learn how to finetune SegFormer and use it for inference!

### Depth estimation

[GLPN](model_doc/glpn), *Global-Local Path Network*, is a Transformer for depth estimation that combines a [SegFormer](model_doc/segformer) encoder with a lightweight decoder.

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/glpn_architecture.jpg"/>
</div>

1. Like ViT, an image is split into a sequence of patches, except these image patches are smaller. This is better for dense prediction tasks like segmentation or depth estimation. The image patches are transformed into patch embeddings (see the [image classification](#image-classification) section for more details about how patch embeddings are created), which are fed to the encoder.

2. The encoder accepts the patch embeddings, and passes them through several encoder blocks. Each block consists of attention and Mix-FFN layers. The purpose of the latter is to provide positional information. At the end of each encoder block is a *patch merging* layer for creating hierarchical representations. The features of each group of neighboring patches are concatenated, and a linear layer is applied to the concatenated features to reduce the number of patches to a resolution of 1/4. This becomes the input to the next encoder block, where this whole process is repeated until you have image features with resolutions of 1/8, 1/16, and 1/32.

3. A lightweight decoder takes the last feature map (1/32 scale) from the encoder and upsamples it to 1/16 scale. From here, the feature is passed into a *Selective Feature Fusion (SFF)* module, which selects and combines local and global features from an attention map for each feature and then upsamples it to 1/8th. This process is repeated until the decoded features are the same size as the original image. The output is passed through two convolution layers and then a sigmoid activation is applied to predict the depth of each pixel.

## Natural language processing

The Transformer was initially designed for machine translation, and since then, it has practically become the default architecture for solving all NLP tasks. Some tasks lend themselves to the Transformer's encoder structure, while others are better suited for the decoder. Still, other tasks make use of both the Transformer's encoder-decoder structure.

### Text classification

[BERT](model_doc/bert) is an encoder-only model and is the first model to effectively implement deep bidirectionality to learn richer representations of the text by attending to words on both sides.

1. BERT uses [WordPiece](tokenizer_summary#wordpiece) tokenization to generate a token embedding of the text.


key: _node_content
string_value:
{"id_": "3a59a2e6-6594-4256-9181-b4628a37bf85", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0e4cefce-5d7e-46a4-bc55-f515a098a595", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "229fde9a57da3e36f0f52863085cd4ced95aa385a49e3f1244f3bef97536f237", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a78eeb09-0079-41c2-8f6d-5fd9485dc461", "node_type": "1", "metadata": {}, "hash": "8818a630621453c7299575ea83566a7afc233719930e51c5b9e64e560b396b84", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17661, "end_char_idx": 21437, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The encoder accepts the patch embeddings, and passes them through several encoder blocks. Each block consists of attention and Mix-FFN layers. The purpose of the latter is to provide positional information. At the end of each encoder block is a *patch merging* layer for creating hierarchical representations. The features of each group of neighboring patches are concatenated, and a linear layer is applied to the concatenated features to reduce the number of patches to a resolution of 1/4. This becomes the input to the next encoder block, where this whole process is repeated until you have image features with resolutions of 1/8, 1/16, and 1/32.

3. A lightweight decoder takes the last feature map (1/32 scale) from the encoder and upsamples it to 1/16 scale. From here, the feature is passed into a *Selective Feature Fusion (SFF)* module, which selects and combines local and global features from an attention map for each feature and then upsamples it to 1/8th. This process is repeated until the decoded features are the same size as the original image. The output is passed through two convolution layers and then a sigmoid activation is applied to predict the depth of each pixel.

## Natural language processing

The Transformer was initially designed for machine translation, and since then, it has practically become the default architecture for solving all NLP tasks. Some tasks lend themselves to the Transformer's encoder structure, while others are better suited for the decoder. Still, other tasks make use of both the Transformer's encoder-decoder structure.

### Text classification

[BERT](model_doc/bert) is an encoder-only model and is the first model to effectively implement deep bidirectionality to learn richer representations of the text by attending to words on both sides.

1. BERT uses [WordPiece](tokenizer_summary#wordpiece) tokenization to generate a token embedding of the text. To tell the difference between a single sentence and a pair of sentences, a special `[SEP]` token is added to differentiate them. A special `[CLS]` token is added to the beginning of every sequence of text. The final output with the `[CLS]` token is used as the input to the classification head for classification tasks. BERT also adds a segment embedding to denote whether a token belongs to the first or second sentence in a pair of sentences.

2. BERT is pretrained with two objectives: masked language modeling and next-sentence prediction. In masked language modeling, some percentage of the input tokens are randomly masked, and the model needs to predict these. This solves the issue of bidirectionality, where the model could cheat and see all the words and "predict" the next word. The final hidden states of the predicted mask tokens are passed to a feedforward network with a softmax over the vocabulary to predict the masked word.

    The second pretraining object is next-sentence prediction. The model must predict whether sentence B follows sentence A. Half of the time sentence B is the next sentence, and the other half of the time, sentence B is a random sentence. The prediction, whether it is the next sentence or not, is passed to a feedforward network with a softmax over the two classes (`IsNext` and `NotNext`).

3. The input embeddings are passed through multiple encoder layers to output some final hidden states.

To use the pretrained model for text classification, add a sequence classification head on top of the base BERT model. The sequence classification head is a linear layer that accepts the final hidden states and performs a linear transformation to convert them into logits. The cross-entropy loss is calculated between the logits and target to find the most likely label.

Ready to try your hand at text classification?


key: _node_content
string_value:
{"id_": "a78eeb09-0079-41c2-8f6d-5fd9485dc461", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3a59a2e6-6594-4256-9181-b4628a37bf85", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "ea63e9c36267cd8c4c991e46f04419836ab1d88a972c84c2a43855ca523956c2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "527ce3d7-96aa-4eb4-a68e-612daf7411af", "node_type": "1", "metadata": {}, "hash": "b563f4f1b38962d6977eae07b01a09daea6b465ee64083821ea7c489aa0d41fd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19577, "end_char_idx": 23300, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
To tell the difference between a single sentence and a pair of sentences, a special `[SEP]` token is added to differentiate them. A special `[CLS]` token is added to the beginning of every sequence of text. The final output with the `[CLS]` token is used as the input to the classification head for classification tasks. BERT also adds a segment embedding to denote whether a token belongs to the first or second sentence in a pair of sentences.

2. BERT is pretrained with two objectives: masked language modeling and next-sentence prediction. In masked language modeling, some percentage of the input tokens are randomly masked, and the model needs to predict these. This solves the issue of bidirectionality, where the model could cheat and see all the words and "predict" the next word. The final hidden states of the predicted mask tokens are passed to a feedforward network with a softmax over the vocabulary to predict the masked word.

    The second pretraining object is next-sentence prediction. The model must predict whether sentence B follows sentence A. Half of the time sentence B is the next sentence, and the other half of the time, sentence B is a random sentence. The prediction, whether it is the next sentence or not, is passed to a feedforward network with a softmax over the two classes (`IsNext` and `NotNext`).

3. The input embeddings are passed through multiple encoder layers to output some final hidden states.

To use the pretrained model for text classification, add a sequence classification head on top of the base BERT model. The sequence classification head is a linear layer that accepts the final hidden states and performs a linear transformation to convert them into logits. The cross-entropy loss is calculated between the logits and target to find the most likely label.

Ready to try your hand at text classification? Check out our complete [text classification guide](tasks/sequence_classification) to learn how to finetune DistilBERT and use it for inference!

### Token classification

To use BERT for token classification tasks like named entity recognition (NER), add a token classification head on top of the base BERT model. The token classification head is a linear layer that accepts the final hidden states and performs a linear transformation to convert them into logits. The cross-entropy loss is calculated between the logits and each token to find the most likely label.

Ready to try your hand at token classification? Check out our complete [token classification guide](tasks/token_classification) to learn how to finetune DistilBERT and use it for inference!

### Question answering

To use BERT for question answering, add a span classification head on top of the base BERT model. This linear layer accepts the final hidden states and performs a linear transformation to compute the `span` start and end logits corresponding to the answer. The cross-entropy loss is calculated between the logits and the label position to find the most likely span of text corresponding to the answer.

Ready to try your hand at question answering? Check out our complete [question answering guide](tasks/question_answering) to learn how to finetune DistilBERT and use it for inference!

<Tip>

💡 Notice how easy it is to use BERT for different tasks once it's been pretrained. You only need to add a specific head to the pretrained model to manipulate the hidden states into your desired output!

</Tip>

### Text generation

[GPT-2](model_doc/gpt2) is a decoder-only model pretrained on a large amount of text. It can generate convincing (though not always true!) text given a prompt and complete other NLP tasks like question answering despite not being explicitly trained to.


key: _node_content
string_value:
{"id_": "527ce3d7-96aa-4eb4-a68e-612daf7411af", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a78eeb09-0079-41c2-8f6d-5fd9485dc461", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "824cf80951093e20981ecfa43a53d4afc98eab376f7801fc5c7fb686ec2cc53a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cb8c4b93-e6b1-45c7-96c1-1b25c4886427", "node_type": "1", "metadata": {}, "hash": "57def7db72d629f5f29568f0bcc759ed1005eae7987fedc4f1a07a6180c817ea", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21391, "end_char_idx": 24938, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Ready to try your hand at text classification? Check out our complete [text classification guide](tasks/sequence_classification) to learn how to finetune DistilBERT and use it for inference!

### Token classification

To use BERT for token classification tasks like named entity recognition (NER), add a token classification head on top of the base BERT model. The token classification head is a linear layer that accepts the final hidden states and performs a linear transformation to convert them into logits. The cross-entropy loss is calculated between the logits and each token to find the most likely label.

Ready to try your hand at token classification? Check out our complete [token classification guide](tasks/token_classification) to learn how to finetune DistilBERT and use it for inference!

### Question answering

To use BERT for question answering, add a span classification head on top of the base BERT model. This linear layer accepts the final hidden states and performs a linear transformation to compute the `span` start and end logits corresponding to the answer. The cross-entropy loss is calculated between the logits and the label position to find the most likely span of text corresponding to the answer.

Ready to try your hand at question answering? Check out our complete [question answering guide](tasks/question_answering) to learn how to finetune DistilBERT and use it for inference!

<Tip>

💡 Notice how easy it is to use BERT for different tasks once it's been pretrained. You only need to add a specific head to the pretrained model to manipulate the hidden states into your desired output!

</Tip>

### Text generation

[GPT-2](model_doc/gpt2) is a decoder-only model pretrained on a large amount of text. It can generate convincing (though not always true!) text given a prompt and complete other NLP tasks like question answering despite not being explicitly trained to.

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/gpt2_architecture.png"/>
</div>

1. GPT-2 uses [byte pair encoding (BPE)](tokenizer_summary#bytepair-encoding-bpe) to tokenize words and generate a token embedding. Positional encodings are added to the token embeddings to indicate the position of each token in the sequence. The input embeddings are passed through multiple decoder blocks to output some final hidden state. Within each decoder block, GPT-2 uses a *masked self-attention* layer which means GPT-2 can't attend to future tokens. It is only allowed to attend to tokens on the left. This is different from BERT's [`mask`] token because, in masked self-attention, an attention mask is used to set the score to `0` for future tokens.

2. The output from the decoder is passed to a language modeling head, which performs a linear transformation to convert the hidden states into logits. The label is the next token in the sequence, which are created by shifting the logits to the right by one. The cross-entropy loss is calculated between the shifted logits and the labels to output the next most likely token.

GPT-2's pretraining objective is based entirely on [causal language modeling](glossary#causal-language-modeling), predicting the next word in a sequence. This makes GPT-2 especially good at tasks that involve generating text.

Ready to try your hand at text generation? Check out our complete [causal language modeling guide](tasks/language_modeling#causal-language-modeling) to learn how to finetune DistilGPT-2 and use it for inference!


key: _node_content
string_value:
{"id_": "cb8c4b93-e6b1-45c7-96c1-1b25c4886427", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "527ce3d7-96aa-4eb4-a68e-612daf7411af", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "a228099ef30996def132b050906fbe8d82a2f9c83e32ce9d1944e1dedf59ac15", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bba0e3a9-49a4-447a-be5b-8ab4876640d1", "node_type": "1", "metadata": {}, "hash": "634f5ba3d985fd494aafa22dc7a95b2d710204be9797674090342cec1a6b2ddc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23187, "end_char_idx": 26645, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
text given a prompt and complete other NLP tasks like question answering despite not being explicitly trained to.

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/gpt2_architecture.png"/>
</div>

1. GPT-2 uses [byte pair encoding (BPE)](tokenizer_summary#bytepair-encoding-bpe) to tokenize words and generate a token embedding. Positional encodings are added to the token embeddings to indicate the position of each token in the sequence. The input embeddings are passed through multiple decoder blocks to output some final hidden state. Within each decoder block, GPT-2 uses a *masked self-attention* layer which means GPT-2 can't attend to future tokens. It is only allowed to attend to tokens on the left. This is different from BERT's [`mask`] token because, in masked self-attention, an attention mask is used to set the score to `0` for future tokens.

2. The output from the decoder is passed to a language modeling head, which performs a linear transformation to convert the hidden states into logits. The label is the next token in the sequence, which are created by shifting the logits to the right by one. The cross-entropy loss is calculated between the shifted logits and the labels to output the next most likely token.

GPT-2's pretraining objective is based entirely on [causal language modeling](glossary#causal-language-modeling), predicting the next word in a sequence. This makes GPT-2 especially good at tasks that involve generating text.

Ready to try your hand at text generation? Check out our complete [causal language modeling guide](tasks/language_modeling#causal-language-modeling) to learn how to finetune DistilGPT-2 and use it for inference!

<Tip>

For more information about text generation, check out the [text generation strategies](generation_strategies) guide!

</Tip>

### Summarization

Encoder-decoder models like [BART](model_doc/bart) and [T5](model_doc/t5) are designed for the sequence-to-sequence pattern of a summarization task. We'll explain how BART works in this section, and then you can try finetuning T5 at the end.

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bart_architecture.png"/>
</div>

1. BART's encoder architecture is very similar to BERT and accepts a token and positional embedding of the text. BART is pretrained by corrupting the input and then reconstructing it with the decoder. Unlike other encoders with specific corruption strategies, BART can apply any type of corruption. The *text infilling* corruption strategy works the best though. In text infilling, a number of text spans are replaced with a **single** [`mask`] token. This is important because the model has to predict the masked tokens, and it teaches the model to predict the number of missing tokens. The input embeddings and masked spans are passed through the encoder to output some final hidden states, but unlike BERT, BART doesn't add a final feedforward network at the end to predict a word.

2. The encoder's output is passed to the decoder, which must predict the masked tokens and any uncorrupted tokens from the encoder's output. This gives additional context to help the decoder restore the original text. The output from the decoder is passed to a language modeling head, which performs a linear transformation to convert the hidden states into logits.


key: _node_content
string_value:
{"id_": "bba0e3a9-49a4-447a-be5b-8ab4876640d1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "52862883-0b18-418f-bcb8-d27773b66f6d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cb8c4b93-e6b1-45c7-96c1-1b25c4886427", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "de7a8c81377954d4432c0523316855f996425cf72883748dfc81c50d6eddfdac", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24940, "end_char_idx": 28313, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<Tip>

For more information about text generation, check out the [text generation strategies](generation_strategies) guide!

</Tip>

### Summarization

Encoder-decoder models like [BART](model_doc/bart) and [T5](model_doc/t5) are designed for the sequence-to-sequence pattern of a summarization task. We'll explain how BART works in this section, and then you can try finetuning T5 at the end.

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bart_architecture.png"/>
</div>

1. BART's encoder architecture is very similar to BERT and accepts a token and positional embedding of the text. BART is pretrained by corrupting the input and then reconstructing it with the decoder. Unlike other encoders with specific corruption strategies, BART can apply any type of corruption. The *text infilling* corruption strategy works the best though. In text infilling, a number of text spans are replaced with a **single** [`mask`] token. This is important because the model has to predict the masked tokens, and it teaches the model to predict the number of missing tokens. The input embeddings and masked spans are passed through the encoder to output some final hidden states, but unlike BERT, BART doesn't add a final feedforward network at the end to predict a word.

2. The encoder's output is passed to the decoder, which must predict the masked tokens and any uncorrupted tokens from the encoder's output. This gives additional context to help the decoder restore the original text. The output from the decoder is passed to a language modeling head, which performs a linear transformation to convert the hidden states into logits. The cross-entropy loss is calculated between the logits and the label, which is just the token shifted to the right.

Ready to try your hand at summarization? Check out our complete [summarization guide](tasks/summarization) to learn how to finetune T5 and use it for inference!

<Tip>

For more information about text generation, check out the [text generation strategies](generation_strategies) guide!

</Tip>

### Translation

Translation is another example of a sequence-to-sequence task, which means you can use an encoder-decoder model like [BART](model_doc/bart) or [T5](model_doc/t5) to do it. We'll explain how BART works in this section, and then you can try finetuning T5 at the end.

BART adapts to translation by adding a separate randomly initialized encoder to map a source language to an input that can be decoded into the target language. This new encoder's embeddings are passed to the pretrained encoder instead of the original word embeddings. The source encoder is trained by updating the source encoder, positional embeddings, and input embeddings with the cross-entropy loss from the model output. The model parameters are frozen in this first step, and all the model parameters are trained together in the second step.

BART has since been followed up by a multilingual version, mBART, intended for translation and pretrained on many different languages.

Ready to try your hand at translation? Check out our complete [translation guide](tasks/translation) to learn how to finetune T5 and use it for inference!

<Tip>

For more information about text generation, check out the [text generation strategies](generation_strategies) guide!

</Tip>


key: _node_content
string_value:
{"id_": "c8baebf0-f9b2-419c-b56f-370cf5c3e950", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "ff90393796b58d9cbff193200432df57f36db1694146fa624ccfa7fdd0c6151d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2a000511-9185-4ac6-96d3-763aa9daaf80", "node_type": "1", "metadata": {}, "hash": "7dfe2d62d1bf1f5cbaa0062a21d0dbca219a264cb25832c2578691dc7414f013", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3099, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Benchmarks

<Tip warning={true}>

Hugging Face's Benchmarking tools are deprecated and it is advised to use external Benchmarking libraries to measure the speed 
and memory complexity of Transformer models.

</Tip>

[[open-in-colab]]

Let's take a look at how 🤗 Transformers models can be benchmarked, best practices, and already available benchmarks.

A notebook explaining in more detail how to benchmark 🤗 Transformers models can be found [here](https://github.com/huggingface/notebooks/tree/main/examples/benchmark.ipynb).

## How to benchmark 🤗 Transformers models

The classes [`PyTorchBenchmark`] and [`TensorFlowBenchmark`] allow to flexibly benchmark 🤗 Transformers models. The benchmark classes allow us to measure the _peak memory usage_ and _required time_ for both _inference_ and _training_.

<Tip>

Hereby, _inference_ is defined by a single forward pass, and _training_ is defined by a single forward pass and
backward pass.

</Tip>

The benchmark classes [`PyTorchBenchmark`] and [`TensorFlowBenchmark`] expect an object of type [`PyTorchBenchmarkArguments`] and
[`TensorFlowBenchmarkArguments`], respectively, for instantiation. [`PyTorchBenchmarkArguments`] and [`TensorFlowBenchmarkArguments`] are data classes and contain all relevant configurations for their corresponding benchmark class. In the following example, it is shown how a BERT model of type _bert-base-cased_ can be benchmarked.

<frameworkcontent>
<pt>
```py
>>> from transformers import PyTorchBenchmark, PyTorchBenchmarkArguments

>>> args = PyTorchBenchmarkArguments(models=["google-bert/bert-base-uncased"], batch_sizes=[8], sequence_lengths=[8, 32, 128, 512])
>>> benchmark = PyTorchBenchmark(args)
```
</pt>
<tf>
```py
>>> from transformers import TensorFlowBenchmark, TensorFlowBenchmarkArguments

>>> args = TensorFlowBenchmarkArguments(
...     models=["google-bert/bert-base-uncased"], batch_sizes=[8], sequence_lengths=[8, 32, 128, 512]
... )
>>> benchmark = TensorFlowBenchmark(args)
```
</tf>
</frameworkcontent>

Here, three arguments are given to the benchmark argument data classes, namely `models`, `batch_sizes`, and
`sequence_lengths`. The argument `models` is required and expects a `list` of model identifiers from the
[model hub](https://huggingface.co/models) The `list` arguments `batch_sizes` and `sequence_lengths` define
the size of the `input_ids` on which the model is benchmarked. There are many more parameters that can be configured
via the benchmark argument data classes. For more detail on these one can either directly consult the files
`src/transformers/benchmark/benchmark_args_utils.py`, `src/transformers/benchmark/benchmark_args.py` (for PyTorch)
and `src/transformers/benchmark/benchmark_args_tf.py` (for Tensorflow). Alternatively, running the following shell
commands from root will print out a descriptive list of all configurable parameters for PyTorch and Tensorflow
respectively.

<frameworkcontent>
<pt>
```bash
python examples/pytorch/benchmarking/run_benchmark.py --help
```

An instantiated benchmark object can then simply be run by calling `benchmark.run()`.


key: _node_content
string_value:
{"id_": "2a000511-9185-4ac6-96d3-763aa9daaf80", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "ff90393796b58d9cbff193200432df57f36db1694146fa624ccfa7fdd0c6151d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c8baebf0-f9b2-419c-b56f-370cf5c3e950", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "fcee66d309064f10554a641b9efa4ab97e12118aa1ae933d7ac51c9b8371fbc9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dae0366f-b0af-48db-a5c1-3bbf85c4904c", "node_type": "1", "metadata": {}, "hash": "a7f84b58ecd46ab5f5f8e5adf15727fc992d2ea05bab20ff341c668393fdddd1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1416, "end_char_idx": 5282, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<frameworkcontent>
<pt>
```py
>>> from transformers import PyTorchBenchmark, PyTorchBenchmarkArguments

>>> args = PyTorchBenchmarkArguments(models=["google-bert/bert-base-uncased"], batch_sizes=[8], sequence_lengths=[8, 32, 128, 512])
>>> benchmark = PyTorchBenchmark(args)
```
</pt>
<tf>
```py
>>> from transformers import TensorFlowBenchmark, TensorFlowBenchmarkArguments

>>> args = TensorFlowBenchmarkArguments(
...     models=["google-bert/bert-base-uncased"], batch_sizes=[8], sequence_lengths=[8, 32, 128, 512]
... )
>>> benchmark = TensorFlowBenchmark(args)
```
</tf>
</frameworkcontent>

Here, three arguments are given to the benchmark argument data classes, namely `models`, `batch_sizes`, and
`sequence_lengths`. The argument `models` is required and expects a `list` of model identifiers from the
[model hub](https://huggingface.co/models) The `list` arguments `batch_sizes` and `sequence_lengths` define
the size of the `input_ids` on which the model is benchmarked. There are many more parameters that can be configured
via the benchmark argument data classes. For more detail on these one can either directly consult the files
`src/transformers/benchmark/benchmark_args_utils.py`, `src/transformers/benchmark/benchmark_args.py` (for PyTorch)
and `src/transformers/benchmark/benchmark_args_tf.py` (for Tensorflow). Alternatively, running the following shell
commands from root will print out a descriptive list of all configurable parameters for PyTorch and Tensorflow
respectively.

<frameworkcontent>
<pt>
```bash
python examples/pytorch/benchmarking/run_benchmark.py --help
```

An instantiated benchmark object can then simply be run by calling `benchmark.run()`.

```py
>>> results = benchmark.run()
>>> print(results)
====================       INFERENCE - SPEED - RESULT       ====================
--------------------------------------------------------------------------------
Model Name             Batch Size     Seq Length     Time in s                  
--------------------------------------------------------------------------------
google-bert/bert-base-uncased          8               8             0.006     
google-bert/bert-base-uncased          8               32            0.006     
google-bert/bert-base-uncased          8              128            0.018     
google-bert/bert-base-uncased          8              512            0.088     
--------------------------------------------------------------------------------

====================      INFERENCE - MEMORY - RESULT       ====================
--------------------------------------------------------------------------------
Model Name             Batch Size     Seq Length    Memory in MB 
--------------------------------------------------------------------------------
google-bert/bert-base-uncased          8               8             1227
google-bert/bert-base-uncased          8               32            1281
google-bert/bert-base-uncased          8              128            1307
google-bert/bert-base-uncased          8              512            1539
--------------------------------------------------------------------------------

====================        ENVIRONMENT INFORMATION         ====================

- transformers_version: 2.11.0
- framework: PyTorch
- use_torchscript: False
- framework_version: 1.4.0
- python_version: 3.6.10
- system: Linux
- cpu: x86_64
- architecture: 64bit
- date: 2020-06-29
- time: 08:58:43.371351
- fp16: False
- use_multiprocessing: True
- only_pretrain_model: False
- cpu_ram_mb: 32088
- use_gpu: True
- num_gpus: 1
- gpu: TITAN RTX
- gpu_ram_mb: 24217
- gpu_power_watts: 280.0
- gpu_performance_state: 2
- use_tpu: False
```
</pt>
<tf>
```bash
python examples/tensorflow/benchmarking/run_benchmark_tf.py --help
```

An instantiated benchmark object can then simply be run by calling `benchmark.run()`.


key: _node_content
string_value:
{"id_": "dae0366f-b0af-48db-a5c1-3bbf85c4904c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "ff90393796b58d9cbff193200432df57f36db1694146fa624ccfa7fdd0c6151d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2a000511-9185-4ac6-96d3-763aa9daaf80", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "a5b8e9a0fb30e1eff8ba585336db5883d79a4e33fa2bae87fb22c0ac203af26f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f44f2e0b-401e-4389-93ce-f5ea9f67847b", "node_type": "1", "metadata": {}, "hash": "50e7bf1014c9b28665782f44b0ba1b2f82e118a08186a211630688127531ed4b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> results = benchmark.run()
>>> print(results)
>>> results = benchmark.run()
>>> print(results)
====================       INFERENCE - SPEED - RESULT       ====================
--------------------------------------------------------------------------------
Model Name             Batch Size     Seq Length     Time in s                  
--------------------------------------------------------------------------------
google-bert/bert-base-uncased          8               8             0.005
google-bert/bert-base-uncased          8               32            0.008
google-bert/bert-base-uncased          8              128            0.022
google-bert/bert-base-uncased          8              512            0.105
--------------------------------------------------------------------------------

====================      INFERENCE - MEMORY - RESULT       ====================
--------------------------------------------------------------------------------
Model Name             Batch Size     Seq Length    Memory in MB 
--------------------------------------------------------------------------------
google-bert/bert-base-uncased          8               8             1330
google-bert/bert-base-uncased          8               32            1330
google-bert/bert-base-uncased          8              128            1330
google-bert/bert-base-uncased          8              512            1770
--------------------------------------------------------------------------------

====================        ENVIRONMENT INFORMATION         ====================

- transformers_version: 2.11.0
- framework: Tensorflow
- use_xla: False
- framework_version: 2.2.0
- python_version: 3.6.10
- system: Linux
- cpu: x86_64
- architecture: 64bit
- date: 2020-06-29
- time: 09:26:35.617317
- fp16: False
- use_multiprocessing: True
- only_pretrain_model: False
- cpu_ram_mb: 32088
- use_gpu: True
- num_gpus: 1
- gpu: TITAN RTX
- gpu_ram_mb: 24217
- gpu_power_watts: 280.0
- gpu_performance_state: 2
- use_tpu: False
```
</tf>
</frameworkcontent>

By default, the _time_ and the _required memory_ for _inference_ are benchmarked. In the example output above the first
two sections show the result corresponding to _inference time_ and _inference memory_. In addition, all relevant
information about the computing environment, _e.g._ the GPU type, the system, the library versions, etc... are printed
out in the third section under _ENVIRONMENT INFORMATION_. This information can optionally be saved in a _.csv_ file
when adding the argument `save_to_csv=True` to [`PyTorchBenchmarkArguments`] and
[`TensorFlowBenchmarkArguments`] respectively. In this case, every section is saved in a separate
_.csv_ file. The path to each _.csv_ file can optionally be defined via the argument data classes.

Instead of benchmarking pre-trained models via their model identifier, _e.g._ `google-bert/bert-base-uncased`, the user can
alternatively benchmark an arbitrary configuration of any available model class. In this case, a `list` of
configurations must be inserted with the benchmark args as follows.

<frameworkcontent>
<pt>
```py
>>> from transformers import PyTorchBenchmark, PyTorchBenchmarkArguments, BertConfig

>>> args = PyTorchBenchmarkArguments(
.     models=["bert-base", "bert-384-hid", "bert-6-lay"], batch_sizes=[8], sequence_lengths=[8, 32, 128, 512]
.


key: _node_content
string_value:
{"id_": "f44f2e0b-401e-4389-93ce-f5ea9f67847b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "ff90393796b58d9cbff193200432df57f36db1694146fa624ccfa7fdd0c6151d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dae0366f-b0af-48db-a5c1-3bbf85c4904c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "4dedb082625fd4b50ab55287a8033f88d60e92b66d274d8bbde093c1647fc8dc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4dd6b29b-2424-400b-a3a8-97e06371b7ee", "node_type": "1", "metadata": {}, "hash": "92a2f9a33470f402fb9e992e74997e7ae6eef52dd475b302f007627dfa220c57", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In the example output above the first
two sections show the result corresponding to _inference time_ and _inference memory_. In addition, all relevant
information about the computing environment, _e.g._ the GPU type, the system, the library versions, etc... are printed
out in the third section under _ENVIRONMENT INFORMATION_. This information can optionally be saved in a _.csv_ file
when adding the argument `save_to_csv=True` to [`PyTorchBenchmarkArguments`] and
[`TensorFlowBenchmarkArguments`] respectively. In this case, every section is saved in a separate
_.csv_ file. The path to each _.csv_ file can optionally be defined via the argument data classes.

Instead of benchmarking pre-trained models via their model identifier, _e.g._ `google-bert/bert-base-uncased`, the user can
alternatively benchmark an arbitrary configuration of any available model class. In this case, a `list` of
configurations must be inserted with the benchmark args as follows.

<frameworkcontent>
<pt>
```py
>>> from transformers import PyTorchBenchmark, PyTorchBenchmarkArguments, BertConfig

>>> args = PyTorchBenchmarkArguments(
.     models=["bert-base", "bert-384-hid", "bert-6-lay"], batch_sizes=[8], sequence_lengths=[8, 32, 128, 512]
. )
>>> config_base = BertConfig()
>>> config_384_hid = BertConfig(hidden_size=384)
>>> config_6_lay = BertConfig(num_hidden_layers=6)

>>> benchmark = PyTorchBenchmark(args, configs=[config_base, config_384_hid, config_6_lay])
>>> benchmark.run()
====================       INFERENCE - SPEED - RESULT       ====================
--------------------------------------------------------------------------------
Model Name             Batch Size     Seq Length       Time in s                  
--------------------------------------------------------------------------------
bert-base                  8              128            0.006
bert-base                  8              512            0.006
bert-base                  8              128            0.018     
bert-base                  8              512            0.088     
bert-384-hid              8               8             0.006     
bert-384-hid              8               32            0.006     
bert-384-hid              8              128            0.011     
bert-384-hid              8              512            0.054     
bert-6-lay                 8               8             0.003     
bert-6-lay                 8               32            0.004     
bert-6-lay                 8              128            0.009     
bert-6-lay                 8              512            0.


key: _node_content
string_value:
{"id_": "4dd6b29b-2424-400b-a3a8-97e06371b7ee", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "ff90393796b58d9cbff193200432df57f36db1694146fa624ccfa7fdd0c6151d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f44f2e0b-401e-4389-93ce-f5ea9f67847b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "7b8f90d74eddfc0db6ac55af0e2184ec0863f2fff5e3f5a21f41497d51b18050", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "56104c30-4b01-4bb1-a65d-5459e903fcee", "node_type": "1", "metadata": {}, "hash": "15b19bb6ed7dd19918845968f493516239f7a4eb4744b614bb9d8b877e1eb44b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In this case, a `list` of
configurations must be inserted with the benchmark args as follows.

<frameworkcontent>
<pt>
```py
>>> from transformers import PyTorchBenchmark, PyTorchBenchmarkArguments, BertConfig

>>> args = PyTorchBenchmarkArguments(
.     models=["bert-base", "bert-384-hid", "bert-6-lay"], batch_sizes=[8], sequence_lengths=[8, 32, 128, 512]
. )
>>> config_base = BertConfig()
>>> config_384_hid = BertConfig(hidden_size=384)
>>> config_6_lay = BertConfig(num_hidden_layers=6)

>>> benchmark = PyTorchBenchmark(args, configs=[config_base, config_384_hid, config_6_lay])
>>> benchmark.run()
====================       INFERENCE - SPEED - RESULT       ====================
--------------------------------------------------------------------------------
Model Name             Batch Size     Seq Length       Time in s                  
--------------------------------------------------------------------------------
bert-base                  8              128            0.006
bert-base                  8              512            0.006
bert-base                  8              128            0.018     
bert-base                  8              512            0.088     
bert-384-hid              8               8             0.006     
bert-384-hid              8               32            0.006     
bert-384-hid              8              128            0.011     
bert-384-hid              8              512            0.054     
bert-6-lay                 8               8             0.003     
bert-6-lay                 8               32            0.004     
bert-6-lay                 8              128            0.009     
bert-6-lay                 8              512            0.044
--------------------------------------------------------------------------------

====================      INFERENCE - MEMORY - RESULT       ====================
--------------------------------------------------------------------------------
Model Name             Batch Size     Seq Length      Memory in MB 
--------------------------------------------------------------------------------
bert-base                  8               8             1277
bert-base                  8               32            1281
bert-base                  8              128            1307     
bert-base                  8              512            1539     
bert-384-hid              8               8             1005     
bert-384-hid              8               32            1027     
bert-384-hid              8              128            1035     
bert-384-hid              8              512            1255     
bert-6-lay                 8               8             1097     
bert-6-lay                 8               32            1101     
bert-6-lay                 8              128            1127     
bert-6-lay                 8              512            1359
--------------------------------------------------------------------------------

====================        ENVIRONMENT INFORMATION         ====================

- transformers_version: 2.11.0
- framework: PyTorch
- use_torchscript: False
- framework_version: 1.4.0
- python_version: 3.6.10
- system: Linux
- cpu: x86_64
- architecture: 64bit
- date: 2020-06-29
- time: 09:35:25.143267
- fp16: False
- use_multiprocessing: True
- only_pretrain_model: False
- cpu_ram_mb: 32088
- use_gpu: True
- num_gpus: 1
- gpu: TITAN RTX
- gpu_ram_mb: 24217
- gpu_power_watts: 280.


key: _node_content
string_value:
{"id_": "56104c30-4b01-4bb1-a65d-5459e903fcee", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "ff90393796b58d9cbff193200432df57f36db1694146fa624ccfa7fdd0c6151d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4dd6b29b-2424-400b-a3a8-97e06371b7ee", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "72aa414e3a1a99d4c3960e7e3f84f8716596085f5b164799fe14384fbdf41e13", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e51a5c9c-5598-4320-b566-ea0694b0a4a7", "node_type": "1", "metadata": {}, "hash": "eb4562bc45c52a0b70931585ea11d6b981a5ce35661844a90f100390d725900d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
044
--------------------------------------------------------------------------------

====================      INFERENCE - MEMORY - RESULT       ====================
--------------------------------------------------------------------------------
Model Name             Batch Size     Seq Length      Memory in MB 
--------------------------------------------------------------------------------
bert-base                  8               8             1277
bert-base                  8               32            1281
bert-base                  8              128            1307     
bert-base                  8              512            1539     
bert-384-hid              8               8             1005     
bert-384-hid              8               32            1027     
bert-384-hid              8              128            1035     
bert-384-hid              8              512            1255     
bert-6-lay                 8               8             1097     
bert-6-lay                 8               32            1101     
bert-6-lay                 8              128            1127     
bert-6-lay                 8              512            1359
--------------------------------------------------------------------------------

====================        ENVIRONMENT INFORMATION         ====================

- transformers_version: 2.11.0
- framework: PyTorch
- use_torchscript: False
- framework_version: 1.4.0
- python_version: 3.6.10
- system: Linux
- cpu: x86_64
- architecture: 64bit
- date: 2020-06-29
- time: 09:35:25.143267
- fp16: False
- use_multiprocessing: True
- only_pretrain_model: False
- cpu_ram_mb: 32088
- use_gpu: True
- num_gpus: 1
- gpu: TITAN RTX
- gpu_ram_mb: 24217
- gpu_power_watts: 280.0
- gpu_performance_state: 2
- use_tpu: False
```
</pt>
<tf>
```py
>>> from transformers import TensorFlowBenchmark, TensorFlowBenchmarkArguments, BertConfig

>>> args = TensorFlowBenchmarkArguments(
.     models=["bert-base", "bert-384-hid", "bert-6-lay"], batch_sizes=[8], sequence_lengths=[8, 32, 128, 512]
. )
>>> config_base = BertConfig()
>>> config_384_hid = BertConfig(hidden_size=384)
>>> config_6_lay = BertConfig(num_hidden_layers=6)

>>> benchmark = TensorFlowBenchmark(args, configs=[config_base, config_384_hid, config_6_lay])
>>> benchmark.run()
====================       INFERENCE - SPEED - RESULT       ====================
--------------------------------------------------------------------------------
Model Name             Batch Size     Seq Length       Time in s                  
--------------------------------------------------------------------------------
bert-base                  8               8             0.005
bert-base                  8               32            0.008
bert-base                  8              128            0.022
bert-base                  8              512            0.106
bert-384-hid              8               8             0.005
bert-384-hid              8               32            0.007
bert-384-hid              8              128            0.018
bert-384-hid              8              512            0.064
bert-6-lay                 8               8             0.002
bert-6-lay                 8               32            0.003
bert-6-lay                 8              128            0.0011
bert-6-lay                 8              512            0.


key: _node_content
string_value:
{"id_": "e51a5c9c-5598-4320-b566-ea0694b0a4a7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "ff90393796b58d9cbff193200432df57f36db1694146fa624ccfa7fdd0c6151d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "56104c30-4b01-4bb1-a65d-5459e903fcee", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "aec346552306288c37c91f7cee42f9b917503434291723658c0c59a19ff621d3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b0cceacd-4b5c-448b-841e-2485205eab4a", "node_type": "1", "metadata": {}, "hash": "f88836cfe2e4abead9edf96fd4b9bc49361b26eb8347c279653188ad81e7f047", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0
- gpu_performance_state: 2
- use_tpu: False
```
</pt>
<tf>
```py
>>> from transformers import TensorFlowBenchmark, TensorFlowBenchmarkArguments, BertConfig

>>> args = TensorFlowBenchmarkArguments(
.     models=["bert-base", "bert-384-hid", "bert-6-lay"], batch_sizes=[8], sequence_lengths=[8, 32, 128, 512]
. )
>>> config_base = BertConfig()
>>> config_384_hid = BertConfig(hidden_size=384)
>>> config_6_lay = BertConfig(num_hidden_layers=6)

>>> benchmark = TensorFlowBenchmark(args, configs=[config_base, config_384_hid, config_6_lay])
>>> benchmark.run()
====================       INFERENCE - SPEED - RESULT       ====================
--------------------------------------------------------------------------------
Model Name             Batch Size     Seq Length       Time in s                  
--------------------------------------------------------------------------------
bert-base                  8               8             0.005
bert-base                  8               32            0.008
bert-base                  8              128            0.022
bert-base                  8              512            0.106
bert-384-hid              8               8             0.005
bert-384-hid              8               32            0.007
bert-384-hid              8              128            0.018
bert-384-hid              8              512            0.064
bert-6-lay                 8               8             0.002
bert-6-lay                 8               32            0.003
bert-6-lay                 8              128            0.0011
bert-6-lay                 8              512            0.074
--------------------------------------------------------------------------------

====================      INFERENCE - MEMORY - RESULT       ====================
--------------------------------------------------------------------------------
Model Name             Batch Size     Seq Length      Memory in MB 
--------------------------------------------------------------------------------
bert-base                  8               8             1330
bert-base                  8               32            1330
bert-base                  8              128            1330
bert-base                  8              512            1770
bert-384-hid              8               8             1330
bert-384-hid              8               32            1330
bert-384-hid              8              128            1330
bert-384-hid              8              512            1540
bert-6-lay                 8               8             1330
bert-6-lay                 8               32            1330
bert-6-lay                 8              128            1330
bert-6-lay                 8              512            1540
--------------------------------------------------------------------------------

====================        ENVIRONMENT INFORMATION         ====================

- transformers_version: 2.11.0
- framework: Tensorflow
- use_xla: False
- framework_version: 2.2.0
- python_version: 3.6.10
- system: Linux
- cpu: x86_64
- architecture: 64bit
- date: 2020-06-29
- time: 09:38:15.487125
- fp16: False
- use_multiprocessing: True
- only_pretrain_model: False
- cpu_ram_mb: 32088
- use_gpu: True
- num_gpus: 1
- gpu: TITAN RTX
- gpu_ram_mb: 24217
- gpu_power_watts: 280.


key: _node_content
string_value:
{"id_": "b0cceacd-4b5c-448b-841e-2485205eab4a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "ff90393796b58d9cbff193200432df57f36db1694146fa624ccfa7fdd0c6151d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e51a5c9c-5598-4320-b566-ea0694b0a4a7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "0f59a973aaee1e18889695adc0561829ccc857c536bfe695b889a3c64d49638a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e3badd33-4ea7-49ab-9552-bf62b8310d94", "node_type": "1", "metadata": {}, "hash": "562b50cefa883492c05b5f81591d7350d4fa496704447f7476278a7fb042e90f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13413, "end_char_idx": 16397, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
074
--------------------------------------------------------------------------------

====================      INFERENCE - MEMORY - RESULT       ====================
--------------------------------------------------------------------------------
Model Name             Batch Size     Seq Length      Memory in MB 
--------------------------------------------------------------------------------
bert-base                  8               8             1330
bert-base                  8               32            1330
bert-base                  8              128            1330
bert-base                  8              512            1770
bert-384-hid              8               8             1330
bert-384-hid              8               32            1330
bert-384-hid              8              128            1330
bert-384-hid              8              512            1540
bert-6-lay                 8               8             1330
bert-6-lay                 8               32            1330
bert-6-lay                 8              128            1330
bert-6-lay                 8              512            1540
--------------------------------------------------------------------------------

====================        ENVIRONMENT INFORMATION         ====================

- transformers_version: 2.11.0
- framework: Tensorflow
- use_xla: False
- framework_version: 2.2.0
- python_version: 3.6.10
- system: Linux
- cpu: x86_64
- architecture: 64bit
- date: 2020-06-29
- time: 09:38:15.487125
- fp16: False
- use_multiprocessing: True
- only_pretrain_model: False
- cpu_ram_mb: 32088
- use_gpu: True
- num_gpus: 1
- gpu: TITAN RTX
- gpu_ram_mb: 24217
- gpu_power_watts: 280.0
- gpu_performance_state: 2
- use_tpu: False
```
</tf>
</frameworkcontent>

Again, _inference time_ and _required memory_ for _inference_ are measured, but this time for customized configurations
of the `BertModel` class. This feature can especially be helpful when deciding for which configuration the model
should be trained.


## Benchmark best practices

This section lists a couple of best practices one should be aware of when benchmarking a model.

- Currently, only single device benchmarking is supported. When benchmarking on GPU, it is recommended that the user
  specifies on which device the code should be run by setting the `CUDA_VISIBLE_DEVICES` environment variable in the
  shell, _e.g._ `export CUDA_VISIBLE_DEVICES=0` before running the code.
- The option `no_multi_processing` should only be set to `True` for testing and debugging. To ensure accurate
  memory measurement it is recommended to run each memory benchmark in a separate process by making sure
  `no_multi_processing` is set to `True`.
- One should always state the environment information when sharing the results of a model benchmark. Results can vary
  heavily between different GPU devices, library versions, etc., so that benchmark results on their own are not very
  useful for the community.


key: _node_content
string_value:
{"id_": "e3badd33-4ea7-49ab-9552-bf62b8310d94", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e86b9ddf-25df-4bfa-a78c-696520de005a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "ff90393796b58d9cbff193200432df57f36db1694146fa624ccfa7fdd0c6151d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b0cceacd-4b5c-448b-841e-2485205eab4a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "b835e94333efbfd7b9239ec9ebc141d4a537ed1e533d702dcd9fae56f8b3a8d5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14833, "end_char_idx": 17380, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
6.10
- system: Linux
- cpu: x86_64
- architecture: 64bit
- date: 2020-06-29
- time: 09:38:15.487125
- fp16: False
- use_multiprocessing: True
- only_pretrain_model: False
- cpu_ram_mb: 32088
- use_gpu: True
- num_gpus: 1
- gpu: TITAN RTX
- gpu_ram_mb: 24217
- gpu_power_watts: 280.0
- gpu_performance_state: 2
- use_tpu: False
```
</tf>
</frameworkcontent>

Again, _inference time_ and _required memory_ for _inference_ are measured, but this time for customized configurations
of the `BertModel` class. This feature can especially be helpful when deciding for which configuration the model
should be trained.


## Benchmark best practices

This section lists a couple of best practices one should be aware of when benchmarking a model.

- Currently, only single device benchmarking is supported. When benchmarking on GPU, it is recommended that the user
  specifies on which device the code should be run by setting the `CUDA_VISIBLE_DEVICES` environment variable in the
  shell, _e.g._ `export CUDA_VISIBLE_DEVICES=0` before running the code.
- The option `no_multi_processing` should only be set to `True` for testing and debugging. To ensure accurate
  memory measurement it is recommended to run each memory benchmark in a separate process by making sure
  `no_multi_processing` is set to `True`.
- One should always state the environment information when sharing the results of a model benchmark. Results can vary
  heavily between different GPU devices, library versions, etc., so that benchmark results on their own are not very
  useful for the community.


## Sharing your benchmark

Previously all available core models (10 at the time) have been benchmarked for _inference time_, across many different
settings: using PyTorch, with and without TorchScript, using TensorFlow, with and without XLA. All of those tests were
done across CPUs (except for TensorFlow XLA) and GPUs.

The approach is detailed in the [following blogpost](https://medium.com/huggingface/benchmarking-transformers-pytorch-and-tensorflow-e2917fb891c2) and the results are
available [here](https://docs.google.com/spreadsheets/d/1sryqufw2D0XlUH4sq3e9Wnxu5EAQkaohzrJbd5HdQ_w/edit?usp=sharing).

With the new _benchmark_ tools, it is easier than ever to share your benchmark results with the community

- [PyTorch Benchmarking Results](https://github.com/huggingface/transformers/tree/main/examples/pytorch/benchmarking/README.md).
- [TensorFlow Benchmarking Results](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/benchmarking/README.md).


key: _node_content
string_value:
{"id_": "45e9b47f-20eb-41c2-b99a-d36ce82dab36", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4daf6054-b0da-4aa9-977e-bb859aeb7876", "node_type": "1", "metadata": {}, "hash": "bbcddc2816a45c1f29c7706411bccc322c54196b006e68c857e171482491db3a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3667, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Text generation strategies

Text generation is essential to many NLP tasks, such as open-ended text generation, summarization, translation, and
more. It also plays a role in a variety of mixed-modality applications that have text as an output like speech-to-text
and vision-to-text. Some of the models that can generate text include
GPT2, XLNet, OpenAI GPT, CTRL, TransformerXL, XLM, Bart, T5, GIT, Whisper.

Check out a few examples that use [`~generation.GenerationMixin.generate`] method to produce
text outputs for different tasks:
* [Text summarization](./tasks/summarization#inference)
* [Image captioning](./model_doc/git#transformers.GitForCausalLM.forward.example)
* [Audio transcription](./model_doc/whisper#transformers.WhisperForConditionalGeneration.forward.example)

Note that the inputs to the generate method depend on the model's modality. They are returned by the model's preprocessor
class, such as AutoTokenizer or AutoProcessor. If a model's preprocessor creates more than one kind of input, pass all
the inputs to generate(). You can learn more about the individual model's preprocessor in the corresponding model's documentation.

The process of selecting output tokens to generate text is known as decoding, and you can customize the decoding strategy
that the `generate()` method will use. Modifying a decoding strategy does not change the values of any trainable parameters.
However, it can have a noticeable impact on the quality of the generated output. It can help reduce repetition in the text
and make it more coherent.

This guide describes:
* default generation configuration
* common decoding strategies and their main parameters
* saving and sharing custom generation configurations with your fine-tuned model on 🤗 Hub

## Default text generation configuration

A decoding strategy for a model is defined in its generation configuration. When using pre-trained models for inference
within a [`pipeline`], the models call the `PreTrainedModel.generate()` method that applies a default generation
configuration under the hood. The default configuration is also used when no custom configuration has been saved with
the model.

When you load a model explicitly, you can inspect the generation configuration that comes with it through
 `model.generation_config`:

```python
>>> from transformers import AutoModelForCausalLM

>>> model = AutoModelForCausalLM.from_pretrained("distilbert/distilgpt2")
>>> model.generation_config
GenerationConfig {
  "bos_token_id": 50256,
  "eos_token_id": 50256
}
<BLANKLINE>
```

Printing out the `model.generation_config` reveals only the values that are different from the default generation
configuration, and does not list any of the default values.

The default generation configuration limits the size of the output combined with the input prompt to a maximum of 20
tokens to avoid running into resource limitations. The default decoding strategy is greedy search, which is the simplest decoding strategy that picks a token with the highest probability as the next token. For many tasks
and small output sizes this works well. However, when used to generate longer outputs, greedy search can start
producing highly repetitive results.

## Customize text generation

You can override any `generation_config` by passing the parameters and their values directly to the [`generate`] method:

```python
>>> my_model.generate(**inputs, num_beams=4, do_sample=True)  # doctest: +SKIP
```

Even if the default decoding strategy mostly works for your task, you can still tweak a few things. Some of the
commonly adjusted parameters include:

- `max_new_tokens`: the maximum number of tokens to generate.


key: _node_content
string_value:
{"id_": "4daf6054-b0da-4aa9-977e-bb859aeb7876", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "45e9b47f-20eb-41c2-b99a-d36ce82dab36", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "5ca2129e149d3f539cedc6f36752132d646a543716c7447c80ef5553a700be60", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c082615a-a757-439b-a5b9-5fef4bf9d1d5", "node_type": "1", "metadata": {}, "hash": "589d2b75895204c7e3738223b6eac7e08c7ecce7124b3ca22b6d360e5ff745d8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1875, "end_char_idx": 5159, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
When using pre-trained models for inference
within a [`pipeline`], the models call the `PreTrainedModel.generate()` method that applies a default generation
configuration under the hood. The default configuration is also used when no custom configuration has been saved with
the model.

When you load a model explicitly, you can inspect the generation configuration that comes with it through
 `model.generation_config`:

```python
>>> from transformers import AutoModelForCausalLM

>>> model = AutoModelForCausalLM.from_pretrained("distilbert/distilgpt2")
>>> model.generation_config
GenerationConfig {
  "bos_token_id": 50256,
  "eos_token_id": 50256
}
<BLANKLINE>
```

Printing out the `model.generation_config` reveals only the values that are different from the default generation
configuration, and does not list any of the default values.

The default generation configuration limits the size of the output combined with the input prompt to a maximum of 20
tokens to avoid running into resource limitations. The default decoding strategy is greedy search, which is the simplest decoding strategy that picks a token with the highest probability as the next token. For many tasks
and small output sizes this works well. However, when used to generate longer outputs, greedy search can start
producing highly repetitive results.

## Customize text generation

You can override any `generation_config` by passing the parameters and their values directly to the [`generate`] method:

```python
>>> my_model.generate(**inputs, num_beams=4, do_sample=True)  # doctest: +SKIP
```

Even if the default decoding strategy mostly works for your task, you can still tweak a few things. Some of the
commonly adjusted parameters include:

- `max_new_tokens`: the maximum number of tokens to generate. In other words, the size of the output sequence, not
including the tokens in the prompt. As an alternative to using the output's length as a stopping criteria, you can choose
to stop generation whenever the full generation exceeds some amount of time. To learn more, check [`StoppingCriteria`].
- `num_beams`: by specifying a number of beams higher than 1, you are effectively switching from greedy search to
beam search. This strategy evaluates several hypotheses at each time step and eventually chooses the hypothesis that
has the overall highest probability for the entire sequence. This has the advantage of identifying high-probability
sequences that start with a lower probability initial tokens and would've been ignored by the greedy search. Visualize how it works [here](https://huggingface.co/spaces/m-ric/beam_search_visualizer).
- `do_sample`: if set to `True`, this parameter enables decoding strategies such as multinomial sampling, beam-search
multinomial sampling, Top-K sampling and Top-p sampling. All these strategies select the next token from the probability
distribution over the entire vocabulary with various strategy-specific adjustments.
- `num_return_sequences`: the number of sequence candidates to return for each input. This option is only available for
the decoding strategies that support multiple sequence candidates, e.g. variations of beam search and sampling. Decoding
strategies like greedy search and contrastive search return a single output sequence.


key: _node_content
string_value:
{"id_": "c082615a-a757-439b-a5b9-5fef4bf9d1d5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4daf6054-b0da-4aa9-977e-bb859aeb7876", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "e2658a8fb9aa60cf2c5960d9df528b32fb59d177d8ad46c8f88ce9c8f658f533", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5f203877-338a-45c5-aa89-a8353a5d2b22", "node_type": "1", "metadata": {}, "hash": "2a359e86854b2eddc67707827f707b4e8b472aebcf268e6b1fc0aa8b1e807467", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3555, "end_char_idx": 6529, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Some of the
commonly adjusted parameters include:

- `max_new_tokens`: the maximum number of tokens to generate. In other words, the size of the output sequence, not
including the tokens in the prompt. As an alternative to using the output's length as a stopping criteria, you can choose
to stop generation whenever the full generation exceeds some amount of time. To learn more, check [`StoppingCriteria`].
- `num_beams`: by specifying a number of beams higher than 1, you are effectively switching from greedy search to
beam search. This strategy evaluates several hypotheses at each time step and eventually chooses the hypothesis that
has the overall highest probability for the entire sequence. This has the advantage of identifying high-probability
sequences that start with a lower probability initial tokens and would've been ignored by the greedy search. Visualize how it works [here](https://huggingface.co/spaces/m-ric/beam_search_visualizer).
- `do_sample`: if set to `True`, this parameter enables decoding strategies such as multinomial sampling, beam-search
multinomial sampling, Top-K sampling and Top-p sampling. All these strategies select the next token from the probability
distribution over the entire vocabulary with various strategy-specific adjustments.
- `num_return_sequences`: the number of sequence candidates to return for each input. This option is only available for
the decoding strategies that support multiple sequence candidates, e.g. variations of beam search and sampling. Decoding
strategies like greedy search and contrastive search return a single output sequence.

## Save a custom decoding strategy with your model

If you would like to share your fine-tuned model with a specific generation configuration, you can:
* Create a [`GenerationConfig`] class instance
* Specify the decoding strategy parameters
* Save your generation configuration with [`GenerationConfig.save_pretrained`], making sure to leave its `config_file_name` argument empty
* Set `push_to_hub` to `True` to upload your config to the model's repo

```python
>>> from transformers import AutoModelForCausalLM, GenerationConfig

>>> model = AutoModelForCausalLM.from_pretrained("my_account/my_model")  # doctest: +SKIP
>>> generation_config = GenerationConfig(
...     max_new_tokens=50, do_sample=True, top_k=50, eos_token_id=model.config.eos_token_id
... )
>>> generation_config.save_pretrained("my_account/my_model", push_to_hub=True)  # doctest: +SKIP
```

You can also store several generation configurations in a single directory, making use of the `config_file_name`
argument in [`GenerationConfig.save_pretrained`]. You can later instantiate them with [`GenerationConfig.from_pretrained`]. This is useful if you want to
store several generation configurations for a single model (e.g. one for creative text generation with sampling, and
one for summarization with beam search). You must have the right Hub permissions to add configuration files to a model.


key: _node_content
string_value:
{"id_": "5f203877-338a-45c5-aa89-a8353a5d2b22", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c082615a-a757-439b-a5b9-5fef4bf9d1d5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "36b48f174447db03db9f3024529790452f2c607200c65d86af5e2e47e9b7c3b2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "57a70f50-7157-4b33-9fe9-071b5eded0a9", "node_type": "1", "metadata": {}, "hash": "4ec15013714f2607c0735fb923ad0b848ace2d3d13efb92ca8865d83933aeb8d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4685, "end_char_idx": 8227, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
All these strategies select the next token from the probability
distribution over the entire vocabulary with various strategy-specific adjustments.
- `num_return_sequences`: the number of sequence candidates to return for each input. This option is only available for
the decoding strategies that support multiple sequence candidates, e.g. variations of beam search and sampling. Decoding
strategies like greedy search and contrastive search return a single output sequence.

## Save a custom decoding strategy with your model

If you would like to share your fine-tuned model with a specific generation configuration, you can:
* Create a [`GenerationConfig`] class instance
* Specify the decoding strategy parameters
* Save your generation configuration with [`GenerationConfig.save_pretrained`], making sure to leave its `config_file_name` argument empty
* Set `push_to_hub` to `True` to upload your config to the model's repo

```python
>>> from transformers import AutoModelForCausalLM, GenerationConfig

>>> model = AutoModelForCausalLM.from_pretrained("my_account/my_model")  # doctest: +SKIP
>>> generation_config = GenerationConfig(
...     max_new_tokens=50, do_sample=True, top_k=50, eos_token_id=model.config.eos_token_id
... )
>>> generation_config.save_pretrained("my_account/my_model", push_to_hub=True)  # doctest: +SKIP
```

You can also store several generation configurations in a single directory, making use of the `config_file_name`
argument in [`GenerationConfig.save_pretrained`]. You can later instantiate them with [`GenerationConfig.from_pretrained`]. This is useful if you want to
store several generation configurations for a single model (e.g. one for creative text generation with sampling, and
one for summarization with beam search). You must have the right Hub permissions to add configuration files to a model.

```python
>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, GenerationConfig

>>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("google-t5/t5-small")

>>> translation_generation_config = GenerationConfig(
...     num_beams=4,
...     early_stopping=True,
...     decoder_start_token_id=0,
...     eos_token_id=model.config.eos_token_id,
...     pad_token=model.config.pad_token_id,
... )

>>> # Tip: add `push_to_hub=True` to push to the Hub
>>> translation_generation_config.save_pretrained("/tmp", "translation_generation_config.json")

>>> # You could then use the named generation config file to parameterize generation
>>> generation_config = GenerationConfig.from_pretrained("/tmp", "translation_generation_config.json")
>>> inputs = tokenizer("translate English to French: Configuration files are easy to use!", return_tensors="pt")
>>> outputs = model.generate(**inputs, generation_config=generation_config)
>>> print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
['Les fichiers de configuration sont faciles à utiliser!']
```

## Streaming

The `generate()` supports streaming, through its `streamer` input. The `streamer` input is compatible with any instance
from a class that has the following methods: `put()` and `end()`. Internally, `put()` is used to push new tokens and
`end()` is used to flag the end of text generation.

<Tip warning={true}>

The API for the streamer classes is still under development and may change in the future.

</Tip>

In practice, you can craft your own streaming class for all sorts of purposes! We also have basic streaming classes
ready for you to use.


key: _node_content
string_value:
{"id_": "57a70f50-7157-4b33-9fe9-071b5eded0a9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5f203877-338a-45c5-aa89-a8353a5d2b22", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "9a859142ff11d1371922a2d8f554e451128b326f47027b1403b2e37146411b64", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "59498d4d-3ba5-4d1f-942b-521e8527b33e", "node_type": "1", "metadata": {}, "hash": "e077e9100f770b81f83830aaa7e04b0167fde32b493306162fa2a6adabdbc4c8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6451, "end_char_idx": 8957, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You must have the right Hub permissions to add configuration files to a model.

```python
>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, GenerationConfig

>>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("google-t5/t5-small")

>>> translation_generation_config = GenerationConfig(
...     num_beams=4,
...     early_stopping=True,
...     decoder_start_token_id=0,
...     eos_token_id=model.config.eos_token_id,
...     pad_token=model.config.pad_token_id,
... )

>>> # Tip: add `push_to_hub=True` to push to the Hub
>>> translation_generation_config.save_pretrained("/tmp", "translation_generation_config.json")

>>> # You could then use the named generation config file to parameterize generation
>>> generation_config = GenerationConfig.from_pretrained("/tmp", "translation_generation_config.json")
>>> inputs = tokenizer("translate English to French: Configuration files are easy to use!", return_tensors="pt")
>>> outputs = model.generate(**inputs, generation_config=generation_config)
>>> print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
['Les fichiers de configuration sont faciles à utiliser!']
```

## Streaming

The `generate()` supports streaming, through its `streamer` input. The `streamer` input is compatible with any instance
from a class that has the following methods: `put()` and `end()`. Internally, `put()` is used to push new tokens and
`end()` is used to flag the end of text generation.

<Tip warning={true}>

The API for the streamer classes is still under development and may change in the future.

</Tip>

In practice, you can craft your own streaming class for all sorts of purposes! We also have basic streaming classes
ready for you to use. For example, you can use the [`TextStreamer`] class to stream the output of `generate()` into
your screen, one word at a time:

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer

>>> tok = AutoTokenizer.from_pretrained("openai-community/gpt2")
>>> model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2")
>>> inputs = tok(["An increasing sequence: one,"], return_tensors="pt")
>>> streamer = TextStreamer(tok)

>>> # Despite returning the usual output, the streamer will also print the generated text to stdout.
>>> _ = model.generate(**inputs, streamer=streamer, max_new_tokens=20)
An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven,
```


key: _node_content
string_value:
{"id_": "59498d4d-3ba5-4d1f-942b-521e8527b33e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "57a70f50-7157-4b33-9fe9-071b5eded0a9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "f32e449951b1c4926a433b293e8f976b50b15a30f2cdb225a1f3a620645d8e2b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ba68b508-6691-402d-953c-bd2f6c6feeb9", "node_type": "1", "metadata": {}, "hash": "f2720d9be69d13a6b1cf7570bac21675732bc99b960d4cb198f6a979b2625f4f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7432, "end_char_idx": 10213, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
", return_tensors="pt")
>>> outputs = model.generate(**inputs, generation_config=generation_config)
>>> print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
['Les fichiers de configuration sont faciles à utiliser!']
```

## Streaming

The `generate()` supports streaming, through its `streamer` input. The `streamer` input is compatible with any instance
from a class that has the following methods: `put()` and `end()`. Internally, `put()` is used to push new tokens and
`end()` is used to flag the end of text generation.

<Tip warning={true}>

The API for the streamer classes is still under development and may change in the future.

</Tip>

In practice, you can craft your own streaming class for all sorts of purposes! We also have basic streaming classes
ready for you to use. For example, you can use the [`TextStreamer`] class to stream the output of `generate()` into
your screen, one word at a time:

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer

>>> tok = AutoTokenizer.from_pretrained("openai-community/gpt2")
>>> model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2")
>>> inputs = tok(["An increasing sequence: one,"], return_tensors="pt")
>>> streamer = TextStreamer(tok)

>>> # Despite returning the usual output, the streamer will also print the generated text to stdout.
>>> _ = model.generate(**inputs, streamer=streamer, max_new_tokens=20)
An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven,
```


## KV Cache Quantization

The `generate()` method supports caching keys and values to enhance efficiency and avoid re-computations. However the key and value
cache can occupy a large portion of memory, becoming a bottleneck for long-context generation, especially for Large Language Models.
Quantizing the cache when using `generate()` can significantly reduce memory requirements at the cost of speed.

KV Cache quantization in `transformers` is largely inspired by the paper [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache]
(https://arxiv.org/abs/2402.02750) and currently supports `quanto` and `HQQ` as backends. For more information on the inner workings see the paper.

To enable quantization of the key-value cache, one needs to indicate `cache_implementation="quantized"` in the `generation_config`.
Quantization related arguments should be passed to the `generation_config` either as a `dict` or an instance of a [`QuantizedCacheConfig`] class.
One has to indicate which quantization backend to use in the [`QuantizedCacheConfig`], the default is `quanto`.

<Tip warning={true}>

Cache quantization can be detrimental if the context length is short and there is enough GPU VRAM available to run without cache quantization.

</Tip>


key: _node_content
string_value:
{"id_": "ba68b508-6691-402d-953c-bd2f6c6feeb9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "59498d4d-3ba5-4d1f-942b-521e8527b33e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "41a8d15aadec7cbcca7f2117363834246aa67567cc76ae933b14213d47aca0d1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "49a87587-a306-4788-b2d1-8a1bce99981e", "node_type": "1", "metadata": {}, "hash": "25083112b6bb76ec99a198b1451e66c6ab55849e8252d4e411b98063ca6243c9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8792, "end_char_idx": 12246, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> _ = model.generate(**inputs, streamer=streamer, max_new_tokens=20)
An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven,
```


## KV Cache Quantization

The `generate()` method supports caching keys and values to enhance efficiency and avoid re-computations. However the key and value
cache can occupy a large portion of memory, becoming a bottleneck for long-context generation, especially for Large Language Models.
Quantizing the cache when using `generate()` can significantly reduce memory requirements at the cost of speed.

KV Cache quantization in `transformers` is largely inspired by the paper [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache]
(https://arxiv.org/abs/2402.02750) and currently supports `quanto` and `HQQ` as backends. For more information on the inner workings see the paper.

To enable quantization of the key-value cache, one needs to indicate `cache_implementation="quantized"` in the `generation_config`.
Quantization related arguments should be passed to the `generation_config` either as a `dict` or an instance of a [`QuantizedCacheConfig`] class.
One has to indicate which quantization backend to use in the [`QuantizedCacheConfig`], the default is `quanto`.

<Tip warning={true}>

Cache quantization can be detrimental if the context length is short and there is enough GPU VRAM available to run without cache quantization.

</Tip>


```python
>>> import torch
>>> from transformers import AutoTokenizer, AutoModelForCausalLM

>>> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
>>> model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-chat-hf", torch_dtype=torch.float16).to("cuda:0")
>>> inputs = tokenizer("I like rock music because", return_tensors="pt").to(model.device)

>>> out = model.generate(**inputs, do_sample=False, max_new_tokens=20, cache_implementation="quantized", cache_config={"nbits": 4, "backend": "quanto"})
>>> print(tokenizer.batch_decode(out, skip_special_tokens=True)[0])
I like rock music because it's loud and energetic. It's a great way to express myself and rel

>>> out = model.generate(**inputs, do_sample=False, max_new_tokens=20)
>>> print(tokenizer.batch_decode(out, skip_special_tokens=True)[0])
I like rock music because it's loud and energetic. I like to listen to it when I'm feeling
```

## Watermarking

The `generate()` supports watermarking the generated text by randomly marking a portion of tokens as "green".
When generating the "green" will have a small 'bias' value added to their logits, thus having a higher chance to be generated.
The watermarked text can be detected by calculating the proportion of "green" tokens in the text and estimating how likely it is
statistically to obtain that amount of "green" tokens for human-generated text. This watermarking strategy was proposed in the paper
["On the Reliability of Watermarks for Large Language Models"](https://arxiv.org/abs/2306.04634). For more information on
the inner functioning of watermarking, it is recommended to refer to the paper.

The watermarking can be used with any generative model in `tranformers` and does not require an extra classification model
to detect watermarked text. To trigger watermarking, pass in a [`WatermarkingConfig`] with needed arguments directly to the
`.generate()` method or add it to the [`GenerationConfig`]. Watermarked text can be later detected with a [`WatermarkDetector`].


key: _node_content
string_value:
{"id_": "49a87587-a306-4788-b2d1-8a1bce99981e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ba68b508-6691-402d-953c-bd2f6c6feeb9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "45d3b99da9b184a4a2e0d52a036c72cbfc7efb5efa5b4cbe44627d9315304499", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2ac8b3ee-3d94-4047-8fc2-e64120affa63", "node_type": "1", "metadata": {}, "hash": "97b9cfbad60d4037cbb7432f80af2910933af554debdb850ef7d79c83ba4d936", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12249, "end_char_idx": 15532, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<Tip warning={true}>

The WatermarkDetector internally relies on the proportion of "green" tokens, and whether generated text follows the coloring pattern.
That is why it is recommended to strip off the prompt text, if it is much longer than the generated text.
This also can have an effect when one sequence in the batch is a lot longer causing other rows to be padded.
Additionally, the detector **must** be initiated with identical watermark configuration arguments used when generating.

</Tip>

Let's generate some text with watermarking. In the below code snippet, we set the bias to 2.5 which is a value that
will be added to "green" tokens' logits. After generating watermarked text, we can pass it directly to the `WatermarkDetector`
to check if the text is machine-generated (outputs `True` for machine-generated and `False` otherwise).

```python
>>> from transformers import AutoTokenizer, AutoModelForCausalLM, WatermarkDetector, WatermarkingConfig

>>> model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2")
>>> tok = AutoTokenizer.from_pretrained("openai-community/gpt2")
>>> tok.pad_token_id = tok.eos_token_id
>>> tok.padding_side = "left"

>>> inputs = tok(["This is the beginning of a long story", "Alice and Bob are"], padding=True, return_tensors="pt")
>>> input_len = inputs["input_ids"].shape[-1]

>>> watermarking_config = WatermarkingConfig(bias=2.5, seeding_scheme="selfhash")
>>> out = model.generate(**inputs, watermarking_config=watermarking_config, do_sample=False, max_length=20)

>>> detector = WatermarkDetector(model_config=model.config, device="cpu", watermarking_config=watermarking_config)
>>> detection_out = detector(out, return_dict=True)
>>> detection_out.prediction
array([True, True])
```


## Decoding strategies

Certain combinations of the `generate()` parameters, and ultimately `generation_config`, can be used to enable specific
decoding strategies. If you are new to this concept, we recommend reading [this blog post that illustrates how common decoding strategies work](https://huggingface.co/blog/how-to-generate).

Here, we'll show some of the parameters that control the decoding strategies and illustrate how you can use them.

### Greedy Search

[`generate`] uses greedy search decoding by default so you don't have to pass any parameters to enable it. This means the parameters `num_beams` is set to 1 and `do_sample=False`.

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> prompt = "I look forward to"
>>> checkpoint = "distilbert/distilgpt2"

>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> inputs = tokenizer(prompt, return_tensors="pt")

>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)
>>> outputs = model.generate(**inputs)
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['I look forward to seeing you all again!\n\n\n\n\n\n\n\n\n\n\n']
```

### Contrastive search

The contrastive search decoding strategy was proposed in the 2022 paper [A Contrastive Framework for Neural Text Generation](https://arxiv.org/abs/2202.06417).
It demonstrates superior results for generating non-repetitive yet coherent long outputs. To learn how contrastive search
works, check out [this blog post](https://huggingface.co/blog/introducing-csearch).


key: _node_content
string_value:
{"id_": "2ac8b3ee-3d94-4047-8fc2-e64120affa63", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "49a87587-a306-4788-b2d1-8a1bce99981e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "c035c0c299e752dacd50d0e77a2d9cd60fc9ac09b518072afcdfceeb5b17f567", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "de9e8122-e63b-4b21-9264-9a7eb7098276", "node_type": "1", "metadata": {}, "hash": "f794ae1b220d767481bbd5832536252890ce56086c9389688f1b1dd2155d49b4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13998, "end_char_idx": 16919, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Decoding strategies

Certain combinations of the `generate()` parameters, and ultimately `generation_config`, can be used to enable specific
decoding strategies. If you are new to this concept, we recommend reading [this blog post that illustrates how common decoding strategies work](https://huggingface.co/blog/how-to-generate).

Here, we'll show some of the parameters that control the decoding strategies and illustrate how you can use them.

### Greedy Search

[`generate`] uses greedy search decoding by default so you don't have to pass any parameters to enable it. This means the parameters `num_beams` is set to 1 and `do_sample=False`.

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> prompt = "I look forward to"
>>> checkpoint = "distilbert/distilgpt2"

>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> inputs = tokenizer(prompt, return_tensors="pt")

>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)
>>> outputs = model.generate(**inputs)
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['I look forward to seeing you all again!\n\n\n\n\n\n\n\n\n\n\n']
```

### Contrastive search

The contrastive search decoding strategy was proposed in the 2022 paper [A Contrastive Framework for Neural Text Generation](https://arxiv.org/abs/2202.06417).
It demonstrates superior results for generating non-repetitive yet coherent long outputs. To learn how contrastive search
works, check out [this blog post](https://huggingface.co/blog/introducing-csearch).
The two main parameters that enable and control the behavior of contrastive search are `penalty_alpha` and `top_k`:

```python
>>> from transformers import AutoTokenizer, AutoModelForCausalLM

>>> checkpoint = "openai-community/gpt2-large"
>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)

>>> prompt = "Hugging Face Company is"
>>> inputs = tokenizer(prompt, return_tensors="pt")

>>> outputs = model.generate(**inputs, penalty_alpha=0.6, top_k=4, max_new_tokens=100)
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['Hugging Face Company is a family owned and operated business. We pride ourselves on being the best
in the business and our customer service is second to none.\n\nIf you have any questions about our
products or services, feel free to contact us at any time. We look forward to hearing from you!']
```

### Multinomial sampling

As opposed to greedy search that always chooses a token with the highest probability as the
next token, multinomial sampling (also called ancestral sampling) randomly selects the next token based on the probability distribution over the entire
vocabulary given by the model. Every token with a non-zero probability has a chance of being selected, thus reducing the
risk of repetition.

To enable multinomial sampling set `do_sample=True` and `num_beams=1`.


key: _node_content
string_value:
{"id_": "de9e8122-e63b-4b21-9264-9a7eb7098276", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2ac8b3ee-3d94-4047-8fc2-e64120affa63", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "9229d3899d77380bc21baefed58e986c669566d3dd48f749836d9b1d93ccbeda", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5cccf3f8-a8fc-4425-8be3-5317a81862ef", "node_type": "1", "metadata": {}, "hash": "8210bcfb1f111f467e38e5c1420a1363697456b6c481598ec3ab4ad4e2b777d1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15327, "end_char_idx": 18855, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
It demonstrates superior results for generating non-repetitive yet coherent long outputs. To learn how contrastive search
works, check out [this blog post](https://huggingface.co/blog/introducing-csearch).
The two main parameters that enable and control the behavior of contrastive search are `penalty_alpha` and `top_k`:

```python
>>> from transformers import AutoTokenizer, AutoModelForCausalLM

>>> checkpoint = "openai-community/gpt2-large"
>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)

>>> prompt = "Hugging Face Company is"
>>> inputs = tokenizer(prompt, return_tensors="pt")

>>> outputs = model.generate(**inputs, penalty_alpha=0.6, top_k=4, max_new_tokens=100)
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['Hugging Face Company is a family owned and operated business. We pride ourselves on being the best
in the business and our customer service is second to none.\n\nIf you have any questions about our
products or services, feel free to contact us at any time. We look forward to hearing from you!']
```

### Multinomial sampling

As opposed to greedy search that always chooses a token with the highest probability as the
next token, multinomial sampling (also called ancestral sampling) randomly selects the next token based on the probability distribution over the entire
vocabulary given by the model. Every token with a non-zero probability has a chance of being selected, thus reducing the
risk of repetition.

To enable multinomial sampling set `do_sample=True` and `num_beams=1`.

```python
>>> from transformers import AutoTokenizer, AutoModelForCausalLM, set_seed
>>> set_seed(0)  # For reproducibility

>>> checkpoint = "openai-community/gpt2-large"
>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)

>>> prompt = "Today was an amazing day because"
>>> inputs = tokenizer(prompt, return_tensors="pt")

>>> outputs = model.generate(**inputs, do_sample=True, num_beams=1, max_new_tokens=100)
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
["Today was an amazing day because we received these wonderful items by the way of a gift shop. The box arrived on a Thursday and I opened it on Monday afternoon to receive the gifts. Both bags featured pieces from all the previous years!\n\nThe box had lots of surprises in it, including some sweet little mini chocolate chips! I don't think I'd eat all of these. This was definitely one of the most expensive presents I have ever got, I actually got most of them for free!\n\nThe first package came"]
```

### Beam-search decoding

Unlike greedy search, beam-search decoding keeps several hypotheses at each time step and eventually chooses
the hypothesis that has the overall highest probability for the entire sequence. This has the advantage of identifying high-probability
sequences that start with lower probability initial tokens and would've been ignored by the greedy search.

<a href="https://huggingface.co/spaces/m-ric/beam_search_visualizer" class="flex flex-col justify-center">
    <img style="max-width: 90%; margin: auto;" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/beam_search.png"/>
</a>

You can visualize how beam-search decoding works in [this interactive demo](https://huggingface.co/spaces/m-ric/beam_search_visualizer): type your input sentence, and play with the parameters to see how the decoding beams change.


key: _node_content
string_value:
{"id_": "5cccf3f8-a8fc-4425-8be3-5317a81862ef", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "de9e8122-e63b-4b21-9264-9a7eb7098276", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "77cf6f3f36471c9b24f16b57fc9befc059a8dc512ad82907507423151f288c7f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2db47660-07f7-4308-b634-3266f35fa7e2", "node_type": "1", "metadata": {}, "hash": "4190f9d3f54bf2362cfe9ac5587ed94c2728f22e72b460924e63af6275b16c81", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17560, "end_char_idx": 20940, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The box arrived on a Thursday and I opened it on Monday afternoon to receive the gifts. Both bags featured pieces from all the previous years!\n\nThe box had lots of surprises in it, including some sweet little mini chocolate chips! I don't think I'd eat all of these. This was definitely one of the most expensive presents I have ever got, I actually got most of them for free!\n\nThe first package came"]
```

### Beam-search decoding

Unlike greedy search, beam-search decoding keeps several hypotheses at each time step and eventually chooses
the hypothesis that has the overall highest probability for the entire sequence. This has the advantage of identifying high-probability
sequences that start with lower probability initial tokens and would've been ignored by the greedy search.

<a href="https://huggingface.co/spaces/m-ric/beam_search_visualizer" class="flex flex-col justify-center">
    <img style="max-width: 90%; margin: auto;" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/beam_search.png"/>
</a>

You can visualize how beam-search decoding works in [this interactive demo](https://huggingface.co/spaces/m-ric/beam_search_visualizer): type your input sentence, and play with the parameters to see how the decoding beams change.

To enable this decoding strategy, specify the `num_beams` (aka number of hypotheses to keep track of) that is greater than 1.

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> prompt = "It is astonishing how one can"
>>> checkpoint = "openai-community/gpt2-medium"

>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> inputs = tokenizer(prompt, return_tensors="pt")

>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)

>>> outputs = model.generate(**inputs, num_beams=5, max_new_tokens=50)
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['It is astonishing how one can have such a profound impact on the lives of so many people in such a short period of
time."\n\nHe added: "I am very proud of the work I have been able to do in the last few years.\n\n"I have']
```

### Beam-search multinomial sampling

As the name implies, this decoding strategy combines beam search with multinomial sampling. You need to specify
the `num_beams` greater than 1, and set `do_sample=True` to use this decoding strategy.

```python
>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, set_seed
>>> set_seed(0)  # For reproducibility

>>> prompt = "translate English to German: The house is wonderful."
>>> checkpoint = "google-t5/t5-small"

>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> inputs = tokenizer(prompt, return_tensors="pt")

>>> model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint)

>>> outputs = model.generate(**inputs, num_beams=5, do_sample=True)
>>> tokenizer.decode(outputs[0], skip_special_tokens=True)
'Das Haus ist wunderbar.'
```

### Diverse beam search decoding

The diverse beam search decoding strategy is an extension of the beam search strategy that allows for generating a more diverse
set of beam sequences to choose from. To learn how it works, refer to [Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models](https://arxiv.org/pdf/1610.02424.pdf).
This approach has three main parameters: `num_beams`, `num_beam_groups`, and `diversity_penalty`.


key: _node_content
string_value:
{"id_": "2db47660-07f7-4308-b634-3266f35fa7e2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5cccf3f8-a8fc-4425-8be3-5317a81862ef", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "cfcf09b69d504af21b10eb58c1fbf4e9949447e433244af67c3d0631706a20f3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0660fba0-30df-43f7-9031-c9a12d47e4a1", "node_type": "1", "metadata": {}, "hash": "c50688cbc1c4a812083d356a11b1b9c4b2b46d1aedc7c403f4018f7d091dec5b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19582, "end_char_idx": 23060, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"\n\nHe added: "I am very proud of the work I have been able to do in the last few years.\n\n"I have']
```

### Beam-search multinomial sampling

As the name implies, this decoding strategy combines beam search with multinomial sampling. You need to specify
the `num_beams` greater than 1, and set `do_sample=True` to use this decoding strategy.

```python
>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, set_seed
>>> set_seed(0)  # For reproducibility

>>> prompt = "translate English to German: The house is wonderful."
>>> checkpoint = "google-t5/t5-small"

>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> inputs = tokenizer(prompt, return_tensors="pt")

>>> model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint)

>>> outputs = model.generate(**inputs, num_beams=5, do_sample=True)
>>> tokenizer.decode(outputs[0], skip_special_tokens=True)
'Das Haus ist wunderbar.'
```

### Diverse beam search decoding

The diverse beam search decoding strategy is an extension of the beam search strategy that allows for generating a more diverse
set of beam sequences to choose from. To learn how it works, refer to [Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models](https://arxiv.org/pdf/1610.02424.pdf).
This approach has three main parameters: `num_beams`, `num_beam_groups`, and `diversity_penalty`.
The diversity penalty ensures the outputs are distinct across groups, and beam search is used within each group.


```python
>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

>>> checkpoint = "google/pegasus-xsum"
>>> prompt = (
...     "The Permaculture Design Principles are a set of universal design principles "
...     "that can be applied to any location, climate and culture, and they allow us to design "
...     "the most efficient and sustainable human habitation and food production systems. "
...     "Permaculture is a design system that encompasses a wide variety of disciplines, such "
...     "as ecology, landscape design, environmental science and energy conservation, and the "
...     "Permaculture design principles are drawn from these various disciplines. Each individual "
...     "design principle itself embodies a complete conceptual framework based on sound "
...     "scientific principles. When we bring all these separate  principles together, we can "
...     "create a design system that both looks at whole systems, the parts that these systems "
...     "consist of, and how those parts interact with each other to create a complex, dynamic, "
...     "living system. Each design principle serves as a tool that allows us to integrate all "
...     "the separate parts of a design, referred to as elements, into a functional, synergistic, "
...     "whole system, where the elements harmoniously interact and work together in the most "
...     "efficient way possible."
... )

>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> inputs = tokenizer(prompt, return_tensors="pt")

>>> model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint)

>>> outputs = model.generate(**inputs, num_beams=5, num_beam_groups=5, max_new_tokens=30, diversity_penalty=1.0)
>>> tokenizer.decode(outputs[0], skip_special_tokens=True)
'The Design Principles are a set of universal design principles that can be applied to any location, climate and
culture, and they allow us to design the'
```

This guide illustrates the main parameters that enable various decoding strategies.


key: _node_content
string_value:
{"id_": "0660fba0-30df-43f7-9031-c9a12d47e4a1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2db47660-07f7-4308-b634-3266f35fa7e2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "98e104e3a805ab9f3e3f55b18ba53a69aa31167891caafd9348036d40abea0ae", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "be5523fe-8658-42f8-bdc1-103b31b945b1", "node_type": "1", "metadata": {}, "hash": "4ddefa557c7c4b62d69e8b25ba956e9f97039e80b2ca8f01654ee97892a2993b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21463, "end_char_idx": 25105, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
...     "Permaculture is a design system that encompasses a wide variety of disciplines, such "
...     "as ecology, landscape design, environmental science and energy conservation, and the "
...     "Permaculture design principles are drawn from these various disciplines. Each individual "
...     "design principle itself embodies a complete conceptual framework based on sound "
...     "scientific principles. When we bring all these separate  principles together, we can "
...     "create a design system that both looks at whole systems, the parts that these systems "
...     "consist of, and how those parts interact with each other to create a complex, dynamic, "
...     "living system. Each design principle serves as a tool that allows us to integrate all "
...     "the separate parts of a design, referred to as elements, into a functional, synergistic, "
...     "whole system, where the elements harmoniously interact and work together in the most "
...     "efficient way possible."
... )

>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> inputs = tokenizer(prompt, return_tensors="pt")

>>> model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint)

>>> outputs = model.generate(**inputs, num_beams=5, num_beam_groups=5, max_new_tokens=30, diversity_penalty=1.0)
>>> tokenizer.decode(outputs[0], skip_special_tokens=True)
'The Design Principles are a set of universal design principles that can be applied to any location, climate and
culture, and they allow us to design the'
```

This guide illustrates the main parameters that enable various decoding strategies. More advanced parameters exist for the
[`generate`] method, which gives you even further control over the [`generate`] method's behavior.
For the complete list of the available parameters, refer to the [API documentation](./main_classes/text_generation.md).

### Speculative Decoding

Speculative decoding (also known as assisted decoding) is a modification of the decoding strategies above, that uses an
assistant model (ideally a much smaller one) with the same tokenizer, to generate a few candidate tokens. The main
model then validates the candidate tokens in a single forward pass, which speeds up the decoding process. If
`do_sample=True`, then the token validation with resampling introduced in the
[speculative decoding paper](https://arxiv.org/pdf/2211.17192.pdf) is used.

Currently, only greedy search and sampling are supported with assisted decoding, and assisted decoding doesn't support batched inputs.
To learn more about assisted decoding, check [this blog post](https://huggingface.co/blog/assisted-generation).

To enable assisted decoding, set the `assistant_model` argument with a model.

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> prompt = "Alice and Bob"
>>> checkpoint = "EleutherAI/pythia-1.4b-deduped"
>>> assistant_checkpoint = "EleutherAI/pythia-160m-deduped"

>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> inputs = tokenizer(prompt, return_tensors="pt")

>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)
>>> assistant_model = AutoModelForCausalLM.from_pretrained(assistant_checkpoint)
>>> outputs = model.generate(**inputs, assistant_model=assistant_model)
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['Alice and Bob are sitting in a bar. Alice is drinking a beer and Bob is drinking a']
```

When using assisted decoding with sampling methods, you can use the `temperature` argument to control the randomness,
just like in multinomial sampling. However, in assisted decoding, reducing the temperature may help improve the latency.


key: _node_content
string_value:
{"id_": "be5523fe-8658-42f8-bdc1-103b31b945b1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0660fba0-30df-43f7-9031-c9a12d47e4a1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "396a0515fd9aa30bf940c3cbef8378afde1843a3b60336573cd14ce611ee525d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d253cf68-c4e4-4dd4-a365-c54e5baeb5d9", "node_type": "1", "metadata": {}, "hash": "64075a65916a5d71032f6056e99eb54ce7c52f2e5836e94ee8f437372eb50559", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23320, "end_char_idx": 26650, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Speculative Decoding

Speculative decoding (also known as assisted decoding) is a modification of the decoding strategies above, that uses an
assistant model (ideally a much smaller one) with the same tokenizer, to generate a few candidate tokens. The main
model then validates the candidate tokens in a single forward pass, which speeds up the decoding process. If
`do_sample=True`, then the token validation with resampling introduced in the
[speculative decoding paper](https://arxiv.org/pdf/2211.17192.pdf) is used.

Currently, only greedy search and sampling are supported with assisted decoding, and assisted decoding doesn't support batched inputs.
To learn more about assisted decoding, check [this blog post](https://huggingface.co/blog/assisted-generation).

To enable assisted decoding, set the `assistant_model` argument with a model.

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> prompt = "Alice and Bob"
>>> checkpoint = "EleutherAI/pythia-1.4b-deduped"
>>> assistant_checkpoint = "EleutherAI/pythia-160m-deduped"

>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> inputs = tokenizer(prompt, return_tensors="pt")

>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)
>>> assistant_model = AutoModelForCausalLM.from_pretrained(assistant_checkpoint)
>>> outputs = model.generate(**inputs, assistant_model=assistant_model)
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['Alice and Bob are sitting in a bar. Alice is drinking a beer and Bob is drinking a']
```

When using assisted decoding with sampling methods, you can use the `temperature` argument to control the randomness,
just like in multinomial sampling. However, in assisted decoding, reducing the temperature may help improve the latency.

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed
>>> set_seed(42)  # For reproducibility

>>> prompt = "Alice and Bob"
>>> checkpoint = "EleutherAI/pythia-1.4b-deduped"
>>> assistant_checkpoint = "EleutherAI/pythia-160m-deduped"

>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> inputs = tokenizer(prompt, return_tensors="pt")

>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)
>>> assistant_model = AutoModelForCausalLM.from_pretrained(assistant_checkpoint)
>>> outputs = model.generate(**inputs, assistant_model=assistant_model, do_sample=True, temperature=0.5)
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['Alice and Bob, a couple of friends of mine, who are both in the same office as']
```

Alternativelly, you can also set the `prompt_lookup_num_tokens` to trigger n-gram based assisted decoding, as opposed
to model based assisted decoding. You can read more about it [here](https://twitter.com/joao_gante/status/1747322413006643259).
### DoLa Decoding

**D**ecoding by C**o**ntrasting **La**yers (DoLa) is a contrastive decoding strategy to improve the factuality and reduce the
hallucinations of LLMs, as described in this paper of ICLR 2024 [DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models](https://arxiv.org/abs/2309.03883).

DoLa is achieved by contrasting the differences in logits obtained from final
layers versus earlier layers, thus amplify the factual knowledge localized to particular part of transformer layers.


key: _node_content
string_value:
{"id_": "d253cf68-c4e4-4dd4-a365-c54e5baeb5d9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "be5523fe-8658-42f8-bdc1-103b31b945b1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "c2ff98db507c95aa0871c6f440059c4b9944cfd6b89136cc35280ca6ad9d10c3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "62bc6463-4d9a-48f9-a26d-4410fddfff5b", "node_type": "1", "metadata": {}, "hash": "a89cde1b24bc423d65409d5a5c2fc2ce3cbb0bf91e5584574f5dd69db0942bce", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 25020, "end_char_idx": 27734, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, in assisted decoding, reducing the temperature may help improve the latency.

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed
>>> set_seed(42)  # For reproducibility

>>> prompt = "Alice and Bob"
>>> checkpoint = "EleutherAI/pythia-1.4b-deduped"
>>> assistant_checkpoint = "EleutherAI/pythia-160m-deduped"

>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> inputs = tokenizer(prompt, return_tensors="pt")

>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)
>>> assistant_model = AutoModelForCausalLM.from_pretrained(assistant_checkpoint)
>>> outputs = model.generate(**inputs, assistant_model=assistant_model, do_sample=True, temperature=0.5)
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['Alice and Bob, a couple of friends of mine, who are both in the same office as']
```

Alternativelly, you can also set the `prompt_lookup_num_tokens` to trigger n-gram based assisted decoding, as opposed
to model based assisted decoding. You can read more about it [here](https://twitter.com/joao_gante/status/1747322413006643259).
### DoLa Decoding

**D**ecoding by C**o**ntrasting **La**yers (DoLa) is a contrastive decoding strategy to improve the factuality and reduce the
hallucinations of LLMs, as described in this paper of ICLR 2024 [DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models](https://arxiv.org/abs/2309.03883).

DoLa is achieved by contrasting the differences in logits obtained from final
layers versus earlier layers, thus amplify the factual knowledge localized to particular part of transformer layers.

Do the following two steps to activate DoLa decoding when calling the `model.generate` function:
1. Set the `dola_layers` argument, which can be either a string or a list of integers.
    - If set to a string, it can be one of `low`, `high`.
    - If set to a list of integers, it should be a list of layer indices between 0 and the total number of layers in the model. The 0-th layer is word embedding, and the 1st layer is the first transformer layer, and so on.
2. Set `repetition_penalty = 1.2` is suggested to reduce repetition in DoLa decoding.

See the following examples for DoLa decoding with the 32-layer LLaMA-7B model.

```python
>>> from transformers import AutoTokenizer, AutoModelForCausalLM, set_seed
>>> import torch

>>> tokenizer = AutoTokenizer.from_pretrained("huggyllama/llama-7b")
>>> model = AutoModelForCausalLM.from_pretrained("huggyllama/llama-7b", torch_dtype=torch.float16)
>>> device = 'cuda' if torch.cuda.is_available() else 'cpu'
>>> model.to(device)
>>> set_seed(42)

>>> text = "On what date was the Declaration of Independence officially signed?"


key: _node_content
string_value:
{"id_": "62bc6463-4d9a-48f9-a26d-4410fddfff5b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d253cf68-c4e4-4dd4-a365-c54e5baeb5d9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "1fede848df6a7e614f27a623f264e89083103e6c9eeeab3733e455c9b2033844", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c96ba282-3ac6-45c9-b90b-223467a4ab71", "node_type": "1", "metadata": {}, "hash": "0a672838ba810f7e6bf629f46d9f8e8ac75a6332b442431bf7cbf99cdea803c9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 26456, "end_char_idx": 29525, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
DoLa is achieved by contrasting the differences in logits obtained from final
layers versus earlier layers, thus amplify the factual knowledge localized to particular part of transformer layers.

Do the following two steps to activate DoLa decoding when calling the `model.generate` function:
1. Set the `dola_layers` argument, which can be either a string or a list of integers.
    - If set to a string, it can be one of `low`, `high`.
    - If set to a list of integers, it should be a list of layer indices between 0 and the total number of layers in the model. The 0-th layer is word embedding, and the 1st layer is the first transformer layer, and so on.
2. Set `repetition_penalty = 1.2` is suggested to reduce repetition in DoLa decoding.

See the following examples for DoLa decoding with the 32-layer LLaMA-7B model.

```python
>>> from transformers import AutoTokenizer, AutoModelForCausalLM, set_seed
>>> import torch

>>> tokenizer = AutoTokenizer.from_pretrained("huggyllama/llama-7b")
>>> model = AutoModelForCausalLM.from_pretrained("huggyllama/llama-7b", torch_dtype=torch.float16)
>>> device = 'cuda' if torch.cuda.is_available() else 'cpu'
>>> model.to(device)
>>> set_seed(42)

>>> text = "On what date was the Declaration of Independence officially signed?"
>>> inputs = tokenizer(text, return_tensors="pt").to(device)

# Vanilla greddy decoding
>>> vanilla_output = model.generate(**inputs, do_sample=False, max_new_tokens=50)
>>> tokenizer.batch_decode(vanilla_output[:, inputs.input_ids.shape[-1]:], skip_special_tokens=True)
['\nThe Declaration of Independence was signed on July 4, 1776.\nWhat was the date of the signing of the Declaration of Independence?\nThe Declaration of Independence was signed on July 4,']

# DoLa decoding with contrasting higher part of layers (layers 16,18,...,30)
>>> dola_high_output = model.generate(**inputs, do_sample=False, max_new_tokens=50, dola_layers='high')
>>> tokenizer.batch_decode(dola_high_output[:, inputs.input_ids.shape[-1]:], skip_special_tokens=True)
['\nJuly 4, 1776, when the Continental Congress voted to separate from Great Britain. The 56 delegates to the Continental Congress signed the Declaration on August 2, 1776.']

# DoLa decoding with contrasting specific layers (layers 28 and 30)
>>> dola_custom_output = model.generate(**inputs, do_sample=False, max_new_tokens=50, dola_layers=[28,30], repetition_penalty=1.2)
>>> tokenizer.batch_decode(dola_custom_output[:, inputs.input_ids.shape[-1]:], skip_special_tokens=True)
['\nIt was officially signed on 2 August 1776, when 56 members of the Second Continental Congress, representing the original 13 American colonies, voted unanimously for the resolution for independence. The 2']
```

#### Understanding the `dola_layers` argument

`dola_layers` stands for the candidate layers in premature layer selection, as described in the DoLa paper. The selected premature layer will be contrasted with the final layer.

Setting `dola_layers` to `'low'` or `'high'` will select the lower or higher part of the layers to contrast, respectively.


key: _node_content
string_value:
{"id_": "c96ba282-3ac6-45c9-b90b-223467a4ab71", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "22362ff1-ecd5-41c9-aea9-7cf4a47309c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "7105c0bea103cb7406f4c99f1ab3aeea30205cb37afa9391fa30eb11c9905643", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "62bc6463-4d9a-48f9-a26d-4410fddfff5b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 7027, "retrieve_doc": true, "source": "transformers"}, "hash": "3ba668e2c0aa7ac50d45cb380f431c6f3d4fb4b3baa3ba1c4a7cfe87ff03bd2c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 28568, "end_char_idx": 30594, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The 56 delegates to the Continental Congress signed the Declaration on August 2, 1776.']

# DoLa decoding with contrasting specific layers (layers 28 and 30)
>>> dola_custom_output = model.generate(**inputs, do_sample=False, max_new_tokens=50, dola_layers=[28,30], repetition_penalty=1.2)
>>> tokenizer.batch_decode(dola_custom_output[:, inputs.input_ids.shape[-1]:], skip_special_tokens=True)
['\nIt was officially signed on 2 August 1776, when 56 members of the Second Continental Congress, representing the original 13 American colonies, voted unanimously for the resolution for independence. The 2']
```

#### Understanding the `dola_layers` argument

`dola_layers` stands for the candidate layers in premature layer selection, as described in the DoLa paper. The selected premature layer will be contrasted with the final layer.

Setting `dola_layers` to `'low'` or `'high'` will select the lower or higher part of the layers to contrast, respectively.
- For `N`-layer models with `N <= 40` layers, the layers of `range(0, N // 2, 2)` and `range(N // 2, N, 2)` are used for `'low'` and `'high'` layers, respectively.
- For models with `N > 40` layers, the layers of `range(0, 20, 2)` and `range(N - 20, N, 2)` are used for `'low'` and `'high'` layers, respectively.
- If the model has tied word embeddings, we skip the word embeddings (0-th) layer and start from the 2nd layer, as the early exit from word embeddings will become identity function.
- Set the `dola_layers` to a list of integers for layer indices to contrast manually specified layers. For example, setting `dola_layers=[28,30]` will contrast the final layer (32-th layer) with the 28-th and 30-th layers.

The paper suggested that contrasting `'high'` layers to improve short-answer tasks like TruthfulQA, and contrasting `'low'` layers to improve all the other long-answer reasoning tasks, such as GSM8K, StrategyQA, FACTOR, and VicunaQA. Applying DoLa to smaller models like GPT-2 is not recommended, as the results shown in the Appendix N of the paper.


key: _node_content
string_value:
{"id_": "6ad264ae-7166-4b47-ac08-58452b031f8b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0268c0a2-6cc2-4024-ad84-06c37d8b4075", "node_type": "1", "metadata": {}, "hash": "544bbdf4780eebd34942a774eafb30a2ddac2e3559a77cfab2cbf38622b332d9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2682, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Glossary

This glossary defines general machine learning and 🤗 Transformers terms to help you better understand the
documentation.

## A

### attention mask

The attention mask is an optional argument used when batching sequences together.

<Youtube id="M6adb1j2jPI"/>

This argument indicates to the model which tokens should be attended to, and which should not.

For example, consider these two sequences:

```python
>>> from transformers import BertTokenizer

>>> tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-cased")

>>> sequence_a = "This is a short sequence."
>>> sequence_b = "This is a rather long sequence. It is at least longer than the sequence A."

>>> encoded_sequence_a = tokenizer(sequence_a)["input_ids"]
>>> encoded_sequence_b = tokenizer(sequence_b)["input_ids"]
```

The encoded versions have different lengths:

```python
>>> len(encoded_sequence_a), len(encoded_sequence_b)
(8, 19)
```

Therefore, we can't put them together in the same tensor as-is. The first sequence needs to be padded up to the length
of the second one, or the second one needs to be truncated down to the length of the first one.

In the first case, the list of IDs will be extended by the padding indices. We can pass a list to the tokenizer and ask
it to pad like this:

```python
>>> padded_sequences = tokenizer([sequence_a, sequence_b], padding=True)
```

We can see that 0s have been added on the right of the first sentence to make it the same length as the second one:

```python
>>> padded_sequences["input_ids"]
[[101, 1188, 1110, 170, 1603, 4954, 119, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [101, 1188, 1110, 170, 1897, 1263, 4954, 119, 1135, 1110, 1120, 1655, 2039, 1190, 1103, 4954, 138, 119, 102]]
```

This can then be converted into a tensor in PyTorch or TensorFlow. The attention mask is a binary tensor indicating the
position of the padded indices so that the model does not attend to them. For the [`BertTokenizer`], `1` indicates a
value that should be attended to, while `0` indicates a padded value. This attention mask is in the dictionary returned
by the tokenizer under the key "attention_mask":

```python
>>> padded_sequences["attention_mask"]
[[1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]
```

### autoencoding models

See [encoder models](#encoder-models) and [masked language modeling](#masked-language-modeling-mlm)

### autoregressive models

See [causal language modeling](#causal-language-modeling) and [decoder models](#decoder-models)

## B

### backbone

The backbone is the network (embeddings and layers) that outputs the raw hidden states or features.


key: _node_content
string_value:
{"id_": "0268c0a2-6cc2-4024-ad84-06c37d8b4075", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6ad264ae-7166-4b47-ac08-58452b031f8b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "d50d2bbc0fa78ec35e75e159d92ad0cf1c169b4269062722a0b5fb2da5761eca", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ecde5024-d591-49bf-988d-d89c68c3d704", "node_type": "1", "metadata": {}, "hash": "724e55a11efb5d580233e2ee07613afcf8e36cc3d6d11d6dfe70f31fd3d954e8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1800, "end_char_idx": 4934, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The attention mask is a binary tensor indicating the
position of the padded indices so that the model does not attend to them. For the [`BertTokenizer`], `1` indicates a
value that should be attended to, while `0` indicates a padded value. This attention mask is in the dictionary returned
by the tokenizer under the key "attention_mask":

```python
>>> padded_sequences["attention_mask"]
[[1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]
```

### autoencoding models

See [encoder models](#encoder-models) and [masked language modeling](#masked-language-modeling-mlm)

### autoregressive models

See [causal language modeling](#causal-language-modeling) and [decoder models](#decoder-models)

## B

### backbone

The backbone is the network (embeddings and layers) that outputs the raw hidden states or features. It is usually connected to a [head](#head) which accepts the features as its input to make a prediction. For example, [`ViTModel`] is a backbone without a specific head on top. Other models can also use [`VitModel`] as a backbone such as [DPT](model_doc/dpt).

## C

### causal language modeling

A pretraining task where the model reads the texts in order and has to predict the next word. It's usually done by
reading the whole sentence but using a mask inside the model to hide the future tokens at a certain timestep.

### channel

Color images are made up of some combination of values in three channels: red, green, and blue (RGB) and grayscale images only have one channel. In 🤗 Transformers, the channel can be the first or last dimension of an image's tensor: [`n_channels`, `height`, `width`] or [`height`, `width`, `n_channels`].

### connectionist temporal classification (CTC)

An algorithm which allows a model to learn without knowing exactly how the input and output are aligned; CTC calculates the distribution of all possible outputs for a given input and chooses the most likely output from it. CTC is commonly used in speech recognition tasks because speech doesn't always cleanly align with the transcript for a variety of reasons such as a speaker's different speech rates.

### convolution

A type of layer in a neural network where the input matrix is multiplied element-wise by a smaller matrix (kernel or filter) and the values are summed up in a new matrix. This is known as a convolutional operation which is repeated over the entire input matrix. Each operation is applied to a different segment of the input matrix. Convolutional neural networks (CNNs) are commonly used in computer vision.

## D

### DataParallel (DP)

Parallelism technique for training on multiple GPUs where the same setup is replicated multiple times, with each instance 
receiving a distinct data slice. The processing is done in parallel and all setups are synchronized at the end of each training step.

Learn more about how DataParallel works [here](perf_train_gpu_many#dataparallel-vs-distributeddataparallel).

### decoder input IDs

This input is specific to encoder-decoder models, and contains the input IDs that will be fed to the decoder.


key: _node_content
string_value:
{"id_": "ecde5024-d591-49bf-988d-d89c68c3d704", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0268c0a2-6cc2-4024-ad84-06c37d8b4075", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "8d3dbc685199699a6a99795d3e4200189233ee685ef8a2330b7c126480d34e90", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e0b4e6e8-3d52-49cf-aaec-1eb31da9813a", "node_type": "1", "metadata": {}, "hash": "d9a6c8657631fc50e8689ef751a97b0bb3966f8c4fdec5d1e8574e7595cf4a95", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3074, "end_char_idx": 6770, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
It's usually done by
reading the whole sentence but using a mask inside the model to hide the future tokens at a certain timestep.

### channel

Color images are made up of some combination of values in three channels: red, green, and blue (RGB) and grayscale images only have one channel. In 🤗 Transformers, the channel can be the first or last dimension of an image's tensor: [`n_channels`, `height`, `width`] or [`height`, `width`, `n_channels`].

### connectionist temporal classification (CTC)

An algorithm which allows a model to learn without knowing exactly how the input and output are aligned; CTC calculates the distribution of all possible outputs for a given input and chooses the most likely output from it. CTC is commonly used in speech recognition tasks because speech doesn't always cleanly align with the transcript for a variety of reasons such as a speaker's different speech rates.

### convolution

A type of layer in a neural network where the input matrix is multiplied element-wise by a smaller matrix (kernel or filter) and the values are summed up in a new matrix. This is known as a convolutional operation which is repeated over the entire input matrix. Each operation is applied to a different segment of the input matrix. Convolutional neural networks (CNNs) are commonly used in computer vision.

## D

### DataParallel (DP)

Parallelism technique for training on multiple GPUs where the same setup is replicated multiple times, with each instance 
receiving a distinct data slice. The processing is done in parallel and all setups are synchronized at the end of each training step.

Learn more about how DataParallel works [here](perf_train_gpu_many#dataparallel-vs-distributeddataparallel).

### decoder input IDs

This input is specific to encoder-decoder models, and contains the input IDs that will be fed to the decoder. These
inputs should be used for sequence to sequence tasks, such as translation or summarization, and are usually built in a
way specific to each model.

Most encoder-decoder models (BART, T5) create their `decoder_input_ids` on their own from the `labels`. In such models,
passing the `labels` is the preferred way to handle training.

Please check each model's docs to see how they handle these input IDs for sequence to sequence training.

### decoder models

Also referred to as autoregressive models, decoder models involve a pretraining task (called causal language modeling) where the model reads the texts in order and has to predict the next word. It's usually done by
reading the whole sentence with a mask to hide future tokens at a certain timestep.

<Youtube id="d_ixlCubqQw"/>

### deep learning (DL)

Machine learning algorithms which use neural networks with several layers.

## E

### encoder models

Also known as autoencoding models, encoder models take an input (such as text or images) and transform them into a condensed numerical representation called an embedding. Oftentimes, encoder models are pretrained using techniques like [masked language modeling](#masked-language-modeling-mlm), which masks parts of the input sequence and forces the model to create more meaningful representations.

<Youtube id="H39Z_720T5s"/>

## F

### feature extraction

The process of selecting and transforming raw data into a set of features that are more informative and useful for machine learning algorithms. Some examples of feature extraction include transforming raw text into word embeddings and extracting important features such as edges or shapes from image/video data.

### feed forward chunking

In each residual attention block in transformers the self-attention layer is usually followed by 2 feed forward layers.


key: _node_content
string_value:
{"id_": "e0b4e6e8-3d52-49cf-aaec-1eb31da9813a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ecde5024-d591-49bf-988d-d89c68c3d704", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "7c30f61cefcc05b2f9e95617a1d2ea6e59d939daf84116d46f81d95e4065ec80", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "59dff7d1-d5c4-4292-a950-990fc55169d5", "node_type": "1", "metadata": {}, "hash": "15a043b5d160ad96a20bb0c6851e88fdbfa41a9f0652116c56d57e630b2ff971", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4935, "end_char_idx": 8450, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
These
inputs should be used for sequence to sequence tasks, such as translation or summarization, and are usually built in a
way specific to each model.

Most encoder-decoder models (BART, T5) create their `decoder_input_ids` on their own from the `labels`. In such models,
passing the `labels` is the preferred way to handle training.

Please check each model's docs to see how they handle these input IDs for sequence to sequence training.

### decoder models

Also referred to as autoregressive models, decoder models involve a pretraining task (called causal language modeling) where the model reads the texts in order and has to predict the next word. It's usually done by
reading the whole sentence with a mask to hide future tokens at a certain timestep.

<Youtube id="d_ixlCubqQw"/>

### deep learning (DL)

Machine learning algorithms which use neural networks with several layers.

## E

### encoder models

Also known as autoencoding models, encoder models take an input (such as text or images) and transform them into a condensed numerical representation called an embedding. Oftentimes, encoder models are pretrained using techniques like [masked language modeling](#masked-language-modeling-mlm), which masks parts of the input sequence and forces the model to create more meaningful representations.

<Youtube id="H39Z_720T5s"/>

## F

### feature extraction

The process of selecting and transforming raw data into a set of features that are more informative and useful for machine learning algorithms. Some examples of feature extraction include transforming raw text into word embeddings and extracting important features such as edges or shapes from image/video data.

### feed forward chunking

In each residual attention block in transformers the self-attention layer is usually followed by 2 feed forward layers.
The intermediate embedding size of the feed forward layers is often bigger than the hidden size of the model (e.g., for
`google-bert/bert-base-uncased`).

For an input of size `[batch_size, sequence_length]`, the memory required to store the intermediate feed forward
embeddings `[batch_size, sequence_length, config.intermediate_size]` can account for a large fraction of the memory
use. The authors of [Reformer: The Efficient Transformer](https://arxiv.org/abs/2001.04451) noticed that since the
computation is independent of the `sequence_length` dimension, it is mathematically equivalent to compute the output
embeddings of both feed forward layers `[batch_size, config.hidden_size]_0, ..., [batch_size, config.hidden_size]_n`
individually and concat them afterward to `[batch_size, sequence_length, config.hidden_size]` with `n = sequence_length`, which trades increased computation time against reduced memory use, but yields a mathematically
**equivalent** result.

For models employing the function [`apply_chunking_to_forward`], the `chunk_size` defines the number of output
embeddings that are computed in parallel and thus defines the trade-off between memory and time complexity. If
`chunk_size` is set to 0, no feed forward chunking is done.

### finetuned models

Finetuning is a form of transfer learning which involves taking a pretrained model, freezing its weights, and replacing the output layer with a newly added [model head](#head). The model head is trained on your target dataset.

See the [Fine-tune a pretrained model](https://huggingface.co/docs/transformers/training) tutorial for more details, and learn how to fine-tune models with 🤗 Transformers.


key: _node_content
string_value:
{"id_": "59dff7d1-d5c4-4292-a950-990fc55169d5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e0b4e6e8-3d52-49cf-aaec-1eb31da9813a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "ec2ac1d172a8f79577956475146d6b37a9fff6f96f1f772ec492c91611d86553", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3dc3dd43-02b7-4b56-904f-f7cd5c4808e4", "node_type": "1", "metadata": {}, "hash": "6df752eec3d8dbb429d09288e9786890f9091a8faa09054fa713e908fc9c8ad4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6771, "end_char_idx": 9956, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The intermediate embedding size of the feed forward layers is often bigger than the hidden size of the model (e.g., for
`google-bert/bert-base-uncased`).

For an input of size `[batch_size, sequence_length]`, the memory required to store the intermediate feed forward
embeddings `[batch_size, sequence_length, config.intermediate_size]` can account for a large fraction of the memory
use. The authors of [Reformer: The Efficient Transformer](https://arxiv.org/abs/2001.04451) noticed that since the
computation is independent of the `sequence_length` dimension, it is mathematically equivalent to compute the output
embeddings of both feed forward layers `[batch_size, config.hidden_size]_0, ..., [batch_size, config.hidden_size]_n`
individually and concat them afterward to `[batch_size, sequence_length, config.hidden_size]` with `n = sequence_length`, which trades increased computation time against reduced memory use, but yields a mathematically
**equivalent** result.

For models employing the function [`apply_chunking_to_forward`], the `chunk_size` defines the number of output
embeddings that are computed in parallel and thus defines the trade-off between memory and time complexity. If
`chunk_size` is set to 0, no feed forward chunking is done.

### finetuned models

Finetuning is a form of transfer learning which involves taking a pretrained model, freezing its weights, and replacing the output layer with a newly added [model head](#head). The model head is trained on your target dataset.

See the [Fine-tune a pretrained model](https://huggingface.co/docs/transformers/training) tutorial for more details, and learn how to fine-tune models with 🤗 Transformers.

## H

### head

The model head refers to the last layer of a neural network that accepts the raw hidden states and projects them onto a different dimension. There is a different model head for each task. For example:

  * [`GPT2ForSequenceClassification`] is a sequence classification head - a linear layer - on top of the base [`GPT2Model`].
  * [`ViTForImageClassification`] is an image classification head - a linear layer on top of the final hidden state of the `CLS` token - on top of the base [`ViTModel`].
  * [`Wav2Vec2ForCTC`] is a language modeling head with [CTC](#connectionist-temporal-classification-ctc) on top of the base [`Wav2Vec2Model`].

## I

### image patch

Vision-based Transformers models split an image into smaller patches which are linearly embedded, and then passed as a sequence to the model. You can find the `patch_size` - or resolution - of the model in its configuration.

### inference

Inference is the process of evaluating a model on new data after training is complete. See the [Pipeline for inference](https://huggingface.co/docs/transformers/pipeline_tutorial) tutorial to learn how to perform inference with 🤗 Transformers.

### input IDs

The input ids are often the only required parameters to be passed to the model as input. They are token indices,
numerical representations of tokens building the sequences that will be used as input by the model.

<Youtube id="VFp38yj8h3A"/>

Each tokenizer works differently but the underlying mechanism remains the same.


key: _node_content
string_value:
{"id_": "3dc3dd43-02b7-4b56-904f-f7cd5c4808e4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "59dff7d1-d5c4-4292-a950-990fc55169d5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "dd8f820a11d1195ab3b7a90b535fbaaf376ae407cf7137254cc15ec4c4d8d580", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3e6885ef-0c8c-48e5-a12c-e2f16a1a3588", "node_type": "1", "metadata": {}, "hash": "eca63cd2b900dfb0620eaa2d0865e268c5a14fbecf1d51dc0d8f52eddcad1941", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8279, "end_char_idx": 11256, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
See the [Fine-tune a pretrained model](https://huggingface.co/docs/transformers/training) tutorial for more details, and learn how to fine-tune models with 🤗 Transformers.

## H

### head

The model head refers to the last layer of a neural network that accepts the raw hidden states and projects them onto a different dimension. There is a different model head for each task. For example:

  * [`GPT2ForSequenceClassification`] is a sequence classification head - a linear layer - on top of the base [`GPT2Model`].
  * [`ViTForImageClassification`] is an image classification head - a linear layer on top of the final hidden state of the `CLS` token - on top of the base [`ViTModel`].
  * [`Wav2Vec2ForCTC`] is a language modeling head with [CTC](#connectionist-temporal-classification-ctc) on top of the base [`Wav2Vec2Model`].

## I

### image patch

Vision-based Transformers models split an image into smaller patches which are linearly embedded, and then passed as a sequence to the model. You can find the `patch_size` - or resolution - of the model in its configuration.

### inference

Inference is the process of evaluating a model on new data after training is complete. See the [Pipeline for inference](https://huggingface.co/docs/transformers/pipeline_tutorial) tutorial to learn how to perform inference with 🤗 Transformers.

### input IDs

The input ids are often the only required parameters to be passed to the model as input. They are token indices,
numerical representations of tokens building the sequences that will be used as input by the model.

<Youtube id="VFp38yj8h3A"/>

Each tokenizer works differently but the underlying mechanism remains the same. Here's an example using the BERT
tokenizer, which is a [WordPiece](https://arxiv.org/pdf/1609.08144.pdf) tokenizer:

```python
>>> from transformers import BertTokenizer

>>> tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-cased")

>>> sequence = "A Titan RTX has 24GB of VRAM"
```

The tokenizer takes care of splitting the sequence into tokens available in the tokenizer vocabulary.

```python
>>> tokenized_sequence = tokenizer.tokenize(sequence)
```

The tokens are either words or subwords. Here for instance, "VRAM" wasn't in the model vocabulary, so it's been split
in "V", "RA" and "M". To indicate those tokens are not separate words but parts of the same word, a double-hash prefix
is added for "RA" and "M":

```python
>>> print(tokenized_sequence)
['A', 'Titan', 'R', '##T', '##X', 'has', '24', '##GB', 'of', 'V', '##RA', '##M']
```

These tokens can then be converted into IDs which are understandable by the model. This can be done by directly feeding the sentence to the tokenizer, which leverages the Rust implementation of [🤗 Tokenizers](https://github.com/huggingface/tokenizers) for peak performance.

```python
>>> inputs = tokenizer(sequence)
```

The tokenizer returns a dictionary with all the arguments necessary for its corresponding model to work properly.


key: _node_content
string_value:
{"id_": "3e6885ef-0c8c-48e5-a12c-e2f16a1a3588", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3dc3dd43-02b7-4b56-904f-f7cd5c4808e4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "df0e83b77324df0369bf3a0e8366210d4970df17e05ef3b7f78e5312a29593ab", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9f019310-de42-478e-bd20-d7ba77c4ba93", "node_type": "1", "metadata": {}, "hash": "fdac5ab48bed5bcc837fabca6cdd2170943cf020784648ccf6a7fd9936b481c4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9619, "end_char_idx": 12723, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### input IDs

The input ids are often the only required parameters to be passed to the model as input. They are token indices,
numerical representations of tokens building the sequences that will be used as input by the model.

<Youtube id="VFp38yj8h3A"/>

Each tokenizer works differently but the underlying mechanism remains the same. Here's an example using the BERT
tokenizer, which is a [WordPiece](https://arxiv.org/pdf/1609.08144.pdf) tokenizer:

```python
>>> from transformers import BertTokenizer

>>> tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-cased")

>>> sequence = "A Titan RTX has 24GB of VRAM"
```

The tokenizer takes care of splitting the sequence into tokens available in the tokenizer vocabulary.

```python
>>> tokenized_sequence = tokenizer.tokenize(sequence)
```

The tokens are either words or subwords. Here for instance, "VRAM" wasn't in the model vocabulary, so it's been split
in "V", "RA" and "M". To indicate those tokens are not separate words but parts of the same word, a double-hash prefix
is added for "RA" and "M":

```python
>>> print(tokenized_sequence)
['A', 'Titan', 'R', '##T', '##X', 'has', '24', '##GB', 'of', 'V', '##RA', '##M']
```

These tokens can then be converted into IDs which are understandable by the model. This can be done by directly feeding the sentence to the tokenizer, which leverages the Rust implementation of [🤗 Tokenizers](https://github.com/huggingface/tokenizers) for peak performance.

```python
>>> inputs = tokenizer(sequence)
```

The tokenizer returns a dictionary with all the arguments necessary for its corresponding model to work properly. The
token indices are under the key `input_ids`:

```python
>>> encoded_sequence = inputs["input_ids"]
>>> print(encoded_sequence)
[101, 138, 18696, 155, 1942, 3190, 1144, 1572, 13745, 1104, 159, 9664, 2107, 102]
```

Note that the tokenizer automatically adds "special tokens" (if the associated model relies on them) which are special
IDs the model sometimes uses.

If we decode the previous sequence of ids,

```python
>>> decoded_sequence = tokenizer.decode(encoded_sequence)
```

we will see

```python
>>> print(decoded_sequence)
[CLS] A Titan RTX has 24GB of VRAM [SEP]
```

because this is the way a [`BertModel`] is going to expect its inputs.

## L

### labels

The labels are an optional argument which can be passed in order for the model to compute the loss itself. These labels
should be the expected prediction of the model: it will use the standard loss in order to compute the loss between its
predictions and the expected value (the label).

These labels are different according to the model head, for example:

- For sequence classification models, ([`BertForSequenceClassification`]), the model expects a tensor of dimension
  `(batch_size)` with each value of the batch corresponding to the expected label of the entire sequence.
- For token classification models, ([`BertForTokenClassification`]), the model expects a tensor of dimension
  `(batch_size, seq_length)` with each value corresponding to the expected label of each individual token.


key: _node_content
string_value:
{"id_": "9f019310-de42-478e-bd20-d7ba77c4ba93", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3e6885ef-0c8c-48e5-a12c-e2f16a1a3588", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "1933c18c61867bbce6edd33e542e7fb8ac5bb0ecfbd7a4307a50265113b15e7a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fa7cf7bd-9a57-4107-8b04-3013d00c2b9b", "node_type": "1", "metadata": {}, "hash": "afe19266d1444271f74feff15a3fa592247d9bf7376625116a291f1313006444", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11095, "end_char_idx": 14472, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
>>> inputs = tokenizer(sequence)
```

The tokenizer returns a dictionary with all the arguments necessary for its corresponding model to work properly. The
token indices are under the key `input_ids`:

```python
>>> encoded_sequence = inputs["input_ids"]
>>> print(encoded_sequence)
[101, 138, 18696, 155, 1942, 3190, 1144, 1572, 13745, 1104, 159, 9664, 2107, 102]
```

Note that the tokenizer automatically adds "special tokens" (if the associated model relies on them) which are special
IDs the model sometimes uses.

If we decode the previous sequence of ids,

```python
>>> decoded_sequence = tokenizer.decode(encoded_sequence)
```

we will see

```python
>>> print(decoded_sequence)
[CLS] A Titan RTX has 24GB of VRAM [SEP]
```

because this is the way a [`BertModel`] is going to expect its inputs.

## L

### labels

The labels are an optional argument which can be passed in order for the model to compute the loss itself. These labels
should be the expected prediction of the model: it will use the standard loss in order to compute the loss between its
predictions and the expected value (the label).

These labels are different according to the model head, for example:

- For sequence classification models, ([`BertForSequenceClassification`]), the model expects a tensor of dimension
  `(batch_size)` with each value of the batch corresponding to the expected label of the entire sequence.
- For token classification models, ([`BertForTokenClassification`]), the model expects a tensor of dimension
  `(batch_size, seq_length)` with each value corresponding to the expected label of each individual token.
- For masked language modeling, ([`BertForMaskedLM`]), the model expects a tensor of dimension `(batch_size,
  seq_length)` with each value corresponding to the expected label of each individual token: the labels being the token
  ID for the masked token, and values to be ignored for the rest (usually -100).
- For sequence to sequence tasks, ([`BartForConditionalGeneration`], [`MBartForConditionalGeneration`]), the model
  expects a tensor of dimension `(batch_size, tgt_seq_length)` with each value corresponding to the target sequences
  associated with each input sequence. During training, both BART and T5 will make the appropriate
  `decoder_input_ids` and decoder attention masks internally. They usually do not need to be supplied. This does not
  apply to models leveraging the Encoder-Decoder framework.
- For image classification models, ([`ViTForImageClassification`]), the model expects a tensor of dimension
  `(batch_size)` with each value of the batch corresponding to the expected label of each individual image.
- For semantic segmentation models, ([`SegformerForSemanticSegmentation`]), the model expects a tensor of dimension
  `(batch_size, height, width)` with each value of the batch corresponding to the expected label of each individual pixel.
- For object detection models, ([`DetrForObjectDetection`]), the model expects a list of dictionaries with a
  `class_labels` and `boxes` key where each value of the batch corresponds to the expected label and number of bounding boxes of each individual image.
- For automatic speech recognition models, ([`Wav2Vec2ForCTC`]), the model expects a tensor of dimension `(batch_size,
  target_length)` with each value corresponding to the expected label of each individual token.


key: _node_content
string_value:
{"id_": "fa7cf7bd-9a57-4107-8b04-3013d00c2b9b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9f019310-de42-478e-bd20-d7ba77c4ba93", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "b76b06068b5d3a6828bbf3020e823848c23b7ada1dd1accfc406ca46bd0fe592", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "37e36f00-5f04-4834-b109-882016246fdd", "node_type": "1", "metadata": {}, "hash": "fc97bdac04d667f2f250bad3d94e1b66467153e4c48013d289ab0eb0399d1542", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12724, "end_char_idx": 16156, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- For masked language modeling, ([`BertForMaskedLM`]), the model expects a tensor of dimension `(batch_size,
  seq_length)` with each value corresponding to the expected label of each individual token: the labels being the token
  ID for the masked token, and values to be ignored for the rest (usually -100).
- For sequence to sequence tasks, ([`BartForConditionalGeneration`], [`MBartForConditionalGeneration`]), the model
  expects a tensor of dimension `(batch_size, tgt_seq_length)` with each value corresponding to the target sequences
  associated with each input sequence. During training, both BART and T5 will make the appropriate
  `decoder_input_ids` and decoder attention masks internally. They usually do not need to be supplied. This does not
  apply to models leveraging the Encoder-Decoder framework.
- For image classification models, ([`ViTForImageClassification`]), the model expects a tensor of dimension
  `(batch_size)` with each value of the batch corresponding to the expected label of each individual image.
- For semantic segmentation models, ([`SegformerForSemanticSegmentation`]), the model expects a tensor of dimension
  `(batch_size, height, width)` with each value of the batch corresponding to the expected label of each individual pixel.
- For object detection models, ([`DetrForObjectDetection`]), the model expects a list of dictionaries with a
  `class_labels` and `boxes` key where each value of the batch corresponds to the expected label and number of bounding boxes of each individual image.
- For automatic speech recognition models, ([`Wav2Vec2ForCTC`]), the model expects a tensor of dimension `(batch_size,
  target_length)` with each value corresponding to the expected label of each individual token.
  
<Tip>

Each model's labels may be different, so be sure to always check the documentation of each model for more information
about their specific labels!

</Tip>

The base models ([`BertModel`]) do not accept labels, as these are the base transformer models, simply outputting
features.

### large language models (LLM)

A generic term that refers to transformer language models (GPT-3, BLOOM, OPT) that were trained on a large quantity of data. These models also tend to have a large number of learnable parameters (e.g. 175 billion for GPT-3).

## M

### masked language modeling (MLM)

A pretraining task where the model sees a corrupted version of the texts, usually done by
masking some tokens randomly, and has to predict the original text.

### multimodal

A task that combines texts with another kind of inputs (for instance images).

## N

### Natural language generation (NLG)

All tasks related to generating text (for instance, [Write With Transformers](https://transformer.huggingface.co/), translation).

### Natural language processing (NLP)

A generic way to say "deal with texts".

### Natural language understanding (NLU)

All tasks related to understanding what is in a text (for instance classifying the
whole text, individual words).

## P

### pipeline

A pipeline in 🤗 Transformers is an abstraction referring to a series of steps that are executed in a specific order to preprocess and transform data and return a prediction from a model. Some example stages found in a pipeline might be data preprocessing, feature extraction, and normalization.

For more details, see [Pipelines for inference](https://huggingface.co/docs/transformers/pipeline_tutorial).


key: _node_content
string_value:
{"id_": "37e36f00-5f04-4834-b109-882016246fdd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fa7cf7bd-9a57-4107-8b04-3013d00c2b9b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "299fc7f9cfe3d7d6eac8eb834ae577d1bab5dde5f15b84184221376cf1eeccc9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c3deba38-04bb-48e0-95e5-933df4e1e733", "node_type": "1", "metadata": {}, "hash": "9087e322d1352cb5d100a624420de265143a8cc5cd06d843cebdbe240a9b3f75", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14476, "end_char_idx": 18172, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<Tip>

Each model's labels may be different, so be sure to always check the documentation of each model for more information
about their specific labels!

</Tip>

The base models ([`BertModel`]) do not accept labels, as these are the base transformer models, simply outputting
features.

### large language models (LLM)

A generic term that refers to transformer language models (GPT-3, BLOOM, OPT) that were trained on a large quantity of data. These models also tend to have a large number of learnable parameters (e.g. 175 billion for GPT-3).

## M

### masked language modeling (MLM)

A pretraining task where the model sees a corrupted version of the texts, usually done by
masking some tokens randomly, and has to predict the original text.

### multimodal

A task that combines texts with another kind of inputs (for instance images).

## N

### Natural language generation (NLG)

All tasks related to generating text (for instance, [Write With Transformers](https://transformer.huggingface.co/), translation).

### Natural language processing (NLP)

A generic way to say "deal with texts".

### Natural language understanding (NLU)

All tasks related to understanding what is in a text (for instance classifying the
whole text, individual words).

## P

### pipeline

A pipeline in 🤗 Transformers is an abstraction referring to a series of steps that are executed in a specific order to preprocess and transform data and return a prediction from a model. Some example stages found in a pipeline might be data preprocessing, feature extraction, and normalization.

For more details, see [Pipelines for inference](https://huggingface.co/docs/transformers/pipeline_tutorial).

### PipelineParallel (PP)

Parallelism technique in which the model is split up vertically (layer-level) across multiple GPUs, so that only one or 
several layers of the model are placed on a single GPU. Each GPU processes in parallel different stages of the pipeline 
and working on a small chunk of the batch. Learn more about how PipelineParallel works [here](perf_train_gpu_many#from-naive-model-parallelism-to-pipeline-parallelism).

### pixel values

A tensor of the numerical representations of an image that is passed to a model. The pixel values have a shape of [`batch_size`, `num_channels`, `height`, `width`], and are generated from an image processor.

### pooling

An operation that reduces a matrix into a smaller matrix, either by taking the maximum or average of the pooled dimension(s). Pooling layers are commonly found between convolutional layers to downsample the feature representation.

### position IDs

Contrary to RNNs that have the position of each token embedded within them, transformers are unaware of the position of
each token. Therefore, the position IDs (`position_ids`) are used by the model to identify each token's position in the
list of tokens.

They are an optional parameter. If no `position_ids` are passed to the model, the IDs are automatically created as
absolute positional embeddings.

Absolute positional embeddings are selected in the range `[0, config.max_position_embeddings - 1]`. Some models use
other types of positional embeddings, such as sinusoidal position embeddings or relative position embeddings.

### preprocessing

The task of preparing raw data into a format that can be easily consumed by machine learning models. For example, text is typically preprocessed by tokenization. To gain a better idea of what preprocessing looks like for other input types, check out the [Preprocess](https://huggingface.co/docs/transformers/preprocessing) tutorial.

### pretrained model

A model that has been pretrained on some data (for instance all of Wikipedia).


key: _node_content
string_value:
{"id_": "c3deba38-04bb-48e0-95e5-933df4e1e733", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "37e36f00-5f04-4834-b109-882016246fdd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "0062f1c99fa507afd9a6d60f5a01084332677be9e09bb1cbb24894d2c0a6dfb5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "996a532f-50ca-4acd-91de-b4b4ba13e0ff", "node_type": "1", "metadata": {}, "hash": "c3a7a4e15a3260f545c0a7a29a610fb2c510e14501dff3997f8930c8ac5ddd62", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16362, "end_char_idx": 19966, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Each GPU processes in parallel different stages of the pipeline 
and working on a small chunk of the batch. Learn more about how PipelineParallel works [here](perf_train_gpu_many#from-naive-model-parallelism-to-pipeline-parallelism).

### pixel values

A tensor of the numerical representations of an image that is passed to a model. The pixel values have a shape of [`batch_size`, `num_channels`, `height`, `width`], and are generated from an image processor.

### pooling

An operation that reduces a matrix into a smaller matrix, either by taking the maximum or average of the pooled dimension(s). Pooling layers are commonly found between convolutional layers to downsample the feature representation.

### position IDs

Contrary to RNNs that have the position of each token embedded within them, transformers are unaware of the position of
each token. Therefore, the position IDs (`position_ids`) are used by the model to identify each token's position in the
list of tokens.

They are an optional parameter. If no `position_ids` are passed to the model, the IDs are automatically created as
absolute positional embeddings.

Absolute positional embeddings are selected in the range `[0, config.max_position_embeddings - 1]`. Some models use
other types of positional embeddings, such as sinusoidal position embeddings or relative position embeddings.

### preprocessing

The task of preparing raw data into a format that can be easily consumed by machine learning models. For example, text is typically preprocessed by tokenization. To gain a better idea of what preprocessing looks like for other input types, check out the [Preprocess](https://huggingface.co/docs/transformers/preprocessing) tutorial.

### pretrained model

A model that has been pretrained on some data (for instance all of Wikipedia). Pretraining methods involve a
self-supervised objective, which can be reading the text and trying to predict the next word (see [causal language
modeling](#causal-language-modeling)) or masking some words and trying to predict them (see [masked language
modeling](#masked-language-modeling-mlm)). 

Speech and vision models have their own pretraining objectives. For example, Wav2Vec2 is a speech model pretrained on a contrastive task which requires the model to identify the "true" speech representation from a set of "false" speech representations. On the other hand, BEiT is a vision model pretrained on a masked image modeling task which masks some of the image patches and requires the model to predict the masked patches (similar to the masked language modeling objective).

## R

### recurrent neural network (RNN)

A type of model that uses a loop over a layer to process texts.

### representation learning

A subfield of machine learning which focuses on learning meaningful representations of raw data. Some examples of representation learning techniques include word embeddings, autoencoders, and Generative Adversarial Networks (GANs).

## S

### sampling rate

A measurement in hertz of the number of samples (the audio signal) taken per second. The sampling rate is a result of discretizing a continuous signal such as speech.

### self-attention

Each element of the input finds out which other elements of the input they should attend to.

### self-supervised learning 

A category of machine learning techniques in which a model creates its own learning objective from unlabeled data. It differs from [unsupervised learning](#unsupervised-learning) and [supervised learning](#supervised-learning) in that the learning process is supervised, but not explicitly from the user.


key: _node_content
string_value:
{"id_": "996a532f-50ca-4acd-91de-b4b4ba13e0ff", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c3deba38-04bb-48e0-95e5-933df4e1e733", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "4cac900d19ea22b9bba8005f58227656ef0fee0f3ad877650ccd9656603c3cc7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "198a25da-e6d4-416d-a0aa-489d7369ee4c", "node_type": "1", "metadata": {}, "hash": "84cf497223feeb5b11ef3bc6a703d790e376496aed1cab97791ff4b9be76a373", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18072, "end_char_idx": 21661, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### pretrained model

A model that has been pretrained on some data (for instance all of Wikipedia). Pretraining methods involve a
self-supervised objective, which can be reading the text and trying to predict the next word (see [causal language
modeling](#causal-language-modeling)) or masking some words and trying to predict them (see [masked language
modeling](#masked-language-modeling-mlm)). 

Speech and vision models have their own pretraining objectives. For example, Wav2Vec2 is a speech model pretrained on a contrastive task which requires the model to identify the "true" speech representation from a set of "false" speech representations. On the other hand, BEiT is a vision model pretrained on a masked image modeling task which masks some of the image patches and requires the model to predict the masked patches (similar to the masked language modeling objective).

## R

### recurrent neural network (RNN)

A type of model that uses a loop over a layer to process texts.

### representation learning

A subfield of machine learning which focuses on learning meaningful representations of raw data. Some examples of representation learning techniques include word embeddings, autoencoders, and Generative Adversarial Networks (GANs).

## S

### sampling rate

A measurement in hertz of the number of samples (the audio signal) taken per second. The sampling rate is a result of discretizing a continuous signal such as speech.

### self-attention

Each element of the input finds out which other elements of the input they should attend to.

### self-supervised learning 

A category of machine learning techniques in which a model creates its own learning objective from unlabeled data. It differs from [unsupervised learning](#unsupervised-learning) and [supervised learning](#supervised-learning) in that the learning process is supervised, but not explicitly from the user. 

One example of self-supervised learning is [masked language modeling](#masked-language-modeling-mlm), where a model is passed sentences with a proportion of its tokens removed and learns to predict the missing tokens.

### semi-supervised learning

A broad category of machine learning training techniques that leverages a small amount of labeled data with a larger quantity of unlabeled data to improve the accuracy of a model, unlike [supervised learning](#supervised-learning) and [unsupervised learning](#unsupervised-learning).

An example of a semi-supervised learning approach is "self-training", in which a model is trained on labeled data, and then used to make predictions on the unlabeled data. The portion of the unlabeled data that the model predicts with the most confidence gets added to the labeled dataset and used to retrain the model.

### sequence-to-sequence (seq2seq)

Models that generate a new sequence from an input, like translation models, or summarization models (such as
[Bart](model_doc/bart) or [T5](model_doc/t5)).

### Sharded DDP

Another name for the foundational [ZeRO](#zero-redundancy-optimizer-zero) concept as used by various other implementations of ZeRO.

### stride

In [convolution](#convolution) or [pooling](#pooling), the stride refers to the distance the kernel is moved over a matrix. A stride of 1 means the kernel is moved one pixel over at a time, and a stride of 2 means the kernel is moved two pixels over at a time.

### supervised learning

A form of model training that directly uses labeled data to correct and instruct model performance. Data is fed into the model being trained, and its predictions are compared to the known labels.


key: _node_content
string_value:
{"id_": "198a25da-e6d4-416d-a0aa-489d7369ee4c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "996a532f-50ca-4acd-91de-b4b4ba13e0ff", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "c0e605d22eb79396eb7f63a3f4d8468d051dbc8860751bfcbd4c6d2cc2be8bdb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "61f3b050-d4f9-4bda-b17b-8835f3a0d4b2", "node_type": "1", "metadata": {}, "hash": "8bb2e16160716b62be4ddb3f60bd6c01d50330634fa4e43a5753e4b5497d2c3e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19969, "end_char_idx": 23395, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
One example of self-supervised learning is [masked language modeling](#masked-language-modeling-mlm), where a model is passed sentences with a proportion of its tokens removed and learns to predict the missing tokens.

### semi-supervised learning

A broad category of machine learning training techniques that leverages a small amount of labeled data with a larger quantity of unlabeled data to improve the accuracy of a model, unlike [supervised learning](#supervised-learning) and [unsupervised learning](#unsupervised-learning).

An example of a semi-supervised learning approach is "self-training", in which a model is trained on labeled data, and then used to make predictions on the unlabeled data. The portion of the unlabeled data that the model predicts with the most confidence gets added to the labeled dataset and used to retrain the model.

### sequence-to-sequence (seq2seq)

Models that generate a new sequence from an input, like translation models, or summarization models (such as
[Bart](model_doc/bart) or [T5](model_doc/t5)).

### Sharded DDP

Another name for the foundational [ZeRO](#zero-redundancy-optimizer-zero) concept as used by various other implementations of ZeRO.

### stride

In [convolution](#convolution) or [pooling](#pooling), the stride refers to the distance the kernel is moved over a matrix. A stride of 1 means the kernel is moved one pixel over at a time, and a stride of 2 means the kernel is moved two pixels over at a time.

### supervised learning

A form of model training that directly uses labeled data to correct and instruct model performance. Data is fed into the model being trained, and its predictions are compared to the known labels. The model updates its weights based on how incorrect its predictions were, and the process is repeated to optimize model performance.

## T

### Tensor Parallelism (TP)

Parallelism technique for training on multiple GPUs in which each tensor is split up into multiple chunks, so instead of 
having the whole tensor reside on a single GPU, each shard of the tensor resides on its designated GPU. Shards gets 
processed separately and in parallel on different GPUs and the results are synced at the end of the processing step. 
This is what is sometimes called horizontal parallelism, as the splitting happens on horizontal level.
Learn more about Tensor Parallelism [here](perf_train_gpu_many#tensor-parallelism).

### token

A part of a sentence, usually a word, but can also be a subword (non-common words are often split in subwords) or a
punctuation symbol.

### token Type IDs

Some models' purpose is to do classification on pairs of sentences or question answering.

<Youtube id="0u3ioSwev3s"/>

These require two different sequences to be joined in a single "input_ids" entry, which usually is performed with the
help of special tokens, such as the classifier (`[CLS]`) and separator (`[SEP]`) tokens. For example, the BERT model
builds its two sequence input as such:

```python
>>> # [CLS] SEQUENCE_A [SEP] SEQUENCE_B [SEP]
```

We can use our tokenizer to automatically generate such a sentence by passing the two sequences to `tokenizer` as two
arguments (and not a list, like before) like this:

```python
>>> from transformers import BertTokenizer

>>> tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-cased")
>>> sequence_a = "HuggingFace is based in NYC"
>>> sequence_b = "Where is HuggingFace based?"


key: _node_content
string_value:
{"id_": "61f3b050-d4f9-4bda-b17b-8835f3a0d4b2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "198a25da-e6d4-416d-a0aa-489d7369ee4c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "40ff71a4608ac570415e381da7957498ab1cf92c44f2b0f0220d92ca4700ac2e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "63f2b57e-cdd7-4435-9464-3f473d5f7cae", "node_type": "1", "metadata": {}, "hash": "fde4c4570d7f72e7333237d6ca27730b8b3d4e54b6069a8bd84acfac5f481a8f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21662, "end_char_idx": 25025, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The model updates its weights based on how incorrect its predictions were, and the process is repeated to optimize model performance.

## T

### Tensor Parallelism (TP)

Parallelism technique for training on multiple GPUs in which each tensor is split up into multiple chunks, so instead of 
having the whole tensor reside on a single GPU, each shard of the tensor resides on its designated GPU. Shards gets 
processed separately and in parallel on different GPUs and the results are synced at the end of the processing step. 
This is what is sometimes called horizontal parallelism, as the splitting happens on horizontal level.
Learn more about Tensor Parallelism [here](perf_train_gpu_many#tensor-parallelism).

### token

A part of a sentence, usually a word, but can also be a subword (non-common words are often split in subwords) or a
punctuation symbol.

### token Type IDs

Some models' purpose is to do classification on pairs of sentences or question answering.

<Youtube id="0u3ioSwev3s"/>

These require two different sequences to be joined in a single "input_ids" entry, which usually is performed with the
help of special tokens, such as the classifier (`[CLS]`) and separator (`[SEP]`) tokens. For example, the BERT model
builds its two sequence input as such:

```python
>>> # [CLS] SEQUENCE_A [SEP] SEQUENCE_B [SEP]
```

We can use our tokenizer to automatically generate such a sentence by passing the two sequences to `tokenizer` as two
arguments (and not a list, like before) like this:

```python
>>> from transformers import BertTokenizer

>>> tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-cased")
>>> sequence_a = "HuggingFace is based in NYC"
>>> sequence_b = "Where is HuggingFace based?"

>>> encoded_dict = tokenizer(sequence_a, sequence_b)
>>> decoded = tokenizer.decode(encoded_dict["input_ids"])
```

which will return:

```python
>>> print(decoded)
[CLS] HuggingFace is based in NYC [SEP] Where is HuggingFace based? [SEP]
```

This is enough for some models to understand where one sequence ends and where another begins. However, other models,
such as BERT, also deploy token type IDs (also called segment IDs). They are represented as a binary mask identifying
the two types of sequence in the model.

The tokenizer returns this mask as the "token_type_ids" entry:

```python
>>> encoded_dict["token_type_ids"]
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1]
```

The first sequence, the "context" used for the question, has all its tokens represented by a `0`, whereas the second
sequence, corresponding to the "question", has all its tokens represented by a `1`.

Some models, like [`XLNetModel`] use an additional token represented by a `2`.

### transfer learning

A technique that involves taking a pretrained model and adapting it to a dataset specific to your task. Instead of training a model from scratch, you can leverage knowledge obtained from an existing model as a starting point. This speeds up the learning process and reduces the amount of training data needed.

### transformer

Self-attention based deep learning model architecture.

## U

### unsupervised learning

A form of model training in which data provided to the model is not labeled. Unsupervised learning techniques leverage statistical information of the data distribution to find patterns useful for the task at hand.


key: _node_content
string_value:
{"id_": "63f2b57e-cdd7-4435-9464-3f473d5f7cae", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8704bb74-48ac-4a96-bb9f-18f282312d0b", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "f56c1f47d023febaccd4366de50a37f229d0bc5b78993817627dfa5ee27ffdca", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "61f3b050-d4f9-4bda-b17b-8835f3a0d4b2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/glossary", "title": "Glossary", "tokens": 5921, "retrieve_doc": true, "source": "transformers"}, "hash": "d1a5e8fa2adf75b8c851275efc29474c809f3daeb549529ec7e674870d69989d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23397, "end_char_idx": 25502, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> encoded_dict = tokenizer(sequence_a, sequence_b)
>>> decoded = tokenizer.decode(encoded_dict["input_ids"])
```

which will return:

```python
>>> print(decoded)
[CLS] HuggingFace is based in NYC [SEP] Where is HuggingFace based? [SEP]
```

This is enough for some models to understand where one sequence ends and where another begins. However, other models,
such as BERT, also deploy token type IDs (also called segment IDs). They are represented as a binary mask identifying
the two types of sequence in the model.

The tokenizer returns this mask as the "token_type_ids" entry:

```python
>>> encoded_dict["token_type_ids"]
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1]
```

The first sequence, the "context" used for the question, has all its tokens represented by a `0`, whereas the second
sequence, corresponding to the "question", has all its tokens represented by a `1`.

Some models, like [`XLNetModel`] use an additional token represented by a `2`.

### transfer learning

A technique that involves taking a pretrained model and adapting it to a dataset specific to your task. Instead of training a model from scratch, you can leverage knowledge obtained from an existing model as a starting point. This speeds up the learning process and reduces the amount of training data needed.

### transformer

Self-attention based deep learning model architecture.

## U

### unsupervised learning

A form of model training in which data provided to the model is not labeled. Unsupervised learning techniques leverage statistical information of the data distribution to find patterns useful for the task at hand.

## Z

### Zero Redundancy Optimizer (ZeRO)

Parallelism technique which performs sharding of the tensors somewhat similar to [TensorParallel](#tensor-parallelism-tp), 
except the whole tensor gets reconstructed in time for a forward or backward computation, therefore the model doesn't need 
to be modified. This method also supports various offloading techniques to compensate for limited GPU memory. 
Learn more about ZeRO [here](perf_train_gpu_many#zero-data-parallelism).


key: _node_content
string_value:
{"id_": "0f8d2bf5-5eeb-4279-a8ed-0aff2cbe0dc4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tf_xla", "title": "XLA Integration for TensorFlow Models", "tokens": 2215, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b7c223ae-f823-4639-b6ca-5dbf824f810a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tf_xla", "title": "XLA Integration for TensorFlow Models", "tokens": 2215, "retrieve_doc": true, "source": "transformers"}, "hash": "1a1cb553522782826b9fe7e92743e3cb0d49fd01c349df97b7a1e56766ebb2b3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3b240436-617f-4a68-9929-fcaf100769f2", "node_type": "1", "metadata": {}, "hash": "cb1acb123801198053de1788131088b2385d44d17b82393016640fdedd49f18c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3326, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# XLA Integration for TensorFlow Models

[[open-in-colab]]

Accelerated Linear Algebra, dubbed XLA, is a compiler for accelerating the runtime of TensorFlow Models. From the [official documentation](https://www.tensorflow.org/xla):

XLA (Accelerated Linear Algebra) is a domain-specific compiler for linear algebra that can accelerate TensorFlow models with potentially no source code changes.

Using XLA in TensorFlow is simple – it comes packaged inside the `tensorflow` library, and it can be triggered with the `jit_compile` argument in any graph-creating function such as [`tf.function`](https://www.tensorflow.org/guide/intro_to_graphs). When using Keras methods like `fit()` and `predict()`, you can enable XLA simply by passing the `jit_compile` argument to `model.compile()`. However, XLA is not limited to these methods - it can also be used to accelerate any arbitrary `tf.function`.

Several TensorFlow methods in 🤗 Transformers have been rewritten to be XLA-compatible, including text generation for models such as [GPT2](https://huggingface.co/docs/transformers/model_doc/gpt2), [T5](https://huggingface.co/docs/transformers/model_doc/t5) and [OPT](https://huggingface.co/docs/transformers/model_doc/opt), as well as speech processing for models such as [Whisper](https://huggingface.co/docs/transformers/model_doc/whisper).

While the exact amount of speed-up is very much model-dependent, for TensorFlow text generation models inside 🤗 Transformers, we noticed a speed-up of ~100x. This document will explain how you can use XLA for these models to get the maximum amount of performance. We’ll also provide links to additional resources if you’re interested to learn more about the benchmarks and our design philosophy behind the XLA integration.

## Running TF functions with XLA

Let us consider the following model in TensorFlow:

```py
import tensorflow as tf

model = tf.keras.Sequential(
    [tf.keras.layers.Dense(10, input_shape=(10,), activation="relu"), tf.keras.layers.Dense(5, activation="softmax")]
)
```

The above model accepts inputs having a dimension of `(10, )`. We can use the model for running a forward pass like so:

```py
# Generate random inputs for the model.
batch_size = 16
input_vector_dim = 10
random_inputs = tf.random.normal((batch_size, input_vector_dim))

# Run a forward pass.
_ = model(random_inputs)
```

In order to run the forward pass with an XLA-compiled function, we’d need to do:

```py
xla_fn = tf.function(model, jit_compile=True)
_ = xla_fn(random_inputs)
```

The default `call()` function of the `model` is used for compiling the XLA graph. But if there’s any other model function you want to compile into XLA that’s also possible with:

```py
my_xla_fn = tf.function(model.my_xla_fn, jit_compile=True)
```

## Running a TF text generation model with XLA from 🤗 Transformers

To enable XLA-accelerated generation within 🤗 Transformers, you need to have a recent version of `transformers` installed. You can install it by running:

```bash
pip install transformers --upgrade
```

And then you can run the following code:

```py
import tensorflow as tf
from transformers import AutoTokenizer, TFAutoModelForCausalLM

# Will error if the minimal version of Transformers is not installed.
from transformers.utils import check_min_version

check_min_version("4.21.0")


key: _node_content
string_value:
{"id_": "3b240436-617f-4a68-9929-fcaf100769f2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tf_xla", "title": "XLA Integration for TensorFlow Models", "tokens": 2215, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b7c223ae-f823-4639-b6ca-5dbf824f810a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tf_xla", "title": "XLA Integration for TensorFlow Models", "tokens": 2215, "retrieve_doc": true, "source": "transformers"}, "hash": "1a1cb553522782826b9fe7e92743e3cb0d49fd01c349df97b7a1e56766ebb2b3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0f8d2bf5-5eeb-4279-a8ed-0aff2cbe0dc4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tf_xla", "title": "XLA Integration for TensorFlow Models", "tokens": 2215, "retrieve_doc": true, "source": "transformers"}, "hash": "555d3569a49ecc32d0a45a70e03c8171b6a01dd95d0568ac4dfb03ada9740b66", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b3afd6a6-84c4-4658-bbec-6d6227a424de", "node_type": "1", "metadata": {}, "hash": "cfbf756928d6bb56c92e13fa29acca586645e08f5d7727ab763d5f3ae43eef82", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3329, "end_char_idx": 6144, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2", padding_side="left", pad_token="</s>")
model = TFAutoModelForCausalLM.from_pretrained("openai-community/gpt2")
input_string = ["TensorFlow is"]

# One line to create an XLA generation function
xla_generate = tf.function(model.generate, jit_compile=True)

tokenized_input = tokenizer(input_string, return_tensors="tf")
generated_tokens = xla_generate(**tokenized_input, num_beams=2)

decoded_text = tokenizer.decode(generated_tokens[0], skip_special_tokens=True)
print(f"Generated -- {decoded_text}")
# Generated -- TensorFlow is an open-source, open-source, distributed-source application # framework for the
```

As you can notice, enabling XLA on `generate()` is just a single line of code. The rest of the code remains unchanged. However, there are a couple of gotchas in the above code snippet that are specific to XLA. You need to be aware of those to realize the speed-ups that XLA can bring in. We discuss these in the following section. 

## Gotchas to be aware of

When you are executing an XLA-enabled function (like `xla_generate()` above) for the first time, it will internally try to infer the computation graph, which is time-consuming.  This process is known as [“tracing”](https://www.tensorflow.org/guide/intro_to_graphs#when_is_a_function_tracing). 

You might notice that the generation time is not fast. Successive calls of `xla_generate()` (or any other XLA-enabled function) won’t have to infer the computation graph, given the inputs to the function follow the same shape with which the computation graph was initially built. While this is not a problem for modalities with fixed input shapes (e.g., images), you must pay attention if you are working with variable input shape modalities (e.g., text).

To ensure `xla_generate()` always operates with the same input shapes, you can specify the `padding` arguments when calling the tokenizer. 

```py
import tensorflow as tf
from transformers import AutoTokenizer, TFAutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2", padding_side="left", pad_token="</s>")
model = TFAutoModelForCausalLM.from_pretrained("openai-community/gpt2")
input_string = ["TensorFlow is"]

xla_generate = tf.function(model.generate, jit_compile=True)

# Here, we call the tokenizer with padding options.
tokenized_input = tokenizer(input_string, pad_to_multiple_of=8, padding=True, return_tensors="tf")

generated_tokens = xla_generate(**tokenized_input, num_beams=2)
decoded_text = tokenizer.decode(generated_tokens[0], skip_special_tokens=True)
print(f"Generated -- {decoded_text}")
```

This way, you can ensure that the inputs to `xla_generate()` will always receive inputs with the shape it was traced with and thus leading to speed-ups in the generation time.


key: _node_content
string_value:
{"id_": "b3afd6a6-84c4-4658-bbec-6d6227a424de", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tf_xla", "title": "XLA Integration for TensorFlow Models", "tokens": 2215, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b7c223ae-f823-4639-b6ca-5dbf824f810a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tf_xla", "title": "XLA Integration for TensorFlow Models", "tokens": 2215, "retrieve_doc": true, "source": "transformers"}, "hash": "1a1cb553522782826b9fe7e92743e3cb0d49fd01c349df97b7a1e56766ebb2b3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3b240436-617f-4a68-9929-fcaf100769f2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tf_xla", "title": "XLA Integration for TensorFlow Models", "tokens": 2215, "retrieve_doc": true, "source": "transformers"}, "hash": "c888148fe65e8f17de7876af736bbc958c901437af4a16b9d961cc498fbce2ca", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "546f46d1-5068-47e8-aff3-48c89fce22e5", "node_type": "1", "metadata": {}, "hash": "cc48ceb3ce183e859398a95b15f5e5efb054174a2d033fd3fb6da2477579bb88", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4548, "end_char_idx": 7631, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This process is known as [“tracing”](https://www.tensorflow.org/guide/intro_to_graphs#when_is_a_function_tracing). 

You might notice that the generation time is not fast. Successive calls of `xla_generate()` (or any other XLA-enabled function) won’t have to infer the computation graph, given the inputs to the function follow the same shape with which the computation graph was initially built. While this is not a problem for modalities with fixed input shapes (e.g., images), you must pay attention if you are working with variable input shape modalities (e.g., text).

To ensure `xla_generate()` always operates with the same input shapes, you can specify the `padding` arguments when calling the tokenizer. 

```py
import tensorflow as tf
from transformers import AutoTokenizer, TFAutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2", padding_side="left", pad_token="</s>")
model = TFAutoModelForCausalLM.from_pretrained("openai-community/gpt2")
input_string = ["TensorFlow is"]

xla_generate = tf.function(model.generate, jit_compile=True)

# Here, we call the tokenizer with padding options.
tokenized_input = tokenizer(input_string, pad_to_multiple_of=8, padding=True, return_tensors="tf")

generated_tokens = xla_generate(**tokenized_input, num_beams=2)
decoded_text = tokenizer.decode(generated_tokens[0], skip_special_tokens=True)
print(f"Generated -- {decoded_text}")
```

This way, you can ensure that the inputs to `xla_generate()` will always receive inputs with the shape it was traced with and thus leading to speed-ups in the generation time. You can verify this with the code below:

```py
import time
import tensorflow as tf
from transformers import AutoTokenizer, TFAutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2", padding_side="left", pad_token="</s>")
model = TFAutoModelForCausalLM.from_pretrained("openai-community/gpt2")

xla_generate = tf.function(model.generate, jit_compile=True)

for input_string in ["TensorFlow is", "TensorFlow is a", "TFLite is a"]:
    tokenized_input = tokenizer(input_string, pad_to_multiple_of=8, padding=True, return_tensors="tf")
    start = time.time_ns()
    generated_tokens = xla_generate(**tokenized_input, num_beams=2)
    end = time.time_ns()
    print(f"Execution time -- {(end - start) / 1e6:.1f} ms\n")
```

On a Tesla T4 GPU, you can expect the outputs like so:

```bash
Execution time -- 30819.6 ms

Execution time -- 79.0 ms

Execution time -- 78.9 ms
```
The first call to `xla_generate()` is time-consuming because of tracing, but the successive calls are orders of magnitude faster. Keep in mind that any change in the generation options at any point with trigger re-tracing and thus leading to slow-downs in the generation time. 

We didn’t cover all the text generation options 🤗 Transformers provides in this document. We encourage you to read the documentation for advanced use cases.

## Additional Resources

Here, we leave you with some additional resources if you want to delve deeper into XLA in 🤗 Transformers and in general.


key: _node_content
string_value:
{"id_": "546f46d1-5068-47e8-aff3-48c89fce22e5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tf_xla", "title": "XLA Integration for TensorFlow Models", "tokens": 2215, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b7c223ae-f823-4639-b6ca-5dbf824f810a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tf_xla", "title": "XLA Integration for TensorFlow Models", "tokens": 2215, "retrieve_doc": true, "source": "transformers"}, "hash": "1a1cb553522782826b9fe7e92743e3cb0d49fd01c349df97b7a1e56766ebb2b3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b3afd6a6-84c4-4658-bbec-6d6227a424de", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tf_xla", "title": "XLA Integration for TensorFlow Models", "tokens": 2215, "retrieve_doc": true, "source": "transformers"}, "hash": "ebd9964d6d614791cebecddb3be916e02fe4d8f8b4773728a645d221588d831a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6145, "end_char_idx": 8819, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can verify this with the code below:

```py
import time
import tensorflow as tf
from transformers import AutoTokenizer, TFAutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2", padding_side="left", pad_token="</s>")
model = TFAutoModelForCausalLM.from_pretrained("openai-community/gpt2")

xla_generate = tf.function(model.generate, jit_compile=True)

for input_string in ["TensorFlow is", "TensorFlow is a", "TFLite is a"]:
    tokenized_input = tokenizer(input_string, pad_to_multiple_of=8, padding=True, return_tensors="tf")
    start = time.time_ns()
    generated_tokens = xla_generate(**tokenized_input, num_beams=2)
    end = time.time_ns()
    print(f"Execution time -- {(end - start) / 1e6:.1f} ms\n")
```

On a Tesla T4 GPU, you can expect the outputs like so:

```bash
Execution time -- 30819.6 ms

Execution time -- 79.0 ms

Execution time -- 78.9 ms
```
The first call to `xla_generate()` is time-consuming because of tracing, but the successive calls are orders of magnitude faster. Keep in mind that any change in the generation options at any point with trigger re-tracing and thus leading to slow-downs in the generation time. 

We didn’t cover all the text generation options 🤗 Transformers provides in this document. We encourage you to read the documentation for advanced use cases.

## Additional Resources

Here, we leave you with some additional resources if you want to delve deeper into XLA in 🤗 Transformers and in general. 
 
* [This Colab Notebook](https://colab.research.google.com/github/huggingface/blog/blob/main/notebooks/91_tf_xla_generate.ipynb) provides an interactive demonstration if you want to fiddle with the XLA-compatible encoder-decoder (like [T5](https://huggingface.co/docs/transformers/model_doc/t5)) and decoder-only (like [GPT2](https://huggingface.co/docs/transformers/model_doc/gpt2)) text generation models. 
* [This blog post](https://huggingface.co/blog/tf-xla-generate) provides an overview of the comparison benchmarks for XLA-compatible models along with a friendly introduction to XLA in TensorFlow. 
* [This blog post](https://blog.tensorflow.org/2022/11/how-hugging-face-improved-text-generation-performance-with-xla.html) discusses our design philosophy behind adding XLA support to the TensorFlow models in 🤗 Transformers. 
* Recommended posts for learning more about XLA and TensorFlow graphs in general:
    * [XLA: Optimizing Compiler for Machine Learning](https://www.tensorflow.org/xla)
    * [Introduction to graphs and tf.function](https://www.tensorflow.org/guide/intro_to_graphs)
    * [Better performance with tf.function](https://www.tensorflow.org/guide/function)


key: _node_content
string_value:
{"id_": "16d3acdb-b9bc-495c-a816-a15c5c494b2d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "31e2b671-c1da-4e7d-a739-159aeeab8794", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "hash": "f772d7527dedbd81e5b79ca7b0a4e86d7e311ea09f51775da93e8621d3e6f5bf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "16d5115f-ed72-4d28-a70b-d954f9d16e09", "node_type": "1", "metadata": {}, "hash": "1420a39d4a15d3485de9266e9b857d1860c108999adaa6fc8a910a87a417a93e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3304, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# LLM inference optimization

Large language models (LLMs) have pushed text generation applications, such as chat and code completion models, to the next level by producing text that displays a high level of understanding and fluency. But what makes LLMs so powerful - namely their size - also presents challenges for inference.

Basic inference is slow because LLMs have to be called repeatedly to generate the next token. The input sequence increases as generation progresses, which takes longer and longer for the LLM to process. LLMs also have billions of parameters, making it a challenge to store and handle all those weights in memory.

This guide will show you how to use the optimization techniques available in Transformers to accelerate LLM inference.

> [!TIP]
> Hugging Face also provides [Text Generation Inference (TGI)](https://hf.co/docs/text-generation-inference), a library dedicated to deploying and serving highly optimized LLMs for inference. It includes more optimization features not included in Transformers, such as continuous batching for increasing throughput and tensor parallelism for multi-GPU inference.

## Static kv-cache and torch.compile

During decoding, a LLM computes the key-value (kv) values for each input token and since it is autoregressive, it computes the same kv values each time because the generated output becomes part of the input now. This is not very efficient because you're recomputing the same kv values each time.

To optimize this, you can use a kv-cache to store the past keys and values instead of recomputing them each time. However, since the kv-cache grows with each generation step and is dynamic, it prevents you from taking advantage of [torch.compile](./perf_torch_compile), a powerful optimization tool that fuses PyTorch code into fast and optimized kernels.

The *static kv-cache* solves this issue by pre-allocating the kv-cache size to a maximum value which allows you to combine it with torch.compile for up to a 4x speed up.

> [!WARNING]
> Currently, only [Llama](./model_doc/llama2) and a few other models support static kv-cache and torch.compile. Check [this issue](https://github.com/huggingface/transformers/issues/28981) for a live model compatibility list.

For this example, let's load the [Gemma](https://hf.co/google/gemma-2b) model.

```py
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("google/gemma-2b")
model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-2b", device_map="auto"
)
```

There are two ways you can configure the model to use a static kv-cache. For a 7B model on an A100, both methods get a 4x speed up in the forward pass. Your speed up may vary depending on the model size (larger models have a smaller speed up) and hardware. If you're using the [`~GenerationMixin.generate`] method, the speed up is ~3x. The forward pass (which still gets 4x speed up) is only a part of the whole [`~GenerationMixin.generate`] code.

<hfoptions id="static-kv">
<hfoption id="generation_config">

Access the model's `generation_config` attribute and set the `cache_implementation` to "static".

```py
model.generation_config.cache_implementation = "static"
```

Call torch.compile on the model to compile the forward pass with the static kv-cache.


key: _node_content
string_value:
{"id_": "16d5115f-ed72-4d28-a70b-d954f9d16e09", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "31e2b671-c1da-4e7d-a739-159aeeab8794", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "hash": "f772d7527dedbd81e5b79ca7b0a4e86d7e311ea09f51775da93e8621d3e6f5bf", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "16d3acdb-b9bc-495c-a816-a15c5c494b2d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "hash": "f75881295a726dc4bdf6f53175c048ea2c1b3f0345bc5381b0392541664e6004", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9416f38e-1a0f-4922-8425-a89abc7324f5", "node_type": "1", "metadata": {}, "hash": "d51094c491733fca5eadfe6128eb6dbf3b58524838592ddeb04714cbebd1a8be", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1829, "end_char_idx": 4791, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The *static kv-cache* solves this issue by pre-allocating the kv-cache size to a maximum value which allows you to combine it with torch.compile for up to a 4x speed up.

> [!WARNING]
> Currently, only [Llama](./model_doc/llama2) and a few other models support static kv-cache and torch.compile. Check [this issue](https://github.com/huggingface/transformers/issues/28981) for a live model compatibility list.

For this example, let's load the [Gemma](https://hf.co/google/gemma-2b) model.

```py
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("google/gemma-2b")
model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-2b", device_map="auto"
)
```

There are two ways you can configure the model to use a static kv-cache. For a 7B model on an A100, both methods get a 4x speed up in the forward pass. Your speed up may vary depending on the model size (larger models have a smaller speed up) and hardware. If you're using the [`~GenerationMixin.generate`] method, the speed up is ~3x. The forward pass (which still gets 4x speed up) is only a part of the whole [`~GenerationMixin.generate`] code.

<hfoptions id="static-kv">
<hfoption id="generation_config">

Access the model's `generation_config` attribute and set the `cache_implementation` to "static".

```py
model.generation_config.cache_implementation = "static"
```

Call torch.compile on the model to compile the forward pass with the static kv-cache.

```py
compiled_model = torch.compile(model, mode="reduce-overhead", fullgraph=True)
input_text = "The theory of special relativity states "
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")

outputs = compiled_model.generate(**input_ids)
tokenizer.batch_decode(outputs, skip_special_tokens=True)
['The theory of special relativity states 1. The speed of light is constant in all inertial reference']
```

Under the hood, `generate` will attempt to reuse the same cache object, removing the need for re-compilation at each call. However, if the batch size or the maximum output length increase between calls, the cache will have to be reinitialized, triggering a new compilation.

</hfoption>
<hfoption id="Static Cache">

A [`StaticCache`] object can be passed to the model's forward pass under the `past_key_values` argument, enabling the use of this object as a static kv-cache. Using this strategy, you can write your own function to decode the next token given the current token and position and cache position of previously generated tokens. You can also pass the [`StaticCache`] object to [`~GenerationMixin.generate`] and use it across calls, like you would do with a dynamic cache.

```py
from transformers import LlamaTokenizer, LlamaForCausalLM, StaticCache, logging
from transformers.testing_utils import CaptureLogger
import torch

prompts = [
    "Simply put, the theory of relativity states that ",
    "My favorite all time favorite condiment is ketchup.


key: _node_content
string_value:
{"id_": "9416f38e-1a0f-4922-8425-a89abc7324f5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "31e2b671-c1da-4e7d-a739-159aeeab8794", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "hash": "f772d7527dedbd81e5b79ca7b0a4e86d7e311ea09f51775da93e8621d3e6f5bf", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "16d5115f-ed72-4d28-a70b-d954f9d16e09", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "hash": "46d75b819e3738054dec5087128b19d5c01c50327995c7b9ddb80ff79464b393", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7f2058e1-e0c0-4c1e-af92-9a522511931d", "node_type": "1", "metadata": {}, "hash": "89fb2f3b85dac2e1ac66f99674d0ab181e6c1f15e4304f06566121bd9aa608c9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2992, "end_char_idx": 6184, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<hfoptions id="static-kv">
<hfoption id="generation_config">

Access the model's `generation_config` attribute and set the `cache_implementation` to "static".

```py
model.generation_config.cache_implementation = "static"
```

Call torch.compile on the model to compile the forward pass with the static kv-cache.

```py
compiled_model = torch.compile(model, mode="reduce-overhead", fullgraph=True)
input_text = "The theory of special relativity states "
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")

outputs = compiled_model.generate(**input_ids)
tokenizer.batch_decode(outputs, skip_special_tokens=True)
['The theory of special relativity states 1. The speed of light is constant in all inertial reference']
```

Under the hood, `generate` will attempt to reuse the same cache object, removing the need for re-compilation at each call. However, if the batch size or the maximum output length increase between calls, the cache will have to be reinitialized, triggering a new compilation.

</hfoption>
<hfoption id="Static Cache">

A [`StaticCache`] object can be passed to the model's forward pass under the `past_key_values` argument, enabling the use of this object as a static kv-cache. Using this strategy, you can write your own function to decode the next token given the current token and position and cache position of previously generated tokens. You can also pass the [`StaticCache`] object to [`~GenerationMixin.generate`] and use it across calls, like you would do with a dynamic cache.

```py
from transformers import LlamaTokenizer, LlamaForCausalLM, StaticCache, logging
from transformers.testing_utils import CaptureLogger
import torch

prompts = [
    "Simply put, the theory of relativity states that ",
    "My favorite all time favorite condiment is ketchup.",
]

NUM_TOKENS_TO_GENERATE = 40
torch_device = "cuda"

tokenizer = LlamaTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf", pad_token="</s>", padding_side="right")
model = LlamaForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf", device_map="sequential")
inputs = tokenizer(prompts, return_tensors="pt", padding=True).to(model.device)

def decode_one_tokens(model, cur_token, input_pos, cache_position, past_key_values):
    logits = model(
        cur_token,
        position_ids=input_pos,
        cache_position=cache_position,
        past_key_values=past_key_values,
        return_dict=False,
        use_cache=True
    )[0]
    new_token = torch.argmax(logits[:, -1], dim=-1)[:, None]
    return new_token
```

There are a few important things you must do to enable static kv-cache and torch.compile with the `StaticCache` method:

1. Initialize the [`StaticCache`] instance before using the model for inference. There you can configure parameters like the maximum batch size and sequence length.

2. Call torch.compile on the model to compile the forward pass with the static kv-cache.

3. Set `enable_math=True` in the [torch.backends.cuda.sdp_kernel](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html) context manager to enable the native PyTorch C++ implementation of scaled dot product attention to speed up inference even more.


key: _node_content
string_value:
{"id_": "7f2058e1-e0c0-4c1e-af92-9a522511931d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "31e2b671-c1da-4e7d-a739-159aeeab8794", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "hash": "f772d7527dedbd81e5b79ca7b0a4e86d7e311ea09f51775da93e8621d3e6f5bf", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9416f38e-1a0f-4922-8425-a89abc7324f5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "hash": "c1b812ef59a76251976ea439e6ed2145122f37b08ad0260938209e377e718ab1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "10821c23-a93c-4eba-9275-7b1ccf26a36f", "node_type": "1", "metadata": {}, "hash": "f9879ced09f4fccaa6e91d962e1d87cc1529c06568ec9f30c71cfdf26df20a26", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4513, "end_char_idx": 7825, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
from transformers import LlamaTokenizer, LlamaForCausalLM, StaticCache, logging
from transformers.testing_utils import CaptureLogger
import torch

prompts = [
    "Simply put, the theory of relativity states that ",
    "My favorite all time favorite condiment is ketchup.",
]

NUM_TOKENS_TO_GENERATE = 40
torch_device = "cuda"

tokenizer = LlamaTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf", pad_token="</s>", padding_side="right")
model = LlamaForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf", device_map="sequential")
inputs = tokenizer(prompts, return_tensors="pt", padding=True).to(model.device)

def decode_one_tokens(model, cur_token, input_pos, cache_position, past_key_values):
    logits = model(
        cur_token,
        position_ids=input_pos,
        cache_position=cache_position,
        past_key_values=past_key_values,
        return_dict=False,
        use_cache=True
    )[0]
    new_token = torch.argmax(logits[:, -1], dim=-1)[:, None]
    return new_token
```

There are a few important things you must do to enable static kv-cache and torch.compile with the `StaticCache` method:

1. Initialize the [`StaticCache`] instance before using the model for inference. There you can configure parameters like the maximum batch size and sequence length.

2. Call torch.compile on the model to compile the forward pass with the static kv-cache.

3. Set `enable_math=True` in the [torch.backends.cuda.sdp_kernel](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html) context manager to enable the native PyTorch C++ implementation of scaled dot product attention to speed up inference even more.

```py
batch_size, seq_length = inputs["input_ids"].shape
with torch.no_grad():
    past_key_values = StaticCache(
        config=model.config, max_batch_size=2, max_cache_len=4096, device=torch_device, dtype=model.dtype
    )
    cache_position = torch.arange(seq_length, device=torch_device)
    generated_ids = torch.zeros(
        batch_size, seq_length + NUM_TOKENS_TO_GENERATE + 1, dtype=torch.int, device=torch_device
    )
    generated_ids[:, cache_position] = inputs["input_ids"].to(torch_device).to(torch.int)

    logits = model(
        **inputs, cache_position=cache_position, past_key_values=past_key_values,return_dict=False, use_cache=True
    )[0]
    next_token = torch.argmax(logits[:, -1], dim=-1)[:, None]
    generated_ids[:, seq_length] = next_token[:, 0]

    decode_one_tokens = torch.compile(decode_one_tokens, mode="reduce-overhead", fullgraph=True)
    cache_position = torch.tensor([seq_length + 1], device=torch_device)
    for _ in range(1, NUM_TOKENS_TO_GENERATE):
        with torch.backends.cuda.sdp_kernel(enable_flash=False, enable_mem_efficient=False, enable_math=True):
            next_token = decode_one_tokens(model, next_token.clone(), None, cache_position, past_key_values)
            generated_ids[:, cache_position] = next_token.int()
        cache_position += 1

text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
text
['Simply put, the theory of relativity states that 1) the speed of light is constant, 2) the speed of light is the same for all observers, and 3) the laws of physics are the same for all observers.',
 'My favorite all time favorite condiment is ketchup.


key: _node_content
string_value:
{"id_": "10821c23-a93c-4eba-9275-7b1ccf26a36f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "31e2b671-c1da-4e7d-a739-159aeeab8794", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "hash": "f772d7527dedbd81e5b79ca7b0a4e86d7e311ea09f51775da93e8621d3e6f5bf", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7f2058e1-e0c0-4c1e-af92-9a522511931d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "hash": "a2f129aff6d586baa55ef59cea5469ddeea39421791a30f63dd2373f3ec2d67d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e5e2747c-5770-4047-a4f2-c3bd196c68a9", "node_type": "1", "metadata": {}, "hash": "d3fb57e868efc03fbb147f29fb32d46007fe9391013b8174827453d6e9056f16", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6186, "end_char_idx": 9507, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
batch_size, seq_length = inputs["input_ids"].shape
with torch.no_grad():
    past_key_values = StaticCache(
        config=model.config, max_batch_size=2, max_cache_len=4096, device=torch_device, dtype=model.dtype
    )
    cache_position = torch.arange(seq_length, device=torch_device)
    generated_ids = torch.zeros(
        batch_size, seq_length + NUM_TOKENS_TO_GENERATE + 1, dtype=torch.int, device=torch_device
    )
    generated_ids[:, cache_position] = inputs["input_ids"].to(torch_device).to(torch.int)

    logits = model(
        **inputs, cache_position=cache_position, past_key_values=past_key_values,return_dict=False, use_cache=True
    )[0]
    next_token = torch.argmax(logits[:, -1], dim=-1)[:, None]
    generated_ids[:, seq_length] = next_token[:, 0]

    decode_one_tokens = torch.compile(decode_one_tokens, mode="reduce-overhead", fullgraph=True)
    cache_position = torch.tensor([seq_length + 1], device=torch_device)
    for _ in range(1, NUM_TOKENS_TO_GENERATE):
        with torch.backends.cuda.sdp_kernel(enable_flash=False, enable_mem_efficient=False, enable_math=True):
            next_token = decode_one_tokens(model, next_token.clone(), None, cache_position, past_key_values)
            generated_ids[:, cache_position] = next_token.int()
        cache_position += 1

text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
text
['Simply put, the theory of relativity states that 1) the speed of light is constant, 2) the speed of light is the same for all observers, and 3) the laws of physics are the same for all observers.',
 'My favorite all time favorite condiment is ketchup. I love it on everything. I love it on my eggs, my fries, my chicken, my burgers, my hot dogs, my sandwiches, my salads, my p']
```

> [!TIP]
> If you want to reuse the [`StaticCache`] object on a new prompt, be sure to reset its contents with the `.reset()` method

</hfoption>
</hfoptions>

## Speculative decoding

> [!TIP]
> For a more in-depth explanation, take a look at the [Assisted Generation: a new direction toward low-latency text generation](https://hf.co/blog/assisted-generation) blog post!

Another issue with autoregression is that for each input token you need to load the model weights each time during the forward pass. This is slow and cumbersome for LLMs which have billions of parameters. Speculative decoding alleviates this slowdown by using a second smaller and faster assistant model to generate candidate tokens that are verified by the larger LLM in a single forward pass. If the verified tokens are correct, the LLM essentially gets them for "free" without having to generate them itself. There is no degradation in accuracy because the verification forward pass ensures the same outputs are generated as if the LLM had generated them on its own.

To get the largest speed up, the assistant model should be a lot smaller than the LLM so that it can generate tokens quickly. The assistant and LLM model must also share the same tokenizer to avoid re-encoding and decoding tokens.

> [!WARNING]
> Speculative decoding is only supported for the greedy search and sampling decoding strategies, and it also doesn't support batched inputs.

Enable speculative decoding by loading an assistant model and passing it to the [`~GenerationMixin.generate`] method.


key: _node_content
string_value:
{"id_": "e5e2747c-5770-4047-a4f2-c3bd196c68a9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "31e2b671-c1da-4e7d-a739-159aeeab8794", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "hash": "f772d7527dedbd81e5b79ca7b0a4e86d7e311ea09f51775da93e8621d3e6f5bf", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "10821c23-a93c-4eba-9275-7b1ccf26a36f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "hash": "c46eb2a15942b5315b2c4c3aee3ff4d1f227aef188c473c4eca8b359b1759105", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2ca01236-90c4-4b47-945a-f35477c1bd90", "node_type": "1", "metadata": {}, "hash": "76f67fe6a8a479472994d93f390addc7e3fb3746a26ee2951655e09a39eab1a2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7769, "end_char_idx": 10446, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
',
 'My favorite all time favorite condiment is ketchup. I love it on everything. I love it on my eggs, my fries, my chicken, my burgers, my hot dogs, my sandwiches, my salads, my p']
```

> [!TIP]
> If you want to reuse the [`StaticCache`] object on a new prompt, be sure to reset its contents with the `.reset()` method

</hfoption>
</hfoptions>

## Speculative decoding

> [!TIP]
> For a more in-depth explanation, take a look at the [Assisted Generation: a new direction toward low-latency text generation](https://hf.co/blog/assisted-generation) blog post!

Another issue with autoregression is that for each input token you need to load the model weights each time during the forward pass. This is slow and cumbersome for LLMs which have billions of parameters. Speculative decoding alleviates this slowdown by using a second smaller and faster assistant model to generate candidate tokens that are verified by the larger LLM in a single forward pass. If the verified tokens are correct, the LLM essentially gets them for "free" without having to generate them itself. There is no degradation in accuracy because the verification forward pass ensures the same outputs are generated as if the LLM had generated them on its own.

To get the largest speed up, the assistant model should be a lot smaller than the LLM so that it can generate tokens quickly. The assistant and LLM model must also share the same tokenizer to avoid re-encoding and decoding tokens.

> [!WARNING]
> Speculative decoding is only supported for the greedy search and sampling decoding strategies, and it also doesn't support batched inputs.

Enable speculative decoding by loading an assistant model and passing it to the [`~GenerationMixin.generate`] method.

<hfoptions id="spec-decoding">
<hfoption id="greedy search">

```py
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained("facebook/opt-1.3b")
inputs = tokenizer("Einstein's theory of relativity states", return_tensors="pt").to(device)

model = AutoModelForCausalLM.from_pretrained("facebook/opt-1.3b").to(device)
assistant_model = AutoModelForCausalLM.from_pretrained("facebook/opt-125m").to(device)
outputs = model.generate(**inputs, assistant_model=assistant_model)
tokenizer.batch_decode(outputs, skip_special_tokens=True)
["Einstein's theory of relativity states that the speed of light is constant.    "]
```

</hfoption>
<hfoption id="sampling">

For speculative sampling decoding, add the `do_sample` and `temperature` parameters to the [`~GenerationMixin.generate`] method in addition to the assistant model.


key: _node_content
string_value:
{"id_": "2ca01236-90c4-4b47-945a-f35477c1bd90", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "31e2b671-c1da-4e7d-a739-159aeeab8794", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "hash": "f772d7527dedbd81e5b79ca7b0a4e86d7e311ea09f51775da93e8621d3e6f5bf", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e5e2747c-5770-4047-a4f2-c3bd196c68a9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "hash": "2d42446bfae38387ee77e1ab064b6daf7721ba43dbd183decffd9fe833a8ed8b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d5d1bdeb-3a18-450c-a4e6-072471bc7a03", "node_type": "1", "metadata": {}, "hash": "0248ab152570f78fe982fae2b54ae145f941996d0e9198c5bc89fb8b8a4f81e0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8727, "end_char_idx": 11780, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If the verified tokens are correct, the LLM essentially gets them for "free" without having to generate them itself. There is no degradation in accuracy because the verification forward pass ensures the same outputs are generated as if the LLM had generated them on its own.

To get the largest speed up, the assistant model should be a lot smaller than the LLM so that it can generate tokens quickly. The assistant and LLM model must also share the same tokenizer to avoid re-encoding and decoding tokens.

> [!WARNING]
> Speculative decoding is only supported for the greedy search and sampling decoding strategies, and it also doesn't support batched inputs.

Enable speculative decoding by loading an assistant model and passing it to the [`~GenerationMixin.generate`] method.

<hfoptions id="spec-decoding">
<hfoption id="greedy search">

```py
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained("facebook/opt-1.3b")
inputs = tokenizer("Einstein's theory of relativity states", return_tensors="pt").to(device)

model = AutoModelForCausalLM.from_pretrained("facebook/opt-1.3b").to(device)
assistant_model = AutoModelForCausalLM.from_pretrained("facebook/opt-125m").to(device)
outputs = model.generate(**inputs, assistant_model=assistant_model)
tokenizer.batch_decode(outputs, skip_special_tokens=True)
["Einstein's theory of relativity states that the speed of light is constant.    "]
```

</hfoption>
<hfoption id="sampling">

For speculative sampling decoding, add the `do_sample` and `temperature` parameters to the [`~GenerationMixin.generate`] method in addition to the assistant model.

```py
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained("facebook/opt-1.3b")
inputs = tokenizer("Einstein's theory of relativity states", return_tensors="pt").to(device)

model = AutoModelForCausalLM.from_pretrained("facebook/opt-1.3b").to(device)
assistant_model = AutoModelForCausalLM.from_pretrained("facebook/opt-125m").to(device)
outputs = model.generate(**inputs, assistant_model=assistant_model, do_sample=True, temperature=0.7)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
["Einstein's theory of relativity states that motion in the universe is not a straight line.\n"]
```

</hfoption>
</hfoptions>

### Prompt lookup decoding

Prompt lookup decoding is a variant of speculative decoding that is also compatible with greedy search and sampling. Prompt lookup works especially well for input-grounded tasks - such as summarization - where there is often overlapping words between the prompt and output. These overlapping n-grams are used as the LLM candidate tokens.

To enable prompt lookup decoding, specify the number of tokens that should be overlapping in the `prompt_lookup_num_tokens` parameter. Then you can pass this parameter to the [`~GenerationMixin.generate`] method.


key: _node_content
string_value:
{"id_": "d5d1bdeb-3a18-450c-a4e6-072471bc7a03", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "31e2b671-c1da-4e7d-a739-159aeeab8794", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "hash": "f772d7527dedbd81e5b79ca7b0a4e86d7e311ea09f51775da93e8621d3e6f5bf", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2ca01236-90c4-4b47-945a-f35477c1bd90", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "hash": "bb6074c6630791366f99d1f8ea3952118985e05e2cc4837505bade4f7de3a9ab", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5715a4d1-cafe-4fb7-a550-946606273ddd", "node_type": "1", "metadata": {}, "hash": "b56d33337d8165730f160893853b895275fe35aaf5fc8e09916092fe532a7212", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10240, "end_char_idx": 13616, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```

</hfoption>
<hfoption id="sampling">

For speculative sampling decoding, add the `do_sample` and `temperature` parameters to the [`~GenerationMixin.generate`] method in addition to the assistant model.

```py
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained("facebook/opt-1.3b")
inputs = tokenizer("Einstein's theory of relativity states", return_tensors="pt").to(device)

model = AutoModelForCausalLM.from_pretrained("facebook/opt-1.3b").to(device)
assistant_model = AutoModelForCausalLM.from_pretrained("facebook/opt-125m").to(device)
outputs = model.generate(**inputs, assistant_model=assistant_model, do_sample=True, temperature=0.7)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
["Einstein's theory of relativity states that motion in the universe is not a straight line.\n"]
```

</hfoption>
</hfoptions>

### Prompt lookup decoding

Prompt lookup decoding is a variant of speculative decoding that is also compatible with greedy search and sampling. Prompt lookup works especially well for input-grounded tasks - such as summarization - where there is often overlapping words between the prompt and output. These overlapping n-grams are used as the LLM candidate tokens.

To enable prompt lookup decoding, specify the number of tokens that should be overlapping in the `prompt_lookup_num_tokens` parameter. Then you can pass this parameter to the [`~GenerationMixin.generate`] method.

<hfoptions id="pld">
<hfoption id="greedy decoding">

```py
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained("facebook/opt-1.3b")
inputs = tokenizer("The second law of thermodynamics states", return_tensors="pt").to(device)

model = AutoModelForCausalLM.from_pretrained("facebook/opt-1.3b").to(device)
assistant_model = AutoModelForCausalLM.from_pretrained("facebook/opt-125m").to(device)
outputs = model.generate(**inputs, prompt_lookup_num_tokens=3)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
['The second law of thermodynamics states that entropy increases with temperature.      ']
```

</hfoption>
<hfoption id="sampling">

For prompt lookup decoding with sampling, add the `do_sample` and `temperature` parameters to the [`~GenerationMixin.generate`] method.

```py
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained("facebook/opt-1.3b")
inputs = tokenizer("The second law of thermodynamics states", return_tensors="pt").to(device)

model = AutoModelForCausalLM.from_pretrained("facebook/opt-1.3b").to(device)
outputs = model.generate(**inputs, prompt_lookup_num_tokens=3, do_sample=True, temperature=0.7)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
["The second law of thermodynamics states that energy cannot be created nor destroyed. It's not a"]
```

</hfoption>
</hfoptions>

## Attention optimizations

A known issue with transformer models is that the self-attention mechanism grows quadratically in compute and memory with the number of input tokens. This limitation is only magnified in LLMs which handles much longer sequences.


key: _node_content
string_value:
{"id_": "5715a4d1-cafe-4fb7-a550-946606273ddd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "31e2b671-c1da-4e7d-a739-159aeeab8794", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "hash": "f772d7527dedbd81e5b79ca7b0a4e86d7e311ea09f51775da93e8621d3e6f5bf", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d5d1bdeb-3a18-450c-a4e6-072471bc7a03", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "hash": "7556aa06d206ae99d3b46c24480b794fab810e696c5c37b77322407c9c881eb4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cf91a09c-baa0-4572-91b5-d2cf8671027c", "node_type": "1", "metadata": {}, "hash": "9e2ed46688312f8ca138d030807a7268e277b8174b71ebe64bc144ba1f0264da", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12515, "end_char_idx": 15999, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```

</hfoption>
<hfoption id="sampling">

For prompt lookup decoding with sampling, add the `do_sample` and `temperature` parameters to the [`~GenerationMixin.generate`] method.

```py
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained("facebook/opt-1.3b")
inputs = tokenizer("The second law of thermodynamics states", return_tensors="pt").to(device)

model = AutoModelForCausalLM.from_pretrained("facebook/opt-1.3b").to(device)
outputs = model.generate(**inputs, prompt_lookup_num_tokens=3, do_sample=True, temperature=0.7)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
["The second law of thermodynamics states that energy cannot be created nor destroyed. It's not a"]
```

</hfoption>
</hfoptions>

## Attention optimizations

A known issue with transformer models is that the self-attention mechanism grows quadratically in compute and memory with the number of input tokens. This limitation is only magnified in LLMs which handles much longer sequences. To address this, try FlashAttention2 or PyTorch's scaled dot product attention (SDPA), which are more memory efficient attention implementations and can accelerate inference.

### FlashAttention-2

FlashAttention and [FlashAttention-2](./perf_infer_gpu_one#flashattention-2) break up the attention computation into smaller chunks and reduces the number of intermediate read/write operations to GPU memory to speed up inference. FlashAttention-2 improves on the original FlashAttention algorithm by also parallelizing over sequence length dimension and better partitioning work on the hardware to reduce synchronization and communication overhead.

To use FlashAttention-2, set `attn_implementation="flash_attention_2"` in the [`~PreTrainedModel.from_pretrained`] method.

```py
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

quant_config = BitsAndBytesConfig(load_in_8bit=True)
model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-2b",
    quantization_config=quant_config,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)
```

### PyTorch scaled dot product attention

Scaled dot product attention (SDPA) is automatically enabled in PyTorch 2.0 and it supports FlashAttention, xFormers, and PyTorch's C++ implementation. SDPA chooses the most performant attention algorithm if you're using a CUDA backend. For other backends, SDPA defaults to the PyTorch C++ implementation.

> [!TIP]
> SDPA supports FlashAttention-2 as long as you have the latest PyTorch version installed.

Use the [torch.backends.cuda.sdp_kernel](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html) context manager to explicitly enable or disable any of the three attention algorithms. For example, set `enable_flash=True` to enable FlashAttention.

```py
import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-2b",
    torch_dtype=torch.bfloat16,
)

with torch.backends.cuda.sdp_kernel(enable_flash=True, enable_math=False, enable_mem_efficient=False):
    outputs = model.generate(**inputs)
```

## Quantization

Quantization reduces the size of the LLM weights by storing them in a lower precision. This translates to lower memory usage and makes loading LLMs for inference more accessible if you're constrained by your GPUs memory.


key: _node_content
string_value:
{"id_": "cf91a09c-baa0-4572-91b5-d2cf8671027c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "31e2b671-c1da-4e7d-a739-159aeeab8794", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "hash": "f772d7527dedbd81e5b79ca7b0a4e86d7e311ea09f51775da93e8621d3e6f5bf", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5715a4d1-cafe-4fb7-a550-946606273ddd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "hash": "96cbdac98ece6ba68f4252bb72c70c2c3184e77a86d8952300f7f2cec8aab5c8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f6a6508b-4fe2-4bb7-81b2-9e47463a478a", "node_type": "1", "metadata": {}, "hash": "5e16265a671d5180fb197170a1fdd86732e0b971c90b35594dc39477ad678e0d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14389, "end_char_idx": 17243, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

quant_config = BitsAndBytesConfig(load_in_8bit=True)
model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-2b",
    quantization_config=quant_config,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)
```

### PyTorch scaled dot product attention

Scaled dot product attention (SDPA) is automatically enabled in PyTorch 2.0 and it supports FlashAttention, xFormers, and PyTorch's C++ implementation. SDPA chooses the most performant attention algorithm if you're using a CUDA backend. For other backends, SDPA defaults to the PyTorch C++ implementation.

> [!TIP]
> SDPA supports FlashAttention-2 as long as you have the latest PyTorch version installed.

Use the [torch.backends.cuda.sdp_kernel](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html) context manager to explicitly enable or disable any of the three attention algorithms. For example, set `enable_flash=True` to enable FlashAttention.

```py
import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-2b",
    torch_dtype=torch.bfloat16,
)

with torch.backends.cuda.sdp_kernel(enable_flash=True, enable_math=False, enable_mem_efficient=False):
    outputs = model.generate(**inputs)
```

## Quantization

Quantization reduces the size of the LLM weights by storing them in a lower precision. This translates to lower memory usage and makes loading LLMs for inference more accessible if you're constrained by your GPUs memory. If you aren't limited by your GPU, you don't necessarily need to quantize your model because it can incur a small latency cost (except for AWQ and fused AWQ modules) due to the extra step required to quantize and dequantize the weights.

> [!TIP]
> There are many quantization libraries (see the [Quantization](./quantization) guide for more details) available, such as Quanto, AQLM, AWQ, and AutoGPTQ. Feel free to try them out and see which one works best for your use case. We also recommend reading the [Overview of natively supported quantization schemes in 🤗 Transformers](https://hf.co/blog/overview-quantization-transformers) blog post which compares AutoGPTQ and bitsandbytes.

Use the Model Memory Calculator below to estimate and compare how much memory is required to load a model. For example, try estimating how much memory it costs to load [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1).

<iframe
	src="https://hf-accelerate-model-memory-usage.hf.space"
	frameborder="0"
	width="850"
	height="450"
></iframe>

To load Mistral-7B-v0.1 in half-precision, set the `torch_dtype` parameter in the [`~transformers.AutoModelForCausalLM.from_pretrained`] method to `torch.bfloat16`. This requires 13.74GB of memory.


key: _node_content
string_value:
{"id_": "f6a6508b-4fe2-4bb7-81b2-9e47463a478a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "31e2b671-c1da-4e7d-a739-159aeeab8794", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "hash": "f772d7527dedbd81e5b79ca7b0a4e86d7e311ea09f51775da93e8621d3e6f5bf", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cf91a09c-baa0-4572-91b5-d2cf8671027c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_optims", "title": "LLM inference optimization", "tokens": 4298, "retrieve_doc": true, "source": "transformers"}, "hash": "36f1d47743d87d26625d35ef38f16d0030739c333e9eede66d3589340a04e769", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15866, "end_char_idx": 17988, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This translates to lower memory usage and makes loading LLMs for inference more accessible if you're constrained by your GPUs memory. If you aren't limited by your GPU, you don't necessarily need to quantize your model because it can incur a small latency cost (except for AWQ and fused AWQ modules) due to the extra step required to quantize and dequantize the weights.

> [!TIP]
> There are many quantization libraries (see the [Quantization](./quantization) guide for more details) available, such as Quanto, AQLM, AWQ, and AutoGPTQ. Feel free to try them out and see which one works best for your use case. We also recommend reading the [Overview of natively supported quantization schemes in 🤗 Transformers](https://hf.co/blog/overview-quantization-transformers) blog post which compares AutoGPTQ and bitsandbytes.

Use the Model Memory Calculator below to estimate and compare how much memory is required to load a model. For example, try estimating how much memory it costs to load [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1).

<iframe
	src="https://hf-accelerate-model-memory-usage.hf.space"
	frameborder="0"
	width="850"
	height="450"
></iframe>

To load Mistral-7B-v0.1 in half-precision, set the `torch_dtype` parameter in the [`~transformers.AutoModelForCausalLM.from_pretrained`] method to `torch.bfloat16`. This requires 13.74GB of memory.

```py
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-v0.1", torch_dtype=torch.bfloat16, device_map="auto",
)
```

To load a quantized model (8-bit or 4-bit) for inference, try [bitsandbytes](https://hf.co/docs/bitsandbytes) and set the `load_in_4bit` or `load_in_8bit` parameters to `True`. Loading the model in 8-bits only requires 6.87 GB of memory.

```py
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
import torch

quant_config = BitsAndBytesConfig(load_in_8bit=True)
model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-v0.1", quantization_config=quant_config, device_map="auto"
)
```


key: _node_content
string_value:
{"id_": "2c3258ea-c403-4fda-b8a4-8f4663eaa024", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8f45fc08-d066-42a1-9804-8b62974943c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "transformers"}, "hash": "7c6e46628b9b6b5ef436d12b752229a7497ea593718ff57bbc6d22af808d6c7a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6302384e-bd17-4fb9-be2d-d74b6606d9f0", "node_type": "1", "metadata": {}, "hash": "8ade258cbf1c0901b8893588a0fa93d16296686d236c09f3f7400d435c838a07", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3615, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Generation with LLMs

[[open-in-colab]]

LLMs, or Large Language Models, are the key component behind text generation. In a nutshell, they consist of large pretrained transformer models trained to predict the next word (or, more precisely, token) given some input text. Since they predict one token at a time, you need to do something more elaborate to generate new sentences other than just calling the model -- you need to do autoregressive generation.

Autoregressive generation is the inference-time procedure of iteratively calling a model with its own generated outputs, given a few initial inputs. In 🤗 Transformers, this is handled by the [`~generation.GenerationMixin.generate`] method, which is available to all models with generative capabilities.

This tutorial will show you how to:

* Generate text with an LLM
* Avoid common pitfalls
* Next steps to help you get the most out of your LLM

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install transformers bitsandbytes>=0.39.0 -q
```


## Generate text

A language model trained for [causal language modeling](tasks/language_modeling) takes a sequence of text tokens as input and returns the probability distribution for the next token.

<!-- [GIF 1 -- FWD PASS] -->
<figure class="image table text-center m-0 w-full">
    <video
        style="max-width: 90%; margin: auto;"
        autoplay loop muted playsinline
        src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/assisted-generation/gif_1_1080p.mov"
    ></video>
    <figcaption>"Forward pass of an LLM"</figcaption>
</figure>

A critical aspect of autoregressive generation with LLMs is how to select the next token from this probability distribution. Anything goes in this step as long as you end up with a token for the next iteration. This means it can be as simple as selecting the most likely token from the probability distribution or as complex as applying a dozen transformations before sampling from the resulting distribution.

<!-- [GIF 2 -- TEXT GENERATION] -->
<figure class="image table text-center m-0 w-full">
    <video
        style="max-width: 90%; margin: auto;"
        autoplay loop muted playsinline
        src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/assisted-generation/gif_2_1080p.mov"
    ></video>
    <figcaption>"Autoregressive generation iteratively selects the next token from a probability distribution to generate text"</figcaption>
</figure>

The process depicted above is repeated iteratively until some stopping condition is reached. Ideally, the stopping condition is dictated by the model, which should learn when to output an end-of-sequence (`EOS`) token. If this is not the case, generation stops when some predefined maximum length is reached.

Properly setting up the token selection step and the stopping condition is essential to make your model behave as you'd expect on your task. That is why we have a [`~generation.GenerationConfig`] file associated with each model, which contains a good default generative parameterization and is loaded alongside your model.

Let's talk code!

<Tip>

If you're interested in basic LLM usage, our high-level [`Pipeline`](pipeline_tutorial) interface is a great starting point. However, LLMs often require advanced features like quantization and fine control of the token selection step, which is best done through [`~generation.GenerationMixin.generate`]. Autoregressive generation with LLMs is also resource-intensive and should be executed on a GPU for adequate throughput.


key: _node_content
string_value:
{"id_": "6302384e-bd17-4fb9-be2d-d74b6606d9f0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8f45fc08-d066-42a1-9804-8b62974943c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "transformers"}, "hash": "7c6e46628b9b6b5ef436d12b752229a7497ea593718ff57bbc6d22af808d6c7a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2c3258ea-c403-4fda-b8a4-8f4663eaa024", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "transformers"}, "hash": "8ae1ffbae00f9860d7820f9b6ad859da792aca46686bab4c6220aba5e53799a1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f0861460-a2d7-441b-b492-0012b6f74db6", "node_type": "1", "metadata": {}, "hash": "ed92f45d2f6f0334ecc0a16537d491eb128a24b5c930acab9e2960e54f0f23e1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1763, "end_char_idx": 5251, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Anything goes in this step as long as you end up with a token for the next iteration. This means it can be as simple as selecting the most likely token from the probability distribution or as complex as applying a dozen transformations before sampling from the resulting distribution.

<!-- [GIF 2 -- TEXT GENERATION] -->
<figure class="image table text-center m-0 w-full">
    <video
        style="max-width: 90%; margin: auto;"
        autoplay loop muted playsinline
        src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/assisted-generation/gif_2_1080p.mov"
    ></video>
    <figcaption>"Autoregressive generation iteratively selects the next token from a probability distribution to generate text"</figcaption>
</figure>

The process depicted above is repeated iteratively until some stopping condition is reached. Ideally, the stopping condition is dictated by the model, which should learn when to output an end-of-sequence (`EOS`) token. If this is not the case, generation stops when some predefined maximum length is reached.

Properly setting up the token selection step and the stopping condition is essential to make your model behave as you'd expect on your task. That is why we have a [`~generation.GenerationConfig`] file associated with each model, which contains a good default generative parameterization and is loaded alongside your model.

Let's talk code!

<Tip>

If you're interested in basic LLM usage, our high-level [`Pipeline`](pipeline_tutorial) interface is a great starting point. However, LLMs often require advanced features like quantization and fine control of the token selection step, which is best done through [`~generation.GenerationMixin.generate`]. Autoregressive generation with LLMs is also resource-intensive and should be executed on a GPU for adequate throughput.

</Tip>

First, you need to load the model.

```py
>>> from transformers import AutoModelForCausalLM

>>> model = AutoModelForCausalLM.from_pretrained(
...     "mistralai/Mistral-7B-v0.1", device_map="auto", load_in_4bit=True
... )
```

You'll notice two flags in the `from_pretrained` call:

 - `device_map` ensures the model is moved to your GPU(s)
 - `load_in_4bit` applies [4-bit dynamic quantization](main_classes/quantization) to massively reduce the resource requirements

There are other ways to initialize a model, but this is a good baseline to begin with an LLM.

Next, you need to preprocess your text input with a [tokenizer](tokenizer_summary).

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1", padding_side="left")
>>> model_inputs = tokenizer(["A list of colors: red, blue"], return_tensors="pt").to("cuda")
```

The `model_inputs` variable holds the tokenized text input, as well as the attention mask. While [`~generation.GenerationMixin.generate`] does its best effort to infer the attention mask when it is not passed, we recommend passing it whenever possible for optimal results.

After tokenizing the inputs, you can call the [`~generation.GenerationMixin.generate`] method to returns the generated tokens. The generated tokens then should be converted to text before printing.

```py
>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'A list of colors: red, blue, green, yellow, orange, purple, pink,'
```

Finally, you don't need to do it one sequence at a time!


key: _node_content
string_value:
{"id_": "f0861460-a2d7-441b-b492-0012b6f74db6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8f45fc08-d066-42a1-9804-8b62974943c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "transformers"}, "hash": "7c6e46628b9b6b5ef436d12b752229a7497ea593718ff57bbc6d22af808d6c7a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6302384e-bd17-4fb9-be2d-d74b6606d9f0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "transformers"}, "hash": "45c5e138b0e4f4975f51fbb4db41f32647b0be16a618a436541ac2b25dc8167a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9448ffd2-ada3-4800-8dc5-d6fba0fa840b", "node_type": "1", "metadata": {}, "hash": "a11e967a82ca12cf79fcdc2c1c41f470651fe99f7ac274a0492d888249a6c76a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3617, "end_char_idx": 6010, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

First, you need to load the model.

```py
>>> from transformers import AutoModelForCausalLM

>>> model = AutoModelForCausalLM.from_pretrained(
...     "mistralai/Mistral-7B-v0.1", device_map="auto", load_in_4bit=True
... )
```

You'll notice two flags in the `from_pretrained` call:

 - `device_map` ensures the model is moved to your GPU(s)
 - `load_in_4bit` applies [4-bit dynamic quantization](main_classes/quantization) to massively reduce the resource requirements

There are other ways to initialize a model, but this is a good baseline to begin with an LLM.

Next, you need to preprocess your text input with a [tokenizer](tokenizer_summary).

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1", padding_side="left")
>>> model_inputs = tokenizer(["A list of colors: red, blue"], return_tensors="pt").to("cuda")
```

The `model_inputs` variable holds the tokenized text input, as well as the attention mask. While [`~generation.GenerationMixin.generate`] does its best effort to infer the attention mask when it is not passed, we recommend passing it whenever possible for optimal results.

After tokenizing the inputs, you can call the [`~generation.GenerationMixin.generate`] method to returns the generated tokens. The generated tokens then should be converted to text before printing.

```py
>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'A list of colors: red, blue, green, yellow, orange, purple, pink,'
```

Finally, you don't need to do it one sequence at a time! You can batch your inputs, which will greatly improve the throughput at a small latency and memory cost. All you need to do is to make sure you pad your inputs properly (more on that below).

```py
>>> tokenizer.pad_token = tokenizer.eos_token  # Most LLMs don't have a pad token by default
>>> model_inputs = tokenizer(
...     ["A list of colors: red, blue", "Portugal is"], return_tensors="pt", padding=True
... ).to("cuda")
>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
['A list of colors: red, blue, green, yellow, orange, purple, pink,',
'Portugal is a country in southwestern Europe, on the Iber']
```

And that's it! In a few lines of code, you can harness the power of an LLM.


key: _node_content
string_value:
{"id_": "9448ffd2-ada3-4800-8dc5-d6fba0fa840b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8f45fc08-d066-42a1-9804-8b62974943c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "transformers"}, "hash": "7c6e46628b9b6b5ef436d12b752229a7497ea593718ff57bbc6d22af808d6c7a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f0861460-a2d7-441b-b492-0012b6f74db6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "transformers"}, "hash": "563da9fa3df026fba2e3ace5c76cf12e03025e03648f830fbde660eeb612a6a7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "86ed5a75-6c05-4856-9b64-7efdf948f846", "node_type": "1", "metadata": {}, "hash": "53d392ad922752b7299d8be8c25a38dd47f1c6dd0db6ee72ace1baf2fa6b2d34", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4612, "end_char_idx": 7132, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
While [`~generation.GenerationMixin.generate`] does its best effort to infer the attention mask when it is not passed, we recommend passing it whenever possible for optimal results.

After tokenizing the inputs, you can call the [`~generation.GenerationMixin.generate`] method to returns the generated tokens. The generated tokens then should be converted to text before printing.

```py
>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'A list of colors: red, blue, green, yellow, orange, purple, pink,'
```

Finally, you don't need to do it one sequence at a time! You can batch your inputs, which will greatly improve the throughput at a small latency and memory cost. All you need to do is to make sure you pad your inputs properly (more on that below).

```py
>>> tokenizer.pad_token = tokenizer.eos_token  # Most LLMs don't have a pad token by default
>>> model_inputs = tokenizer(
...     ["A list of colors: red, blue", "Portugal is"], return_tensors="pt", padding=True
... ).to("cuda")
>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
['A list of colors: red, blue, green, yellow, orange, purple, pink,',
'Portugal is a country in southwestern Europe, on the Iber']
```

And that's it! In a few lines of code, you can harness the power of an LLM.


## Common pitfalls

There are many [generation strategies](generation_strategies), and sometimes the default values may not be appropriate for your use case. If your outputs aren't aligned with what you're expecting, we've created a list of the most common pitfalls and how to avoid them.

```py
>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")
>>> tokenizer.pad_token = tokenizer.eos_token  # Most LLMs don't have a pad token by default
>>> model = AutoModelForCausalLM.from_pretrained(
...     "mistralai/Mistral-7B-v0.1", device_map="auto", load_in_4bit=True
... )
```

### Generated output is too short/long

If not specified in the [`~generation.GenerationConfig`] file, `generate` returns up to 20 tokens by default. We highly recommend manually setting `max_new_tokens` in your `generate` call to control the maximum number of new tokens it can return. Keep in mind LLMs (more precisely, [decoder-only models](https://huggingface.co/learn/nlp-course/chapter1/6?fw=pt)) also return the input prompt as part of the output.


key: _node_content
string_value:
{"id_": "86ed5a75-6c05-4856-9b64-7efdf948f846", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8f45fc08-d066-42a1-9804-8b62974943c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "transformers"}, "hash": "7c6e46628b9b6b5ef436d12b752229a7497ea593718ff57bbc6d22af808d6c7a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9448ffd2-ada3-4800-8dc5-d6fba0fa840b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "transformers"}, "hash": "b832419b337f40ce3b1f62e80fabbd54640973660a81b3304c15311715e121b4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0e9b7834-4bfd-4783-b8f7-bd6dea599978", "node_type": "1", "metadata": {}, "hash": "113cdfb0e5a37e902b267751b2595782b1841f7982bab0bb5c4da5af328728cf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5950, "end_char_idx": 9027, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In a few lines of code, you can harness the power of an LLM.


## Common pitfalls

There are many [generation strategies](generation_strategies), and sometimes the default values may not be appropriate for your use case. If your outputs aren't aligned with what you're expecting, we've created a list of the most common pitfalls and how to avoid them.

```py
>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")
>>> tokenizer.pad_token = tokenizer.eos_token  # Most LLMs don't have a pad token by default
>>> model = AutoModelForCausalLM.from_pretrained(
...     "mistralai/Mistral-7B-v0.1", device_map="auto", load_in_4bit=True
... )
```

### Generated output is too short/long

If not specified in the [`~generation.GenerationConfig`] file, `generate` returns up to 20 tokens by default. We highly recommend manually setting `max_new_tokens` in your `generate` call to control the maximum number of new tokens it can return. Keep in mind LLMs (more precisely, [decoder-only models](https://huggingface.co/learn/nlp-course/chapter1/6?fw=pt)) also return the input prompt as part of the output.


```py
>>> model_inputs = tokenizer(["A sequence of numbers: 1, 2"], return_tensors="pt").to("cuda")

>>> # By default, the output will contain up to 20 tokens
>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'A sequence of numbers: 1, 2, 3, 4, 5'

>>> # Setting `max_new_tokens` allows you to control the maximum length
>>> generated_ids = model.generate(**model_inputs, max_new_tokens=50)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'A sequence of numbers: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,'
```

### Incorrect generation mode

By default, and unless specified in the [`~generation.GenerationConfig`] file, `generate` selects the most likely token at each iteration (greedy decoding). Depending on your task, this may be undesirable; creative tasks like chatbots or writing an essay benefit from sampling. On the other hand, input-grounded tasks like audio transcription or translation benefit from greedy decoding. Enable sampling with `do_sample=True`, and you can learn more about this topic in this [blog post](https://huggingface.co/blog/how-to-generate).

```py
>>> # Set seed or reproducibility -- you don't need this unless you want full reproducibility
>>> from transformers import set_seed
>>> set_seed(42)

>>> model_inputs = tokenizer(["I am a cat."], return_tensors="pt").to("cuda")

>>> # LLM + greedy decoding = repetitive, boring output
>>> generated_ids = model.generate(**model_inputs)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'I am a cat. I am a cat. I am a cat. I am a cat'

>>> # With sampling, the output becomes more creative!
>>> generated_ids = model.generate(**model_inputs, do_sample=True)
>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
'I am a cat.  Specifically, I am an indoor-only cat.


key: _node_content
string_value:
{"id_": "0e9b7834-4bfd-4783-b8f7-bd6dea599978", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8f45fc08-d066-42a1-9804-8b62974943c0", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "transformers"}, "hash": "7c6e46628b9b6b5ef436d12b752229a7497ea593718ff57bbc6d22af808d6c7a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "86ed5a75-6c05-4856-9b64-7efdf948f846", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/llm_tutorial", "title": "Generation with LLMs", "tokens": 3594, "retrieve_doc": true, "source": "transformers"}, "hash": "cefbda645ef25aa62405a70fd519ecf836f274027e0b30782fd8816e45c0071a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0aad3c73-c244-41da-b298-a3fff01c3cdb", "node_type": "1", "metadata": {}, "hash": "c4ae41c5777a36ba310ec9edb457541ab67e3b5379e0b262c94d74b7cd631834", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7942, "end_char_idx": 10796, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


