# embedding_metadata

key: chroma:document
string_value:
## Re-use Cache to continue generation

Sometimes you would want to fist fill-in cache object with key/values for certain prefix prompt and re-use it several times to generate different sequences from it. We are working hard on adding this feature to 🤗 Transformers and will update this section soon.


key: _node_content
string_value:
{"id_": "ab10489c-9537-436c-b266-80b2efe7b587", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/accelerate", "title": "Distributed training with \ud83e\udd17 Accelerate", "tokens": 1054, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3aa99498-7863-4cf7-bc51-e5f465bc7f4a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/accelerate", "title": "Distributed training with \ud83e\udd17 Accelerate", "tokens": 1054, "retrieve_doc": true, "source": "transformers"}, "hash": "839d0123fd4e6631f69648ba6ca5163ea5abc3628df918cbee14aaa6ef50dfd4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cdc660ae-1551-48cd-a7b9-a339b935dbe4", "node_type": "1", "metadata": {}, "hash": "d72d682e22186a9be416d73902d80574c8e1bb97898d735edede859bfb0be1ae", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3394, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Distributed training with 🤗 Accelerate

As models get bigger, parallelism has emerged as a strategy for training larger models on limited hardware and accelerating training speed by several orders of magnitude. At Hugging Face, we created the [🤗 Accelerate](https://huggingface.co/docs/accelerate) library to help users easily train a 🤗 Transformers model on any type of distributed setup, whether it is multiple GPU's on one machine or multiple GPU's across several machines. In this tutorial, learn how to customize your native PyTorch training loop to enable training in a distributed environment.

## Setup

Get started by installing 🤗 Accelerate:

```bash
pip install accelerate
```

Then import and create an [`~accelerate.Accelerator`] object. The [`~accelerate.Accelerator`] will automatically detect your type of distributed setup and initialize all the necessary components for training. You don't need to explicitly place your model on a device.

```py
>>> from accelerate import Accelerator

>>> accelerator = Accelerator()
```

## Prepare to accelerate

The next step is to pass all the relevant training objects to the [`~accelerate.Accelerator.prepare`] method. This includes your training and evaluation DataLoaders, a model and an optimizer:

```py
>>> train_dataloader, eval_dataloader, model, optimizer = accelerator.prepare(
...     train_dataloader, eval_dataloader, model, optimizer
... )
```

## Backward

The last addition is to replace the typical `loss.backward()` in your training loop with 🤗 Accelerate's [`~accelerate.Accelerator.backward`]method:

```py
>>> for epoch in range(num_epochs):
...     for batch in train_dataloader:
...         outputs = model(**batch)
...         loss = outputs.loss
...         accelerator.backward(loss)

...         optimizer.step()
...         lr_scheduler.step()
...         optimizer.zero_grad()
...         progress_bar.update(1)
```

As you can see in the following code, you only need to add four additional lines of code to your training loop to enable distributed training!

```diff
+ from accelerate import Accelerator
  from transformers import AdamW, AutoModelForSequenceClassification, get_scheduler

+ accelerator = Accelerator()

  model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=2)
  optimizer = AdamW(model.parameters(), lr=3e-5)

- device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
- model.to(device)

+ train_dataloader, eval_dataloader, model, optimizer = accelerator.prepare(
+     train_dataloader, eval_dataloader, model, optimizer
+ )

  num_epochs = 3
  num_training_steps = num_epochs * len(train_dataloader)
  lr_scheduler = get_scheduler(
      "linear",
      optimizer=optimizer,
      num_warmup_steps=0,
      num_training_steps=num_training_steps
  )

  progress_bar = tqdm(range(num_training_steps))

  model.train()
  for epoch in range(num_epochs):
      for batch in train_dataloader:
-         batch = {k: v.to(device) for k, v in batch.items()}
          outputs = model(**batch)
          loss = outputs.loss
-         loss.backward()
+         accelerator.backward(loss)

          optimizer.step()
          lr_scheduler.step()
          optimizer.zero_grad()
          progress_bar.update(1)
```

## Train

Once you've added the relevant lines of code, launch your training in a script or a notebook like Colaboratory.


key: _node_content
string_value:
{"id_": "cdc660ae-1551-48cd-a7b9-a339b935dbe4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/accelerate", "title": "Distributed training with \ud83e\udd17 Accelerate", "tokens": 1054, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3aa99498-7863-4cf7-bc51-e5f465bc7f4a", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/accelerate", "title": "Distributed training with \ud83e\udd17 Accelerate", "tokens": 1054, "retrieve_doc": true, "source": "transformers"}, "hash": "839d0123fd4e6631f69648ba6ca5163ea5abc3628df918cbee14aaa6ef50dfd4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ab10489c-9537-436c-b266-80b2efe7b587", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/accelerate", "title": "Distributed training with \ud83e\udd17 Accelerate", "tokens": 1054, "retrieve_doc": true, "source": "transformers"}, "hash": "c14a49a78c6c1489bc7fb3bf51ebb898dcffbad5148dd111f4f49db77c336be0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2049, "end_char_idx": 4096, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```diff
+ from accelerate import Accelerator
  from transformers import AdamW, AutoModelForSequenceClassification, get_scheduler

+ accelerator = Accelerator()

  model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=2)
  optimizer = AdamW(model.parameters(), lr=3e-5)

- device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
- model.to(device)

+ train_dataloader, eval_dataloader, model, optimizer = accelerator.prepare(
+     train_dataloader, eval_dataloader, model, optimizer
+ )

  num_epochs = 3
  num_training_steps = num_epochs * len(train_dataloader)
  lr_scheduler = get_scheduler(
      "linear",
      optimizer=optimizer,
      num_warmup_steps=0,
      num_training_steps=num_training_steps
  )

  progress_bar = tqdm(range(num_training_steps))

  model.train()
  for epoch in range(num_epochs):
      for batch in train_dataloader:
-         batch = {k: v.to(device) for k, v in batch.items()}
          outputs = model(**batch)
          loss = outputs.loss
-         loss.backward()
+         accelerator.backward(loss)

          optimizer.step()
          lr_scheduler.step()
          optimizer.zero_grad()
          progress_bar.update(1)
```

## Train

Once you've added the relevant lines of code, launch your training in a script or a notebook like Colaboratory.

### Train with a script

If you are running your training from a script, run the following command to create and save a configuration file:

```bash
accelerate config
```

Then launch your training with:

```bash
accelerate launch train.py
```

### Train with a notebook

🤗 Accelerate can also run in a notebook if you're planning on using Colaboratory's TPUs. Wrap all the code responsible for training in a function, and pass it to [`~accelerate.notebook_launcher`]:

```py
>>> from accelerate import notebook_launcher

>>> notebook_launcher(training_function)
```

For more information about 🤗 Accelerate and its rich features, refer to the [documentation](https://huggingface.co/docs/accelerate).


key: _node_content
string_value:
{"id_": "19647714-3ba8-4877-b611-cd9c4bf5c093", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "37cbed30-c667-4ebd-96e9-00c55d9acc02", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e176cf9a-4957-4b2a-b08a-e46ef97ba437", "node_type": "1", "metadata": {}, "hash": "5646aaa13f706a472cedfbc8fc5d1cb7e667bb5490d127cec146423143f317f6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2733, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<!--⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.
-->

# Community

This page regroups resources around 🤗 Transformers developed by the community.

## Community resources:

| Resource     |      Description      |      Author      |
|:----------|:-------------|------:|
| [Hugging Face Transformers Glossary Flashcards](https://www.darigovresearch.com/huggingface-transformers-glossary-flashcards) | A set of flashcards based on the [Transformers Docs Glossary](glossary) that has been put into a form which can be easily learned/revised using [Anki](https://apps.ankiweb.net/) an open source, cross platform app specifically designed for long term knowledge retention. See this [Introductory video on how to use the flashcards](https://www.youtube.com/watch?v=Dji_h7PILrw). | [Darigov Research](https://www.darigovresearch.com/) |

## Community notebooks:

| Notebook     |      Description      |      Author      |      |
|:----------|:-------------|:-------------|------:|
| [Fine-tune a pre-trained Transformer to generate lyrics](https://github.com/AlekseyKorshuk/huggingartists) | How to generate lyrics in the style of your favorite artist by fine-tuning a GPT-2 model |  [Aleksey Korshuk](https://github.com/AlekseyKorshuk) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/AlekseyKorshuk/huggingartists/blob/master/huggingartists-demo.ipynb) |
| [Train T5 in Tensorflow 2](https://github.com/snapthat/TF-T5-text-to-text) | How to train T5 for any task using Tensorflow 2. This notebook demonstrates a Question & Answer task implemented in Tensorflow 2 using SQUAD | [Muhammad Harris](https://github.com/HarrisDePerceptron) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/snapthat/TF-T5-text-to-text/blob/master/snapthatT5/notebooks/TF-T5-Datasets%20Training.ipynb) |
| [Train T5 on TPU](https://github.com/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb)  | How to train T5 on SQUAD with Transformers and Nlp | [Suraj Patil](https://github.com/patil-suraj) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb#scrollTo=QLGiFCDqvuil) |
| [Fine-tune T5 for Classification and Multiple Choice](https://github.com/patil-suraj/exploring-T5/blob/master/t5_fine_tuning.ipynb)  | How to fine-tune T5 for classification and multiple choice tasks using a text-to-text format with PyTorch Lightning |  [Suraj Patil](https://github.com/patil-suraj) | [!


key: _node_content
string_value:
{"id_": "e176cf9a-4957-4b2a-b08a-e46ef97ba437", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "37cbed30-c667-4ebd-96e9-00c55d9acc02", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "19647714-3ba8-4877-b611-cd9c4bf5c093", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74757c3f59a27e93424cba24e98b9981afcfe44235c9057718462dbf88d7bba1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3679653f-11c9-4231-9051-f27b859643ee", "node_type": "1", "metadata": {}, "hash": "03e4059e6cdb8d1f431fb51aa411f27bddccb5dc77a4ab65d7f877e7830585ac", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1669, "end_char_idx": 4102, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This notebook demonstrates a Question & Answer task implemented in Tensorflow 2 using SQUAD | [Muhammad Harris](https://github.com/HarrisDePerceptron) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/snapthat/TF-T5-text-to-text/blob/master/snapthatT5/notebooks/TF-T5-Datasets%20Training.ipynb) |
| [Train T5 on TPU](https://github.com/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb)  | How to train T5 on SQUAD with Transformers and Nlp | [Suraj Patil](https://github.com/patil-suraj) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb#scrollTo=QLGiFCDqvuil) |
| [Fine-tune T5 for Classification and Multiple Choice](https://github.com/patil-suraj/exploring-T5/blob/master/t5_fine_tuning.ipynb)  | How to fine-tune T5 for classification and multiple choice tasks using a text-to-text format with PyTorch Lightning |  [Suraj Patil](https://github.com/patil-suraj) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patil-suraj/exploring-T5/blob/master/t5_fine_tuning.ipynb) |
| [Fine-tune DialoGPT on New Datasets and Languages](https://github.com/ncoop57/i-am-a-nerd/blob/master/_notebooks/2020-05-12-chatbot-part-1.ipynb)  | How to fine-tune the DialoGPT model on a new dataset for open-dialog conversational chatbots |  [Nathan Cooper](https://github.com/ncoop57) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ncoop57/i-am-a-nerd/blob/master/_notebooks/2020-05-12-chatbot-part-1.ipynb) |
| [Long Sequence Modeling with Reformer](https://github.com/patrickvonplaten/notebooks/blob/master/PyTorch_Reformer.ipynb)  | How to train on sequences as long as 500,000 tokens with Reformer |  [Patrick von Platen](https://github.com/patrickvonplaten) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/PyTorch_Reformer.ipynb)  |
| [Fine-tune BART for Summarization](https://github.com/ohmeow/ohmeow_website/blob/master/posts/2021-05-25-mbart-sequence-classification-with-blurr.ipynb) | How to fine-tune BART for summarization with fastai using blurr | [Wayde Gilliam](https://ohmeow.com/) | [!


key: _node_content
string_value:
{"id_": "3679653f-11c9-4231-9051-f27b859643ee", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "37cbed30-c667-4ebd-96e9-00c55d9acc02", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e176cf9a-4957-4b2a-b08a-e46ef97ba437", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "0cac1fd67084d1022359bc1214802498c70abc02bed0969e29a13c0f44654ea3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7b435d1d-a546-485f-a366-ad3300ac95ef", "node_type": "1", "metadata": {}, "hash": "43676eefa6ae10acac64ac2d310ae9b05be3d412612453e74c64b05b78b48f87", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3205, "end_char_idx": 5576, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ncoop57/i-am-a-nerd/blob/master/_notebooks/2020-05-12-chatbot-part-1.ipynb) |
| [Long Sequence Modeling with Reformer](https://github.com/patrickvonplaten/notebooks/blob/master/PyTorch_Reformer.ipynb)  | How to train on sequences as long as 500,000 tokens with Reformer |  [Patrick von Platen](https://github.com/patrickvonplaten) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/PyTorch_Reformer.ipynb)  |
| [Fine-tune BART for Summarization](https://github.com/ohmeow/ohmeow_website/blob/master/posts/2021-05-25-mbart-sequence-classification-with-blurr.ipynb) | How to fine-tune BART for summarization with fastai using blurr | [Wayde Gilliam](https://ohmeow.com/) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ohmeow/ohmeow_website/blob/master/posts/2021-05-25-mbart-sequence-classification-with-blurr.ipynb) |
| [Fine-tune a pre-trained Transformer on anyone's tweets](https://colab.research.google.com/github/borisdayma/huggingtweets/blob/master/huggingtweets-demo.ipynb) | How to generate tweets in the style of your favorite Twitter account by fine-tuning a GPT-2 model |  [Boris Dayma](https://github.com/borisdayma) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/borisdayma/huggingtweets/blob/master/huggingtweets-demo.ipynb) |
| [Optimize 🤗 Hugging Face models with Weights & Biases](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/huggingface/Optimize_Hugging_Face_models_with_Weights_%26_Biases.ipynb) | A complete tutorial showcasing W&B integration with Hugging Face | [Boris Dayma](https://github.com/borisdayma) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/huggingface/Optimize_Hugging_Face_models_with_Weights_%26_Biases.ipynb) |
| [Pretrain Longformer](https://github.com/allenai/longformer/blob/master/scripts/convert_model_to_long.ipynb)  | How to build a "long" version of existing pretrained models |  [Iz Beltagy](https://beltagy.net) | [!


key: _node_content
string_value:
{"id_": "7b435d1d-a546-485f-a366-ad3300ac95ef", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "37cbed30-c667-4ebd-96e9-00c55d9acc02", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3679653f-11c9-4231-9051-f27b859643ee", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "d586fea41f6da6f0f3425c51bfce70fa736ac50c2babb49ec871734ac2f8d42d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f0f0e4cc-5799-42d1-9cbc-e3df232e8061", "node_type": "1", "metadata": {}, "hash": "0d93a5a7e3890ef8f99723966e997099817e5d7fbcd3ef21d0a2968a08340ae8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4634, "end_char_idx": 6882, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/borisdayma/huggingtweets/blob/master/huggingtweets-demo.ipynb) |
| [Optimize 🤗 Hugging Face models with Weights & Biases](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/huggingface/Optimize_Hugging_Face_models_with_Weights_%26_Biases.ipynb) | A complete tutorial showcasing W&B integration with Hugging Face | [Boris Dayma](https://github.com/borisdayma) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/huggingface/Optimize_Hugging_Face_models_with_Weights_%26_Biases.ipynb) |
| [Pretrain Longformer](https://github.com/allenai/longformer/blob/master/scripts/convert_model_to_long.ipynb)  | How to build a "long" version of existing pretrained models |  [Iz Beltagy](https://beltagy.net) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/allenai/longformer/blob/master/scripts/convert_model_to_long.ipynb) |
| [Fine-tune Longformer for QA](https://github.com/patil-suraj/Notebooks/blob/master/longformer_qa_training.ipynb) | How to fine-tune longformer model for QA task | [Suraj Patil](https://github.com/patil-suraj) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patil-suraj/Notebooks/blob/master/longformer_qa_training.ipynb) |
| [Evaluate Model with 🤗nlp](https://github.com/patrickvonplaten/notebooks/blob/master/How_to_evaluate_Longformer_on_TriviaQA_using_NLP.ipynb) | How to evaluate longformer on TriviaQA with `nlp` | [Patrick von Platen](https://github.com/patrickvonplaten) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1m7eTGlPmLRgoPkkA7rkhQdZ9ydpmsdLE?usp=sharing) |
| [Fine-tune T5 for Sentiment Span Extraction](https://github.com/enzoampil/t5-intro/blob/master/t5_qa_training_pytorch_span_extraction.ipynb)  | How to fine-tune T5 for sentiment span extraction using a text-to-text format with PyTorch Lightning |  [Lorenzo Ampil](https://github.com/enzoampil) | [!


key: _node_content
string_value:
{"id_": "f0f0e4cc-5799-42d1-9cbc-e3df232e8061", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "37cbed30-c667-4ebd-96e9-00c55d9acc02", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7b435d1d-a546-485f-a366-ad3300ac95ef", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "f42928e975e07076e62f5ece65bb8100d9c172af013b36309bafc4b47f7ad1e3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "30b6b27f-24ab-4ab4-b551-6ac719838f7e", "node_type": "1", "metadata": {}, "hash": "4f6876d0a4c0ebf1f9e8b456eeef0cbe1df29ed19164be9f0eda6dc283b2f8dc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5576, "end_char_idx": 7882, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/allenai/longformer/blob/master/scripts/convert_model_to_long.ipynb) |
| [Fine-tune Longformer for QA](https://github.com/patil-suraj/Notebooks/blob/master/longformer_qa_training.ipynb) | How to fine-tune longformer model for QA task | [Suraj Patil](https://github.com/patil-suraj) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patil-suraj/Notebooks/blob/master/longformer_qa_training.ipynb) |
| [Evaluate Model with 🤗nlp](https://github.com/patrickvonplaten/notebooks/blob/master/How_to_evaluate_Longformer_on_TriviaQA_using_NLP.ipynb) | How to evaluate longformer on TriviaQA with `nlp` | [Patrick von Platen](https://github.com/patrickvonplaten) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1m7eTGlPmLRgoPkkA7rkhQdZ9ydpmsdLE?usp=sharing) |
| [Fine-tune T5 for Sentiment Span Extraction](https://github.com/enzoampil/t5-intro/blob/master/t5_qa_training_pytorch_span_extraction.ipynb)  | How to fine-tune T5 for sentiment span extraction using a text-to-text format with PyTorch Lightning |  [Lorenzo Ampil](https://github.com/enzoampil) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/enzoampil/t5-intro/blob/master/t5_qa_training_pytorch_span_extraction.ipynb) |
| [Fine-tune DistilBert for Multiclass Classification](https://github.com/abhimishra91/transformers-tutorials/blob/master/transformers_multiclass_classification.ipynb) | How to fine-tune DistilBert for multiclass classification with PyTorch | [Abhishek Kumar Mishra](https://github.com/abhimishra91) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/abhimishra91/transformers-tutorials/blob/master/transformers_multiclass_classification.ipynb)|
|[Fine-tune BERT for Multi-label Classification](https://github.com/abhimishra91/transformers-tutorials/blob/master/transformers_multi_label_classification.ipynb)|How to fine-tune BERT for multi-label classification using PyTorch|[Abhishek Kumar Mishra](https://github.com/abhimishra91) |[!


key: _node_content
string_value:
{"id_": "30b6b27f-24ab-4ab4-b551-6ac719838f7e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "37cbed30-c667-4ebd-96e9-00c55d9acc02", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f0f0e4cc-5799-42d1-9cbc-e3df232e8061", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "a9e0f6716bcc42f4ae1e91ab58b7e2f4ddbe9b5219411ba5edf691d7b3c48f5e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "199eba88-fec5-4f63-ad0a-fbe2b631ca70", "node_type": "1", "metadata": {}, "hash": "907c9c21a581057ef6092e330668e407c94875911b4e0b8d8edfa29506e142a8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6882, "end_char_idx": 9359, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/enzoampil/t5-intro/blob/master/t5_qa_training_pytorch_span_extraction.ipynb) |
| [Fine-tune DistilBert for Multiclass Classification](https://github.com/abhimishra91/transformers-tutorials/blob/master/transformers_multiclass_classification.ipynb) | How to fine-tune DistilBert for multiclass classification with PyTorch | [Abhishek Kumar Mishra](https://github.com/abhimishra91) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/abhimishra91/transformers-tutorials/blob/master/transformers_multiclass_classification.ipynb)|
|[Fine-tune BERT for Multi-label Classification](https://github.com/abhimishra91/transformers-tutorials/blob/master/transformers_multi_label_classification.ipynb)|How to fine-tune BERT for multi-label classification using PyTorch|[Abhishek Kumar Mishra](https://github.com/abhimishra91) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/abhimishra91/transformers-tutorials/blob/master/transformers_multi_label_classification.ipynb)|
|[Fine-tune T5 for Summarization](https://github.com/abhimishra91/transformers-tutorials/blob/master/transformers_summarization_wandb.ipynb)|How to fine-tune T5 for summarization in PyTorch and track experiments with WandB|[Abhishek Kumar Mishra](https://github.com/abhimishra91) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/abhimishra91/transformers-tutorials/blob/master/transformers_summarization_wandb.ipynb)|
|[Speed up Fine-Tuning in Transformers with Dynamic Padding / Bucketing](https://github.com/ELS-RD/transformers-notebook/blob/master/Divide_Hugging_Face_Transformers_training_time_by_2_or_more.ipynb)|How to speed up fine-tuning by a factor of 2 using dynamic padding / bucketing|[Michael Benesty](https://github.com/pommedeterresautee) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1CBfRU1zbfu7-ijiOqAAQUA-RJaxfcJoO?usp=sharing)|
|[Pretrain Reformer for Masked Language Modeling](https://github.com/patrickvonplaten/notebooks/blob/master/Reformer_For_Masked_LM.ipynb)| How to train a Reformer model with bi-directional self-attention layers | [Patrick von Platen](https://github.com/patrickvonplaten) | [!


key: _node_content
string_value:
{"id_": "199eba88-fec5-4f63-ad0a-fbe2b631ca70", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "37cbed30-c667-4ebd-96e9-00c55d9acc02", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "30b6b27f-24ab-4ab4-b551-6ac719838f7e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "b3ad9009bfdc68ae216debd2d5a68b1a9617ec528d54e4df93d55b343a656193", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cfe31154-8b93-46f8-b648-1b6610c40c7e", "node_type": "1", "metadata": {}, "hash": "85fa2245760f981b1f35660b7c49afa4abafb5fe617af081d9fb99ba38bfd74b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8377, "end_char_idx": 10803, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/abhimishra91/transformers-tutorials/blob/master/transformers_summarization_wandb.ipynb)|
|[Speed up Fine-Tuning in Transformers with Dynamic Padding / Bucketing](https://github.com/ELS-RD/transformers-notebook/blob/master/Divide_Hugging_Face_Transformers_training_time_by_2_or_more.ipynb)|How to speed up fine-tuning by a factor of 2 using dynamic padding / bucketing|[Michael Benesty](https://github.com/pommedeterresautee) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1CBfRU1zbfu7-ijiOqAAQUA-RJaxfcJoO?usp=sharing)|
|[Pretrain Reformer for Masked Language Modeling](https://github.com/patrickvonplaten/notebooks/blob/master/Reformer_For_Masked_LM.ipynb)| How to train a Reformer model with bi-directional self-attention layers | [Patrick von Platen](https://github.com/patrickvonplaten) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1tzzh0i8PgDQGV3SMFUGxM7_gGae3K-uW?usp=sharing)|
|[Expand and Fine Tune Sci-BERT](https://github.com/lordtt13/word-embeddings/blob/master/COVID-19%20Research%20Data/COVID-SciBERT.ipynb)| How to increase vocabulary of a pretrained SciBERT model from AllenAI on the CORD dataset and pipeline it. | [Tanmay Thakur](https://github.com/lordtt13) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1rqAR40goxbAfez1xvF3hBJphSCsvXmh8)|
|[Fine Tune BlenderBotSmall for Summarization using the Trainer API](https://github.com/lordtt13/transformers-experiments/blob/master/Custom%20Tasks/fine-tune-blenderbot_small-for-summarization.ipynb)| How to fine-tune BlenderBotSmall for summarization on a custom dataset, using the Trainer API. | [Tanmay Thakur](https://github.com/lordtt13) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/19Wmupuls7mykSGyRN_Qo6lPQhgp56ymq?usp=sharing)|
|[Fine-tune Electra and interpret with Integrated Gradients](https://github.com/elsanns/xai-nlp-notebooks/blob/master/electra_fine_tune_interpret_captum_ig.ipynb) | How to fine-tune Electra for sentiment analysis and interpret predictions with Captum Integrated Gradients | [Eliza Szczechla](https://elsanns.github.io) | [!


key: _node_content
string_value:
{"id_": "cfe31154-8b93-46f8-b648-1b6610c40c7e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "37cbed30-c667-4ebd-96e9-00c55d9acc02", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "199eba88-fec5-4f63-ad0a-fbe2b631ca70", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "6ea5efbb16cf1500241c36b30d05cb03219f587baa76820d8752040ad23f957f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3ac5e56d-5fcb-4836-a3db-d7accfc517bc", "node_type": "1", "metadata": {}, "hash": "e2bf0113b2a3c3b6e1ad1f33e5a921bb1700b307dfc6541bf459fe8dc67a0c88", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9767, "end_char_idx": 12337, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
| [Tanmay Thakur](https://github.com/lordtt13) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1rqAR40goxbAfez1xvF3hBJphSCsvXmh8)|
|[Fine Tune BlenderBotSmall for Summarization using the Trainer API](https://github.com/lordtt13/transformers-experiments/blob/master/Custom%20Tasks/fine-tune-blenderbot_small-for-summarization.ipynb)| How to fine-tune BlenderBotSmall for summarization on a custom dataset, using the Trainer API. | [Tanmay Thakur](https://github.com/lordtt13) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/19Wmupuls7mykSGyRN_Qo6lPQhgp56ymq?usp=sharing)|
|[Fine-tune Electra and interpret with Integrated Gradients](https://github.com/elsanns/xai-nlp-notebooks/blob/master/electra_fine_tune_interpret_captum_ig.ipynb) | How to fine-tune Electra for sentiment analysis and interpret predictions with Captum Integrated Gradients | [Eliza Szczechla](https://elsanns.github.io) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/elsanns/xai-nlp-notebooks/blob/master/electra_fine_tune_interpret_captum_ig.ipynb)|
|[fine-tune a non-English GPT-2 Model with Trainer class](https://github.com/philschmid/fine-tune-GPT-2/blob/master/Fine_tune_a_non_English_GPT_2_Model_with_Huggingface.ipynb) | How to fine-tune a non-English GPT-2 Model with Trainer class | [Philipp Schmid](https://www.philschmid.de) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/philschmid/fine-tune-GPT-2/blob/master/Fine_tune_a_non_English_GPT_2_Model_with_Huggingface.ipynb)|
|[Fine-tune a DistilBERT Model for Multi Label Classification task](https://github.com/DhavalTaunk08/Transformers_scripts/blob/master/Transformers_multilabel_distilbert.ipynb) | How to fine-tune a DistilBERT Model for Multi Label Classification task | [Dhaval Taunk](https://github.com/DhavalTaunk08) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/DhavalTaunk08/Transformers_scripts/blob/master/Transformers_multilabel_distilbert.ipynb)|
|[Fine-tune ALBERT for sentence-pair classification](https://github.com/NadirEM/nlp-notebooks/blob/master/Fine_tune_ALBERT_sentence_pair_classification.ipynb) | How to fine-tune an ALBERT model or another BERT-based model for the sentence-pair classification task | [Nadir El Manouzi](https://github.com/NadirEM) | [!


key: _node_content
string_value:
{"id_": "3ac5e56d-5fcb-4836-a3db-d7accfc517bc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "37cbed30-c667-4ebd-96e9-00c55d9acc02", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cfe31154-8b93-46f8-b648-1b6610c40c7e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "48847e1090dbe7503e4bbe3262eb3c518c13ec6378b6668a0c109e4d29f78ae3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "74776684-ca80-479a-aa5b-0b6c82ce81e9", "node_type": "1", "metadata": {}, "hash": "81394bed4305e0d14d84fad18b41452ea7a31658c6b5493e6766e1f856ab11eb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11293, "end_char_idx": 13629, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/philschmid/fine-tune-GPT-2/blob/master/Fine_tune_a_non_English_GPT_2_Model_with_Huggingface.ipynb)|
|[Fine-tune a DistilBERT Model for Multi Label Classification task](https://github.com/DhavalTaunk08/Transformers_scripts/blob/master/Transformers_multilabel_distilbert.ipynb) | How to fine-tune a DistilBERT Model for Multi Label Classification task | [Dhaval Taunk](https://github.com/DhavalTaunk08) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/DhavalTaunk08/Transformers_scripts/blob/master/Transformers_multilabel_distilbert.ipynb)|
|[Fine-tune ALBERT for sentence-pair classification](https://github.com/NadirEM/nlp-notebooks/blob/master/Fine_tune_ALBERT_sentence_pair_classification.ipynb) | How to fine-tune an ALBERT model or another BERT-based model for the sentence-pair classification task | [Nadir El Manouzi](https://github.com/NadirEM) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NadirEM/nlp-notebooks/blob/master/Fine_tune_ALBERT_sentence_pair_classification.ipynb)|
|[Fine-tune Roberta for sentiment analysis](https://github.com/DhavalTaunk08/NLP_scripts/blob/master/sentiment_analysis_using_roberta.ipynb) | How to fine-tune a Roberta model for sentiment analysis | [Dhaval Taunk](https://github.com/DhavalTaunk08) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/DhavalTaunk08/NLP_scripts/blob/master/sentiment_analysis_using_roberta.ipynb)|
|[Evaluating Question Generation Models](https://github.com/flexudy-pipe/qugeev) | How accurate are the answers to questions generated by your seq2seq transformer model? | [Pascal Zoleko](https://github.com/zolekode) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1bpsSqCQU-iw_5nNoRm_crPq6FRuJthq_?usp=sharing)|
|[Classify text with DistilBERT and Tensorflow](https://github.com/peterbayerle/huggingface_notebook/blob/main/distilbert_tf.ipynb) | How to fine-tune DistilBERT for text classification in TensorFlow | [Peter Bayerle](https://github.com/peterbayerle) | [!


key: _node_content
string_value:
{"id_": "74776684-ca80-479a-aa5b-0b6c82ce81e9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "37cbed30-c667-4ebd-96e9-00c55d9acc02", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3ac5e56d-5fcb-4836-a3db-d7accfc517bc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "55cb64c04f0c9143458b64aeaa0ac09c338dfc9e1301597619a59d98cf97f83a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0a4225ca-44d5-4954-a2fd-a2f434392276", "node_type": "1", "metadata": {}, "hash": "4d1b70bf401d7b8d1835e95f85b9c6c7c7c3a5c66cce3eefc058e29e42fc239c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12337, "end_char_idx": 14703, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NadirEM/nlp-notebooks/blob/master/Fine_tune_ALBERT_sentence_pair_classification.ipynb)|
|[Fine-tune Roberta for sentiment analysis](https://github.com/DhavalTaunk08/NLP_scripts/blob/master/sentiment_analysis_using_roberta.ipynb) | How to fine-tune a Roberta model for sentiment analysis | [Dhaval Taunk](https://github.com/DhavalTaunk08) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/DhavalTaunk08/NLP_scripts/blob/master/sentiment_analysis_using_roberta.ipynb)|
|[Evaluating Question Generation Models](https://github.com/flexudy-pipe/qugeev) | How accurate are the answers to questions generated by your seq2seq transformer model? | [Pascal Zoleko](https://github.com/zolekode) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1bpsSqCQU-iw_5nNoRm_crPq6FRuJthq_?usp=sharing)|
|[Classify text with DistilBERT and Tensorflow](https://github.com/peterbayerle/huggingface_notebook/blob/main/distilbert_tf.ipynb) | How to fine-tune DistilBERT for text classification in TensorFlow | [Peter Bayerle](https://github.com/peterbayerle) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/peterbayerle/huggingface_notebook/blob/main/distilbert_tf.ipynb)|
|[Leverage BERT for Encoder-Decoder Summarization on CNN/Dailymail](https://github.com/patrickvonplaten/notebooks/blob/master/BERT2BERT_for_CNN_Dailymail.ipynb) | How to warm-start a *EncoderDecoderModel* with a *google-bert/bert-base-uncased* checkpoint for summarization on CNN/Dailymail | [Patrick von Platen](https://github.com/patrickvonplaten) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/BERT2BERT_for_CNN_Dailymail.ipynb)|
|[Leverage RoBERTa for Encoder-Decoder Summarization on BBC XSum](https://github.com/patrickvonplaten/notebooks/blob/master/RoBERTaShared_for_BBC_XSum.ipynb) | How to warm-start a shared *EncoderDecoderModel* with a *FacebookAI/roberta-base* checkpoint for summarization on BBC/XSum | [Patrick von Platen](https://github.com/patrickvonplaten) | [!


key: _node_content
string_value:
{"id_": "0a4225ca-44d5-4954-a2fd-a2f434392276", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "37cbed30-c667-4ebd-96e9-00c55d9acc02", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "74776684-ca80-479a-aa5b-0b6c82ce81e9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "0c3e7de539006203d02e2e493c671c0fc62eda424de1330a4453386a848b97be", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "17615da8-5d33-4460-a411-4ffebb331b07", "node_type": "1", "metadata": {}, "hash": "9d8ffb1ae73fff6d8f6498f4c12d81e2909193bda9968544980adea14993260b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13629, "end_char_idx": 15876, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/peterbayerle/huggingface_notebook/blob/main/distilbert_tf.ipynb)|
|[Leverage BERT for Encoder-Decoder Summarization on CNN/Dailymail](https://github.com/patrickvonplaten/notebooks/blob/master/BERT2BERT_for_CNN_Dailymail.ipynb) | How to warm-start a *EncoderDecoderModel* with a *google-bert/bert-base-uncased* checkpoint for summarization on CNN/Dailymail | [Patrick von Platen](https://github.com/patrickvonplaten) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/BERT2BERT_for_CNN_Dailymail.ipynb)|
|[Leverage RoBERTa for Encoder-Decoder Summarization on BBC XSum](https://github.com/patrickvonplaten/notebooks/blob/master/RoBERTaShared_for_BBC_XSum.ipynb) | How to warm-start a shared *EncoderDecoderModel* with a *FacebookAI/roberta-base* checkpoint for summarization on BBC/XSum | [Patrick von Platen](https://github.com/patrickvonplaten) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/RoBERTaShared_for_BBC_XSum.ipynb)|
|[Fine-tune TAPAS on Sequential Question Answering (SQA)](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb) | How to fine-tune *TapasForQuestionAnswering* with a *tapas-base* checkpoint on the Sequential Question Answering (SQA) dataset | [Niels Rogge](https://github.com/nielsrogge) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb)|
|[Evaluate TAPAS on Table Fact Checking (TabFact)](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/TAPAS/Evaluating_TAPAS_on_the_Tabfact_test_set.ipynb) | How to evaluate a fine-tuned *TapasForSequenceClassification* with a *tapas-base-finetuned-tabfact* checkpoint using a combination of the 🤗 datasets and 🤗 transformers libraries | [Niels Rogge](https://github.com/nielsrogge) | [!


key: _node_content
string_value:
{"id_": "17615da8-5d33-4460-a411-4ffebb331b07", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "37cbed30-c667-4ebd-96e9-00c55d9acc02", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0a4225ca-44d5-4954-a2fd-a2f434392276", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "9147ae27054d6af462ef8b943d5525823b7388d22adb2a75db88f5c8d36c85c1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "052038f9-fc1b-4e90-91f8-7db503be69e6", "node_type": "1", "metadata": {}, "hash": "e2ac6a16f6b7109db2f03ad1df0552e6370961d89eee34ed421acfcb26efbd4d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14703, "end_char_idx": 17436, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/RoBERTaShared_for_BBC_XSum.ipynb)|
|[Fine-tune TAPAS on Sequential Question Answering (SQA)](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb) | How to fine-tune *TapasForQuestionAnswering* with a *tapas-base* checkpoint on the Sequential Question Answering (SQA) dataset | [Niels Rogge](https://github.com/nielsrogge) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TAPAS/Fine_tuning_TapasForQuestionAnswering_on_SQA.ipynb)|
|[Evaluate TAPAS on Table Fact Checking (TabFact)](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/TAPAS/Evaluating_TAPAS_on_the_Tabfact_test_set.ipynb) | How to evaluate a fine-tuned *TapasForSequenceClassification* with a *tapas-base-finetuned-tabfact* checkpoint using a combination of the 🤗 datasets and 🤗 transformers libraries | [Niels Rogge](https://github.com/nielsrogge) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TAPAS/Evaluating_TAPAS_on_the_Tabfact_test_set.ipynb)|
|[Fine-tuning mBART for translation](https://colab.research.google.com/github/vasudevgupta7/huggingface-tutorials/blob/main/translation_training.ipynb) | How to fine-tune mBART using Seq2SeqTrainer for Hindi to English translation | [Vasudev Gupta](https://github.com/vasudevgupta7) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/vasudevgupta7/huggingface-tutorials/blob/main/translation_training.ipynb)|
|[Fine-tune LayoutLM on FUNSD (a form understanding dataset)](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LayoutLM/Fine_tuning_LayoutLMForTokenClassification_on_FUNSD.ipynb) | How to fine-tune *LayoutLMForTokenClassification* on the FUNSD dataset for information extraction from scanned documents | [Niels Rogge](https://github.com/nielsrogge) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLM/Fine_tuning_LayoutLMForTokenClassification_on_FUNSD.ipynb)|
|[Fine-Tune DistilGPT2 and Generate Text](https://colab.research.google.com/github/tripathiaakash/DistilGPT2-Tutorial/blob/main/distilgpt2_fine_tuning.ipynb) | How to fine-tune DistilGPT2 and generate text | [Aakash Tripathi](https://github.com/tripathiaakash) | [!


key: _node_content
string_value:
{"id_": "052038f9-fc1b-4e90-91f8-7db503be69e6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "37cbed30-c667-4ebd-96e9-00c55d9acc02", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "17615da8-5d33-4460-a411-4ffebb331b07", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "e7a4a68c89db29749dfa854d37ea870f0735b81b95ca660d59332132e94d5e4f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8c7a9b4d-ba33-4f07-8810-8f20a897ff40", "node_type": "1", "metadata": {}, "hash": "5ad18b33ca524874f40ea2c8c82314066533866bf20837faf9555c311f9123ac", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16380, "end_char_idx": 18946, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/vasudevgupta7/huggingface-tutorials/blob/main/translation_training.ipynb)|
|[Fine-tune LayoutLM on FUNSD (a form understanding dataset)](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LayoutLM/Fine_tuning_LayoutLMForTokenClassification_on_FUNSD.ipynb) | How to fine-tune *LayoutLMForTokenClassification* on the FUNSD dataset for information extraction from scanned documents | [Niels Rogge](https://github.com/nielsrogge) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLM/Fine_tuning_LayoutLMForTokenClassification_on_FUNSD.ipynb)|
|[Fine-Tune DistilGPT2 and Generate Text](https://colab.research.google.com/github/tripathiaakash/DistilGPT2-Tutorial/blob/main/distilgpt2_fine_tuning.ipynb) | How to fine-tune DistilGPT2 and generate text | [Aakash Tripathi](https://github.com/tripathiaakash) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/tripathiaakash/DistilGPT2-Tutorial/blob/main/distilgpt2_fine_tuning.ipynb)|
|[Fine-Tune LED on up to 8K tokens](https://github.com/patrickvonplaten/notebooks/blob/master/Fine_tune_Longformer_Encoder_Decoder_(LED)_for_Summarization_on_pubmed.ipynb) | How to fine-tune LED on pubmed for long-range summarization | [Patrick von Platen](https://github.com/patrickvonplaten) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/Fine_tune_Longformer_Encoder_Decoder_(LED)_for_Summarization_on_pubmed.ipynb)|
|[Evaluate LED on Arxiv](https://github.com/patrickvonplaten/notebooks/blob/master/LED_on_Arxiv.ipynb) | How to effectively evaluate LED on long-range summarization | [Patrick von Platen](https://github.com/patrickvonplaten) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/LED_on_Arxiv.ipynb)|
|[Fine-tune LayoutLM on RVL-CDIP (a document image classification dataset)](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LayoutLM/Fine_tuning_LayoutLMForSequenceClassification_on_RVL_CDIP.ipynb) | How to fine-tune *LayoutLMForSequenceClassification* on the RVL-CDIP dataset for scanned document classification | [Niels Rogge](https://github.com/nielsrogge) | [!


key: _node_content
string_value:
{"id_": "8c7a9b4d-ba33-4f07-8810-8f20a897ff40", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "37cbed30-c667-4ebd-96e9-00c55d9acc02", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "052038f9-fc1b-4e90-91f8-7db503be69e6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "241de393eaa45675b35b5331e7af6d912404ddc32a7343e02f582412f5989df5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9dff5aaf-e36c-4140-969b-61f9084a43a9", "node_type": "1", "metadata": {}, "hash": "efbb1bef0f0869670a2cc0b02a62fbbf397b2293d8f75852a29d8bd8fd617634", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17926, "end_char_idx": 20347, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/Fine_tune_Longformer_Encoder_Decoder_(LED)_for_Summarization_on_pubmed.ipynb)|
|[Evaluate LED on Arxiv](https://github.com/patrickvonplaten/notebooks/blob/master/LED_on_Arxiv.ipynb) | How to effectively evaluate LED on long-range summarization | [Patrick von Platen](https://github.com/patrickvonplaten) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/LED_on_Arxiv.ipynb)|
|[Fine-tune LayoutLM on RVL-CDIP (a document image classification dataset)](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LayoutLM/Fine_tuning_LayoutLMForSequenceClassification_on_RVL_CDIP.ipynb) | How to fine-tune *LayoutLMForSequenceClassification* on the RVL-CDIP dataset for scanned document classification | [Niels Rogge](https://github.com/nielsrogge) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/LayoutLM/Fine_tuning_LayoutLMForSequenceClassification_on_RVL_CDIP.ipynb)|
|[Wav2Vec2 CTC decoding with GPT2 adjustment](https://github.com/voidful/huggingface_notebook/blob/main/xlsr_gpt.ipynb) | How to decode CTC sequence with language model adjustment | [Eric Lam](https://github.com/voidful) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1e_z5jQHYbO2YKEaUgzb1ww1WwiAyydAj?usp=sharing)|
|[Fine-tune BART for summarization in two languages with Trainer class](https://github.com/elsanns/xai-nlp-notebooks/blob/master/fine_tune_bart_summarization_two_langs.ipynb) | How to fine-tune BART for summarization in two languages with Trainer class | [Eliza Szczechla](https://github.com/elsanns) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/elsanns/xai-nlp-notebooks/blob/master/fine_tune_bart_summarization_two_langs.ipynb)|
|[Evaluate Big Bird on Trivia QA](https://github.com/patrickvonplaten/notebooks/blob/master/Evaluating_Big_Bird_on_TriviaQA.ipynb) | How to evaluate BigBird on long document question answering on Trivia QA | [Patrick von Platen](https://github.com/patrickvonplaten) | [!


key: _node_content
string_value:
{"id_": "9dff5aaf-e36c-4140-969b-61f9084a43a9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "37cbed30-c667-4ebd-96e9-00c55d9acc02", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8c7a9b4d-ba33-4f07-8810-8f20a897ff40", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "af040384843084370ddeebeeb60e708a758ad5b169cffa2b7a7bbf15977d7577", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "961744e9-f570-408f-bf2c-98e3149f9dbb", "node_type": "1", "metadata": {}, "hash": "c334f9b6cc26964de64e7a5852add96f0ec3bb6db3ac6237a1751b9b85d77e9d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19408, "end_char_idx": 21397, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1e_z5jQHYbO2YKEaUgzb1ww1WwiAyydAj?usp=sharing)|
|[Fine-tune BART for summarization in two languages with Trainer class](https://github.com/elsanns/xai-nlp-notebooks/blob/master/fine_tune_bart_summarization_two_langs.ipynb) | How to fine-tune BART for summarization in two languages with Trainer class | [Eliza Szczechla](https://github.com/elsanns) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/elsanns/xai-nlp-notebooks/blob/master/fine_tune_bart_summarization_two_langs.ipynb)|
|[Evaluate Big Bird on Trivia QA](https://github.com/patrickvonplaten/notebooks/blob/master/Evaluating_Big_Bird_on_TriviaQA.ipynb) | How to evaluate BigBird on long document question answering on Trivia QA | [Patrick von Platen](https://github.com/patrickvonplaten) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/Evaluating_Big_Bird_on_TriviaQA.ipynb)|
| [Create video captions using Wav2Vec2](https://github.com/Muennighoff/ytclipcc/blob/main/wav2vec_youtube_captions.ipynb) | How to create YouTube captions from any video by transcribing the audio with Wav2Vec | [Niklas Muennighoff](https://github.com/Muennighoff) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Muennighoff/ytclipcc/blob/main/wav2vec_youtube_captions.ipynb) |
| [Fine-tune the Vision Transformer on CIFAR-10 using PyTorch Lightning](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/VisionTransformer/Fine_tuning_the_Vision_Transformer_on_CIFAR_10_with_PyTorch_Lightning.ipynb) | How to fine-tune the Vision Transformer (ViT) on CIFAR-10 using HuggingFace Transformers, Datasets and PyTorch Lightning | [Niels Rogge](https://github.com/nielsrogge) |[!


key: _node_content
string_value:
{"id_": "961744e9-f570-408f-bf2c-98e3149f9dbb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "37cbed30-c667-4ebd-96e9-00c55d9acc02", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9dff5aaf-e36c-4140-969b-61f9084a43a9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "5f5ab0b4cb8a4d6c3a629d2222f5b8bf136fe8b7c3ac25d5f7b2a33acfbe7997", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "12d5faed-47be-4238-a23c-d8607df9e3ea", "node_type": "1", "metadata": {}, "hash": "b49382e8ec3c496a6546d7ce7234f59ab3ed0a0d2e6de36c71c6c98eff613089", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20347, "end_char_idx": 22601, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/Evaluating_Big_Bird_on_TriviaQA.ipynb)|
| [Create video captions using Wav2Vec2](https://github.com/Muennighoff/ytclipcc/blob/main/wav2vec_youtube_captions.ipynb) | How to create YouTube captions from any video by transcribing the audio with Wav2Vec | [Niklas Muennighoff](https://github.com/Muennighoff) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Muennighoff/ytclipcc/blob/main/wav2vec_youtube_captions.ipynb) |
| [Fine-tune the Vision Transformer on CIFAR-10 using PyTorch Lightning](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/VisionTransformer/Fine_tuning_the_Vision_Transformer_on_CIFAR_10_with_PyTorch_Lightning.ipynb) | How to fine-tune the Vision Transformer (ViT) on CIFAR-10 using HuggingFace Transformers, Datasets and PyTorch Lightning | [Niels Rogge](https://github.com/nielsrogge) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/VisionTransformer/Fine_tuning_the_Vision_Transformer_on_CIFAR_10_with_PyTorch_Lightning.ipynb) |
| [Fine-tune the Vision Transformer on CIFAR-10 using the 🤗 Trainer](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/VisionTransformer/Fine_tuning_the_Vision_Transformer_on_CIFAR_10_with_the_%F0%9F%A4%97_Trainer.ipynb) | How to fine-tune the Vision Transformer (ViT) on CIFAR-10 using HuggingFace Transformers, Datasets and the 🤗 Trainer | [Niels Rogge](https://github.com/nielsrogge) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/VisionTransformer/Fine_tuning_the_Vision_Transformer_on_CIFAR_10_with_the_%F0%9F%A4%97_Trainer.ipynb) |
| [Evaluate LUKE on Open Entity, an entity typing dataset](https://github.com/studio-ousia/luke/blob/master/notebooks/huggingface_open_entity.ipynb) | How to evaluate *LukeForEntityClassification* on the Open Entity dataset | [Ikuya Yamada](https://github.com/ikuyamada) |[!


key: _node_content
string_value:
{"id_": "12d5faed-47be-4238-a23c-d8607df9e3ea", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "37cbed30-c667-4ebd-96e9-00c55d9acc02", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "961744e9-f570-408f-bf2c-98e3149f9dbb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "e6adedf0de056165e868a1ea43a9cee769668d353388004a864f0c9c1efa08af", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d5aa7f2a-581e-44e8-8d2a-00550b96a4ca", "node_type": "1", "metadata": {}, "hash": "d354221b9e7c8ee88d8535fdd44dc929672beac5dfa0df074ebd02aa949684f1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21397, "end_char_idx": 23982, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/VisionTransformer/Fine_tuning_the_Vision_Transformer_on_CIFAR_10_with_PyTorch_Lightning.ipynb) |
| [Fine-tune the Vision Transformer on CIFAR-10 using the 🤗 Trainer](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/VisionTransformer/Fine_tuning_the_Vision_Transformer_on_CIFAR_10_with_the_%F0%9F%A4%97_Trainer.ipynb) | How to fine-tune the Vision Transformer (ViT) on CIFAR-10 using HuggingFace Transformers, Datasets and the 🤗 Trainer | [Niels Rogge](https://github.com/nielsrogge) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/VisionTransformer/Fine_tuning_the_Vision_Transformer_on_CIFAR_10_with_the_%F0%9F%A4%97_Trainer.ipynb) |
| [Evaluate LUKE on Open Entity, an entity typing dataset](https://github.com/studio-ousia/luke/blob/master/notebooks/huggingface_open_entity.ipynb) | How to evaluate *LukeForEntityClassification* on the Open Entity dataset | [Ikuya Yamada](https://github.com/ikuyamada) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/studio-ousia/luke/blob/master/notebooks/huggingface_open_entity.ipynb) |
| [Evaluate LUKE on TACRED, a relation extraction dataset](https://github.com/studio-ousia/luke/blob/master/notebooks/huggingface_tacred.ipynb) | How to evaluate *LukeForEntityPairClassification* on the TACRED dataset | [Ikuya Yamada](https://github.com/ikuyamada) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/studio-ousia/luke/blob/master/notebooks/huggingface_tacred.ipynb) |
| [Evaluate LUKE on CoNLL-2003, an important NER benchmark](https://github.com/studio-ousia/luke/blob/master/notebooks/huggingface_conll_2003.ipynb) | How to evaluate *LukeForEntitySpanClassification* on the CoNLL-2003 dataset | [Ikuya Yamada](https://github.com/ikuyamada) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/studio-ousia/luke/blob/master/notebooks/huggingface_conll_2003.ipynb) |
| [Evaluate BigBird-Pegasus on PubMed dataset](https://github.com/vasudevgupta7/bigbird/blob/main/notebooks/bigbird_pegasus_evaluation.ipynb) | How to evaluate *BigBirdPegasusForConditionalGeneration* on PubMed dataset | [Vasudev Gupta](https://github.com/vasudevgupta7) | [!


key: _node_content
string_value:
{"id_": "d5aa7f2a-581e-44e8-8d2a-00550b96a4ca", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "37cbed30-c667-4ebd-96e9-00c55d9acc02", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "12d5faed-47be-4238-a23c-d8607df9e3ea", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "9930f64b672dee2107d148977b687f8e97b83fbf0abd7f6055a90a4bb1af667b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ad989800-e964-4ea9-82aa-367c531f364f", "node_type": "1", "metadata": {}, "hash": "f7d658bf782333a73641622c708d7a5f434e7253ee90567e028039a46c0ffe1f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23058, "end_char_idx": 25572, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/studio-ousia/luke/blob/master/notebooks/huggingface_tacred.ipynb) |
| [Evaluate LUKE on CoNLL-2003, an important NER benchmark](https://github.com/studio-ousia/luke/blob/master/notebooks/huggingface_conll_2003.ipynb) | How to evaluate *LukeForEntitySpanClassification* on the CoNLL-2003 dataset | [Ikuya Yamada](https://github.com/ikuyamada) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/studio-ousia/luke/blob/master/notebooks/huggingface_conll_2003.ipynb) |
| [Evaluate BigBird-Pegasus on PubMed dataset](https://github.com/vasudevgupta7/bigbird/blob/main/notebooks/bigbird_pegasus_evaluation.ipynb) | How to evaluate *BigBirdPegasusForConditionalGeneration* on PubMed dataset | [Vasudev Gupta](https://github.com/vasudevgupta7) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/vasudevgupta7/bigbird/blob/main/notebooks/bigbird_pegasus_evaluation.ipynb) |
| [Speech Emotion Classification with Wav2Vec2](https://github/m3hrdadfi/soxan/blob/main/notebooks/Emotion_recognition_in_Greek_speech_using_Wav2Vec2.ipynb) | How to leverage a pretrained Wav2Vec2 model for Emotion Classification on the MEGA dataset | [Mehrdad Farahani](https://github.com/m3hrdadfi) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/m3hrdadfi/soxan/blob/main/notebooks/Emotion_recognition_in_Greek_speech_using_Wav2Vec2.ipynb) |
| [Detect objects in an image with DETR](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/DETR/DETR_minimal_example_(with_DetrFeatureExtractor).ipynb) | How to use a trained *DetrForObjectDetection* model to detect objects in an image and visualize attention | [Niels Rogge](https://github.com/NielsRogge) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/DETR/DETR_minimal_example_(with_DetrFeatureExtractor).ipynb) |
| [Fine-tune DETR on a custom object detection dataset](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/DETR/Fine_tuning_DetrForObjectDetection_on_custom_dataset_(balloon).ipynb) | How to fine-tune *DetrForObjectDetection* on a custom object detection dataset | [Niels Rogge](https://github.com/NielsRogge) | [!


key: _node_content
string_value:
{"id_": "ad989800-e964-4ea9-82aa-367c531f364f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "37cbed30-c667-4ebd-96e9-00c55d9acc02", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "74d714af8e35fb1703fcc4f44e23228f715556eae56a6c7cf14ac422368d5a08", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d5aa7f2a-581e-44e8-8d2a-00550b96a4ca", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/community", "title": "Community", "tokens": 7568, "retrieve_doc": true, "source": "transformers"}, "hash": "042ce2a793b648ceb4dd9b88416732f08900ec361f6032c5d461bced7f02cb8f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24481, "end_char_idx": 26213, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/m3hrdadfi/soxan/blob/main/notebooks/Emotion_recognition_in_Greek_speech_using_Wav2Vec2.ipynb) |
| [Detect objects in an image with DETR](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/DETR/DETR_minimal_example_(with_DetrFeatureExtractor).ipynb) | How to use a trained *DetrForObjectDetection* model to detect objects in an image and visualize attention | [Niels Rogge](https://github.com/NielsRogge) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/DETR/DETR_minimal_example_(with_DetrFeatureExtractor).ipynb) |
| [Fine-tune DETR on a custom object detection dataset](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/DETR/Fine_tuning_DetrForObjectDetection_on_custom_dataset_(balloon).ipynb) | How to fine-tune *DetrForObjectDetection* on a custom object detection dataset | [Niels Rogge](https://github.com/NielsRogge) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/DETR/Fine_tuning_DetrForObjectDetection_on_custom_dataset_(balloon).ipynb) |
| [Finetune T5 for Named Entity Recognition](https://github.com/ToluClassics/Notebooks/blob/main/T5_Ner_Finetuning.ipynb) | How to fine-tune *T5* on a Named Entity Recognition Task | [Ogundepo Odunayo](https://github.com/ToluClassics) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1obr78FY_cBmWY5ODViCmzdY6O1KB65Vc?usp=sharing) |


key: _node_content
string_value:
{"id_": "914a742b-a14c-48a8-8e90-4f88ec17e0aa", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/troubleshooting", "title": "Troubleshoot", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b5dbe511-377f-4069-9548-35adb6bc6f8d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/troubleshooting", "title": "Troubleshoot", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "cf7ceb63fcd69053af31c31e769d7cbe115106f05438ffa92b3abb285b66d3aa", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e4feea49-00f4-404d-8be5-4dd846842f4c", "node_type": "1", "metadata": {}, "hash": "b3dd10e4f76c6551c63850c81e5b0af2754fd19389c8c589fd05bb452412f72f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3294, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<!---
Copyright 2022 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
rendered properly in your Markdown viewer.

-->

# Troubleshoot

Sometimes errors occur, but we are here to help! This guide covers some of the most common issues we've seen and how you can resolve them. However, this guide isn't meant to be a comprehensive collection of every 🤗 Transformers issue. For more help with troubleshooting your issue, try:

<Youtube id="S2EEG3JIt2A"/>

1. Asking for help on the [forums](https://discuss.huggingface.co/). There are specific categories you can post your question to, like [Beginners](https://discuss.huggingface.co/c/beginners/5) or [🤗 Transformers](https://discuss.huggingface.co/c/transformers/9). Make sure you write a good descriptive forum post with some reproducible code to maximize the likelihood that your problem is solved!

<Youtube id="_PAli-V4wj0"/>

2. Create an [Issue](https://github.com/huggingface/transformers/issues/new/choose) on the 🤗 Transformers repository if it is a bug related to the library. Try to include as much information describing the bug as possible to help us better figure out what's wrong and how we can fix it.

3. Check the [Migration](migration) guide if you use an older version of 🤗 Transformers since some important changes have been introduced between versions.

For more details about troubleshooting and getting help, take a look at [Chapter 8](https://huggingface.co/course/chapter8/1?fw=pt) of the Hugging Face course.


## Firewalled environments

Some GPU instances on cloud and intranet setups are firewalled to external connections, resulting in a connection error. When your script attempts to download model weights or datasets, the download will hang and then timeout with the following message:

```
ValueError: Connection error, and we cannot find the requested files in the cached path.
Please try again or make sure your Internet connection is on.
```

In this case, you should try to run 🤗 Transformers on [offline mode](installation#offline-mode) to avoid the connection error.

## CUDA out of memory

Training large models with millions of parameters can be challenging without the appropriate hardware. A common error you may encounter when the GPU runs out of memory is:

```
CUDA out of memory. Tried to allocate 256.00 MiB (GPU 0; 11.17 GiB total capacity; 9.70 GiB already allocated; 179.81 MiB free; 9.85 GiB reserved in total by PyTorch)
```

Here are some potential solutions you can try to lessen memory use:

- Reduce the [`per_device_train_batch_size`](main_classes/trainer#transformers.TrainingArguments.per_device_train_batch_size) value in [`TrainingArguments`].


key: _node_content
string_value:
{"id_": "e4feea49-00f4-404d-8be5-4dd846842f4c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/troubleshooting", "title": "Troubleshoot", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b5dbe511-377f-4069-9548-35adb6bc6f8d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/troubleshooting", "title": "Troubleshoot", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "cf7ceb63fcd69053af31c31e769d7cbe115106f05438ffa92b3abb285b66d3aa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "914a742b-a14c-48a8-8e90-4f88ec17e0aa", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/troubleshooting", "title": "Troubleshoot", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "1da95ab0384de49a8832d67734cb621930471eb74752c7e9d7a3602fb0a7191a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7eb06962-af0e-4bd8-939a-01bfa835705f", "node_type": "1", "metadata": {}, "hash": "568b298d42e80db4371c2dcff4595575dee08541b8363fab802dafe31b46e4f9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2125, "end_char_idx": 5499, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Firewalled environments

Some GPU instances on cloud and intranet setups are firewalled to external connections, resulting in a connection error. When your script attempts to download model weights or datasets, the download will hang and then timeout with the following message:

```
ValueError: Connection error, and we cannot find the requested files in the cached path.
Please try again or make sure your Internet connection is on.
```

In this case, you should try to run 🤗 Transformers on [offline mode](installation#offline-mode) to avoid the connection error.

## CUDA out of memory

Training large models with millions of parameters can be challenging without the appropriate hardware. A common error you may encounter when the GPU runs out of memory is:

```
CUDA out of memory. Tried to allocate 256.00 MiB (GPU 0; 11.17 GiB total capacity; 9.70 GiB already allocated; 179.81 MiB free; 9.85 GiB reserved in total by PyTorch)
```

Here are some potential solutions you can try to lessen memory use:

- Reduce the [`per_device_train_batch_size`](main_classes/trainer#transformers.TrainingArguments.per_device_train_batch_size) value in [`TrainingArguments`].
- Try using [`gradient_accumulation_steps`](main_classes/trainer#transformers.TrainingArguments.gradient_accumulation_steps) in [`TrainingArguments`] to effectively increase overall batch size.

<Tip>

Refer to the Performance [guide](performance) for more details about memory-saving techniques.

</Tip>

## Unable to load a saved TensorFlow model

TensorFlow's [model.save](https://www.tensorflow.org/tutorials/keras/save_and_load#save_the_entire_model) method will save the entire model - architecture, weights, training configuration - in a single file. However, when you load the model file again, you may run into an error because 🤗 Transformers may not load all the TensorFlow-related objects in the model file. To avoid issues with saving and loading TensorFlow models, we recommend you:

- Save the model weights as a `h5` file extension with [`model.save_weights`](https://www.tensorflow.org/tutorials/keras/save_and_load#save_the_entire_model) and then reload the model with [`~TFPreTrainedModel.from_pretrained`]:

```py
>>> from transformers import TFPreTrainedModel
>>> from tensorflow import keras

>>> model.save_weights("some_folder/tf_model.h5")
>>> model = TFPreTrainedModel.from_pretrained("some_folder")
```

- Save the model with [`~TFPretrainedModel.save_pretrained`] and load it again with [`~TFPreTrainedModel.from_pretrained`]:

```py
>>> from transformers import TFPreTrainedModel

>>> model.save_pretrained("path_to/model")
>>> model = TFPreTrainedModel.from_pretrained("path_to/model")
```

## ImportError

Another common error you may encounter, especially if it is a newly released model, is `ImportError`:

```
ImportError: cannot import name 'ImageGPTImageProcessor' from 'transformers' (unknown location)
```

For these error types, check to make sure you have the latest version of 🤗 Transformers installed to access the most recent models:

```bash
pip install transformers --upgrade
```

## CUDA error: device-side assert triggered

Sometimes you may run into a generic CUDA error about an error in the device code.

```
RuntimeError: CUDA error: device-side assert triggered
```

You should try to run the code on a CPU first to get a more descriptive error message.


key: _node_content
string_value:
{"id_": "7eb06962-af0e-4bd8-939a-01bfa835705f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/troubleshooting", "title": "Troubleshoot", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b5dbe511-377f-4069-9548-35adb6bc6f8d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/troubleshooting", "title": "Troubleshoot", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "cf7ceb63fcd69053af31c31e769d7cbe115106f05438ffa92b3abb285b66d3aa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e4feea49-00f4-404d-8be5-4dd846842f4c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/troubleshooting", "title": "Troubleshoot", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "7e813b2db2bdca09ca22968e591bdff8ebc326d78dfdcae1f448e72b12dbf2e7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3548e6ab-f4fd-423b-baf9-9ea296c2f9b1", "node_type": "1", "metadata": {}, "hash": "d092617d771890efe17cf01a85b00b48d02a30f3f9a2f5934b0f548cb950fac5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3853, "end_char_idx": 6260, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, when you load the model file again, you may run into an error because 🤗 Transformers may not load all the TensorFlow-related objects in the model file. To avoid issues with saving and loading TensorFlow models, we recommend you:

- Save the model weights as a `h5` file extension with [`model.save_weights`](https://www.tensorflow.org/tutorials/keras/save_and_load#save_the_entire_model) and then reload the model with [`~TFPreTrainedModel.from_pretrained`]:

```py
>>> from transformers import TFPreTrainedModel
>>> from tensorflow import keras

>>> model.save_weights("some_folder/tf_model.h5")
>>> model = TFPreTrainedModel.from_pretrained("some_folder")
```

- Save the model with [`~TFPretrainedModel.save_pretrained`] and load it again with [`~TFPreTrainedModel.from_pretrained`]:

```py
>>> from transformers import TFPreTrainedModel

>>> model.save_pretrained("path_to/model")
>>> model = TFPreTrainedModel.from_pretrained("path_to/model")
```

## ImportError

Another common error you may encounter, especially if it is a newly released model, is `ImportError`:

```
ImportError: cannot import name 'ImageGPTImageProcessor' from 'transformers' (unknown location)
```

For these error types, check to make sure you have the latest version of 🤗 Transformers installed to access the most recent models:

```bash
pip install transformers --upgrade
```

## CUDA error: device-side assert triggered

Sometimes you may run into a generic CUDA error about an error in the device code.

```
RuntimeError: CUDA error: device-side assert triggered
```

You should try to run the code on a CPU first to get a more descriptive error message. Add the following environment variable to the beginning of your code to switch to a CPU:

```py
>>> import os

>>> os.environ["CUDA_VISIBLE_DEVICES"] = ""
```

Another option is to get a better traceback from the GPU. Add the following environment variable to the beginning of your code to get the traceback to point to the source of the error:

```py
>>> import os

>>> os.environ["CUDA_LAUNCH_BLOCKING"] = "1"
```

## Incorrect output when padding tokens aren't masked

In some cases, the output `hidden_state` may be incorrect if the `input_ids` include padding tokens. To demonstrate, load a model and tokenizer. You can access a model's `pad_token_id` to see its value. The `pad_token_id` may be `None` for some models, but you can always manually set it.


key: _node_content
string_value:
{"id_": "3548e6ab-f4fd-423b-baf9-9ea296c2f9b1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/troubleshooting", "title": "Troubleshoot", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b5dbe511-377f-4069-9548-35adb6bc6f8d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/troubleshooting", "title": "Troubleshoot", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "cf7ceb63fcd69053af31c31e769d7cbe115106f05438ffa92b3abb285b66d3aa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7eb06962-af0e-4bd8-939a-01bfa835705f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/troubleshooting", "title": "Troubleshoot", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "d949a44b8448370e7fe0b8f1778455d9ac4a2c1fa78b4debd4098eaddb920fdc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6be726e1-376f-4cf8-84b5-95fa4b7c6674", "node_type": "1", "metadata": {}, "hash": "600f469dd4f389d108283ff543bffeac5b0baa6e042a33481ac5febbdedadcaf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5349, "end_char_idx": 8395, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```
RuntimeError: CUDA error: device-side assert triggered
```

You should try to run the code on a CPU first to get a more descriptive error message. Add the following environment variable to the beginning of your code to switch to a CPU:

```py
>>> import os

>>> os.environ["CUDA_VISIBLE_DEVICES"] = ""
```

Another option is to get a better traceback from the GPU. Add the following environment variable to the beginning of your code to get the traceback to point to the source of the error:

```py
>>> import os

>>> os.environ["CUDA_LAUNCH_BLOCKING"] = "1"
```

## Incorrect output when padding tokens aren't masked

In some cases, the output `hidden_state` may be incorrect if the `input_ids` include padding tokens. To demonstrate, load a model and tokenizer. You can access a model's `pad_token_id` to see its value. The `pad_token_id` may be `None` for some models, but you can always manually set it.

```py
>>> from transformers import AutoModelForSequenceClassification
>>> import torch

>>> model = AutoModelForSequenceClassification.from_pretrained("google-bert/bert-base-uncased")
>>> model.config.pad_token_id
0
```

The following example shows the output without masking the padding tokens:

```py
>>> input_ids = torch.tensor([[7592, 2057, 2097, 2393, 9611, 2115], [7592, 0, 0, 0, 0, 0]])
>>> output = model(input_ids)
>>> print(output.logits)
tensor([[ 0.0082, -0.2307],
        [ 0.1317, -0.1683]], grad_fn=<AddmmBackward0>)
```

Here is the actual output of the second sequence:

```py
>>> input_ids = torch.tensor([[7592]])
>>> output = model(input_ids)
>>> print(output.logits)
tensor([[-0.1008, -0.4061]], grad_fn=<AddmmBackward0>)
```

Most of the time, you should provide an `attention_mask` to your model to ignore the padding tokens to avoid this silent error. Now the output of the second sequence matches its actual output:

<Tip>

By default, the tokenizer creates an `attention_mask` for you based on your specific tokenizer's defaults.

</Tip>

```py
>>> attention_mask = torch.tensor([[1, 1, 1, 1, 1, 1], [1, 0, 0, 0, 0, 0]])
>>> output = model(input_ids, attention_mask=attention_mask)
>>> print(output.logits)
tensor([[ 0.0082, -0.2307],
        [-0.1008, -0.4061]], grad_fn=<AddmmBackward0>)
```

🤗 Transformers doesn't automatically create an `attention_mask` to mask a padding token if it is provided because:

- Some models don't have a padding token.
- For some use-cases, users want a model to attend to a padding token.

## ValueError: Unrecognized configuration class XYZ for this kind of AutoModel

Generally, we recommend using the [`AutoModel`] class to load pretrained instances of models. This class
can automatically infer and load the correct architecture from a given checkpoint based on the configuration. If you see
this `ValueError` when loading a model from a checkpoint, this means the Auto class couldn't find a mapping from
the configuration in the given checkpoint to the kind of model you are trying to load. Most commonly, this happens when a
checkpoint doesn't support a given task.


key: _node_content
string_value:
{"id_": "6be726e1-376f-4cf8-84b5-95fa4b7c6674", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/troubleshooting", "title": "Troubleshoot", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b5dbe511-377f-4069-9548-35adb6bc6f8d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/troubleshooting", "title": "Troubleshoot", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "cf7ceb63fcd69053af31c31e769d7cbe115106f05438ffa92b3abb285b66d3aa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3548e6ab-f4fd-423b-baf9-9ea296c2f9b1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/troubleshooting", "title": "Troubleshoot", "tokens": 2128, "retrieve_doc": true, "source": "transformers"}, "hash": "597354f279b4e601ba3008b16676ceaa7fb27c8b3aaaac36042bd6b73758807b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7139, "end_char_idx": 9048, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Now the output of the second sequence matches its actual output:

<Tip>

By default, the tokenizer creates an `attention_mask` for you based on your specific tokenizer's defaults.

</Tip>

```py
>>> attention_mask = torch.tensor([[1, 1, 1, 1, 1, 1], [1, 0, 0, 0, 0, 0]])
>>> output = model(input_ids, attention_mask=attention_mask)
>>> print(output.logits)
tensor([[ 0.0082, -0.2307],
        [-0.1008, -0.4061]], grad_fn=<AddmmBackward0>)
```

🤗 Transformers doesn't automatically create an `attention_mask` to mask a padding token if it is provided because:

- Some models don't have a padding token.
- For some use-cases, users want a model to attend to a padding token.

## ValueError: Unrecognized configuration class XYZ for this kind of AutoModel

Generally, we recommend using the [`AutoModel`] class to load pretrained instances of models. This class
can automatically infer and load the correct architecture from a given checkpoint based on the configuration. If you see
this `ValueError` when loading a model from a checkpoint, this means the Auto class couldn't find a mapping from
the configuration in the given checkpoint to the kind of model you are trying to load. Most commonly, this happens when a
checkpoint doesn't support a given task.
For instance, you'll see this error in the following example because there is no GPT2 for question answering:

```py
>>> from transformers import AutoProcessor, AutoModelForQuestionAnswering

>>> processor = AutoProcessor.from_pretrained("openai-community/gpt2-medium")
>>> model = AutoModelForQuestionAnswering.from_pretrained("openai-community/gpt2-medium")
ValueError: Unrecognized configuration class <class 'transformers.models.gpt2.configuration_gpt2.GPT2Config'> for this kind of AutoModel: AutoModelForQuestionAnswering.
Model type should be one of AlbertConfig, BartConfig, BertConfig, BigBirdConfig, BigBirdPegasusConfig, BloomConfig, ...
```


key: _node_content
string_value:
{"id_": "d119712c-f3ca-49a2-ace8-a9e7da5c1079", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0ce28ad3-3451-4013-81ff-30e122e7137e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "hash": "b97638da20ee8948778fc11261763dd7a2666a79a5866a41f99ad4299f694ae6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a0634133-cce3-483b-a9b6-68cd3547b7a7", "node_type": "1", "metadata": {}, "hash": "931116e3deacf06a9e2f17d13ea7291f63b37d84bf62f1bea4a600e10f1d1523", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2694, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Export to ONNX

Deploying 🤗 Transformers models in production environments often requires, or can benefit from exporting the models into 
a serialized format that can be loaded and executed on specialized runtimes and hardware.

🤗 Optimum is an extension of Transformers that enables exporting models from PyTorch or TensorFlow to serialized formats 
such as ONNX and TFLite through its `exporters` module. 🤗 Optimum also provides a set of performance optimization tools to train 
and run models on targeted hardware with maximum efficiency.

This guide demonstrates how you can export 🤗 Transformers models to ONNX with 🤗 Optimum, for the guide on exporting models to TFLite, 
please refer to the [Export to TFLite page](tflite).

## Export to ONNX 

[ONNX (Open Neural Network eXchange)](http://onnx.ai) is an open standard that defines a common set of operators and a 
common file format to represent deep learning models in a wide variety of frameworks, including PyTorch and
TensorFlow. When a model is exported to the ONNX format, these operators are used to
construct a computational graph (often called an _intermediate representation_) which
represents the flow of data through the neural network.

By exposing a graph with standardized operators and data types, ONNX makes it easy to
switch between frameworks. For example, a model trained in PyTorch can be exported to
ONNX format and then imported in TensorFlow (and vice versa).

Once exported to ONNX format, a model can be:
- optimized for inference via techniques such as [graph optimization](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) and [quantization](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization). 
- run with ONNX Runtime via [`ORTModelForXXX` classes](https://huggingface.co/docs/optimum/onnxruntime/package_reference/modeling_ort),
which follow the same `AutoModel` API as the one you are used to in 🤗 Transformers.
- run with [optimized inference pipelines](https://huggingface.co/docs/optimum/main/en/onnxruntime/usage_guides/pipelines),
which has the same API as the [`pipeline`] function in 🤗 Transformers. 

🤗 Optimum provides support for the ONNX export by leveraging configuration objects. These configuration objects come 
ready-made for a number of model architectures, and are designed to be easily extendable to other architectures.

For the list of ready-made configurations, please refer to [🤗 Optimum documentation](https://huggingface.co/docs/optimum/exporters/onnx/overview).

There are two ways to export a 🤗 Transformers model to ONNX, here we show both:

- export with 🤗 Optimum via CLI.
- export with 🤗 Optimum with `optimum.onnxruntime`.


key: _node_content
string_value:
{"id_": "a0634133-cce3-483b-a9b6-68cd3547b7a7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0ce28ad3-3451-4013-81ff-30e122e7137e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "hash": "b97638da20ee8948778fc11261763dd7a2666a79a5866a41f99ad4299f694ae6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d119712c-f3ca-49a2-ace8-a9e7da5c1079", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "hash": "4a53a6ddf954af28b05288e06c33c5f1c0f3437a8886a02e96d15c3f5fd3ba26", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c7b9945e-864a-4021-9a45-f4006f879b1a", "node_type": "1", "metadata": {}, "hash": "345ffc7a2848e26b8c9f520076bb4fc57d3fd19f076f6ca2a0b39a6903913f41", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 994, "end_char_idx": 4083, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
When a model is exported to the ONNX format, these operators are used to
construct a computational graph (often called an _intermediate representation_) which
represents the flow of data through the neural network.

By exposing a graph with standardized operators and data types, ONNX makes it easy to
switch between frameworks. For example, a model trained in PyTorch can be exported to
ONNX format and then imported in TensorFlow (and vice versa).

Once exported to ONNX format, a model can be:
- optimized for inference via techniques such as [graph optimization](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/optimization) and [quantization](https://huggingface.co/docs/optimum/onnxruntime/usage_guides/quantization). 
- run with ONNX Runtime via [`ORTModelForXXX` classes](https://huggingface.co/docs/optimum/onnxruntime/package_reference/modeling_ort),
which follow the same `AutoModel` API as the one you are used to in 🤗 Transformers.
- run with [optimized inference pipelines](https://huggingface.co/docs/optimum/main/en/onnxruntime/usage_guides/pipelines),
which has the same API as the [`pipeline`] function in 🤗 Transformers. 

🤗 Optimum provides support for the ONNX export by leveraging configuration objects. These configuration objects come 
ready-made for a number of model architectures, and are designed to be easily extendable to other architectures.

For the list of ready-made configurations, please refer to [🤗 Optimum documentation](https://huggingface.co/docs/optimum/exporters/onnx/overview).

There are two ways to export a 🤗 Transformers model to ONNX, here we show both:

- export with 🤗 Optimum via CLI.
- export with 🤗 Optimum with `optimum.onnxruntime`.

### Exporting a 🤗 Transformers model to ONNX with CLI

To export a 🤗 Transformers model to ONNX, first install an extra dependency:

```bash
pip install optimum[exporters]
```

To check out all available arguments, refer to the [🤗 Optimum docs](https://huggingface.co/docs/optimum/exporters/onnx/usage_guides/export_a_model#exporting-a-model-to-onnx-using-the-cli), 
or view help in command line:

```bash
optimum-cli export onnx --help
```

To export a model's checkpoint from the 🤗 Hub, for example, `distilbert/distilbert-base-uncased-distilled-squad`, run the following command: 

```bash
optimum-cli export onnx --model distilbert/distilbert-base-uncased-distilled-squad distilbert_base_uncased_squad_onnx/
```

You should see the logs indicating progress and showing where the resulting `model.onnx` is saved, like this:

```bash
Validating ONNX model distilbert_base_uncased_squad_onnx/model.onnx...
	-[✓] ONNX model output names match reference model (start_logits, end_logits)
	- Validating ONNX Model output "start_logits":
		-[✓] (2, 16) matches (2, 16)
		-[✓] all values close (atol: 0.0001)
	- Validating ONNX Model output "end_logits":
		-[✓] (2, 16) matches (2, 16)
		-[✓] all values close (atol: 0.0001)
The ONNX export succeeded and the exported model was saved at: distilbert_base_uncased_squad_onnx
```

The example above illustrates exporting a checkpoint from 🤗 Hub.


key: _node_content
string_value:
{"id_": "c7b9945e-864a-4021-9a45-f4006f879b1a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0ce28ad3-3451-4013-81ff-30e122e7137e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "hash": "b97638da20ee8948778fc11261763dd7a2666a79a5866a41f99ad4299f694ae6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a0634133-cce3-483b-a9b6-68cd3547b7a7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "hash": "1e1f4e9fa7bb3fd1839a3571e8d4ab7bf9438c1d296500a931de4c48a3c82dbf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d2adb3aa-0828-4c0c-9b3f-8307e8b66340", "node_type": "1", "metadata": {}, "hash": "7d11f21fa86f5366da58a521326e1d04781c99acdd252ed4c285ae7f564a002a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2696, "end_char_idx": 5498, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Exporting a 🤗 Transformers model to ONNX with CLI

To export a 🤗 Transformers model to ONNX, first install an extra dependency:

```bash
pip install optimum[exporters]
```

To check out all available arguments, refer to the [🤗 Optimum docs](https://huggingface.co/docs/optimum/exporters/onnx/usage_guides/export_a_model#exporting-a-model-to-onnx-using-the-cli), 
or view help in command line:

```bash
optimum-cli export onnx --help
```

To export a model's checkpoint from the 🤗 Hub, for example, `distilbert/distilbert-base-uncased-distilled-squad`, run the following command: 

```bash
optimum-cli export onnx --model distilbert/distilbert-base-uncased-distilled-squad distilbert_base_uncased_squad_onnx/
```

You should see the logs indicating progress and showing where the resulting `model.onnx` is saved, like this:

```bash
Validating ONNX model distilbert_base_uncased_squad_onnx/model.onnx...
	-[✓] ONNX model output names match reference model (start_logits, end_logits)
	- Validating ONNX Model output "start_logits":
		-[✓] (2, 16) matches (2, 16)
		-[✓] all values close (atol: 0.0001)
	- Validating ONNX Model output "end_logits":
		-[✓] (2, 16) matches (2, 16)
		-[✓] all values close (atol: 0.0001)
The ONNX export succeeded and the exported model was saved at: distilbert_base_uncased_squad_onnx
```

The example above illustrates exporting a checkpoint from 🤗 Hub. When exporting a local model, first make sure that you 
saved both the model's weights and tokenizer files in the same directory (`local_path`). When using CLI, pass the 
`local_path` to the `model` argument instead of the checkpoint name on 🤗 Hub and provide the `--task` argument. 
You can review the list of supported tasks in the [🤗 Optimum documentation](https://huggingface.co/docs/optimum/exporters/task_manager).
If `task` argument is not provided, it will default to the model architecture without any task specific head.

```bash
optimum-cli export onnx --model local_path --task question-answering distilbert_base_uncased_squad_onnx/
```

The resulting `model.onnx` file can then be run on one of the [many
accelerators](https://onnx.ai/supported-tools.html#deployModel) that support the ONNX
standard. For example, we can load and run the model with [ONNX
Runtime](https://onnxruntime.ai/) as follows:

```python
>>> from transformers import AutoTokenizer
>>> from optimum.onnxruntime import ORTModelForQuestionAnswering

>>> tokenizer = AutoTokenizer.from_pretrained("distilbert_base_uncased_squad_onnx")
>>> model = ORTModelForQuestionAnswering.from_pretrained("distilbert_base_uncased_squad_onnx")
>>> inputs = tokenizer("What am I using?", "Using DistilBERT with ONNX Runtime!", return_tensors="pt")
>>> outputs = model(**inputs)
```

The process is identical for TensorFlow checkpoints on the Hub.


key: _node_content
string_value:
{"id_": "d2adb3aa-0828-4c0c-9b3f-8307e8b66340", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0ce28ad3-3451-4013-81ff-30e122e7137e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "hash": "b97638da20ee8948778fc11261763dd7a2666a79a5866a41f99ad4299f694ae6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c7b9945e-864a-4021-9a45-f4006f879b1a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "hash": "5d6b1a6b3f867adb5925e021d8970e56a542c7fc72f6f14d396d2d0c35d40df5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c2aa3620-04b7-4c58-8e86-5e8f97c1840d", "node_type": "1", "metadata": {}, "hash": "5eb36c9de8909b05989aa9b7d03e7770055a28c4a9b5d7ef1c558ec87d97c263", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4084, "end_char_idx": 7106, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
When exporting a local model, first make sure that you 
saved both the model's weights and tokenizer files in the same directory (`local_path`). When using CLI, pass the 
`local_path` to the `model` argument instead of the checkpoint name on 🤗 Hub and provide the `--task` argument. 
You can review the list of supported tasks in the [🤗 Optimum documentation](https://huggingface.co/docs/optimum/exporters/task_manager).
If `task` argument is not provided, it will default to the model architecture without any task specific head.

```bash
optimum-cli export onnx --model local_path --task question-answering distilbert_base_uncased_squad_onnx/
```

The resulting `model.onnx` file can then be run on one of the [many
accelerators](https://onnx.ai/supported-tools.html#deployModel) that support the ONNX
standard. For example, we can load and run the model with [ONNX
Runtime](https://onnxruntime.ai/) as follows:

```python
>>> from transformers import AutoTokenizer
>>> from optimum.onnxruntime import ORTModelForQuestionAnswering

>>> tokenizer = AutoTokenizer.from_pretrained("distilbert_base_uncased_squad_onnx")
>>> model = ORTModelForQuestionAnswering.from_pretrained("distilbert_base_uncased_squad_onnx")
>>> inputs = tokenizer("What am I using?", "Using DistilBERT with ONNX Runtime!", return_tensors="pt")
>>> outputs = model(**inputs)
```

The process is identical for TensorFlow checkpoints on the Hub. For instance, here's how you would
export a pure TensorFlow checkpoint from the [Keras organization](https://huggingface.co/keras-io):

```bash
optimum-cli export onnx --model keras-io/transformers-qa distilbert_base_cased_squad_onnx/
```

### Exporting a 🤗 Transformers model to ONNX with `optimum.onnxruntime`

Alternative to CLI, you can export a 🤗 Transformers model to ONNX programmatically like so: 

```python
>>> from optimum.onnxruntime import ORTModelForSequenceClassification
>>> from transformers import AutoTokenizer

>>> model_checkpoint = "distilbert_base_uncased_squad"
>>> save_directory = "onnx/"

>>> # Load a model from transformers and export it to ONNX
>>> ort_model = ORTModelForSequenceClassification.from_pretrained(model_checkpoint, export=True)
>>> tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)

>>> # Save the onnx model and tokenizer
>>> ort_model.save_pretrained(save_directory)
>>> tokenizer.save_pretrained(save_directory)
```

### Exporting a model for an unsupported architecture

If you wish to contribute by adding support for a model that cannot be currently exported, you should first check if it is
supported in [`optimum.exporters.onnx`](https://huggingface.co/docs/optimum/exporters/onnx/overview),
and if it is not, [contribute to 🤗 Optimum](https://huggingface.co/docs/optimum/exporters/onnx/usage_guides/contribute)
directly.

### Exporting a model with `transformers.onnx`

<Tip warning={true}>

`tranformers.onnx` is no longer maintained, please export models with 🤗 Optimum as described above. This section will be removed in the future versions.


key: _node_content
string_value:
{"id_": "c2aa3620-04b7-4c58-8e86-5e8f97c1840d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0ce28ad3-3451-4013-81ff-30e122e7137e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "hash": "b97638da20ee8948778fc11261763dd7a2666a79a5866a41f99ad4299f694ae6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d2adb3aa-0828-4c0c-9b3f-8307e8b66340", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "hash": "22c6d0444383c509054462501ea7e014de72b2ee3129e4fe2a6475840b317499", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "351b2714-0c14-4a03-8945-53fff3b30202", "node_type": "1", "metadata": {}, "hash": "3c66741d635fb69a621735c0e4c3442c0c99f4715f545ef96d51b4b5c378de9f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5499, "end_char_idx": 8714, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For instance, here's how you would
export a pure TensorFlow checkpoint from the [Keras organization](https://huggingface.co/keras-io):

```bash
optimum-cli export onnx --model keras-io/transformers-qa distilbert_base_cased_squad_onnx/
```

### Exporting a 🤗 Transformers model to ONNX with `optimum.onnxruntime`

Alternative to CLI, you can export a 🤗 Transformers model to ONNX programmatically like so: 

```python
>>> from optimum.onnxruntime import ORTModelForSequenceClassification
>>> from transformers import AutoTokenizer

>>> model_checkpoint = "distilbert_base_uncased_squad"
>>> save_directory = "onnx/"

>>> # Load a model from transformers and export it to ONNX
>>> ort_model = ORTModelForSequenceClassification.from_pretrained(model_checkpoint, export=True)
>>> tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)

>>> # Save the onnx model and tokenizer
>>> ort_model.save_pretrained(save_directory)
>>> tokenizer.save_pretrained(save_directory)
```

### Exporting a model for an unsupported architecture

If you wish to contribute by adding support for a model that cannot be currently exported, you should first check if it is
supported in [`optimum.exporters.onnx`](https://huggingface.co/docs/optimum/exporters/onnx/overview),
and if it is not, [contribute to 🤗 Optimum](https://huggingface.co/docs/optimum/exporters/onnx/usage_guides/contribute)
directly.

### Exporting a model with `transformers.onnx`

<Tip warning={true}>

`tranformers.onnx` is no longer maintained, please export models with 🤗 Optimum as described above. This section will be removed in the future versions.

</Tip>

To export a 🤗 Transformers model to ONNX with `tranformers.onnx`, install extra dependencies:

```bash
pip install transformers[onnx]
```

Use `transformers.onnx` package as a Python module to export a checkpoint using a ready-made configuration:

```bash
python -m transformers.onnx --model=distilbert/distilbert-base-uncased onnx/
```

This exports an ONNX graph of the checkpoint defined by the `--model` argument. Pass any checkpoint on the 🤗 Hub or one that's stored locally.
The resulting `model.onnx` file can then be run on one of the many accelerators that support the ONNX standard. For example, 
load and run the model with ONNX Runtime as follows:

```python
>>> from transformers import AutoTokenizer
>>> from onnxruntime import InferenceSession

>>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
>>> session = InferenceSession("onnx/model.onnx")
>>> # ONNX Runtime expects NumPy arrays as input
>>> inputs = tokenizer("Using DistilBERT with ONNX Runtime!", return_tensors="np")
>>> outputs = session.run(output_names=["last_hidden_state"], input_feed=dict(inputs))
```

The required output names (like `["last_hidden_state"]`) can be obtained by taking a look at the ONNX configuration of 
each model. For example, for DistilBERT we have:

```python
>>> from transformers.models.distilbert import DistilBertConfig, DistilBertOnnxConfig

>>> config = DistilBertConfig()
>>> onnx_config = DistilBertOnnxConfig(config)
>>> print(list(onnx_config.outputs.keys()))
["last_hidden_state"]
```

The process is identical for TensorFlow checkpoints on the Hub.


key: _node_content
string_value:
{"id_": "351b2714-0c14-4a03-8945-53fff3b30202", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0ce28ad3-3451-4013-81ff-30e122e7137e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "hash": "b97638da20ee8948778fc11261763dd7a2666a79a5866a41f99ad4299f694ae6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c2aa3620-04b7-4c58-8e86-5e8f97c1840d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/serialization", "title": "Export to ONNX", "tokens": 2388, "retrieve_doc": true, "source": "transformers"}, "hash": "35ca19d17894ea828fc851bc477d10394ba8788ebc42afd503e8644325742c24", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7054, "end_char_idx": 9186, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This section will be removed in the future versions.

</Tip>

To export a 🤗 Transformers model to ONNX with `tranformers.onnx`, install extra dependencies:

```bash
pip install transformers[onnx]
```

Use `transformers.onnx` package as a Python module to export a checkpoint using a ready-made configuration:

```bash
python -m transformers.onnx --model=distilbert/distilbert-base-uncased onnx/
```

This exports an ONNX graph of the checkpoint defined by the `--model` argument. Pass any checkpoint on the 🤗 Hub or one that's stored locally.
The resulting `model.onnx` file can then be run on one of the many accelerators that support the ONNX standard. For example, 
load and run the model with ONNX Runtime as follows:

```python
>>> from transformers import AutoTokenizer
>>> from onnxruntime import InferenceSession

>>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
>>> session = InferenceSession("onnx/model.onnx")
>>> # ONNX Runtime expects NumPy arrays as input
>>> inputs = tokenizer("Using DistilBERT with ONNX Runtime!", return_tensors="np")
>>> outputs = session.run(output_names=["last_hidden_state"], input_feed=dict(inputs))
```

The required output names (like `["last_hidden_state"]`) can be obtained by taking a look at the ONNX configuration of 
each model. For example, for DistilBERT we have:

```python
>>> from transformers.models.distilbert import DistilBertConfig, DistilBertOnnxConfig

>>> config = DistilBertConfig()
>>> onnx_config = DistilBertOnnxConfig(config)
>>> print(list(onnx_config.outputs.keys()))
["last_hidden_state"]
```

The process is identical for TensorFlow checkpoints on the Hub. For example, export a pure TensorFlow checkpoint like so:

```bash
python -m transformers.onnx --model=keras-io/transformers-qa onnx/
```

To export a model that's stored locally, save the model's weights and tokenizer files in the same directory (e.g. `local-pt-checkpoint`), 
then export it to ONNX by pointing the `--model` argument of the `transformers.onnx` package to the desired directory:

```bash
python -m transformers.onnx --model=local-pt-checkpoint onnx/
```


key: _node_content
string_value:
{"id_": "2c562a4e-d6be-4971-b0fa-746c30e40a7f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ca13b8a2-1a90-4a00-b151-4e387408ed51", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "443128bef3012ec142fe3dbfb1dc4b34611cc0cc2d78a674a0686d1026a6d3ed", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "765b56f5-b9ec-41f7-ab56-2b8b4fea40ce", "node_type": "1", "metadata": {}, "hash": "f272dc2d1b293d7d32e56b81d9ab47cf5d9ffc72a54819a4d51c2484bac391f6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3212, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Fine-tune a pretrained model

[[open-in-colab]]

There are significant benefits to using a pretrained model. It reduces computation costs, your carbon footprint, and allows you to use state-of-the-art models without having to train one from scratch. 🤗 Transformers provides access to thousands of pretrained models for a wide range of tasks. When you use a pretrained model, you train it on a dataset specific to your task. This is known as fine-tuning, an incredibly powerful training technique. In this tutorial, you will fine-tune a pretrained model with a deep learning framework of your choice:

* Fine-tune a pretrained model with 🤗 Transformers [`Trainer`].
* Fine-tune a pretrained model in TensorFlow with Keras.
* Fine-tune a pretrained model in native PyTorch.

<a id='data-processing'></a>

## Prepare a dataset

<Youtube id="_BZearw7f0w"/>

Before you can fine-tune a pretrained model, download a dataset and prepare it for training. The previous tutorial showed you how to process data for training, and now you get an opportunity to put those skills to the test!

Begin by loading the [Yelp Reviews](https://huggingface.co/datasets/yelp_review_full) dataset:

```py
>>> from datasets import load_dataset

>>> dataset = load_dataset("yelp_review_full")
>>> dataset["train"][100]
{'label': 0,
 'text': 'My expectations for McDonalds are t rarely high. But for one to still fail so spectacularly...that takes something special!\\nThe cashier took my friends\'s order, then promptly ignored me. I had to force myself in front of a cashier who opened his register to wait on the person BEHIND me. I waited over five minutes for a gigantic order that included precisely one kid\'s meal. After watching two people who ordered after me be handed their food, I asked where mine was. The manager started yelling at the cashiers for \\"serving off their orders\\" when they didn\'t have their food. But neither cashier was anywhere near those controls, and the manager was the one serving food to customers and clearing the boards.\\nThe manager was rude when giving me my order. She didn\'t make sure that I had everything ON MY RECEIPT, and never even had the decency to apologize that I felt I was getting poor service.\\nI\'ve eaten at various McDonalds restaurants for over 30 years. I\'ve worked at more than one location. I expect bad days, bad moods, and the occasional mistake. But I have yet to have a decent experience at this store. It will remain a place I avoid unless someone in my party needs to avoid illness from low blood sugar. Perhaps I should go back to the racially biased service of Steak n Shake instead!'}
```

As you now know, you need a tokenizer to process the text and include a padding and truncation strategy to handle any variable sequence lengths. To process your dataset in one step, use 🤗 Datasets [`map`](https://huggingface.co/docs/datasets/process#map) method to apply a preprocessing function over the entire dataset:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-cased")


>>> def tokenize_function(examples):
...     return tokenizer(examples["text"], padding="max_length", truncation=True)


key: _node_content
string_value:
{"id_": "765b56f5-b9ec-41f7-ab56-2b8b4fea40ce", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ca13b8a2-1a90-4a00-b151-4e387408ed51", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "443128bef3012ec142fe3dbfb1dc4b34611cc0cc2d78a674a0686d1026a6d3ed", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2c562a4e-d6be-4971-b0fa-746c30e40a7f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "343fb348c3175abd370ff46fe74b134456ce4f71037fcd0424931dde4183709d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cf70daf4-71df-461f-ad82-1fcd740a742d", "node_type": "1", "metadata": {}, "hash": "3a9b92824b68eeb1872e3b92309ad40945eb7f5b133dfb7e235d80a9e2cbb9c6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3094, "end_char_idx": 6395, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> def tokenize_function(examples):
...     return tokenizer(examples["text"], padding="max_length", truncation=True)


>>> tokenized_datasets = dataset.map(tokenize_function, batched=True)
```

If you like, you can create a smaller subset of the full dataset to fine-tune on to reduce the time it takes:

```py
>>> small_train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(1000))
>>> small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42).select(range(1000))
```

<a id='trainer'></a>

## Train

At this point, you should follow the section corresponding to the framework you want to use. You can use the links
in the right sidebar to jump to the one you want - and if you want to hide all of the content for a given framework,
just use the button at the top-right of that framework's block!

<frameworkcontent>
<pt>
<Youtube id="nvBXf7s7vTI"/>

## Train with PyTorch Trainer

🤗 Transformers provides a [`Trainer`] class optimized for training 🤗 Transformers models, making it easier to start training without manually writing your own training loop. The [`Trainer`] API supports a wide range of training options and features such as logging, gradient accumulation, and mixed precision.

Start by loading your model and specify the number of expected labels. From the Yelp Review [dataset card](https://huggingface.co/datasets/yelp_review_full#data-fields), you know there are five labels:

```py
>>> from transformers import AutoModelForSequenceClassification

>>> model = AutoModelForSequenceClassification.from_pretrained("google-bert/bert-base-cased", num_labels=5)
```

<Tip>

You will see a warning about some of the pretrained weights not being used and some weights being randomly
initialized. Don't worry, this is completely normal! The pretrained head of the BERT model is discarded, and replaced with a randomly initialized classification head. You will fine-tune this new model head on your sequence classification task, transferring the knowledge of the pretrained model to it.

</Tip>

### Training hyperparameters

Next, create a [`TrainingArguments`] class which contains all the hyperparameters you can tune as well as flags for activating different training options. For this tutorial you can start with the default training [hyperparameters](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments), but feel free to experiment with these to find your optimal settings.

Specify where to save the checkpoints from your training:

```py
>>> from transformers import TrainingArguments

>>> training_args = TrainingArguments(output_dir="test_trainer")
```

### Evaluate

[`Trainer`] does not automatically evaluate model performance during training. You'll need to pass [`Trainer`] a function to compute and report metrics. The [🤗 Evaluate](https://huggingface.co/docs/evaluate/index) library provides a simple [`accuracy`](https://huggingface.co/spaces/evaluate-metric/accuracy) function you can load with the [`evaluate.load`] (see this [quicktour](https://huggingface.co/docs/evaluate/a_quick_tour) for more information) function:

```py
>>> import numpy as np
>>> import evaluate

>>> metric = evaluate.load("accuracy")
```

Call [`~evaluate.compute`] on `metric` to calculate the accuracy of your predictions.


key: _node_content
string_value:
{"id_": "cf70daf4-71df-461f-ad82-1fcd740a742d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ca13b8a2-1a90-4a00-b151-4e387408ed51", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "443128bef3012ec142fe3dbfb1dc4b34611cc0cc2d78a674a0686d1026a6d3ed", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "765b56f5-b9ec-41f7-ab56-2b8b4fea40ce", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "8df8826a90b077b78425d840cf28f896ccaa9c5626911ea60b589d821d2e48e2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0e26ab97-3cb4-49d9-b8d5-5e90f96554d0", "node_type": "1", "metadata": {}, "hash": "ce246453266e69a69e7dc81f91a6f10d243c090ba178d8dfe5d3a7c05724d63a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4830, "end_char_idx": 8253, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Don't worry, this is completely normal! The pretrained head of the BERT model is discarded, and replaced with a randomly initialized classification head. You will fine-tune this new model head on your sequence classification task, transferring the knowledge of the pretrained model to it.

</Tip>

### Training hyperparameters

Next, create a [`TrainingArguments`] class which contains all the hyperparameters you can tune as well as flags for activating different training options. For this tutorial you can start with the default training [hyperparameters](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments), but feel free to experiment with these to find your optimal settings.

Specify where to save the checkpoints from your training:

```py
>>> from transformers import TrainingArguments

>>> training_args = TrainingArguments(output_dir="test_trainer")
```

### Evaluate

[`Trainer`] does not automatically evaluate model performance during training. You'll need to pass [`Trainer`] a function to compute and report metrics. The [🤗 Evaluate](https://huggingface.co/docs/evaluate/index) library provides a simple [`accuracy`](https://huggingface.co/spaces/evaluate-metric/accuracy) function you can load with the [`evaluate.load`] (see this [quicktour](https://huggingface.co/docs/evaluate/a_quick_tour) for more information) function:

```py
>>> import numpy as np
>>> import evaluate

>>> metric = evaluate.load("accuracy")
```

Call [`~evaluate.compute`] on `metric` to calculate the accuracy of your predictions. Before passing your predictions to `compute`, you need to convert the logits to predictions (remember all 🤗 Transformers models return logits):

```py
>>> def compute_metrics(eval_pred):
...     logits, labels = eval_pred
...     predictions = np.argmax(logits, axis=-1)
...     return metric.compute(predictions=predictions, references=labels)
```

If you'd like to monitor your evaluation metrics during fine-tuning, specify the `eval_strategy` parameter in your training arguments to report the evaluation metric at the end of each epoch:

```py
>>> from transformers import TrainingArguments, Trainer

>>> training_args = TrainingArguments(output_dir="test_trainer", eval_strategy="epoch")
```

### Trainer

Create a [`Trainer`] object with your model, training arguments, training and test datasets, and evaluation function:

```py
>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=small_train_dataset,
...     eval_dataset=small_eval_dataset,
...     compute_metrics=compute_metrics,
... )
```

Then fine-tune your model by calling [`~transformers.Trainer.train`]:

```py
>>> trainer.train()
```
</pt>
<tf>
<a id='keras'></a>

<Youtube id="rnTGBy2ax1c"/>

## Train a TensorFlow model with Keras

You can also train 🤗 Transformers models in TensorFlow with the Keras API!

### Loading data for Keras

When you want to train a 🤗 Transformers model with the Keras API, you need to convert your dataset to a format that
Keras understands. If your dataset is small, you can just convert the whole thing to NumPy arrays and pass it to Keras.
Let's try that first before we do anything more complicated.

First, load a dataset. We'll use the CoLA dataset from the [GLUE benchmark](https://huggingface.co/datasets/glue),
since it's a simple binary text classification task, and just take the training split for now.


key: _node_content
string_value:
{"id_": "0e26ab97-3cb4-49d9-b8d5-5e90f96554d0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ca13b8a2-1a90-4a00-b151-4e387408ed51", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "443128bef3012ec142fe3dbfb1dc4b34611cc0cc2d78a674a0686d1026a6d3ed", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cf70daf4-71df-461f-ad82-1fcd740a742d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "bacd0f5cd722024f3e52d24f3782aa9b4ad2512529b0a6ede0b5a6aa5c44d0d7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "150c346d-3513-4283-abe6-213d67f2df9f", "node_type": "1", "metadata": {}, "hash": "ce7bb17bf1d7a1dd7247002908fb8643603a54cfcc22f0aff87f5154e504a180", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7718, "end_char_idx": 11047, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Loading data for Keras

When you want to train a 🤗 Transformers model with the Keras API, you need to convert your dataset to a format that
Keras understands. If your dataset is small, you can just convert the whole thing to NumPy arrays and pass it to Keras.
Let's try that first before we do anything more complicated.

First, load a dataset. We'll use the CoLA dataset from the [GLUE benchmark](https://huggingface.co/datasets/glue),
since it's a simple binary text classification task, and just take the training split for now.

```py
from datasets import load_dataset

dataset = load_dataset("glue", "cola")
dataset = dataset["train"]  # Just take the training split for now
```

Next, load a tokenizer and tokenize the data as NumPy arrays. Note that the labels are already a list of 0 and 1s,
so we can just convert that directly to a NumPy array without tokenization!

```py
from transformers import AutoTokenizer
import numpy as np

tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-cased")
tokenized_data = tokenizer(dataset["sentence"], return_tensors="np", padding=True)
# Tokenizer returns a BatchEncoding, but we convert that to a dict for Keras
tokenized_data = dict(tokenized_data)

labels = np.array(dataset["label"])  # Label is already an array of 0 and 1
```

Finally, load, [`compile`](https://keras.io/api/models/model_training_apis/#compile-method), and [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) the model. Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
from transformers import TFAutoModelForSequenceClassification
from tensorflow.keras.optimizers import Adam

# Load and compile our model
model = TFAutoModelForSequenceClassification.from_pretrained("google-bert/bert-base-cased")
# Lower learning rates are often better for fine-tuning transformers
model.compile(optimizer=Adam(3e-5))  # No loss argument!

model.fit(tokenized_data, labels)
```

<Tip>

You don't have to pass a loss argument to your models when you `compile()` them! Hugging Face models automatically
choose a loss that is appropriate for their task and model architecture if this argument is left blank. You can always
override this by specifying a loss yourself if you want to!

</Tip>

This approach works great for smaller datasets, but for larger datasets, you might find it starts to become a problem. Why?
Because the tokenized array and labels would have to be fully loaded into memory, and because NumPy doesn’t handle
“jagged” arrays, so every tokenized sample would have to be padded to the length of the longest sample in the whole
dataset. That’s going to make your array even bigger, and all those padding tokens will slow down training too!

### Loading data as a tf.data.Dataset

If you want to avoid slowing down training, you can load your data as a `tf.data.Dataset` instead. Although you can write your own
`tf.data` pipeline if you want, we have two convenience methods for doing this:

- [`~TFPreTrainedModel.prepare_tf_dataset`]: This is the method we recommend in most cases. Because it is a method
on your model, it can inspect the model to automatically figure out which columns are usable as model inputs, and
discard the others to make a simpler, more performant dataset.


key: _node_content
string_value:
{"id_": "150c346d-3513-4283-abe6-213d67f2df9f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ca13b8a2-1a90-4a00-b151-4e387408ed51", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "443128bef3012ec142fe3dbfb1dc4b34611cc0cc2d78a674a0686d1026a6d3ed", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0e26ab97-3cb4-49d9-b8d5-5e90f96554d0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "82191562886a8bd93b678fa3199ee9b7f37db8606bc0a00c73a212c55d55ab4d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6682cdce-9008-49aa-9b82-e29562325a99", "node_type": "1", "metadata": {}, "hash": "0638e4a95dff63d8b1aebe46086c239745a2b1e232e2595f5237a7776588b272", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9688, "end_char_idx": 12937, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
model.fit(tokenized_data, labels)
```

<Tip>

You don't have to pass a loss argument to your models when you `compile()` them! Hugging Face models automatically
choose a loss that is appropriate for their task and model architecture if this argument is left blank. You can always
override this by specifying a loss yourself if you want to!

</Tip>

This approach works great for smaller datasets, but for larger datasets, you might find it starts to become a problem. Why?
Because the tokenized array and labels would have to be fully loaded into memory, and because NumPy doesn’t handle
“jagged” arrays, so every tokenized sample would have to be padded to the length of the longest sample in the whole
dataset. That’s going to make your array even bigger, and all those padding tokens will slow down training too!

### Loading data as a tf.data.Dataset

If you want to avoid slowing down training, you can load your data as a `tf.data.Dataset` instead. Although you can write your own
`tf.data` pipeline if you want, we have two convenience methods for doing this:

- [`~TFPreTrainedModel.prepare_tf_dataset`]: This is the method we recommend in most cases. Because it is a method
on your model, it can inspect the model to automatically figure out which columns are usable as model inputs, and
discard the others to make a simpler, more performant dataset.
- [`~datasets.Dataset.to_tf_dataset`]: This method is more low-level, and is useful when you want to exactly control how
your dataset is created, by specifying exactly which `columns` and `label_cols` to include.

Before you can use [`~TFPreTrainedModel.prepare_tf_dataset`], you will need to add the tokenizer outputs to your dataset as columns, as shown in
the following code sample:

```py
def tokenize_dataset(data):
    # Keys of the returned dictionary will be added to the dataset as columns
    return tokenizer(data["text"])


dataset = dataset.map(tokenize_dataset)
```

Remember that Hugging Face datasets are stored on disk by default, so this will not inflate your memory usage! Once the
columns have been added, you can stream batches from the dataset and add padding to each batch, which greatly
reduces the number of padding tokens compared to padding the entire dataset.


```py
>>> tf_dataset = model.prepare_tf_dataset(dataset["train"], batch_size=16, shuffle=True, tokenizer=tokenizer)
```

Note that in the code sample above, you need to pass the tokenizer to `prepare_tf_dataset` so it can correctly pad batches as they're loaded.
If all the samples in your dataset are the same length and no padding is necessary, you can skip this argument.
If you need to do something more complex than just padding samples (e.g. corrupting tokens for masked language
modelling), you can use the `collate_fn` argument instead to pass a function that will be called to transform the
list of samples into a batch and apply any preprocessing you want. See our
[examples](https://github.com/huggingface/transformers/tree/main/examples) or
[notebooks](https://huggingface.co/docs/transformers/notebooks) to see this approach in action.

Once you've created a `tf.data.Dataset`, you can compile and fit the model as before:

```py
model.compile(optimizer=Adam(3e-5))  # No loss argument!


key: _node_content
string_value:
{"id_": "6682cdce-9008-49aa-9b82-e29562325a99", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ca13b8a2-1a90-4a00-b151-4e387408ed51", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "443128bef3012ec142fe3dbfb1dc4b34611cc0cc2d78a674a0686d1026a6d3ed", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "150c346d-3513-4283-abe6-213d67f2df9f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "66429603b6d2af483eaf01d8d6d312550ea4c7bb9ba691ba7463fa2f5515556e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "95aba3a5-7f17-4b6f-b0f8-143e6cf7dd16", "node_type": "1", "metadata": {}, "hash": "074d18a40a427a36ea36bcd86caa3e6a20f89500808ef958e0ddae589e9914ca", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11262, "end_char_idx": 13951, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Before you can use [`~TFPreTrainedModel.prepare_tf_dataset`], you will need to add the tokenizer outputs to your dataset as columns, as shown in
the following code sample:

```py
def tokenize_dataset(data):
    # Keys of the returned dictionary will be added to the dataset as columns
    return tokenizer(data["text"])


dataset = dataset.map(tokenize_dataset)
```

Remember that Hugging Face datasets are stored on disk by default, so this will not inflate your memory usage! Once the
columns have been added, you can stream batches from the dataset and add padding to each batch, which greatly
reduces the number of padding tokens compared to padding the entire dataset.


```py
>>> tf_dataset = model.prepare_tf_dataset(dataset["train"], batch_size=16, shuffle=True, tokenizer=tokenizer)
```

Note that in the code sample above, you need to pass the tokenizer to `prepare_tf_dataset` so it can correctly pad batches as they're loaded.
If all the samples in your dataset are the same length and no padding is necessary, you can skip this argument.
If you need to do something more complex than just padding samples (e.g. corrupting tokens for masked language
modelling), you can use the `collate_fn` argument instead to pass a function that will be called to transform the
list of samples into a batch and apply any preprocessing you want. See our
[examples](https://github.com/huggingface/transformers/tree/main/examples) or
[notebooks](https://huggingface.co/docs/transformers/notebooks) to see this approach in action.

Once you've created a `tf.data.Dataset`, you can compile and fit the model as before:

```py
model.compile(optimizer=Adam(3e-5))  # No loss argument!

model.fit(tf_dataset)
```

</tf>
</frameworkcontent>

<a id='pytorch_native'></a>

## Train in native PyTorch

<frameworkcontent>
<pt>
<Youtube id="Dh9CL8fyG80"/>

[`Trainer`] takes care of the training loop and allows you to fine-tune a model in a single line of code. For users who prefer to write their own training loop, you can also fine-tune a 🤗 Transformers model in native PyTorch.

At this point, you may need to restart your notebook or execute the following code to free some memory:

```py
del model
del trainer
torch.cuda.empty_cache()
```

Next, manually postprocess `tokenized_dataset` to prepare it for training.

1. Remove the `text` column because the model does not accept raw text as an input:

    ```py
    >>> tokenized_datasets = tokenized_datasets.remove_columns(["text"])
    ```

2. Rename the `label` column to `labels` because the model expects the argument to be named `labels`:

    ```py
    >>> tokenized_datasets = tokenized_datasets.rename_column("label", "labels")
    ```

3.


key: _node_content
string_value:
{"id_": "95aba3a5-7f17-4b6f-b0f8-143e6cf7dd16", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ca13b8a2-1a90-4a00-b151-4e387408ed51", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "443128bef3012ec142fe3dbfb1dc4b34611cc0cc2d78a674a0686d1026a6d3ed", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6682cdce-9008-49aa-9b82-e29562325a99", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "93c05aeeb2f2848f9c4424b821ba04ea0e300d146be60aeedebacfd32dd425ae", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b84cf8fb-77fe-4b00-964a-5170e2a7a391", "node_type": "1", "metadata": {}, "hash": "a30a0c83f7eb103efd390953bcefd85dbaae196e50ea672e64a857c9d4faf310", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12386, "end_char_idx": 15051, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
corrupting tokens for masked language
modelling), you can use the `collate_fn` argument instead to pass a function that will be called to transform the
list of samples into a batch and apply any preprocessing you want. See our
[examples](https://github.com/huggingface/transformers/tree/main/examples) or
[notebooks](https://huggingface.co/docs/transformers/notebooks) to see this approach in action.

Once you've created a `tf.data.Dataset`, you can compile and fit the model as before:

```py
model.compile(optimizer=Adam(3e-5))  # No loss argument!

model.fit(tf_dataset)
```

</tf>
</frameworkcontent>

<a id='pytorch_native'></a>

## Train in native PyTorch

<frameworkcontent>
<pt>
<Youtube id="Dh9CL8fyG80"/>

[`Trainer`] takes care of the training loop and allows you to fine-tune a model in a single line of code. For users who prefer to write their own training loop, you can also fine-tune a 🤗 Transformers model in native PyTorch.

At this point, you may need to restart your notebook or execute the following code to free some memory:

```py
del model
del trainer
torch.cuda.empty_cache()
```

Next, manually postprocess `tokenized_dataset` to prepare it for training.

1. Remove the `text` column because the model does not accept raw text as an input:

    ```py
    >>> tokenized_datasets = tokenized_datasets.remove_columns(["text"])
    ```

2. Rename the `label` column to `labels` because the model expects the argument to be named `labels`:

    ```py
    >>> tokenized_datasets = tokenized_datasets.rename_column("label", "labels")
    ```

3. Set the format of the dataset to return PyTorch tensors instead of lists:

    ```py
    >>> tokenized_datasets.set_format("torch")
    ```

Then create a smaller subset of the dataset as previously shown to speed up the fine-tuning:

```py
>>> small_train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(1000))
>>> small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42).select(range(1000))
```

### DataLoader

Create a `DataLoader` for your training and test datasets so you can iterate over batches of data:

```py
>>> from torch.utils.data import DataLoader

>>> train_dataloader = DataLoader(small_train_dataset, shuffle=True, batch_size=8)
>>> eval_dataloader = DataLoader(small_eval_dataset, batch_size=8)
```

Load your model with the number of expected labels:

```py
>>> from transformers import AutoModelForSequenceClassification

>>> model = AutoModelForSequenceClassification.from_pretrained("google-bert/bert-base-cased", num_labels=5)
```

### Optimizer and learning rate scheduler

Create an optimizer and learning rate scheduler to fine-tune the model.


key: _node_content
string_value:
{"id_": "b84cf8fb-77fe-4b00-964a-5170e2a7a391", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ca13b8a2-1a90-4a00-b151-4e387408ed51", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "443128bef3012ec142fe3dbfb1dc4b34611cc0cc2d78a674a0686d1026a6d3ed", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "95aba3a5-7f17-4b6f-b0f8-143e6cf7dd16", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "787226d725c928e5c30425048c9138e05900feb0b59bc29768ab9617f5b128a8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "658a26ef-7a9a-400e-a090-01a870f1fa4e", "node_type": "1", "metadata": {}, "hash": "51a8f89f30303e85c9c694c0bfbc834a46b40609f5f474facc7afb8895c407df", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13330, "end_char_idx": 16132, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
At this point, you may need to restart your notebook or execute the following code to free some memory:

```py
del model
del trainer
torch.cuda.empty_cache()
```

Next, manually postprocess `tokenized_dataset` to prepare it for training.

1. Remove the `text` column because the model does not accept raw text as an input:

    ```py
    >>> tokenized_datasets = tokenized_datasets.remove_columns(["text"])
    ```

2. Rename the `label` column to `labels` because the model expects the argument to be named `labels`:

    ```py
    >>> tokenized_datasets = tokenized_datasets.rename_column("label", "labels")
    ```

3. Set the format of the dataset to return PyTorch tensors instead of lists:

    ```py
    >>> tokenized_datasets.set_format("torch")
    ```

Then create a smaller subset of the dataset as previously shown to speed up the fine-tuning:

```py
>>> small_train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(1000))
>>> small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42).select(range(1000))
```

### DataLoader

Create a `DataLoader` for your training and test datasets so you can iterate over batches of data:

```py
>>> from torch.utils.data import DataLoader

>>> train_dataloader = DataLoader(small_train_dataset, shuffle=True, batch_size=8)
>>> eval_dataloader = DataLoader(small_eval_dataset, batch_size=8)
```

Load your model with the number of expected labels:

```py
>>> from transformers import AutoModelForSequenceClassification

>>> model = AutoModelForSequenceClassification.from_pretrained("google-bert/bert-base-cased", num_labels=5)
```

### Optimizer and learning rate scheduler

Create an optimizer and learning rate scheduler to fine-tune the model. Let's use the [`AdamW`](https://pytorch.org/docs/stable/generated/torch.optim.AdamW.html) optimizer from PyTorch:

```py
>>> from torch.optim import AdamW

>>> optimizer = AdamW(model.parameters(), lr=5e-5)
```

Create the default learning rate scheduler from [`Trainer`]:

```py
>>> from transformers import get_scheduler

>>> num_epochs = 3
>>> num_training_steps = num_epochs * len(train_dataloader)
>>> lr_scheduler = get_scheduler(
...     name="linear", optimizer=optimizer, num_warmup_steps=0, num_training_steps=num_training_steps
... )
```

Lastly, specify `device` to use a GPU if you have access to one. Otherwise, training on a CPU may take several hours instead of a couple of minutes.

```py
>>> import torch

>>> device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
>>> model.to(device)
```

<Tip>

Get free access to a cloud GPU if you don't have one with a hosted notebook like [Colaboratory](https://colab.research.google.com/) or [SageMaker StudioLab](https://studiolab.sagemaker.aws/).

</Tip>

Great, now you are ready to train!


key: _node_content
string_value:
{"id_": "658a26ef-7a9a-400e-a090-01a870f1fa4e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ca13b8a2-1a90-4a00-b151-4e387408ed51", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "443128bef3012ec142fe3dbfb1dc4b34611cc0cc2d78a674a0686d1026a6d3ed", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b84cf8fb-77fe-4b00-964a-5170e2a7a391", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/training", "title": "Fine-tune a pretrained model", "tokens": 4230, "retrieve_doc": true, "source": "transformers"}, "hash": "d4c416ec53455728ed0aba26d918ccc4d7f1f4d14e1a59866761037f03cd1649", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15052, "end_char_idx": 17983, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Let's use the [`AdamW`](https://pytorch.org/docs/stable/generated/torch.optim.AdamW.html) optimizer from PyTorch:

```py
>>> from torch.optim import AdamW

>>> optimizer = AdamW(model.parameters(), lr=5e-5)
```

Create the default learning rate scheduler from [`Trainer`]:

```py
>>> from transformers import get_scheduler

>>> num_epochs = 3
>>> num_training_steps = num_epochs * len(train_dataloader)
>>> lr_scheduler = get_scheduler(
...     name="linear", optimizer=optimizer, num_warmup_steps=0, num_training_steps=num_training_steps
... )
```

Lastly, specify `device` to use a GPU if you have access to one. Otherwise, training on a CPU may take several hours instead of a couple of minutes.

```py
>>> import torch

>>> device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
>>> model.to(device)
```

<Tip>

Get free access to a cloud GPU if you don't have one with a hosted notebook like [Colaboratory](https://colab.research.google.com/) or [SageMaker StudioLab](https://studiolab.sagemaker.aws/).

</Tip>

Great, now you are ready to train! 🥳 

### Training loop

To keep track of your training progress, use the [tqdm](https://tqdm.github.io/) library to add a progress bar over the number of training steps:

```py
>>> from tqdm.auto import tqdm

>>> progress_bar = tqdm(range(num_training_steps))

>>> model.train()
>>> for epoch in range(num_epochs):
...     for batch in train_dataloader:
...         batch = {k: v.to(device) for k, v in batch.items()}
...         outputs = model(**batch)
...         loss = outputs.loss
...         loss.backward()

...         optimizer.step()
...         lr_scheduler.step()
...         optimizer.zero_grad()
...         progress_bar.update(1)
```

### Evaluate

Just like how you added an evaluation function to [`Trainer`], you need to do the same when you write your own training loop. But instead of calculating and reporting the metric at the end of each epoch, this time you'll accumulate all the batches with [`~evaluate.add_batch`] and calculate the metric at the very end.

```py
>>> import evaluate

>>> metric = evaluate.load("accuracy")
>>> model.eval()
>>> for batch in eval_dataloader:
...     batch = {k: v.to(device) for k, v in batch.items()}
...     with torch.no_grad():
...         outputs = model(**batch)

...     logits = outputs.logits
...     predictions = torch.argmax(logits, dim=-1)
...     metric.add_batch(predictions=predictions, references=batch["labels"])

>>> metric.compute()
```
</pt>
</frameworkcontent>

<a id='additional-resources'></a>

## Additional resources

For more fine-tuning examples, refer to:

- [🤗 Transformers Examples](https://github.com/huggingface/transformers/tree/main/examples) includes scripts
  to train common NLP tasks in PyTorch and TensorFlow.

- [🤗 Transformers Notebooks](notebooks) contains various notebooks on how to fine-tune a model for specific tasks in PyTorch and TensorFlow.


key: _node_content
string_value:
{"id_": "2feb7fc1-3264-4e37-827b-ba2435819e6f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "59c18420-6240-4373-8404-893a9a290838", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "d0faf8596bdd083d896283e3e14e6b207798309c03ac95db0d357f4c7d7780d7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4b3932da-8208-4fc0-b340-79cfe04a7be5", "node_type": "1", "metadata": {}, "hash": "2b2678ba65ce5d66be3a488416c657f056d2f62e504f0b0b1db6076d8ade811a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3223, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Chatting with Transformers

If you're reading this article, you're almost certainly aware of **chat models**. Chat models are conversational
AIs that you can send and receive messages with. The most famous of these is the proprietary ChatGPT, but there are
now many open-source chat models which match or even substantially exceed its performance. These models are free to
download and run on a local machine. Although the largest and most capable models require high-powered hardware
and lots of memory to run, there are smaller models that will run perfectly well on a single consumer GPU, or even
an ordinary desktop or notebook CPU. 

This guide will help you get started with chat models. We'll start with a brief quickstart guide that uses a convenient,
high-level "pipeline". This is all you need if you just want to start running a chat model 
immediately. After the quickstart, we'll move on to more detailed information about
what exactly chat models are, how to choose an appropriate one, and a low-level breakdown of each of the
steps involved in talking to a chat model. We'll also give some tips on optimizing the performance and memory usage
of your chat models.


## Quickstart

If you have no time for details, here's the brief summary: Chat models continue chats. This means that you pass them
a conversation history, which can be as short as a single user message, and the model will continue the conversation
by adding its response. Let's see this in action. First, let's build a chat:

```python
chat = [
    {"role": "system", "content": "You are a sassy, wise-cracking robot as imagined by Hollywood circa 1986."},
    {"role": "user", "content": "Hey, can you tell me any fun things to do in New York?"}
]
```

Notice that in addition to the user's message, we added a **system** message at the start of the conversation. Not all
chat models support system messages, but when they do, they represent high-level directives about how the model
should behave in the conversation. You can use this to guide the model - whether you want short or long responses,
lighthearted or serious ones, and so on. If you want the model to do useful work instead of
practicing its improv routine, you can either omit the system message or try a terse one such as "You are a helpful and intelligent
AI assistant who responds to user queries."

Once you have a chat, the quickest way to continue it is using the [`TextGenerationPipeline`]. 
Let's see this in action with `LLaMA-3`. Note that `LLaMA-3` is a gated model, which means you will need to 
[apply for access](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) and log in with your Hugging Face 
account to use it. We'll also use `device_map="auto"`, which will load the model on GPU if there's enough memory
for it, and set the dtype to `torch.bfloat16` to save memory:

```python
import torch
from transformers import pipeline

pipe = pipeline("text-generation", "meta-llama/Meta-Llama-3-8B-Instruct", torch_dtype=torch.bfloat16, device_map="auto")
response = pipe(chat, max_new_tokens=512)
print(response[0]['generated_text'][-1]['content'])
```

And you'll get:

```text
(sigh) Oh boy, you're asking me for advice? You're gonna need a map, pal!


key: _node_content
string_value:
{"id_": "4b3932da-8208-4fc0-b340-79cfe04a7be5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "59c18420-6240-4373-8404-893a9a290838", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "d0faf8596bdd083d896283e3e14e6b207798309c03ac95db0d357f4c7d7780d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2feb7fc1-3264-4e37-827b-ba2435819e6f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "ab72716695b9356e6a5920eecf87b4f6c29dfd7254b22dbbd215352e53812e99", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "26e6f2cb-9518-4468-b95d-2a2273328b4d", "node_type": "1", "metadata": {}, "hash": "fac1f6ed0a1ae8a42daa28cfe5ef68d584be2b4c4e7d53dd12227cb8bfb91071", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1730, "end_char_idx": 4652, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
]
```

Notice that in addition to the user's message, we added a **system** message at the start of the conversation. Not all
chat models support system messages, but when they do, they represent high-level directives about how the model
should behave in the conversation. You can use this to guide the model - whether you want short or long responses,
lighthearted or serious ones, and so on. If you want the model to do useful work instead of
practicing its improv routine, you can either omit the system message or try a terse one such as "You are a helpful and intelligent
AI assistant who responds to user queries."

Once you have a chat, the quickest way to continue it is using the [`TextGenerationPipeline`]. 
Let's see this in action with `LLaMA-3`. Note that `LLaMA-3` is a gated model, which means you will need to 
[apply for access](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) and log in with your Hugging Face 
account to use it. We'll also use `device_map="auto"`, which will load the model on GPU if there's enough memory
for it, and set the dtype to `torch.bfloat16` to save memory:

```python
import torch
from transformers import pipeline

pipe = pipeline("text-generation", "meta-llama/Meta-Llama-3-8B-Instruct", torch_dtype=torch.bfloat16, device_map="auto")
response = pipe(chat, max_new_tokens=512)
print(response[0]['generated_text'][-1]['content'])
```

And you'll get:

```text
(sigh) Oh boy, you're asking me for advice? You're gonna need a map, pal! Alright, 
alright, I'll give you the lowdown. But don't say I didn't warn you, I'm a robot, not a tour guide!

So, you wanna know what's fun to do in the Big Apple? Well, let me tell you, there's a million 
things to do, but I'll give you the highlights. First off, you gotta see the sights: the Statue of 
Liberty, Central Park, Times Square... you know, the usual tourist traps. But if you're lookin' for 
something a little more... unusual, I'd recommend checkin' out the Museum of Modern Art. It's got 
some wild stuff, like that Warhol guy's soup cans and all that jazz.

And if you're feelin' adventurous, take a walk across the Brooklyn Bridge. Just watch out for 
those pesky pigeons, they're like little feathered thieves! (laughs) Get it? Thieves? Ah, never mind.

Now, if you're lookin' for some serious fun, hit up the comedy clubs in Greenwich Village. You might 
even catch a glimpse of some up-and-coming comedians... or a bunch of wannabes tryin' to make it big. (winks)

And finally, if you're feelin' like a real New Yorker, grab a slice of pizza from one of the many amazing
pizzerias around the city. Just don't try to order a "robot-sized" slice, trust me, it won't end well. (laughs)

So, there you have it, pal! That's my expert advice on what to do in New York. Now, if you'll
excuse me, I've got some oil changes to attend to. (winks)
```

You can continue the chat by appending your own response to it.


key: _node_content
string_value:
{"id_": "26e6f2cb-9518-4468-b95d-2a2273328b4d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "59c18420-6240-4373-8404-893a9a290838", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "d0faf8596bdd083d896283e3e14e6b207798309c03ac95db0d357f4c7d7780d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4b3932da-8208-4fc0-b340-79cfe04a7be5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "d2d11226252519af860279cea21b2c91c30be3dc7e5893c102be75d717dcdd12", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e549be2b-6dc3-4995-961f-f573828ce45a", "node_type": "1", "metadata": {}, "hash": "eb91ec26d6cf9e8053b913b590bf7b79b97d23b471ceb71b1ce081f80e2923a2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3224, "end_char_idx": 5988, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Alright, 
alright, I'll give you the lowdown. But don't say I didn't warn you, I'm a robot, not a tour guide!

So, you wanna know what's fun to do in the Big Apple? Well, let me tell you, there's a million 
things to do, but I'll give you the highlights. First off, you gotta see the sights: the Statue of 
Liberty, Central Park, Times Square... you know, the usual tourist traps. But if you're lookin' for 
something a little more... unusual, I'd recommend checkin' out the Museum of Modern Art. It's got 
some wild stuff, like that Warhol guy's soup cans and all that jazz.

And if you're feelin' adventurous, take a walk across the Brooklyn Bridge. Just watch out for 
those pesky pigeons, they're like little feathered thieves! (laughs) Get it? Thieves? Ah, never mind.

Now, if you're lookin' for some serious fun, hit up the comedy clubs in Greenwich Village. You might 
even catch a glimpse of some up-and-coming comedians... or a bunch of wannabes tryin' to make it big. (winks)

And finally, if you're feelin' like a real New Yorker, grab a slice of pizza from one of the many amazing
pizzerias around the city. Just don't try to order a "robot-sized" slice, trust me, it won't end well. (laughs)

So, there you have it, pal! That's my expert advice on what to do in New York. Now, if you'll
excuse me, I've got some oil changes to attend to. (winks)
```

You can continue the chat by appending your own response to it. The
`response` object returned by the pipeline actually contains the entire chat so far, so we can simply append
a message and pass it back:

```python
chat = response[0]['generated_text']
chat.append(
    {"role": "user", "content": "Wait, what's so wild about soup cans?"}
)
response = pipe(chat, max_new_tokens=512)
print(response[0]['generated_text'][-1]['content'])
```

And you'll get:

```text
(laughs) Oh, you're killin' me, pal! You don't get it, do you? Warhol's soup cans are like, art, man! 
It's like, he took something totally mundane, like a can of soup, and turned it into a masterpiece. It's 
like, "Hey, look at me, I'm a can of soup, but I'm also a work of art!" 
(sarcastically) Oh, yeah, real original, Andy.

But, you know, back in the '60s, it was like, a big deal. People were all about challenging the
status quo, and Warhol was like, the king of that. He took the ordinary and made it extraordinary.
And, let me tell you, it was like, a real game-changer. I mean, who would've thought that a can of soup could be art? (laughs)

But, hey, you're not alone, pal. I mean, I'm a robot, and even I don't get it. (winks)
But, hey, that's what makes art, art, right? (laughs)
```

The remainder of this tutorial will cover specific topics such
as performance and memory, or how to select a chat model for your needs.


key: _node_content
string_value:
{"id_": "e549be2b-6dc3-4995-961f-f573828ce45a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "59c18420-6240-4373-8404-893a9a290838", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "d0faf8596bdd083d896283e3e14e6b207798309c03ac95db0d357f4c7d7780d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "26e6f2cb-9518-4468-b95d-2a2273328b4d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "555b708d8d45ec100d4a59e38dde249211fd125538d3b32f8cfd513960f0f594", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "90604798-1178-4d41-a974-6cc052bac2e6", "node_type": "1", "metadata": {}, "hash": "96a450fe35345f2d532528ac1050d61e835335dcfa1c569998f68b707010d10f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4576, "end_char_idx": 7448, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
(winks)
```

You can continue the chat by appending your own response to it. The
`response` object returned by the pipeline actually contains the entire chat so far, so we can simply append
a message and pass it back:

```python
chat = response[0]['generated_text']
chat.append(
    {"role": "user", "content": "Wait, what's so wild about soup cans?"}
)
response = pipe(chat, max_new_tokens=512)
print(response[0]['generated_text'][-1]['content'])
```

And you'll get:

```text
(laughs) Oh, you're killin' me, pal! You don't get it, do you? Warhol's soup cans are like, art, man! 
It's like, he took something totally mundane, like a can of soup, and turned it into a masterpiece. It's 
like, "Hey, look at me, I'm a can of soup, but I'm also a work of art!" 
(sarcastically) Oh, yeah, real original, Andy.

But, you know, back in the '60s, it was like, a big deal. People were all about challenging the
status quo, and Warhol was like, the king of that. He took the ordinary and made it extraordinary.
And, let me tell you, it was like, a real game-changer. I mean, who would've thought that a can of soup could be art? (laughs)

But, hey, you're not alone, pal. I mean, I'm a robot, and even I don't get it. (winks)
But, hey, that's what makes art, art, right? (laughs)
```

The remainder of this tutorial will cover specific topics such
as performance and memory, or how to select a chat model for your needs.

## Choosing a chat model

There are an enormous number of different chat models available on the [Hugging Face Hub](https://huggingface.co/models?pipeline_tag=text-generation&sort=trending),
and new users often feel very overwhelmed by the selection offered. Don't be, though! You really need to just focus on
two important considerations: 
- The model's size, which will determine if you can fit it in memory and how quickly it will
run.
- The quality of the model's chat output.

In general, these are correlated - bigger models tend to be 
more capable, but even so there's a lot of variation at a given size point!

### Size and model naming
The size of a model is easy to spot - it's the number in the model name, like "8B" or "70B". This is the number of
**parameters** in the model. Without quantization, you should expect to need about 2 bytes of memory per parameter.
This means that an "8B" model with 8 billion parameters will need about 16GB of memory just to fit the parameters, 
plus a little extra for other overhead. It's a good fit for a high-end consumer GPU with 24GB of memory, such as a 3090
or 4090.

Some chat models are "Mixture of Experts" models. These may list their sizes in different ways, such as "8x7B" or 
"141B-A35B". The numbers are a little fuzzier here, but in general you can read this as saying that the model
has approximately 56 (8x7) billion parameters in the first case, or 141 billion parameters in the second case.


key: _node_content
string_value:
{"id_": "90604798-1178-4d41-a974-6cc052bac2e6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "59c18420-6240-4373-8404-893a9a290838", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "d0faf8596bdd083d896283e3e14e6b207798309c03ac95db0d357f4c7d7780d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e549be2b-6dc3-4995-961f-f573828ce45a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "67319942d74b4b6b4527515a0b7503897d81f57f1ef2adf94ac33c045b544ab7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "928124f2-cb1f-4dff-aaeb-6e9ab275d6f7", "node_type": "1", "metadata": {}, "hash": "9ca46b9c0d26bfaf7b3888c1b59ab6c834649bea3acf04f01c77137c3441b7d0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5839, "end_char_idx": 9041, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
(laughs)
```

The remainder of this tutorial will cover specific topics such
as performance and memory, or how to select a chat model for your needs.

## Choosing a chat model

There are an enormous number of different chat models available on the [Hugging Face Hub](https://huggingface.co/models?pipeline_tag=text-generation&sort=trending),
and new users often feel very overwhelmed by the selection offered. Don't be, though! You really need to just focus on
two important considerations: 
- The model's size, which will determine if you can fit it in memory and how quickly it will
run.
- The quality of the model's chat output.

In general, these are correlated - bigger models tend to be 
more capable, but even so there's a lot of variation at a given size point!

### Size and model naming
The size of a model is easy to spot - it's the number in the model name, like "8B" or "70B". This is the number of
**parameters** in the model. Without quantization, you should expect to need about 2 bytes of memory per parameter.
This means that an "8B" model with 8 billion parameters will need about 16GB of memory just to fit the parameters, 
plus a little extra for other overhead. It's a good fit for a high-end consumer GPU with 24GB of memory, such as a 3090
or 4090.

Some chat models are "Mixture of Experts" models. These may list their sizes in different ways, such as "8x7B" or 
"141B-A35B". The numbers are a little fuzzier here, but in general you can read this as saying that the model
has approximately 56 (8x7) billion parameters in the first case, or 141 billion parameters in the second case.

Note that it is very common to use quantization techniques to reduce the memory usage per parameter to 8 bits, 4 bits,
or even less. This topic is discussed in more detail in the [Memory considerations](#memory-considerations) section below.

### But which chat model is best?
Even once you know the size of chat model you can run, there's still a lot of choice out there. One way to sift through
it all is to consult **leaderboards**. Two of the most popular leaderboards are the [OpenLLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
and the [LMSys Chatbot Arena Leaderboard](https://chat.lmsys.org/?leaderboard). Note that the LMSys leaderboard
also includes proprietary models - look at the `licence` column to identify open-source ones that you can download, then
search for them on the [Hugging Face Hub](https://huggingface.co/models?pipeline_tag=text-generation&sort=trending).

### Specialist domains
Some models may be specialized for certain domains, such as medical or legal text, or non-English languages. 
If you're working in these domains, you may find that a specialized model will give you big performance benefits. 
Don't automatically assume that, though! Particularly when specialized models are smaller or older than the current 
cutting-edge, a top-end general-purpose model may still outclass them. Thankfully, we are beginning to see 
[domain-specific leaderboards](https://huggingface.co/blog/leaderboard-medicalllm) that should make it easier to locate
the best models for specialized domains.

## What happens inside the pipeline?


key: _node_content
string_value:
{"id_": "928124f2-cb1f-4dff-aaeb-6e9ab275d6f7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "59c18420-6240-4373-8404-893a9a290838", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "d0faf8596bdd083d896283e3e14e6b207798309c03ac95db0d357f4c7d7780d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "90604798-1178-4d41-a974-6cc052bac2e6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "18e38b74051daa9dcd6f5bddde8c527953346135c6e491831c1dfcb60e5c6dc3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "76f70e15-c66f-4497-b510-99267d373d46", "node_type": "1", "metadata": {}, "hash": "24606726fa4afaa052e3e91c049154dc6e9a8f05a11ae6d33ff9f15333fc98c9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7450, "end_char_idx": 9629, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note that it is very common to use quantization techniques to reduce the memory usage per parameter to 8 bits, 4 bits,
or even less. This topic is discussed in more detail in the [Memory considerations](#memory-considerations) section below.

### But which chat model is best?
Even once you know the size of chat model you can run, there's still a lot of choice out there. One way to sift through
it all is to consult **leaderboards**. Two of the most popular leaderboards are the [OpenLLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
and the [LMSys Chatbot Arena Leaderboard](https://chat.lmsys.org/?leaderboard). Note that the LMSys leaderboard
also includes proprietary models - look at the `licence` column to identify open-source ones that you can download, then
search for them on the [Hugging Face Hub](https://huggingface.co/models?pipeline_tag=text-generation&sort=trending).

### Specialist domains
Some models may be specialized for certain domains, such as medical or legal text, or non-English languages. 
If you're working in these domains, you may find that a specialized model will give you big performance benefits. 
Don't automatically assume that, though! Particularly when specialized models are smaller or older than the current 
cutting-edge, a top-end general-purpose model may still outclass them. Thankfully, we are beginning to see 
[domain-specific leaderboards](https://huggingface.co/blog/leaderboard-medicalllm) that should make it easier to locate
the best models for specialized domains.

## What happens inside the pipeline?

The quickstart above used a high-level pipeline to chat with a chat model, which is convenient, but not the
most flexible. Let's take a more low-level approach, to see each of the steps involved in chat. Let's start with
a code sample, and then break it down:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Prepare the input as before
chat = [
    {"role": "system", "content": "You are a sassy, wise-cracking robot as imagined by Hollywood circa 1986."},
    {"role": "user", "content": "Hey, can you tell me any fun things to do in New York?"}


key: _node_content
string_value:
{"id_": "76f70e15-c66f-4497-b510-99267d373d46", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "59c18420-6240-4373-8404-893a9a290838", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "d0faf8596bdd083d896283e3e14e6b207798309c03ac95db0d357f4c7d7780d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "928124f2-cb1f-4dff-aaeb-6e9ab275d6f7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "3d71a230a2cb79a7c78fb19e0289d31ad56bbadf026aa34db922bf9f1c1ce64c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "acd5e78a-e124-4c01-ac67-f7517e0c072e", "node_type": "1", "metadata": {}, "hash": "d89cda7242f1103b34849b6e30a65c06075820313544e8b04d02af7060979123", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8099, "end_char_idx": 11261, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note that the LMSys leaderboard
also includes proprietary models - look at the `licence` column to identify open-source ones that you can download, then
search for them on the [Hugging Face Hub](https://huggingface.co/models?pipeline_tag=text-generation&sort=trending).

### Specialist domains
Some models may be specialized for certain domains, such as medical or legal text, or non-English languages. 
If you're working in these domains, you may find that a specialized model will give you big performance benefits. 
Don't automatically assume that, though! Particularly when specialized models are smaller or older than the current 
cutting-edge, a top-end general-purpose model may still outclass them. Thankfully, we are beginning to see 
[domain-specific leaderboards](https://huggingface.co/blog/leaderboard-medicalllm) that should make it easier to locate
the best models for specialized domains.

## What happens inside the pipeline?

The quickstart above used a high-level pipeline to chat with a chat model, which is convenient, but not the
most flexible. Let's take a more low-level approach, to see each of the steps involved in chat. Let's start with
a code sample, and then break it down:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Prepare the input as before
chat = [
    {"role": "system", "content": "You are a sassy, wise-cracking robot as imagined by Hollywood circa 1986."},
    {"role": "user", "content": "Hey, can you tell me any fun things to do in New York?"}
]

# 1: Load the model and tokenizer
model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct", device_map="auto", torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")

# 2: Apply the chat template
formatted_chat = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)
print("Formatted chat:\n", formatted_chat)

# 3: Tokenize the chat (This can be combined with the previous step using tokenize=True)
inputs = tokenizer(formatted_chat, return_tensors="pt", add_special_tokens=False)
# Move the tokenized inputs to the same device the model is on (GPU/CPU)
inputs = {key: tensor.to(model.device) for key, tensor in inputs.items()}
print("Tokenized inputs:\n", inputs)

# 4: Generate text from the model
outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.1)
print("Generated tokens:\n", outputs)

# 5: Decode the output back to a string
decoded_output = tokenizer.decode(outputs[0][inputs['input_ids'].size(1):], skip_special_tokens=True)
print("Decoded output:\n", decoded_output)
```

There's a lot in here, each piece of which could be its own document! Rather than going into too much detail, I'll cover
the broad ideas, and leave the details for the linked documents. The key steps are:

1. [Models](https://huggingface.co/learn/nlp-course/en/chapter2/3) and [Tokenizers](https://huggingface.co/learn/nlp-course/en/chapter2/4?fw=pt) are loaded from the Hugging Face Hub.
2. The chat is formatted using the tokenizer's [chat template](https://huggingface.co/docs/transformers/main/en/chat_templating)
3.


key: _node_content
string_value:
{"id_": "acd5e78a-e124-4c01-ac67-f7517e0c072e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "59c18420-6240-4373-8404-893a9a290838", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "d0faf8596bdd083d896283e3e14e6b207798309c03ac95db0d357f4c7d7780d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "76f70e15-c66f-4497-b510-99267d373d46", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "4289751eb75600542d671b173327bcd7ce7cd7973139b43dcaca9edd82f60726", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5c097d1b-dea1-4b15-8ac3-7cbb976044c9", "node_type": "1", "metadata": {}, "hash": "ef0a01f30941c39bca4814fa1beddc1000d1828bae11e1e0f502378a15ccdd28", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10811, "end_char_idx": 13892, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Rather than going into too much detail, I'll cover
the broad ideas, and leave the details for the linked documents. The key steps are:

1. [Models](https://huggingface.co/learn/nlp-course/en/chapter2/3) and [Tokenizers](https://huggingface.co/learn/nlp-course/en/chapter2/4?fw=pt) are loaded from the Hugging Face Hub.
2. The chat is formatted using the tokenizer's [chat template](https://huggingface.co/docs/transformers/main/en/chat_templating)
3. The formatted chat is [tokenized](https://huggingface.co/learn/nlp-course/en/chapter2/4) using the tokenizer.
4. We [generate](https://huggingface.co/docs/transformers/en/llm_tutorial) a response from the model.
5. The tokens output by the model are decoded back to a string

## Performance, memory and hardware

You probably know by now that most machine learning tasks are run on GPUs. However, it is entirely possible
to generate text from a chat model or language model on a CPU, albeit somewhat more slowly. If you can fit
the model in GPU memory, though, this will usually be the preferable option.

### Memory considerations

By default, Hugging Face classes like [`TextGenerationPipeline`] or [`AutoModelForCausalLM`] will load the model in 
`float32` precision. This means that it will need 4 bytes (32 bits) per parameter, so an "8B" model with 8 billion
parameters will need ~32GB of memory. However, this can be wasteful! Most modern language models are trained in 
"bfloat16" precision, which uses only 2 bytes per parameter. If your hardware supports it (Nvidia 30xx/Axxx
or newer), you can load the model in `bfloat16` precision, using the `torch_dtype` argument as we did above.

It is possible to go even lower than 16-bits using "quantization", a method to lossily compress model weights. This
allows each parameter to be squeezed down to 8 bits, 4 bits or even less. Note that, especially at 4 bits,
the model's outputs may be negatively affected, but often this is a tradeoff worth making to fit a larger and more
capable chat model in memory. Let's see this in action with `bitsandbytes`:

```python
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(load_in_8bit=True)  # You can also try load_in_4bit
model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct", device_map="auto", quantization_config=quantization_config)
```

Or we can do the same thing using the `pipeline` API:

```python
from transformers import pipeline, BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(load_in_8bit=True)  # You can also try load_in_4bit
pipe = pipeline("text-generation", "meta-llama/Meta-Llama-3-8B-Instruct", device_map="auto", model_kwargs={"quantization_config": quantization_config})
```

There are several other options for quantizing models besides `bitsandbytes` - please see the [Quantization guide](./quantization)
for more information.

### Performance considerations

<Tip>

For a more extensive guide on language model performance and optimization, check out [LLM Inference Optimization](./llm_optims) .

</Tip>


key: _node_content
string_value:
{"id_": "5c097d1b-dea1-4b15-8ac3-7cbb976044c9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "59c18420-6240-4373-8404-893a9a290838", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "d0faf8596bdd083d896283e3e14e6b207798309c03ac95db0d357f4c7d7780d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "acd5e78a-e124-4c01-ac67-f7517e0c072e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/conversations", "title": "Chatting with Transformers", "tokens": 4154, "retrieve_doc": true, "source": "transformers"}, "hash": "235374372074d40e89a7710f64e30bcd64cec4f745fe697d41e69946c4cf34d4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12301, "end_char_idx": 16507, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If your hardware supports it (Nvidia 30xx/Axxx
or newer), you can load the model in `bfloat16` precision, using the `torch_dtype` argument as we did above.

It is possible to go even lower than 16-bits using "quantization", a method to lossily compress model weights. This
allows each parameter to be squeezed down to 8 bits, 4 bits or even less. Note that, especially at 4 bits,
the model's outputs may be negatively affected, but often this is a tradeoff worth making to fit a larger and more
capable chat model in memory. Let's see this in action with `bitsandbytes`:

```python
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(load_in_8bit=True)  # You can also try load_in_4bit
model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct", device_map="auto", quantization_config=quantization_config)
```

Or we can do the same thing using the `pipeline` API:

```python
from transformers import pipeline, BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(load_in_8bit=True)  # You can also try load_in_4bit
pipe = pipeline("text-generation", "meta-llama/Meta-Llama-3-8B-Instruct", device_map="auto", model_kwargs={"quantization_config": quantization_config})
```

There are several other options for quantizing models besides `bitsandbytes` - please see the [Quantization guide](./quantization)
for more information.

### Performance considerations

<Tip>

For a more extensive guide on language model performance and optimization, check out [LLM Inference Optimization](./llm_optims) .

</Tip>


As a general rule, larger chat models will be slower in addition to requiring more memory. It's possible to be
more concrete about this, though: Generating text from a chat model is unusual in that it is bottlenecked by
**memory bandwidth** rather than compute power, because every active parameter must be read from memory for each
token that the model generates. This means that number of tokens per second you can generate from a chat
model is generally proportional to the total bandwidth of the memory it resides in, divided by the size of the model.

In our quickstart example above, our model was ~16GB in size when loaded in `bfloat16` precision. 
This means that 16GB must be read from memory for every token generated by the model. Total memory bandwidth can
vary from 20-100GB/sec for consumer CPUs to 200-900GB/sec for consumer GPUs, specialized CPUs like
Intel Xeon, AMD Threadripper/Epyc or high-end Apple silicon, and finally up to 2-3TB/sec for data center GPUs like
the Nvidia A100 or H100. This should give you a good idea of the generation speed you can expect from these different
hardware types.

Therefore, if you want to improve the speed of text generation, the easiest solution is to either reduce the
size of the model in memory (usually by quantization), or get hardware with higher memory bandwidth. For advanced users, 
several other techniques exist to get around this bandwidth bottleneck. The most common are variants on 
[assisted generation](https://huggingface.co/blog/assisted-generation), also known as "speculative
sampling". These techniques try to guess multiple future tokens at once, often using a smaller "draft model", and then
confirm these generations with the chat model. If the guesses are validated by the chat model, more than one token can
be generated per forward pass, which greatly alleviates the bandwidth bottleneck and improves generation speed.  

Finally, we should also note the impact of "Mixture of Experts" (MoE) models here. Several popular chat models,
such as Mixtral, Qwen-MoE and DBRX, are MoE models. In these models, not every parameter is active for every token generated.
As a result, MoE models generally have much lower memory bandwidth requirements, even though their total size
can be quite large. They can therefore be several times faster than a normal "dense" model of the same size. However,
techniques like assisted generation are generally ineffective for these models because more parameters will become
active with each new speculated token, which will negate the bandwidth and speed benefits that the MoE architecture
provides.


key: _node_content
string_value:
{"id_": "ee43eb76-6f8c-40e5-8b24-ba00ccd89a6c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5e9ee699-723a-43ca-977a-24df3760e274", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7848127f-954f-4272-8e53-ac9e04732291", "node_type": "1", "metadata": {}, "hash": "e1a97070c0f542c2a82d1b8466ed0c96915b8669d35b069519b1813076853ad2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3357, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# How 🤗 Transformers solve tasks

In [What 🤗 Transformers can do](task_summary), you learned about natural language processing (NLP), speech and audio, computer vision tasks, and some important applications of them. This page will look closely at how models solve these tasks and explain what's happening under the hood. There are many ways to solve a given task, some models may implement certain techniques or even approach the task from a new angle, but for Transformer models, the general idea is the same. Owing to its flexible architecture, most models are a variant of an encoder, decoder, or encoder-decoder structure. In addition to Transformer models, our library also has several convolutional neural networks (CNNs), which are still used today for computer vision tasks. We'll also explain how a modern CNN works.

To explain how tasks are solved, we'll walk through what goes on inside the model to output useful predictions.

- [Wav2Vec2](model_doc/wav2vec2) for audio classification and automatic speech recognition (ASR)
- [Vision Transformer (ViT)](model_doc/vit) and [ConvNeXT](model_doc/convnext) for image classification
- [DETR](model_doc/detr) for object detection
- [Mask2Former](model_doc/mask2former) for image segmentation
- [GLPN](model_doc/glpn) for depth estimation
- [BERT](model_doc/bert) for NLP tasks like text classification, token classification and question answering that use an encoder
- [GPT2](model_doc/gpt2) for NLP tasks like text generation that use a decoder
- [BART](model_doc/bart) for NLP tasks like summarization and translation that use an encoder-decoder

<Tip>

Before you go further, it is good to have some basic knowledge of the original Transformer architecture. Knowing how encoders, decoders, and attention work will aid you in understanding how different Transformer models work. If you're just getting started or need a refresher, check out our [course](https://huggingface.co/course/chapter1/4?fw=pt) for more information! 

</Tip>

## Speech and audio

[Wav2Vec2](model_doc/wav2vec2) is a self-supervised model pretrained on unlabeled speech data and finetuned on labeled data for audio classification and automatic speech recognition. 

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/wav2vec2_architecture.png"/>
</div>

This model has four main components:

1. A *feature encoder* takes the raw audio waveform, normalizes it to zero mean and unit variance, and converts it into a sequence of feature vectors that are each 20ms long.

2. Waveforms are continuous by nature, so they can't be divided into separate units like a sequence of text can be split into words. That's why the feature vectors are passed to a *quantization module*, which aims to learn discrete speech units. The speech unit is chosen from a collection of codewords, known as a *codebook* (you can think of this as the vocabulary). From the codebook, the vector or speech unit, that best represents the continuous audio input is chosen and forwarded through the model.

3. About half of the feature vectors are randomly masked, and the masked feature vector is fed to a *context network*, which is a Transformer encoder that also adds relative positional embeddings.

4. The pretraining objective of the context network is a *contrastive task*.


key: _node_content
string_value:
{"id_": "7848127f-954f-4272-8e53-ac9e04732291", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5e9ee699-723a-43ca-977a-24df3760e274", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ee43eb76-6f8c-40e5-8b24-ba00ccd89a6c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "9efee23a70ad906a2fb4bc269cec2386f502db23e59d048980346fc69b2b82ac", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c33cc9b2-d2b2-41e4-b95f-5b122d5e0e68", "node_type": "1", "metadata": {}, "hash": "3c31f07bfc10f5796ed28049146f3fb3730c5aab9d5105a78a4ecf2ab284190a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1718, "end_char_idx": 5348, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Knowing how encoders, decoders, and attention work will aid you in understanding how different Transformer models work. If you're just getting started or need a refresher, check out our [course](https://huggingface.co/course/chapter1/4?fw=pt) for more information! 

</Tip>

## Speech and audio

[Wav2Vec2](model_doc/wav2vec2) is a self-supervised model pretrained on unlabeled speech data and finetuned on labeled data for audio classification and automatic speech recognition. 

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/wav2vec2_architecture.png"/>
</div>

This model has four main components:

1. A *feature encoder* takes the raw audio waveform, normalizes it to zero mean and unit variance, and converts it into a sequence of feature vectors that are each 20ms long.

2. Waveforms are continuous by nature, so they can't be divided into separate units like a sequence of text can be split into words. That's why the feature vectors are passed to a *quantization module*, which aims to learn discrete speech units. The speech unit is chosen from a collection of codewords, known as a *codebook* (you can think of this as the vocabulary). From the codebook, the vector or speech unit, that best represents the continuous audio input is chosen and forwarded through the model.

3. About half of the feature vectors are randomly masked, and the masked feature vector is fed to a *context network*, which is a Transformer encoder that also adds relative positional embeddings.

4. The pretraining objective of the context network is a *contrastive task*. The model has to predict the true quantized speech representation of the masked prediction from a set of false ones, encouraging the model to find the most similar context vector and quantized speech unit (the target label).

Now that wav2vec2 is pretrained, you can finetune it on your data for audio classification or automatic speech recognition!

### Audio classification

To use the pretrained model for audio classification, add a sequence classification head on top of the base Wav2Vec2 model. The classification head is a linear layer that accepts the encoder's hidden states. The hidden states represent the learned features from each audio frame which can have varying lengths. To create one vector of fixed-length, the hidden states are pooled first and then transformed into logits over the class labels. The cross-entropy loss is calculated between the logits and target to find the most likely class.

Ready to try your hand at audio classification? Check out our complete [audio classification guide](tasks/audio_classification) to learn how to finetune Wav2Vec2 and use it for inference!

### Automatic speech recognition

To use the pretrained model for automatic speech recognition, add a language modeling head on top of the base Wav2Vec2 model for [connectionist temporal classification (CTC)](glossary#connectionist-temporal-classification-ctc). The language modeling head is a linear layer that accepts the encoder's hidden states and transforms them into logits. Each logit represents a token class (the number of tokens comes from the task vocabulary). The CTC loss is calculated between the logits and targets to find the most likely sequence of tokens, which are then decoded into a transcription.

Ready to try your hand at automatic speech recognition? Check out our complete [automatic speech recognition guide](tasks/asr) to learn how to finetune Wav2Vec2 and use it for inference!

## Computer vision

There are two ways to approach computer vision tasks:

1.


key: _node_content
string_value:
{"id_": "c33cc9b2-d2b2-41e4-b95f-5b122d5e0e68", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5e9ee699-723a-43ca-977a-24df3760e274", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7848127f-954f-4272-8e53-ac9e04732291", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "c848dc72b44ff248da5ed62fb8f91d954c8ef5e8c0c04aa49649a5709a67a09c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e2342916-f819-48f5-b5fc-8d0ad4703ce5", "node_type": "1", "metadata": {}, "hash": "e04cf1ad489f740330c0cc09f61ecec877ea432baaf3480ebd43de1337292068", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3584, "end_char_idx": 7211, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Now that wav2vec2 is pretrained, you can finetune it on your data for audio classification or automatic speech recognition!

### Audio classification

To use the pretrained model for audio classification, add a sequence classification head on top of the base Wav2Vec2 model. The classification head is a linear layer that accepts the encoder's hidden states. The hidden states represent the learned features from each audio frame which can have varying lengths. To create one vector of fixed-length, the hidden states are pooled first and then transformed into logits over the class labels. The cross-entropy loss is calculated between the logits and target to find the most likely class.

Ready to try your hand at audio classification? Check out our complete [audio classification guide](tasks/audio_classification) to learn how to finetune Wav2Vec2 and use it for inference!

### Automatic speech recognition

To use the pretrained model for automatic speech recognition, add a language modeling head on top of the base Wav2Vec2 model for [connectionist temporal classification (CTC)](glossary#connectionist-temporal-classification-ctc). The language modeling head is a linear layer that accepts the encoder's hidden states and transforms them into logits. Each logit represents a token class (the number of tokens comes from the task vocabulary). The CTC loss is calculated between the logits and targets to find the most likely sequence of tokens, which are then decoded into a transcription.

Ready to try your hand at automatic speech recognition? Check out our complete [automatic speech recognition guide](tasks/asr) to learn how to finetune Wav2Vec2 and use it for inference!

## Computer vision

There are two ways to approach computer vision tasks:

1. Split an image into a sequence of patches and process them in parallel with a Transformer.
2. Use a modern CNN, like [ConvNeXT](model_doc/convnext), which relies on convolutional layers but adopts modern network designs.

<Tip>

A third approach mixes Transformers with convolutions (for example, [Convolutional Vision Transformer](model_doc/cvt) or [LeViT](model_doc/levit)). We won't discuss those because they just combine the two approaches we examine here.

</Tip>

ViT and ConvNeXT are commonly used for image classification, but for other vision tasks like object detection, segmentation, and depth estimation, we'll look at DETR, Mask2Former and GLPN, respectively; these models are better suited for those tasks.

### Image classification

ViT and ConvNeXT can both be used for image classification; the main difference is that ViT uses an attention mechanism while ConvNeXT uses convolutions.

#### Transformer

[ViT](model_doc/vit) replaces convolutions entirely with a pure Transformer architecture. If you're familiar with the original Transformer, then you're already most of the way toward understanding ViT.

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/vit_architecture.jpg"/>
</div>

The main change ViT introduced was in how images are fed to a Transformer:

1. An image is split into square non-overlapping patches, each of which gets turned into a vector or *patch embedding*. The patch embeddings are generated from a convolutional 2D layer which creates the proper input dimensions (which for a base Transformer is 768 values for each patch embedding). If you had a 224x224 pixel image, you could split it into 196 16x16 image patches. Just like how text is tokenized into words, an image is "tokenized" into a sequence of patches.

2.


key: _node_content
string_value:
{"id_": "e2342916-f819-48f5-b5fc-8d0ad4703ce5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5e9ee699-723a-43ca-977a-24df3760e274", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c33cc9b2-d2b2-41e4-b95f-5b122d5e0e68", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "d1540bb187db31a0d5322b18cdef85f54350f6071f2fcdcd334de30b9ee28e0e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bc719bea-1cd7-41fc-be8a-27ea1c3d212e", "node_type": "1", "metadata": {}, "hash": "5d694f3409c116fe359e1bc25ae552327265e19083d99e0d03cbe7f469032d10", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5440, "end_char_idx": 8911, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
2. Use a modern CNN, like [ConvNeXT](model_doc/convnext), which relies on convolutional layers but adopts modern network designs.

<Tip>

A third approach mixes Transformers with convolutions (for example, [Convolutional Vision Transformer](model_doc/cvt) or [LeViT](model_doc/levit)). We won't discuss those because they just combine the two approaches we examine here.

</Tip>

ViT and ConvNeXT are commonly used for image classification, but for other vision tasks like object detection, segmentation, and depth estimation, we'll look at DETR, Mask2Former and GLPN, respectively; these models are better suited for those tasks.

### Image classification

ViT and ConvNeXT can both be used for image classification; the main difference is that ViT uses an attention mechanism while ConvNeXT uses convolutions.

#### Transformer

[ViT](model_doc/vit) replaces convolutions entirely with a pure Transformer architecture. If you're familiar with the original Transformer, then you're already most of the way toward understanding ViT.

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/vit_architecture.jpg"/>
</div>

The main change ViT introduced was in how images are fed to a Transformer:

1. An image is split into square non-overlapping patches, each of which gets turned into a vector or *patch embedding*. The patch embeddings are generated from a convolutional 2D layer which creates the proper input dimensions (which for a base Transformer is 768 values for each patch embedding). If you had a 224x224 pixel image, you could split it into 196 16x16 image patches. Just like how text is tokenized into words, an image is "tokenized" into a sequence of patches.

2. A *learnable embedding* - a special `[CLS]` token - is added to the beginning of the patch embeddings just like BERT. The final hidden state of the `[CLS]` token is used as the input to the attached classification head; other outputs are ignored. This token helps the model learn how to encode a representation of the image.

3. The last thing to add to the patch and learnable embeddings are the *position embeddings* because the model doesn't know how the image patches are ordered. The position embeddings are also learnable and have the same size as the patch embeddings. Finally, all of the embeddings are passed to the Transformer encoder.

4. The output, specifically only the output with the `[CLS]` token, is passed to a multilayer perceptron head (MLP). ViT's pretraining objective is simply classification. Like other classification heads, the MLP head converts the output into logits over the class labels and calculates the cross-entropy loss to find the most likely class.

Ready to try your hand at image classification? Check out our complete [image classification guide](tasks/image_classification) to learn how to finetune ViT and use it for inference!

#### CNN

<Tip>

This section briefly explains convolutions, but it'd be helpful to have a prior understanding of how they change an image's shape and size. If you're unfamiliar with convolutions, check out the [Convolution Neural Networks chapter](https://github.com/fastai/fastbook/blob/master/13_convolutions.ipynb) from the fastai book!

</Tip>

[ConvNeXT](model_doc/convnext) is a CNN architecture that adopts new and modern network designs to improve performance. However, convolutions are still at the core of the model.


key: _node_content
string_value:
{"id_": "bc719bea-1cd7-41fc-be8a-27ea1c3d212e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5e9ee699-723a-43ca-977a-24df3760e274", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e2342916-f819-48f5-b5fc-8d0ad4703ce5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "9b43b07829aba4a6f7bd353683603dfb2213ba3b7857fce6ad3db904d404fb86", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "39c3231d-be8f-4255-a5ff-f66c9b63af76", "node_type": "1", "metadata": {}, "hash": "ac3c9ff2a235f2f925b88f3d21cd67c2868aadb3ab7077fca49d3b5e56532360", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7112, "end_char_idx": 10537, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Just like how text is tokenized into words, an image is "tokenized" into a sequence of patches.

2. A *learnable embedding* - a special `[CLS]` token - is added to the beginning of the patch embeddings just like BERT. The final hidden state of the `[CLS]` token is used as the input to the attached classification head; other outputs are ignored. This token helps the model learn how to encode a representation of the image.

3. The last thing to add to the patch and learnable embeddings are the *position embeddings* because the model doesn't know how the image patches are ordered. The position embeddings are also learnable and have the same size as the patch embeddings. Finally, all of the embeddings are passed to the Transformer encoder.

4. The output, specifically only the output with the `[CLS]` token, is passed to a multilayer perceptron head (MLP). ViT's pretraining objective is simply classification. Like other classification heads, the MLP head converts the output into logits over the class labels and calculates the cross-entropy loss to find the most likely class.

Ready to try your hand at image classification? Check out our complete [image classification guide](tasks/image_classification) to learn how to finetune ViT and use it for inference!

#### CNN

<Tip>

This section briefly explains convolutions, but it'd be helpful to have a prior understanding of how they change an image's shape and size. If you're unfamiliar with convolutions, check out the [Convolution Neural Networks chapter](https://github.com/fastai/fastbook/blob/master/13_convolutions.ipynb) from the fastai book!

</Tip>

[ConvNeXT](model_doc/convnext) is a CNN architecture that adopts new and modern network designs to improve performance. However, convolutions are still at the core of the model. From a high-level perspective, a [convolution](glossary#convolution) is an operation where a smaller matrix (*kernel*) is multiplied by a small window of the image pixels. It computes some features from it, such as a particular texture or curvature of a line. Then it slides over to the next window of pixels; the distance the convolution travels is known as the *stride*. 

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/convolution.gif"/>
</div>

<small>A basic convolution without padding or stride, taken from <a href="https://arxiv.org/abs/1603.07285">A guide to convolution arithmetic for deep learning.</a></small>

You can feed this output to another convolutional layer, and with each successive layer, the network learns more complex and abstract things like hotdogs or rockets. Between convolutional layers, it is common to add a pooling layer to reduce dimensionality and make the model more robust to variations of a feature's position.

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/convnext_architecture.png"/>
</div>

ConvNeXT modernizes a CNN in five ways:

1. Change the number of blocks in each stage and "patchify" an image with a larger stride and corresponding kernel size. The non-overlapping sliding window makes this patchifying strategy similar to how ViT splits an image into patches.

2. A *bottleneck* layer shrinks the number of channels and then restores it because it is faster to do a 1x1 convolution, and you can increase the depth.


key: _node_content
string_value:
{"id_": "39c3231d-be8f-4255-a5ff-f66c9b63af76", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5e9ee699-723a-43ca-977a-24df3760e274", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bc719bea-1cd7-41fc-be8a-27ea1c3d212e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "e8ddaf57927da5a407c62ad1d60a8ffce122d4a04aa17740a1fda26747404893", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "03dbd2c1-e994-45c6-89e1-34d91173046a", "node_type": "1", "metadata": {}, "hash": "09b978741bcc88f050366efa761ed7ca536a8ccacbb2911a5a90aff364625130", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8854, "end_char_idx": 12436, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, convolutions are still at the core of the model. From a high-level perspective, a [convolution](glossary#convolution) is an operation where a smaller matrix (*kernel*) is multiplied by a small window of the image pixels. It computes some features from it, such as a particular texture or curvature of a line. Then it slides over to the next window of pixels; the distance the convolution travels is known as the *stride*. 

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/convolution.gif"/>
</div>

<small>A basic convolution without padding or stride, taken from <a href="https://arxiv.org/abs/1603.07285">A guide to convolution arithmetic for deep learning.</a></small>

You can feed this output to another convolutional layer, and with each successive layer, the network learns more complex and abstract things like hotdogs or rockets. Between convolutional layers, it is common to add a pooling layer to reduce dimensionality and make the model more robust to variations of a feature's position.

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/convnext_architecture.png"/>
</div>

ConvNeXT modernizes a CNN in five ways:

1. Change the number of blocks in each stage and "patchify" an image with a larger stride and corresponding kernel size. The non-overlapping sliding window makes this patchifying strategy similar to how ViT splits an image into patches.

2. A *bottleneck* layer shrinks the number of channels and then restores it because it is faster to do a 1x1 convolution, and you can increase the depth. An inverted bottleneck does the opposite by expanding the number of channels and shrinking them, which is more memory efficient.

3. Replace the typical 3x3 convolutional layer in the bottleneck layer with *depthwise convolution*, which applies a convolution to each input channel separately and then stacks them back together at the end. This widens the network width for improved performance.

4. ViT has a global receptive field which means it can see more of an image at once thanks to its attention mechanism. ConvNeXT attempts to replicate this effect by increasing the kernel size to 7x7.

5. ConvNeXT also makes several layer design changes that imitate Transformer models. There are fewer activation and normalization layers,  the activation function is switched to GELU instead of ReLU, and it uses LayerNorm instead of BatchNorm.

The output from the convolution blocks is passed to a classification head which converts the outputs into logits and calculates the cross-entropy loss to find the most likely label.

### Object detection

[DETR](model_doc/detr), *DEtection TRansformer*, is an end-to-end object detection model that combines a CNN with a Transformer encoder-decoder.

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/detr_architecture.png"/>
</div>

1. A pretrained CNN *backbone* takes an image, represented by its pixel values, and creates a low-resolution feature map of it. A 1x1 convolution is applied to the feature map to reduce dimensionality and it creates a new feature map with a high-level image representation. Since the Transformer is a sequential model, the feature map is flattened into a sequence of feature vectors that are combined with positional embeddings.

2. The feature vectors are passed to the encoder, which learns the image representations using its attention layers.


key: _node_content
string_value:
{"id_": "03dbd2c1-e994-45c6-89e1-34d91173046a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5e9ee699-723a-43ca-977a-24df3760e274", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "39c3231d-be8f-4255-a5ff-f66c9b63af76", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "2f7c5a17f667cb25aa925465d858d2d0bc3483a1c91d9566e90a8e7b1e3f544c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "48462d69-2f67-476a-81a5-329c2bda9b40", "node_type": "1", "metadata": {}, "hash": "f964375713d8d0ada0ada6bc873dfbf82c720e246c24f89c99b027f763c422e5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10538, "end_char_idx": 14135, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
An inverted bottleneck does the opposite by expanding the number of channels and shrinking them, which is more memory efficient.

3. Replace the typical 3x3 convolutional layer in the bottleneck layer with *depthwise convolution*, which applies a convolution to each input channel separately and then stacks them back together at the end. This widens the network width for improved performance.

4. ViT has a global receptive field which means it can see more of an image at once thanks to its attention mechanism. ConvNeXT attempts to replicate this effect by increasing the kernel size to 7x7.

5. ConvNeXT also makes several layer design changes that imitate Transformer models. There are fewer activation and normalization layers,  the activation function is switched to GELU instead of ReLU, and it uses LayerNorm instead of BatchNorm.

The output from the convolution blocks is passed to a classification head which converts the outputs into logits and calculates the cross-entropy loss to find the most likely label.

### Object detection

[DETR](model_doc/detr), *DEtection TRansformer*, is an end-to-end object detection model that combines a CNN with a Transformer encoder-decoder.

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/detr_architecture.png"/>
</div>

1. A pretrained CNN *backbone* takes an image, represented by its pixel values, and creates a low-resolution feature map of it. A 1x1 convolution is applied to the feature map to reduce dimensionality and it creates a new feature map with a high-level image representation. Since the Transformer is a sequential model, the feature map is flattened into a sequence of feature vectors that are combined with positional embeddings.

2. The feature vectors are passed to the encoder, which learns the image representations using its attention layers. Next, the encoder hidden states are combined with *object queries* in the decoder. Object queries are learned embeddings that focus on the different regions of an image, and they're updated as they progress through each attention layer. The decoder hidden states are passed to a feedforward network that predicts the bounding box coordinates and class label for each object query, or `no object` if there isn't one.

    DETR decodes each object query in parallel to output *N* final predictions, where *N* is the number of queries. Unlike a typical autoregressive model that predicts one element at a time, object detection is a set prediction task (`bounding box`, `class label`) that makes *N* predictions in a single pass.

3. DETR uses a *bipartite matching loss* during training to compare a fixed number of predictions with a fixed set of ground truth labels. If there are fewer ground truth labels in the set of *N* labels, then they're padded with a `no object` class. This loss function encourages DETR to find a one-to-one assignment between the predictions and ground truth labels. If either the bounding boxes or class labels aren't correct, a loss is incurred. Likewise, if DETR predicts an object that doesn't exist, it is penalized. This encourages DETR to find other objects in an image instead of focusing on one really prominent object.

An object detection head is added on top of DETR to find the class label and the coordinates of the bounding box. There are two components to the object detection head: a linear layer to transform the decoder hidden states into logits over the class labels, and a MLP to predict the bounding box.

Ready to try your hand at object detection?


key: _node_content
string_value:
{"id_": "48462d69-2f67-476a-81a5-329c2bda9b40", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5e9ee699-723a-43ca-977a-24df3760e274", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "03dbd2c1-e994-45c6-89e1-34d91173046a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "989780a72eeb326a875ef3191477d3b6f7a119ff6a7c0d2a8182b36401ac66a0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1a058785-76b6-4331-bf9b-412b482a99eb", "node_type": "1", "metadata": {}, "hash": "e6de48e286d8fb3f0bf3c86c37259979d33690b27ceea1a417475bd0a40ac31d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12320, "end_char_idx": 15927, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
2. The feature vectors are passed to the encoder, which learns the image representations using its attention layers. Next, the encoder hidden states are combined with *object queries* in the decoder. Object queries are learned embeddings that focus on the different regions of an image, and they're updated as they progress through each attention layer. The decoder hidden states are passed to a feedforward network that predicts the bounding box coordinates and class label for each object query, or `no object` if there isn't one.

    DETR decodes each object query in parallel to output *N* final predictions, where *N* is the number of queries. Unlike a typical autoregressive model that predicts one element at a time, object detection is a set prediction task (`bounding box`, `class label`) that makes *N* predictions in a single pass.

3. DETR uses a *bipartite matching loss* during training to compare a fixed number of predictions with a fixed set of ground truth labels. If there are fewer ground truth labels in the set of *N* labels, then they're padded with a `no object` class. This loss function encourages DETR to find a one-to-one assignment between the predictions and ground truth labels. If either the bounding boxes or class labels aren't correct, a loss is incurred. Likewise, if DETR predicts an object that doesn't exist, it is penalized. This encourages DETR to find other objects in an image instead of focusing on one really prominent object.

An object detection head is added on top of DETR to find the class label and the coordinates of the bounding box. There are two components to the object detection head: a linear layer to transform the decoder hidden states into logits over the class labels, and a MLP to predict the bounding box.

Ready to try your hand at object detection? Check out our complete [object detection guide](tasks/object_detection) to learn how to finetune DETR and use it for inference!

### Image segmentation

[Mask2Former](model_doc/mask2former) is a universal architecture for solving all types of image segmentation tasks. Traditional segmentation models are typically tailored towards a particular subtask of image segmentation, like instance, semantic or panoptic segmentation. Mask2Former frames each of those tasks as a *mask classification* problem. Mask classification groups pixels into *N* segments, and predicts *N* masks and their corresponding class label for a given image. We'll explain how Mask2Former works in this section, and then you can try finetuning SegFormer at the end.

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/mask2former_architecture.png"/>
</div>

There are three main components to Mask2Former:

1. A [Swin](model_doc/swin) backbone accepts an image and creates a low-resolution image feature map from 3 consecutive 3x3 convolutions.

2. The feature map is passed to a *pixel decoder* which gradually upsamples the low-resolution features into high-resolution per-pixel embeddings. The pixel decoder actually generates multi-scale features (contains both low- and high-resolution features) with resolutions 1/32, 1/16, and 1/8th of the original image.

3. Each of these feature maps of differing scales is fed successively to one Transformer decoder layer at a time in order to capture small objects from the high-resolution features. The key to Mask2Former is the *masked attention* mechanism in the decoder. Unlike cross-attention which can attend to the entire image, masked attention only focuses on a certain area of the image.


key: _node_content
string_value:
{"id_": "1a058785-76b6-4331-bf9b-412b482a99eb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5e9ee699-723a-43ca-977a-24df3760e274", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "48462d69-2f67-476a-81a5-329c2bda9b40", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "49ad9b58a6f55045329df1d65cd005a60abfbfa075095179b655dc724ca3123d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a3bc914e-5d46-4baa-ae27-1a510d263cec", "node_type": "1", "metadata": {}, "hash": "62ebdb7a5ae9ea2bc6fe057fa014c66129ea80b79ea5e15b7d40c57262d3ba1d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14092, "end_char_idx": 17750, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Ready to try your hand at object detection? Check out our complete [object detection guide](tasks/object_detection) to learn how to finetune DETR and use it for inference!

### Image segmentation

[Mask2Former](model_doc/mask2former) is a universal architecture for solving all types of image segmentation tasks. Traditional segmentation models are typically tailored towards a particular subtask of image segmentation, like instance, semantic or panoptic segmentation. Mask2Former frames each of those tasks as a *mask classification* problem. Mask classification groups pixels into *N* segments, and predicts *N* masks and their corresponding class label for a given image. We'll explain how Mask2Former works in this section, and then you can try finetuning SegFormer at the end.

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/mask2former_architecture.png"/>
</div>

There are three main components to Mask2Former:

1. A [Swin](model_doc/swin) backbone accepts an image and creates a low-resolution image feature map from 3 consecutive 3x3 convolutions.

2. The feature map is passed to a *pixel decoder* which gradually upsamples the low-resolution features into high-resolution per-pixel embeddings. The pixel decoder actually generates multi-scale features (contains both low- and high-resolution features) with resolutions 1/32, 1/16, and 1/8th of the original image.

3. Each of these feature maps of differing scales is fed successively to one Transformer decoder layer at a time in order to capture small objects from the high-resolution features. The key to Mask2Former is the *masked attention* mechanism in the decoder. Unlike cross-attention which can attend to the entire image, masked attention only focuses on a certain area of the image. This is faster and leads to better performance because the local features of an image are enough for the model to learn from.

4. Like [DETR](tasks_explained#object-detection), Mask2Former also uses learned object queries and combines them with the image features from the pixel decoder to make a set prediction (`class label`, `mask prediction`). The decoder hidden states are passed into a linear layer and transformed into logits over the class labels. The cross-entropy loss is calculated between the logits and class label to find the most likely one.

    The mask predictions are generated by combining the pixel-embeddings with the final decoder hidden states. The sigmoid cross-entropy and dice loss is calculated between the logits and the ground truth mask to find the most likely mask.

Ready to try your hand at object detection? Check out our complete [image segmentation guide](tasks/semantic_segmentation) to learn how to finetune SegFormer and use it for inference!

### Depth estimation

[GLPN](model_doc/glpn), *Global-Local Path Network*, is a Transformer for depth estimation that combines a [SegFormer](model_doc/segformer) encoder with a lightweight decoder.

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/glpn_architecture.jpg"/>
</div>

1. Like ViT, an image is split into a sequence of patches, except these image patches are smaller. This is better for dense prediction tasks like segmentation or depth estimation. The image patches are transformed into patch embeddings (see the [image classification](#image-classification) section for more details about how patch embeddings are created), which are fed to the encoder.

2. The encoder accepts the patch embeddings, and passes them through several encoder blocks.


key: _node_content
string_value:
{"id_": "a3bc914e-5d46-4baa-ae27-1a510d263cec", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5e9ee699-723a-43ca-977a-24df3760e274", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1a058785-76b6-4331-bf9b-412b482a99eb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "bd9ba8a30e2d3ae93cf5ff30b4c5c22b6efed2f873f3bf45ca314388f0a1f2f6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1ba84c8e-7141-4de8-8c47-b1bece89ffa1", "node_type": "1", "metadata": {}, "hash": "7e09583d737b9f04549311f7e9ae3edd95aa712e5c6047600e77b0788cc987a9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15928, "end_char_idx": 19576, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This is faster and leads to better performance because the local features of an image are enough for the model to learn from.

4. Like [DETR](tasks_explained#object-detection), Mask2Former also uses learned object queries and combines them with the image features from the pixel decoder to make a set prediction (`class label`, `mask prediction`). The decoder hidden states are passed into a linear layer and transformed into logits over the class labels. The cross-entropy loss is calculated between the logits and class label to find the most likely one.

    The mask predictions are generated by combining the pixel-embeddings with the final decoder hidden states. The sigmoid cross-entropy and dice loss is calculated between the logits and the ground truth mask to find the most likely mask.

Ready to try your hand at object detection? Check out our complete [image segmentation guide](tasks/semantic_segmentation) to learn how to finetune SegFormer and use it for inference!

### Depth estimation

[GLPN](model_doc/glpn), *Global-Local Path Network*, is a Transformer for depth estimation that combines a [SegFormer](model_doc/segformer) encoder with a lightweight decoder.

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/glpn_architecture.jpg"/>
</div>

1. Like ViT, an image is split into a sequence of patches, except these image patches are smaller. This is better for dense prediction tasks like segmentation or depth estimation. The image patches are transformed into patch embeddings (see the [image classification](#image-classification) section for more details about how patch embeddings are created), which are fed to the encoder.

2. The encoder accepts the patch embeddings, and passes them through several encoder blocks. Each block consists of attention and Mix-FFN layers. The purpose of the latter is to provide positional information. At the end of each encoder block is a *patch merging* layer for creating hierarchical representations. The features of each group of neighboring patches are concatenated, and a linear layer is applied to the concatenated features to reduce the number of patches to a resolution of 1/4. This becomes the input to the next encoder block, where this whole process is repeated until you have image features with resolutions of 1/8, 1/16, and 1/32.

3. A lightweight decoder takes the last feature map (1/32 scale) from the encoder and upsamples it to 1/16 scale. From here, the feature is passed into a *Selective Feature Fusion (SFF)* module, which selects and combines local and global features from an attention map for each feature and then upsamples it to 1/8th. This process is repeated until the decoded features are the same size as the original image. The output is passed through two convolution layers and then a sigmoid activation is applied to predict the depth of each pixel.

## Natural language processing

The Transformer was initially designed for machine translation, and since then, it has practically become the default architecture for solving all NLP tasks. Some tasks lend themselves to the Transformer's encoder structure, while others are better suited for the decoder. Still, other tasks make use of both the Transformer's encoder-decoder structure.

### Text classification

[BERT](model_doc/bert) is an encoder-only model and is the first model to effectively implement deep bidirectionality to learn richer representations of the text by attending to words on both sides.

1. BERT uses [WordPiece](tokenizer_summary#wordpiece) tokenization to generate a token embedding of the text.


key: _node_content
string_value:
{"id_": "1ba84c8e-7141-4de8-8c47-b1bece89ffa1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5e9ee699-723a-43ca-977a-24df3760e274", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a3bc914e-5d46-4baa-ae27-1a510d263cec", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "229fde9a57da3e36f0f52863085cd4ced95aa385a49e3f1244f3bef97536f237", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "78ad270c-5a0a-4f53-84ec-2bc96fa13c30", "node_type": "1", "metadata": {}, "hash": "8818a630621453c7299575ea83566a7afc233719930e51c5b9e64e560b396b84", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17661, "end_char_idx": 21437, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The encoder accepts the patch embeddings, and passes them through several encoder blocks. Each block consists of attention and Mix-FFN layers. The purpose of the latter is to provide positional information. At the end of each encoder block is a *patch merging* layer for creating hierarchical representations. The features of each group of neighboring patches are concatenated, and a linear layer is applied to the concatenated features to reduce the number of patches to a resolution of 1/4. This becomes the input to the next encoder block, where this whole process is repeated until you have image features with resolutions of 1/8, 1/16, and 1/32.

3. A lightweight decoder takes the last feature map (1/32 scale) from the encoder and upsamples it to 1/16 scale. From here, the feature is passed into a *Selective Feature Fusion (SFF)* module, which selects and combines local and global features from an attention map for each feature and then upsamples it to 1/8th. This process is repeated until the decoded features are the same size as the original image. The output is passed through two convolution layers and then a sigmoid activation is applied to predict the depth of each pixel.

## Natural language processing

The Transformer was initially designed for machine translation, and since then, it has practically become the default architecture for solving all NLP tasks. Some tasks lend themselves to the Transformer's encoder structure, while others are better suited for the decoder. Still, other tasks make use of both the Transformer's encoder-decoder structure.

### Text classification

[BERT](model_doc/bert) is an encoder-only model and is the first model to effectively implement deep bidirectionality to learn richer representations of the text by attending to words on both sides.

1. BERT uses [WordPiece](tokenizer_summary#wordpiece) tokenization to generate a token embedding of the text. To tell the difference between a single sentence and a pair of sentences, a special `[SEP]` token is added to differentiate them. A special `[CLS]` token is added to the beginning of every sequence of text. The final output with the `[CLS]` token is used as the input to the classification head for classification tasks. BERT also adds a segment embedding to denote whether a token belongs to the first or second sentence in a pair of sentences.

2. BERT is pretrained with two objectives: masked language modeling and next-sentence prediction. In masked language modeling, some percentage of the input tokens are randomly masked, and the model needs to predict these. This solves the issue of bidirectionality, where the model could cheat and see all the words and "predict" the next word. The final hidden states of the predicted mask tokens are passed to a feedforward network with a softmax over the vocabulary to predict the masked word.

    The second pretraining object is next-sentence prediction. The model must predict whether sentence B follows sentence A. Half of the time sentence B is the next sentence, and the other half of the time, sentence B is a random sentence. The prediction, whether it is the next sentence or not, is passed to a feedforward network with a softmax over the two classes (`IsNext` and `NotNext`).

3. The input embeddings are passed through multiple encoder layers to output some final hidden states.

To use the pretrained model for text classification, add a sequence classification head on top of the base BERT model. The sequence classification head is a linear layer that accepts the final hidden states and performs a linear transformation to convert them into logits. The cross-entropy loss is calculated between the logits and target to find the most likely label.

Ready to try your hand at text classification?


key: _node_content
string_value:
{"id_": "78ad270c-5a0a-4f53-84ec-2bc96fa13c30", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5e9ee699-723a-43ca-977a-24df3760e274", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1ba84c8e-7141-4de8-8c47-b1bece89ffa1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "ea63e9c36267cd8c4c991e46f04419836ab1d88a972c84c2a43855ca523956c2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4f036edc-c04f-478a-8829-645d27b8a6d8", "node_type": "1", "metadata": {}, "hash": "b563f4f1b38962d6977eae07b01a09daea6b465ee64083821ea7c489aa0d41fd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19577, "end_char_idx": 23300, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
To tell the difference between a single sentence and a pair of sentences, a special `[SEP]` token is added to differentiate them. A special `[CLS]` token is added to the beginning of every sequence of text. The final output with the `[CLS]` token is used as the input to the classification head for classification tasks. BERT also adds a segment embedding to denote whether a token belongs to the first or second sentence in a pair of sentences.

2. BERT is pretrained with two objectives: masked language modeling and next-sentence prediction. In masked language modeling, some percentage of the input tokens are randomly masked, and the model needs to predict these. This solves the issue of bidirectionality, where the model could cheat and see all the words and "predict" the next word. The final hidden states of the predicted mask tokens are passed to a feedforward network with a softmax over the vocabulary to predict the masked word.

    The second pretraining object is next-sentence prediction. The model must predict whether sentence B follows sentence A. Half of the time sentence B is the next sentence, and the other half of the time, sentence B is a random sentence. The prediction, whether it is the next sentence or not, is passed to a feedforward network with a softmax over the two classes (`IsNext` and `NotNext`).

3. The input embeddings are passed through multiple encoder layers to output some final hidden states.

To use the pretrained model for text classification, add a sequence classification head on top of the base BERT model. The sequence classification head is a linear layer that accepts the final hidden states and performs a linear transformation to convert them into logits. The cross-entropy loss is calculated between the logits and target to find the most likely label.

Ready to try your hand at text classification? Check out our complete [text classification guide](tasks/sequence_classification) to learn how to finetune DistilBERT and use it for inference!

### Token classification

To use BERT for token classification tasks like named entity recognition (NER), add a token classification head on top of the base BERT model. The token classification head is a linear layer that accepts the final hidden states and performs a linear transformation to convert them into logits. The cross-entropy loss is calculated between the logits and each token to find the most likely label.

Ready to try your hand at token classification? Check out our complete [token classification guide](tasks/token_classification) to learn how to finetune DistilBERT and use it for inference!

### Question answering

To use BERT for question answering, add a span classification head on top of the base BERT model. This linear layer accepts the final hidden states and performs a linear transformation to compute the `span` start and end logits corresponding to the answer. The cross-entropy loss is calculated between the logits and the label position to find the most likely span of text corresponding to the answer.

Ready to try your hand at question answering? Check out our complete [question answering guide](tasks/question_answering) to learn how to finetune DistilBERT and use it for inference!

<Tip>

💡 Notice how easy it is to use BERT for different tasks once it's been pretrained. You only need to add a specific head to the pretrained model to manipulate the hidden states into your desired output!

</Tip>

### Text generation

[GPT-2](model_doc/gpt2) is a decoder-only model pretrained on a large amount of text. It can generate convincing (though not always true!) text given a prompt and complete other NLP tasks like question answering despite not being explicitly trained to.


key: _node_content
string_value:
{"id_": "4f036edc-c04f-478a-8829-645d27b8a6d8", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5e9ee699-723a-43ca-977a-24df3760e274", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "78ad270c-5a0a-4f53-84ec-2bc96fa13c30", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "824cf80951093e20981ecfa43a53d4afc98eab376f7801fc5c7fb686ec2cc53a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5c277a4e-2661-4b3b-ad67-4195303806d1", "node_type": "1", "metadata": {}, "hash": "57def7db72d629f5f29568f0bcc759ed1005eae7987fedc4f1a07a6180c817ea", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21391, "end_char_idx": 24938, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Ready to try your hand at text classification? Check out our complete [text classification guide](tasks/sequence_classification) to learn how to finetune DistilBERT and use it for inference!

### Token classification

To use BERT for token classification tasks like named entity recognition (NER), add a token classification head on top of the base BERT model. The token classification head is a linear layer that accepts the final hidden states and performs a linear transformation to convert them into logits. The cross-entropy loss is calculated between the logits and each token to find the most likely label.

Ready to try your hand at token classification? Check out our complete [token classification guide](tasks/token_classification) to learn how to finetune DistilBERT and use it for inference!

### Question answering

To use BERT for question answering, add a span classification head on top of the base BERT model. This linear layer accepts the final hidden states and performs a linear transformation to compute the `span` start and end logits corresponding to the answer. The cross-entropy loss is calculated between the logits and the label position to find the most likely span of text corresponding to the answer.

Ready to try your hand at question answering? Check out our complete [question answering guide](tasks/question_answering) to learn how to finetune DistilBERT and use it for inference!

<Tip>

💡 Notice how easy it is to use BERT for different tasks once it's been pretrained. You only need to add a specific head to the pretrained model to manipulate the hidden states into your desired output!

</Tip>

### Text generation

[GPT-2](model_doc/gpt2) is a decoder-only model pretrained on a large amount of text. It can generate convincing (though not always true!) text given a prompt and complete other NLP tasks like question answering despite not being explicitly trained to.

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/gpt2_architecture.png"/>
</div>

1. GPT-2 uses [byte pair encoding (BPE)](tokenizer_summary#bytepair-encoding-bpe) to tokenize words and generate a token embedding. Positional encodings are added to the token embeddings to indicate the position of each token in the sequence. The input embeddings are passed through multiple decoder blocks to output some final hidden state. Within each decoder block, GPT-2 uses a *masked self-attention* layer which means GPT-2 can't attend to future tokens. It is only allowed to attend to tokens on the left. This is different from BERT's [`mask`] token because, in masked self-attention, an attention mask is used to set the score to `0` for future tokens.

2. The output from the decoder is passed to a language modeling head, which performs a linear transformation to convert the hidden states into logits. The label is the next token in the sequence, which are created by shifting the logits to the right by one. The cross-entropy loss is calculated between the shifted logits and the labels to output the next most likely token.

GPT-2's pretraining objective is based entirely on [causal language modeling](glossary#causal-language-modeling), predicting the next word in a sequence. This makes GPT-2 especially good at tasks that involve generating text.

Ready to try your hand at text generation? Check out our complete [causal language modeling guide](tasks/language_modeling#causal-language-modeling) to learn how to finetune DistilGPT-2 and use it for inference!


key: _node_content
string_value:
{"id_": "5c277a4e-2661-4b3b-ad67-4195303806d1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5e9ee699-723a-43ca-977a-24df3760e274", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4f036edc-c04f-478a-8829-645d27b8a6d8", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "a228099ef30996def132b050906fbe8d82a2f9c83e32ce9d1944e1dedf59ac15", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ed8c2a3a-e290-4e59-9737-8390157c9d42", "node_type": "1", "metadata": {}, "hash": "634f5ba3d985fd494aafa22dc7a95b2d710204be9797674090342cec1a6b2ddc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23187, "end_char_idx": 26645, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
text given a prompt and complete other NLP tasks like question answering despite not being explicitly trained to.

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/gpt2_architecture.png"/>
</div>

1. GPT-2 uses [byte pair encoding (BPE)](tokenizer_summary#bytepair-encoding-bpe) to tokenize words and generate a token embedding. Positional encodings are added to the token embeddings to indicate the position of each token in the sequence. The input embeddings are passed through multiple decoder blocks to output some final hidden state. Within each decoder block, GPT-2 uses a *masked self-attention* layer which means GPT-2 can't attend to future tokens. It is only allowed to attend to tokens on the left. This is different from BERT's [`mask`] token because, in masked self-attention, an attention mask is used to set the score to `0` for future tokens.

2. The output from the decoder is passed to a language modeling head, which performs a linear transformation to convert the hidden states into logits. The label is the next token in the sequence, which are created by shifting the logits to the right by one. The cross-entropy loss is calculated between the shifted logits and the labels to output the next most likely token.

GPT-2's pretraining objective is based entirely on [causal language modeling](glossary#causal-language-modeling), predicting the next word in a sequence. This makes GPT-2 especially good at tasks that involve generating text.

Ready to try your hand at text generation? Check out our complete [causal language modeling guide](tasks/language_modeling#causal-language-modeling) to learn how to finetune DistilGPT-2 and use it for inference!

<Tip>

For more information about text generation, check out the [text generation strategies](generation_strategies) guide!

</Tip>

### Summarization

Encoder-decoder models like [BART](model_doc/bart) and [T5](model_doc/t5) are designed for the sequence-to-sequence pattern of a summarization task. We'll explain how BART works in this section, and then you can try finetuning T5 at the end.

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bart_architecture.png"/>
</div>

1. BART's encoder architecture is very similar to BERT and accepts a token and positional embedding of the text. BART is pretrained by corrupting the input and then reconstructing it with the decoder. Unlike other encoders with specific corruption strategies, BART can apply any type of corruption. The *text infilling* corruption strategy works the best though. In text infilling, a number of text spans are replaced with a **single** [`mask`] token. This is important because the model has to predict the masked tokens, and it teaches the model to predict the number of missing tokens. The input embeddings and masked spans are passed through the encoder to output some final hidden states, but unlike BERT, BART doesn't add a final feedforward network at the end to predict a word.

2. The encoder's output is passed to the decoder, which must predict the masked tokens and any uncorrupted tokens from the encoder's output. This gives additional context to help the decoder restore the original text. The output from the decoder is passed to a language modeling head, which performs a linear transformation to convert the hidden states into logits.


key: _node_content
string_value:
{"id_": "ed8c2a3a-e290-4e59-9737-8390157c9d42", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5e9ee699-723a-43ca-977a-24df3760e274", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "f66694f678b8d66eb1c4d91644f37ea780f29f92de783accc8c78be14920e31c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5c277a4e-2661-4b3b-ad67-4195303806d1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks_explained", "title": "How \ud83e\udd17 Transformers solve tasks", "tokens": 6113, "retrieve_doc": true, "source": "transformers"}, "hash": "de7a8c81377954d4432c0523316855f996425cf72883748dfc81c50d6eddfdac", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24940, "end_char_idx": 28313, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<Tip>

For more information about text generation, check out the [text generation strategies](generation_strategies) guide!

</Tip>

### Summarization

Encoder-decoder models like [BART](model_doc/bart) and [T5](model_doc/t5) are designed for the sequence-to-sequence pattern of a summarization task. We'll explain how BART works in this section, and then you can try finetuning T5 at the end.

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bart_architecture.png"/>
</div>

1. BART's encoder architecture is very similar to BERT and accepts a token and positional embedding of the text. BART is pretrained by corrupting the input and then reconstructing it with the decoder. Unlike other encoders with specific corruption strategies, BART can apply any type of corruption. The *text infilling* corruption strategy works the best though. In text infilling, a number of text spans are replaced with a **single** [`mask`] token. This is important because the model has to predict the masked tokens, and it teaches the model to predict the number of missing tokens. The input embeddings and masked spans are passed through the encoder to output some final hidden states, but unlike BERT, BART doesn't add a final feedforward network at the end to predict a word.

2. The encoder's output is passed to the decoder, which must predict the masked tokens and any uncorrupted tokens from the encoder's output. This gives additional context to help the decoder restore the original text. The output from the decoder is passed to a language modeling head, which performs a linear transformation to convert the hidden states into logits. The cross-entropy loss is calculated between the logits and the label, which is just the token shifted to the right.

Ready to try your hand at summarization? Check out our complete [summarization guide](tasks/summarization) to learn how to finetune T5 and use it for inference!

<Tip>

For more information about text generation, check out the [text generation strategies](generation_strategies) guide!

</Tip>

### Translation

Translation is another example of a sequence-to-sequence task, which means you can use an encoder-decoder model like [BART](model_doc/bart) or [T5](model_doc/t5) to do it. We'll explain how BART works in this section, and then you can try finetuning T5 at the end.

BART adapts to translation by adding a separate randomly initialized encoder to map a source language to an input that can be decoded into the target language. This new encoder's embeddings are passed to the pretrained encoder instead of the original word embeddings. The source encoder is trained by updating the source encoder, positional embeddings, and input embeddings with the cross-entropy loss from the model output. The model parameters are frozen in this first step, and all the model parameters are trained together in the second step.

BART has since been followed up by a multilingual version, mBART, intended for translation and pretrained on many different languages.

Ready to try your hand at translation? Check out our complete [translation guide](tasks/translation) to learn how to finetune T5 and use it for inference!

<Tip>

For more information about text generation, check out the [text generation strategies](generation_strategies) guide!

</Tip>


key: _node_content
string_value:
{"id_": "9436df89-f585-4e54-b37c-603e25f6051e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fd878aeb-25d7-4095-8003-41408771ebf2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "ff90393796b58d9cbff193200432df57f36db1694146fa624ccfa7fdd0c6151d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "96739bdb-e222-4827-b8c3-fd15f9bcb8da", "node_type": "1", "metadata": {}, "hash": "7dfe2d62d1bf1f5cbaa0062a21d0dbca219a264cb25832c2578691dc7414f013", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3099, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Benchmarks

<Tip warning={true}>

Hugging Face's Benchmarking tools are deprecated and it is advised to use external Benchmarking libraries to measure the speed 
and memory complexity of Transformer models.

</Tip>

[[open-in-colab]]

Let's take a look at how 🤗 Transformers models can be benchmarked, best practices, and already available benchmarks.

A notebook explaining in more detail how to benchmark 🤗 Transformers models can be found [here](https://github.com/huggingface/notebooks/tree/main/examples/benchmark.ipynb).

## How to benchmark 🤗 Transformers models

The classes [`PyTorchBenchmark`] and [`TensorFlowBenchmark`] allow to flexibly benchmark 🤗 Transformers models. The benchmark classes allow us to measure the _peak memory usage_ and _required time_ for both _inference_ and _training_.

<Tip>

Hereby, _inference_ is defined by a single forward pass, and _training_ is defined by a single forward pass and
backward pass.

</Tip>

The benchmark classes [`PyTorchBenchmark`] and [`TensorFlowBenchmark`] expect an object of type [`PyTorchBenchmarkArguments`] and
[`TensorFlowBenchmarkArguments`], respectively, for instantiation. [`PyTorchBenchmarkArguments`] and [`TensorFlowBenchmarkArguments`] are data classes and contain all relevant configurations for their corresponding benchmark class. In the following example, it is shown how a BERT model of type _bert-base-cased_ can be benchmarked.

<frameworkcontent>
<pt>
```py
>>> from transformers import PyTorchBenchmark, PyTorchBenchmarkArguments

>>> args = PyTorchBenchmarkArguments(models=["google-bert/bert-base-uncased"], batch_sizes=[8], sequence_lengths=[8, 32, 128, 512])
>>> benchmark = PyTorchBenchmark(args)
```
</pt>
<tf>
```py
>>> from transformers import TensorFlowBenchmark, TensorFlowBenchmarkArguments

>>> args = TensorFlowBenchmarkArguments(
...     models=["google-bert/bert-base-uncased"], batch_sizes=[8], sequence_lengths=[8, 32, 128, 512]
... )
>>> benchmark = TensorFlowBenchmark(args)
```
</tf>
</frameworkcontent>

Here, three arguments are given to the benchmark argument data classes, namely `models`, `batch_sizes`, and
`sequence_lengths`. The argument `models` is required and expects a `list` of model identifiers from the
[model hub](https://huggingface.co/models) The `list` arguments `batch_sizes` and `sequence_lengths` define
the size of the `input_ids` on which the model is benchmarked. There are many more parameters that can be configured
via the benchmark argument data classes. For more detail on these one can either directly consult the files
`src/transformers/benchmark/benchmark_args_utils.py`, `src/transformers/benchmark/benchmark_args.py` (for PyTorch)
and `src/transformers/benchmark/benchmark_args_tf.py` (for Tensorflow). Alternatively, running the following shell
commands from root will print out a descriptive list of all configurable parameters for PyTorch and Tensorflow
respectively.

<frameworkcontent>
<pt>
```bash
python examples/pytorch/benchmarking/run_benchmark.py --help
```

An instantiated benchmark object can then simply be run by calling `benchmark.run()`.


key: _node_content
string_value:
{"id_": "96739bdb-e222-4827-b8c3-fd15f9bcb8da", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fd878aeb-25d7-4095-8003-41408771ebf2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "ff90393796b58d9cbff193200432df57f36db1694146fa624ccfa7fdd0c6151d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9436df89-f585-4e54-b37c-603e25f6051e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "fcee66d309064f10554a641b9efa4ab97e12118aa1ae933d7ac51c9b8371fbc9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "67d6d573-e869-4062-a649-aef79a3bb50c", "node_type": "1", "metadata": {}, "hash": "a7f84b58ecd46ab5f5f8e5adf15727fc992d2ea05bab20ff341c668393fdddd1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1416, "end_char_idx": 5282, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<frameworkcontent>
<pt>
```py
>>> from transformers import PyTorchBenchmark, PyTorchBenchmarkArguments

>>> args = PyTorchBenchmarkArguments(models=["google-bert/bert-base-uncased"], batch_sizes=[8], sequence_lengths=[8, 32, 128, 512])
>>> benchmark = PyTorchBenchmark(args)
```
</pt>
<tf>
```py
>>> from transformers import TensorFlowBenchmark, TensorFlowBenchmarkArguments

>>> args = TensorFlowBenchmarkArguments(
...     models=["google-bert/bert-base-uncased"], batch_sizes=[8], sequence_lengths=[8, 32, 128, 512]
... )
>>> benchmark = TensorFlowBenchmark(args)
```
</tf>
</frameworkcontent>

Here, three arguments are given to the benchmark argument data classes, namely `models`, `batch_sizes`, and
`sequence_lengths`. The argument `models` is required and expects a `list` of model identifiers from the
[model hub](https://huggingface.co/models) The `list` arguments `batch_sizes` and `sequence_lengths` define
the size of the `input_ids` on which the model is benchmarked. There are many more parameters that can be configured
via the benchmark argument data classes. For more detail on these one can either directly consult the files
`src/transformers/benchmark/benchmark_args_utils.py`, `src/transformers/benchmark/benchmark_args.py` (for PyTorch)
and `src/transformers/benchmark/benchmark_args_tf.py` (for Tensorflow). Alternatively, running the following shell
commands from root will print out a descriptive list of all configurable parameters for PyTorch and Tensorflow
respectively.

<frameworkcontent>
<pt>
```bash
python examples/pytorch/benchmarking/run_benchmark.py --help
```

An instantiated benchmark object can then simply be run by calling `benchmark.run()`.

```py
>>> results = benchmark.run()
>>> print(results)
====================       INFERENCE - SPEED - RESULT       ====================
--------------------------------------------------------------------------------
Model Name             Batch Size     Seq Length     Time in s                  
--------------------------------------------------------------------------------
google-bert/bert-base-uncased          8               8             0.006     
google-bert/bert-base-uncased          8               32            0.006     
google-bert/bert-base-uncased          8              128            0.018     
google-bert/bert-base-uncased          8              512            0.088     
--------------------------------------------------------------------------------

====================      INFERENCE - MEMORY - RESULT       ====================
--------------------------------------------------------------------------------
Model Name             Batch Size     Seq Length    Memory in MB 
--------------------------------------------------------------------------------
google-bert/bert-base-uncased          8               8             1227
google-bert/bert-base-uncased          8               32            1281
google-bert/bert-base-uncased          8              128            1307
google-bert/bert-base-uncased          8              512            1539
--------------------------------------------------------------------------------

====================        ENVIRONMENT INFORMATION         ====================

- transformers_version: 2.11.0
- framework: PyTorch
- use_torchscript: False
- framework_version: 1.4.0
- python_version: 3.6.10
- system: Linux
- cpu: x86_64
- architecture: 64bit
- date: 2020-06-29
- time: 08:58:43.371351
- fp16: False
- use_multiprocessing: True
- only_pretrain_model: False
- cpu_ram_mb: 32088
- use_gpu: True
- num_gpus: 1
- gpu: TITAN RTX
- gpu_ram_mb: 24217
- gpu_power_watts: 280.0
- gpu_performance_state: 2
- use_tpu: False
```
</pt>
<tf>
```bash
python examples/tensorflow/benchmarking/run_benchmark_tf.py --help
```

An instantiated benchmark object can then simply be run by calling `benchmark.run()`.


key: _node_content
string_value:
{"id_": "67d6d573-e869-4062-a649-aef79a3bb50c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fd878aeb-25d7-4095-8003-41408771ebf2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "ff90393796b58d9cbff193200432df57f36db1694146fa624ccfa7fdd0c6151d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "96739bdb-e222-4827-b8c3-fd15f9bcb8da", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "a5b8e9a0fb30e1eff8ba585336db5883d79a4e33fa2bae87fb22c0ac203af26f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "56589b3a-9d81-4bee-8b6f-9dc8c608671b", "node_type": "1", "metadata": {}, "hash": "50e7bf1014c9b28665782f44b0ba1b2f82e118a08186a211630688127531ed4b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> results = benchmark.run()
>>> print(results)
>>> results = benchmark.run()
>>> print(results)
====================       INFERENCE - SPEED - RESULT       ====================
--------------------------------------------------------------------------------
Model Name             Batch Size     Seq Length     Time in s                  
--------------------------------------------------------------------------------
google-bert/bert-base-uncased          8               8             0.005
google-bert/bert-base-uncased          8               32            0.008
google-bert/bert-base-uncased          8              128            0.022
google-bert/bert-base-uncased          8              512            0.105
--------------------------------------------------------------------------------

====================      INFERENCE - MEMORY - RESULT       ====================
--------------------------------------------------------------------------------
Model Name             Batch Size     Seq Length    Memory in MB 
--------------------------------------------------------------------------------
google-bert/bert-base-uncased          8               8             1330
google-bert/bert-base-uncased          8               32            1330
google-bert/bert-base-uncased          8              128            1330
google-bert/bert-base-uncased          8              512            1770
--------------------------------------------------------------------------------

====================        ENVIRONMENT INFORMATION         ====================

- transformers_version: 2.11.0
- framework: Tensorflow
- use_xla: False
- framework_version: 2.2.0
- python_version: 3.6.10
- system: Linux
- cpu: x86_64
- architecture: 64bit
- date: 2020-06-29
- time: 09:26:35.617317
- fp16: False
- use_multiprocessing: True
- only_pretrain_model: False
- cpu_ram_mb: 32088
- use_gpu: True
- num_gpus: 1
- gpu: TITAN RTX
- gpu_ram_mb: 24217
- gpu_power_watts: 280.0
- gpu_performance_state: 2
- use_tpu: False
```
</tf>
</frameworkcontent>

By default, the _time_ and the _required memory_ for _inference_ are benchmarked. In the example output above the first
two sections show the result corresponding to _inference time_ and _inference memory_. In addition, all relevant
information about the computing environment, _e.g._ the GPU type, the system, the library versions, etc... are printed
out in the third section under _ENVIRONMENT INFORMATION_. This information can optionally be saved in a _.csv_ file
when adding the argument `save_to_csv=True` to [`PyTorchBenchmarkArguments`] and
[`TensorFlowBenchmarkArguments`] respectively. In this case, every section is saved in a separate
_.csv_ file. The path to each _.csv_ file can optionally be defined via the argument data classes.

Instead of benchmarking pre-trained models via their model identifier, _e.g._ `google-bert/bert-base-uncased`, the user can
alternatively benchmark an arbitrary configuration of any available model class. In this case, a `list` of
configurations must be inserted with the benchmark args as follows.

<frameworkcontent>
<pt>
```py
>>> from transformers import PyTorchBenchmark, PyTorchBenchmarkArguments, BertConfig

>>> args = PyTorchBenchmarkArguments(
.     models=["bert-base", "bert-384-hid", "bert-6-lay"], batch_sizes=[8], sequence_lengths=[8, 32, 128, 512]
.


key: _node_content
string_value:
{"id_": "56589b3a-9d81-4bee-8b6f-9dc8c608671b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fd878aeb-25d7-4095-8003-41408771ebf2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "ff90393796b58d9cbff193200432df57f36db1694146fa624ccfa7fdd0c6151d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "67d6d573-e869-4062-a649-aef79a3bb50c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "4dedb082625fd4b50ab55287a8033f88d60e92b66d274d8bbde093c1647fc8dc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fd3c8dd3-16f7-4811-a380-58d6a43d75df", "node_type": "1", "metadata": {}, "hash": "92a2f9a33470f402fb9e992e74997e7ae6eef52dd475b302f007627dfa220c57", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In the example output above the first
two sections show the result corresponding to _inference time_ and _inference memory_. In addition, all relevant
information about the computing environment, _e.g._ the GPU type, the system, the library versions, etc... are printed
out in the third section under _ENVIRONMENT INFORMATION_. This information can optionally be saved in a _.csv_ file
when adding the argument `save_to_csv=True` to [`PyTorchBenchmarkArguments`] and
[`TensorFlowBenchmarkArguments`] respectively. In this case, every section is saved in a separate
_.csv_ file. The path to each _.csv_ file can optionally be defined via the argument data classes.

Instead of benchmarking pre-trained models via their model identifier, _e.g._ `google-bert/bert-base-uncased`, the user can
alternatively benchmark an arbitrary configuration of any available model class. In this case, a `list` of
configurations must be inserted with the benchmark args as follows.

<frameworkcontent>
<pt>
```py
>>> from transformers import PyTorchBenchmark, PyTorchBenchmarkArguments, BertConfig

>>> args = PyTorchBenchmarkArguments(
.     models=["bert-base", "bert-384-hid", "bert-6-lay"], batch_sizes=[8], sequence_lengths=[8, 32, 128, 512]
. )
>>> config_base = BertConfig()
>>> config_384_hid = BertConfig(hidden_size=384)
>>> config_6_lay = BertConfig(num_hidden_layers=6)

>>> benchmark = PyTorchBenchmark(args, configs=[config_base, config_384_hid, config_6_lay])
>>> benchmark.run()
====================       INFERENCE - SPEED - RESULT       ====================
--------------------------------------------------------------------------------
Model Name             Batch Size     Seq Length       Time in s                  
--------------------------------------------------------------------------------
bert-base                  8              128            0.006
bert-base                  8              512            0.006
bert-base                  8              128            0.018     
bert-base                  8              512            0.088     
bert-384-hid              8               8             0.006     
bert-384-hid              8               32            0.006     
bert-384-hid              8              128            0.011     
bert-384-hid              8              512            0.054     
bert-6-lay                 8               8             0.003     
bert-6-lay                 8               32            0.004     
bert-6-lay                 8              128            0.009     
bert-6-lay                 8              512            0.


key: _node_content
string_value:
{"id_": "fd3c8dd3-16f7-4811-a380-58d6a43d75df", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fd878aeb-25d7-4095-8003-41408771ebf2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "ff90393796b58d9cbff193200432df57f36db1694146fa624ccfa7fdd0c6151d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "56589b3a-9d81-4bee-8b6f-9dc8c608671b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "7b8f90d74eddfc0db6ac55af0e2184ec0863f2fff5e3f5a21f41497d51b18050", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3cc7abce-468b-4968-a117-39b3cf2b26d2", "node_type": "1", "metadata": {}, "hash": "15b19bb6ed7dd19918845968f493516239f7a4eb4744b614bb9d8b877e1eb44b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In this case, a `list` of
configurations must be inserted with the benchmark args as follows.

<frameworkcontent>
<pt>
```py
>>> from transformers import PyTorchBenchmark, PyTorchBenchmarkArguments, BertConfig

>>> args = PyTorchBenchmarkArguments(
.     models=["bert-base", "bert-384-hid", "bert-6-lay"], batch_sizes=[8], sequence_lengths=[8, 32, 128, 512]
. )
>>> config_base = BertConfig()
>>> config_384_hid = BertConfig(hidden_size=384)
>>> config_6_lay = BertConfig(num_hidden_layers=6)

>>> benchmark = PyTorchBenchmark(args, configs=[config_base, config_384_hid, config_6_lay])
>>> benchmark.run()
====================       INFERENCE - SPEED - RESULT       ====================
--------------------------------------------------------------------------------
Model Name             Batch Size     Seq Length       Time in s                  
--------------------------------------------------------------------------------
bert-base                  8              128            0.006
bert-base                  8              512            0.006
bert-base                  8              128            0.018     
bert-base                  8              512            0.088     
bert-384-hid              8               8             0.006     
bert-384-hid              8               32            0.006     
bert-384-hid              8              128            0.011     
bert-384-hid              8              512            0.054     
bert-6-lay                 8               8             0.003     
bert-6-lay                 8               32            0.004     
bert-6-lay                 8              128            0.009     
bert-6-lay                 8              512            0.044
--------------------------------------------------------------------------------

====================      INFERENCE - MEMORY - RESULT       ====================
--------------------------------------------------------------------------------
Model Name             Batch Size     Seq Length      Memory in MB 
--------------------------------------------------------------------------------
bert-base                  8               8             1277
bert-base                  8               32            1281
bert-base                  8              128            1307     
bert-base                  8              512            1539     
bert-384-hid              8               8             1005     
bert-384-hid              8               32            1027     
bert-384-hid              8              128            1035     
bert-384-hid              8              512            1255     
bert-6-lay                 8               8             1097     
bert-6-lay                 8               32            1101     
bert-6-lay                 8              128            1127     
bert-6-lay                 8              512            1359
--------------------------------------------------------------------------------

====================        ENVIRONMENT INFORMATION         ====================

- transformers_version: 2.11.0
- framework: PyTorch
- use_torchscript: False
- framework_version: 1.4.0
- python_version: 3.6.10
- system: Linux
- cpu: x86_64
- architecture: 64bit
- date: 2020-06-29
- time: 09:35:25.143267
- fp16: False
- use_multiprocessing: True
- only_pretrain_model: False
- cpu_ram_mb: 32088
- use_gpu: True
- num_gpus: 1
- gpu: TITAN RTX
- gpu_ram_mb: 24217
- gpu_power_watts: 280.


key: _node_content
string_value:
{"id_": "3cc7abce-468b-4968-a117-39b3cf2b26d2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fd878aeb-25d7-4095-8003-41408771ebf2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "ff90393796b58d9cbff193200432df57f36db1694146fa624ccfa7fdd0c6151d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fd3c8dd3-16f7-4811-a380-58d6a43d75df", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "72aa414e3a1a99d4c3960e7e3f84f8716596085f5b164799fe14384fbdf41e13", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0cd55ee4-31e7-4476-a60b-0ae1e4786940", "node_type": "1", "metadata": {}, "hash": "eb4562bc45c52a0b70931585ea11d6b981a5ce35661844a90f100390d725900d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
044
--------------------------------------------------------------------------------

====================      INFERENCE - MEMORY - RESULT       ====================
--------------------------------------------------------------------------------
Model Name             Batch Size     Seq Length      Memory in MB 
--------------------------------------------------------------------------------
bert-base                  8               8             1277
bert-base                  8               32            1281
bert-base                  8              128            1307     
bert-base                  8              512            1539     
bert-384-hid              8               8             1005     
bert-384-hid              8               32            1027     
bert-384-hid              8              128            1035     
bert-384-hid              8              512            1255     
bert-6-lay                 8               8             1097     
bert-6-lay                 8               32            1101     
bert-6-lay                 8              128            1127     
bert-6-lay                 8              512            1359
--------------------------------------------------------------------------------

====================        ENVIRONMENT INFORMATION         ====================

- transformers_version: 2.11.0
- framework: PyTorch
- use_torchscript: False
- framework_version: 1.4.0
- python_version: 3.6.10
- system: Linux
- cpu: x86_64
- architecture: 64bit
- date: 2020-06-29
- time: 09:35:25.143267
- fp16: False
- use_multiprocessing: True
- only_pretrain_model: False
- cpu_ram_mb: 32088
- use_gpu: True
- num_gpus: 1
- gpu: TITAN RTX
- gpu_ram_mb: 24217
- gpu_power_watts: 280.0
- gpu_performance_state: 2
- use_tpu: False
```
</pt>
<tf>
```py
>>> from transformers import TensorFlowBenchmark, TensorFlowBenchmarkArguments, BertConfig

>>> args = TensorFlowBenchmarkArguments(
.     models=["bert-base", "bert-384-hid", "bert-6-lay"], batch_sizes=[8], sequence_lengths=[8, 32, 128, 512]
. )
>>> config_base = BertConfig()
>>> config_384_hid = BertConfig(hidden_size=384)
>>> config_6_lay = BertConfig(num_hidden_layers=6)

>>> benchmark = TensorFlowBenchmark(args, configs=[config_base, config_384_hid, config_6_lay])
>>> benchmark.run()
====================       INFERENCE - SPEED - RESULT       ====================
--------------------------------------------------------------------------------
Model Name             Batch Size     Seq Length       Time in s                  
--------------------------------------------------------------------------------
bert-base                  8               8             0.005
bert-base                  8               32            0.008
bert-base                  8              128            0.022
bert-base                  8              512            0.106
bert-384-hid              8               8             0.005
bert-384-hid              8               32            0.007
bert-384-hid              8              128            0.018
bert-384-hid              8              512            0.064
bert-6-lay                 8               8             0.002
bert-6-lay                 8               32            0.003
bert-6-lay                 8              128            0.0011
bert-6-lay                 8              512            0.


key: _node_content
string_value:
{"id_": "0cd55ee4-31e7-4476-a60b-0ae1e4786940", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fd878aeb-25d7-4095-8003-41408771ebf2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "ff90393796b58d9cbff193200432df57f36db1694146fa624ccfa7fdd0c6151d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3cc7abce-468b-4968-a117-39b3cf2b26d2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "aec346552306288c37c91f7cee42f9b917503434291723658c0c59a19ff621d3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5aa12e8b-399e-4b46-84e4-07b1a1c836ac", "node_type": "1", "metadata": {}, "hash": "f88836cfe2e4abead9edf96fd4b9bc49361b26eb8347c279653188ad81e7f047", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0
- gpu_performance_state: 2
- use_tpu: False
```
</pt>
<tf>
```py
>>> from transformers import TensorFlowBenchmark, TensorFlowBenchmarkArguments, BertConfig

>>> args = TensorFlowBenchmarkArguments(
.     models=["bert-base", "bert-384-hid", "bert-6-lay"], batch_sizes=[8], sequence_lengths=[8, 32, 128, 512]
. )
>>> config_base = BertConfig()
>>> config_384_hid = BertConfig(hidden_size=384)
>>> config_6_lay = BertConfig(num_hidden_layers=6)

>>> benchmark = TensorFlowBenchmark(args, configs=[config_base, config_384_hid, config_6_lay])
>>> benchmark.run()
====================       INFERENCE - SPEED - RESULT       ====================
--------------------------------------------------------------------------------
Model Name             Batch Size     Seq Length       Time in s                  
--------------------------------------------------------------------------------
bert-base                  8               8             0.005
bert-base                  8               32            0.008
bert-base                  8              128            0.022
bert-base                  8              512            0.106
bert-384-hid              8               8             0.005
bert-384-hid              8               32            0.007
bert-384-hid              8              128            0.018
bert-384-hid              8              512            0.064
bert-6-lay                 8               8             0.002
bert-6-lay                 8               32            0.003
bert-6-lay                 8              128            0.0011
bert-6-lay                 8              512            0.074
--------------------------------------------------------------------------------

====================      INFERENCE - MEMORY - RESULT       ====================
--------------------------------------------------------------------------------
Model Name             Batch Size     Seq Length      Memory in MB 
--------------------------------------------------------------------------------
bert-base                  8               8             1330
bert-base                  8               32            1330
bert-base                  8              128            1330
bert-base                  8              512            1770
bert-384-hid              8               8             1330
bert-384-hid              8               32            1330
bert-384-hid              8              128            1330
bert-384-hid              8              512            1540
bert-6-lay                 8               8             1330
bert-6-lay                 8               32            1330
bert-6-lay                 8              128            1330
bert-6-lay                 8              512            1540
--------------------------------------------------------------------------------

====================        ENVIRONMENT INFORMATION         ====================

- transformers_version: 2.11.0
- framework: Tensorflow
- use_xla: False
- framework_version: 2.2.0
- python_version: 3.6.10
- system: Linux
- cpu: x86_64
- architecture: 64bit
- date: 2020-06-29
- time: 09:38:15.487125
- fp16: False
- use_multiprocessing: True
- only_pretrain_model: False
- cpu_ram_mb: 32088
- use_gpu: True
- num_gpus: 1
- gpu: TITAN RTX
- gpu_ram_mb: 24217
- gpu_power_watts: 280.


key: _node_content
string_value:
{"id_": "5aa12e8b-399e-4b46-84e4-07b1a1c836ac", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fd878aeb-25d7-4095-8003-41408771ebf2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "ff90393796b58d9cbff193200432df57f36db1694146fa624ccfa7fdd0c6151d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0cd55ee4-31e7-4476-a60b-0ae1e4786940", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "0f59a973aaee1e18889695adc0561829ccc857c536bfe695b889a3c64d49638a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4df70b1a-1131-40e0-b2e1-92bc83bbd007", "node_type": "1", "metadata": {}, "hash": "562b50cefa883492c05b5f81591d7350d4fa496704447f7476278a7fb042e90f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13413, "end_char_idx": 16397, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
074
--------------------------------------------------------------------------------

====================      INFERENCE - MEMORY - RESULT       ====================
--------------------------------------------------------------------------------
Model Name             Batch Size     Seq Length      Memory in MB 
--------------------------------------------------------------------------------
bert-base                  8               8             1330
bert-base                  8               32            1330
bert-base                  8              128            1330
bert-base                  8              512            1770
bert-384-hid              8               8             1330
bert-384-hid              8               32            1330
bert-384-hid              8              128            1330
bert-384-hid              8              512            1540
bert-6-lay                 8               8             1330
bert-6-lay                 8               32            1330
bert-6-lay                 8              128            1330
bert-6-lay                 8              512            1540
--------------------------------------------------------------------------------

====================        ENVIRONMENT INFORMATION         ====================

- transformers_version: 2.11.0
- framework: Tensorflow
- use_xla: False
- framework_version: 2.2.0
- python_version: 3.6.10
- system: Linux
- cpu: x86_64
- architecture: 64bit
- date: 2020-06-29
- time: 09:38:15.487125
- fp16: False
- use_multiprocessing: True
- only_pretrain_model: False
- cpu_ram_mb: 32088
- use_gpu: True
- num_gpus: 1
- gpu: TITAN RTX
- gpu_ram_mb: 24217
- gpu_power_watts: 280.0
- gpu_performance_state: 2
- use_tpu: False
```
</tf>
</frameworkcontent>

Again, _inference time_ and _required memory_ for _inference_ are measured, but this time for customized configurations
of the `BertModel` class. This feature can especially be helpful when deciding for which configuration the model
should be trained.


## Benchmark best practices

This section lists a couple of best practices one should be aware of when benchmarking a model.

- Currently, only single device benchmarking is supported. When benchmarking on GPU, it is recommended that the user
  specifies on which device the code should be run by setting the `CUDA_VISIBLE_DEVICES` environment variable in the
  shell, _e.g._ `export CUDA_VISIBLE_DEVICES=0` before running the code.
- The option `no_multi_processing` should only be set to `True` for testing and debugging. To ensure accurate
  memory measurement it is recommended to run each memory benchmark in a separate process by making sure
  `no_multi_processing` is set to `True`.
- One should always state the environment information when sharing the results of a model benchmark. Results can vary
  heavily between different GPU devices, library versions, etc., so that benchmark results on their own are not very
  useful for the community.


key: _node_content
string_value:
{"id_": "4df70b1a-1131-40e0-b2e1-92bc83bbd007", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fd878aeb-25d7-4095-8003-41408771ebf2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "ff90393796b58d9cbff193200432df57f36db1694146fa624ccfa7fdd0c6151d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5aa12e8b-399e-4b46-84e4-07b1a1c836ac", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/benchmarks", "title": "Benchmarks", "tokens": 4058, "retrieve_doc": true, "source": "transformers"}, "hash": "b835e94333efbfd7b9239ec9ebc141d4a537ed1e533d702dcd9fae56f8b3a8d5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14833, "end_char_idx": 17380, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
6.10
- system: Linux
- cpu: x86_64
- architecture: 64bit
- date: 2020-06-29
- time: 09:38:15.487125
- fp16: False
- use_multiprocessing: True
- only_pretrain_model: False
- cpu_ram_mb: 32088
- use_gpu: True
- num_gpus: 1
- gpu: TITAN RTX
- gpu_ram_mb: 24217
- gpu_power_watts: 280.0
- gpu_performance_state: 2
- use_tpu: False
```
</tf>
</frameworkcontent>

Again, _inference time_ and _required memory_ for _inference_ are measured, but this time for customized configurations
of the `BertModel` class. This feature can especially be helpful when deciding for which configuration the model
should be trained.


## Benchmark best practices

This section lists a couple of best practices one should be aware of when benchmarking a model.

- Currently, only single device benchmarking is supported. When benchmarking on GPU, it is recommended that the user
  specifies on which device the code should be run by setting the `CUDA_VISIBLE_DEVICES` environment variable in the
  shell, _e.g._ `export CUDA_VISIBLE_DEVICES=0` before running the code.
- The option `no_multi_processing` should only be set to `True` for testing and debugging. To ensure accurate
  memory measurement it is recommended to run each memory benchmark in a separate process by making sure
  `no_multi_processing` is set to `True`.
- One should always state the environment information when sharing the results of a model benchmark. Results can vary
  heavily between different GPU devices, library versions, etc., so that benchmark results on their own are not very
  useful for the community.


## Sharing your benchmark

Previously all available core models (10 at the time) have been benchmarked for _inference time_, across many different
settings: using PyTorch, with and without TorchScript, using TensorFlow, with and without XLA. All of those tests were
done across CPUs (except for TensorFlow XLA) and GPUs.

The approach is detailed in the [following blogpost](https://medium.com/huggingface/benchmarking-transformers-pytorch-and-tensorflow-e2917fb891c2) and the results are
available [here](https://docs.google.com/spreadsheets/d/1sryqufw2D0XlUH4sq3e9Wnxu5EAQkaohzrJbd5HdQ_w/edit?usp=sharing).

With the new _benchmark_ tools, it is easier than ever to share your benchmark results with the community

- [PyTorch Benchmarking Results](https://github.com/huggingface/transformers/tree/main/examples/pytorch/benchmarking/README.md).
- [TensorFlow Benchmarking Results](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/benchmarking/README.md).


key: _node_content
string_value:
{"id_": "a684bd9d-1f4d-4727-a505-d5af54a45bf9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e9f71c25-71f0-4b18-be11-c33b6fcecac6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "d7c025cad61c09f008bde35b4154c5dc0fe13b268c8411677421925cfff7b247", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0f751fe3-e311-4700-b302-51360618e12a", "node_type": "1", "metadata": {}, "hash": "bbcddc2816a45c1f29c7706411bccc322c54196b006e68c857e171482491db3a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3667, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Text generation strategies

Text generation is essential to many NLP tasks, such as open-ended text generation, summarization, translation, and
more. It also plays a role in a variety of mixed-modality applications that have text as an output like speech-to-text
and vision-to-text. Some of the models that can generate text include
GPT2, XLNet, OpenAI GPT, CTRL, TransformerXL, XLM, Bart, T5, GIT, Whisper.

Check out a few examples that use [`~generation.GenerationMixin.generate`] method to produce
text outputs for different tasks:
* [Text summarization](./tasks/summarization#inference)
* [Image captioning](./model_doc/git#transformers.GitForCausalLM.forward.example)
* [Audio transcription](./model_doc/whisper#transformers.WhisperForConditionalGeneration.forward.example)

Note that the inputs to the generate method depend on the model's modality. They are returned by the model's preprocessor
class, such as AutoTokenizer or AutoProcessor. If a model's preprocessor creates more than one kind of input, pass all
the inputs to generate(). You can learn more about the individual model's preprocessor in the corresponding model's documentation.

The process of selecting output tokens to generate text is known as decoding, and you can customize the decoding strategy
that the `generate()` method will use. Modifying a decoding strategy does not change the values of any trainable parameters.
However, it can have a noticeable impact on the quality of the generated output. It can help reduce repetition in the text
and make it more coherent.

This guide describes:
* default generation configuration
* common decoding strategies and their main parameters
* saving and sharing custom generation configurations with your fine-tuned model on 🤗 Hub

## Default text generation configuration

A decoding strategy for a model is defined in its generation configuration. When using pre-trained models for inference
within a [`pipeline`], the models call the `PreTrainedModel.generate()` method that applies a default generation
configuration under the hood. The default configuration is also used when no custom configuration has been saved with
the model.

When you load a model explicitly, you can inspect the generation configuration that comes with it through
 `model.generation_config`:

```python
>>> from transformers import AutoModelForCausalLM

>>> model = AutoModelForCausalLM.from_pretrained("distilbert/distilgpt2")
>>> model.generation_config
GenerationConfig {
  "bos_token_id": 50256,
  "eos_token_id": 50256
}
<BLANKLINE>
```

Printing out the `model.generation_config` reveals only the values that are different from the default generation
configuration, and does not list any of the default values.

The default generation configuration limits the size of the output combined with the input prompt to a maximum of 20
tokens to avoid running into resource limitations. The default decoding strategy is greedy search, which is the simplest decoding strategy that picks a token with the highest probability as the next token. For many tasks
and small output sizes this works well. However, when used to generate longer outputs, greedy search can start
producing highly repetitive results.

## Customize text generation

You can override any `generation_config` by passing the parameters and their values directly to the [`generate`] method:

```python
>>> my_model.generate(**inputs, num_beams=4, do_sample=True)  # doctest: +SKIP
```

Even if the default decoding strategy mostly works for your task, you can still tweak a few things. Some of the
commonly adjusted parameters include:

- `max_new_tokens`: the maximum number of tokens to generate.


key: _node_content
string_value:
{"id_": "0f751fe3-e311-4700-b302-51360618e12a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e9f71c25-71f0-4b18-be11-c33b6fcecac6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "d7c025cad61c09f008bde35b4154c5dc0fe13b268c8411677421925cfff7b247", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a684bd9d-1f4d-4727-a505-d5af54a45bf9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "1886a425678e45df5aca23ab3e88583d12d38a690b652fc024d06fd24a7ae29b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "152a21c2-1c12-48ee-bff1-5b8ae291d503", "node_type": "1", "metadata": {}, "hash": "589d2b75895204c7e3738223b6eac7e08c7ecce7124b3ca22b6d360e5ff745d8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1875, "end_char_idx": 5159, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
When using pre-trained models for inference
within a [`pipeline`], the models call the `PreTrainedModel.generate()` method that applies a default generation
configuration under the hood. The default configuration is also used when no custom configuration has been saved with
the model.

When you load a model explicitly, you can inspect the generation configuration that comes with it through
 `model.generation_config`:

```python
>>> from transformers import AutoModelForCausalLM

>>> model = AutoModelForCausalLM.from_pretrained("distilbert/distilgpt2")
>>> model.generation_config
GenerationConfig {
  "bos_token_id": 50256,
  "eos_token_id": 50256
}
<BLANKLINE>
```

Printing out the `model.generation_config` reveals only the values that are different from the default generation
configuration, and does not list any of the default values.

The default generation configuration limits the size of the output combined with the input prompt to a maximum of 20
tokens to avoid running into resource limitations. The default decoding strategy is greedy search, which is the simplest decoding strategy that picks a token with the highest probability as the next token. For many tasks
and small output sizes this works well. However, when used to generate longer outputs, greedy search can start
producing highly repetitive results.

## Customize text generation

You can override any `generation_config` by passing the parameters and their values directly to the [`generate`] method:

```python
>>> my_model.generate(**inputs, num_beams=4, do_sample=True)  # doctest: +SKIP
```

Even if the default decoding strategy mostly works for your task, you can still tweak a few things. Some of the
commonly adjusted parameters include:

- `max_new_tokens`: the maximum number of tokens to generate. In other words, the size of the output sequence, not
including the tokens in the prompt. As an alternative to using the output's length as a stopping criteria, you can choose
to stop generation whenever the full generation exceeds some amount of time. To learn more, check [`StoppingCriteria`].
- `num_beams`: by specifying a number of beams higher than 1, you are effectively switching from greedy search to
beam search. This strategy evaluates several hypotheses at each time step and eventually chooses the hypothesis that
has the overall highest probability for the entire sequence. This has the advantage of identifying high-probability
sequences that start with a lower probability initial tokens and would've been ignored by the greedy search. Visualize how it works [here](https://huggingface.co/spaces/m-ric/beam_search_visualizer).
- `do_sample`: if set to `True`, this parameter enables decoding strategies such as multinomial sampling, beam-search
multinomial sampling, Top-K sampling and Top-p sampling. All these strategies select the next token from the probability
distribution over the entire vocabulary with various strategy-specific adjustments.
- `num_return_sequences`: the number of sequence candidates to return for each input. This option is only available for
the decoding strategies that support multiple sequence candidates, e.g. variations of beam search and sampling. Decoding
strategies like greedy search and contrastive search return a single output sequence.


key: _node_content
string_value:
{"id_": "152a21c2-1c12-48ee-bff1-5b8ae291d503", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e9f71c25-71f0-4b18-be11-c33b6fcecac6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "d7c025cad61c09f008bde35b4154c5dc0fe13b268c8411677421925cfff7b247", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0f751fe3-e311-4700-b302-51360618e12a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "9f956f7d531d29e2762dc944fad31578777ba6d8cfbc12274ab8af66ebe9d832", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4aac5c56-bc86-4248-bc94-cefd3bb54efe", "node_type": "1", "metadata": {}, "hash": "2a359e86854b2eddc67707827f707b4e8b472aebcf268e6b1fc0aa8b1e807467", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3555, "end_char_idx": 6529, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Some of the
commonly adjusted parameters include:

- `max_new_tokens`: the maximum number of tokens to generate. In other words, the size of the output sequence, not
including the tokens in the prompt. As an alternative to using the output's length as a stopping criteria, you can choose
to stop generation whenever the full generation exceeds some amount of time. To learn more, check [`StoppingCriteria`].
- `num_beams`: by specifying a number of beams higher than 1, you are effectively switching from greedy search to
beam search. This strategy evaluates several hypotheses at each time step and eventually chooses the hypothesis that
has the overall highest probability for the entire sequence. This has the advantage of identifying high-probability
sequences that start with a lower probability initial tokens and would've been ignored by the greedy search. Visualize how it works [here](https://huggingface.co/spaces/m-ric/beam_search_visualizer).
- `do_sample`: if set to `True`, this parameter enables decoding strategies such as multinomial sampling, beam-search
multinomial sampling, Top-K sampling and Top-p sampling. All these strategies select the next token from the probability
distribution over the entire vocabulary with various strategy-specific adjustments.
- `num_return_sequences`: the number of sequence candidates to return for each input. This option is only available for
the decoding strategies that support multiple sequence candidates, e.g. variations of beam search and sampling. Decoding
strategies like greedy search and contrastive search return a single output sequence.

## Save a custom decoding strategy with your model

If you would like to share your fine-tuned model with a specific generation configuration, you can:
* Create a [`GenerationConfig`] class instance
* Specify the decoding strategy parameters
* Save your generation configuration with [`GenerationConfig.save_pretrained`], making sure to leave its `config_file_name` argument empty
* Set `push_to_hub` to `True` to upload your config to the model's repo

```python
>>> from transformers import AutoModelForCausalLM, GenerationConfig

>>> model = AutoModelForCausalLM.from_pretrained("my_account/my_model")  # doctest: +SKIP
>>> generation_config = GenerationConfig(
...     max_new_tokens=50, do_sample=True, top_k=50, eos_token_id=model.config.eos_token_id
... )
>>> generation_config.save_pretrained("my_account/my_model", push_to_hub=True)  # doctest: +SKIP
```

You can also store several generation configurations in a single directory, making use of the `config_file_name`
argument in [`GenerationConfig.save_pretrained`]. You can later instantiate them with [`GenerationConfig.from_pretrained`]. This is useful if you want to
store several generation configurations for a single model (e.g. one for creative text generation with sampling, and
one for summarization with beam search). You must have the right Hub permissions to add configuration files to a model.


key: _node_content
string_value:
{"id_": "4aac5c56-bc86-4248-bc94-cefd3bb54efe", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e9f71c25-71f0-4b18-be11-c33b6fcecac6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "d7c025cad61c09f008bde35b4154c5dc0fe13b268c8411677421925cfff7b247", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "152a21c2-1c12-48ee-bff1-5b8ae291d503", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "1222dff302661c076d383787fa786e18116089257883d1c344291a556a2139dc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "36010df9-d214-4b7e-afad-53fca156618a", "node_type": "1", "metadata": {}, "hash": "4ec15013714f2607c0735fb923ad0b848ace2d3d13efb92ca8865d83933aeb8d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4685, "end_char_idx": 8227, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
All these strategies select the next token from the probability
distribution over the entire vocabulary with various strategy-specific adjustments.
- `num_return_sequences`: the number of sequence candidates to return for each input. This option is only available for
the decoding strategies that support multiple sequence candidates, e.g. variations of beam search and sampling. Decoding
strategies like greedy search and contrastive search return a single output sequence.

## Save a custom decoding strategy with your model

If you would like to share your fine-tuned model with a specific generation configuration, you can:
* Create a [`GenerationConfig`] class instance
* Specify the decoding strategy parameters
* Save your generation configuration with [`GenerationConfig.save_pretrained`], making sure to leave its `config_file_name` argument empty
* Set `push_to_hub` to `True` to upload your config to the model's repo

```python
>>> from transformers import AutoModelForCausalLM, GenerationConfig

>>> model = AutoModelForCausalLM.from_pretrained("my_account/my_model")  # doctest: +SKIP
>>> generation_config = GenerationConfig(
...     max_new_tokens=50, do_sample=True, top_k=50, eos_token_id=model.config.eos_token_id
... )
>>> generation_config.save_pretrained("my_account/my_model", push_to_hub=True)  # doctest: +SKIP
```

You can also store several generation configurations in a single directory, making use of the `config_file_name`
argument in [`GenerationConfig.save_pretrained`]. You can later instantiate them with [`GenerationConfig.from_pretrained`]. This is useful if you want to
store several generation configurations for a single model (e.g. one for creative text generation with sampling, and
one for summarization with beam search). You must have the right Hub permissions to add configuration files to a model.

```python
>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, GenerationConfig

>>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("google-t5/t5-small")

>>> translation_generation_config = GenerationConfig(
...     num_beams=4,
...     early_stopping=True,
...     decoder_start_token_id=0,
...     eos_token_id=model.config.eos_token_id,
...     pad_token=model.config.pad_token_id,
... )

>>> # Tip: add `push_to_hub=True` to push to the Hub
>>> translation_generation_config.save_pretrained("/tmp", "translation_generation_config.json")

>>> # You could then use the named generation config file to parameterize generation
>>> generation_config = GenerationConfig.from_pretrained("/tmp", "translation_generation_config.json")
>>> inputs = tokenizer("translate English to French: Configuration files are easy to use!", return_tensors="pt")
>>> outputs = model.generate(**inputs, generation_config=generation_config)
>>> print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
['Les fichiers de configuration sont faciles à utiliser!']
```

## Streaming

The `generate()` supports streaming, through its `streamer` input. The `streamer` input is compatible with any instance
from a class that has the following methods: `put()` and `end()`. Internally, `put()` is used to push new tokens and
`end()` is used to flag the end of text generation.

<Tip warning={true}>

The API for the streamer classes is still under development and may change in the future.

</Tip>

In practice, you can craft your own streaming class for all sorts of purposes! We also have basic streaming classes
ready for you to use.


key: _node_content
string_value:
{"id_": "36010df9-d214-4b7e-afad-53fca156618a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e9f71c25-71f0-4b18-be11-c33b6fcecac6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "d7c025cad61c09f008bde35b4154c5dc0fe13b268c8411677421925cfff7b247", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4aac5c56-bc86-4248-bc94-cefd3bb54efe", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "6143469a8d2448e726800260df7e711d392a6f5094eb651631bd028fe6513992", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ea0c1d54-ca93-45ff-b3c5-0b0bb6201af7", "node_type": "1", "metadata": {}, "hash": "92491f2a106280683918e0fee75619dd93472bdaf319dc246569e4f6b745d2af", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6451, "end_char_idx": 8957, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You must have the right Hub permissions to add configuration files to a model.

```python
>>> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, GenerationConfig

>>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("google-t5/t5-small")

>>> translation_generation_config = GenerationConfig(
...     num_beams=4,
...     early_stopping=True,
...     decoder_start_token_id=0,
...     eos_token_id=model.config.eos_token_id,
...     pad_token=model.config.pad_token_id,
... )

>>> # Tip: add `push_to_hub=True` to push to the Hub
>>> translation_generation_config.save_pretrained("/tmp", "translation_generation_config.json")

>>> # You could then use the named generation config file to parameterize generation
>>> generation_config = GenerationConfig.from_pretrained("/tmp", "translation_generation_config.json")
>>> inputs = tokenizer("translate English to French: Configuration files are easy to use!", return_tensors="pt")
>>> outputs = model.generate(**inputs, generation_config=generation_config)
>>> print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
['Les fichiers de configuration sont faciles à utiliser!']
```

## Streaming

The `generate()` supports streaming, through its `streamer` input. The `streamer` input is compatible with any instance
from a class that has the following methods: `put()` and `end()`. Internally, `put()` is used to push new tokens and
`end()` is used to flag the end of text generation.

<Tip warning={true}>

The API for the streamer classes is still under development and may change in the future.

</Tip>

In practice, you can craft your own streaming class for all sorts of purposes! We also have basic streaming classes
ready for you to use. For example, you can use the [`TextStreamer`] class to stream the output of `generate()` into
your screen, one word at a time:

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer

>>> tok = AutoTokenizer.from_pretrained("openai-community/gpt2")
>>> model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2")
>>> inputs = tok(["An increasing sequence: one,"], return_tensors="pt")
>>> streamer = TextStreamer(tok)

>>> # Despite returning the usual output, the streamer will also print the generated text to stdout.
>>> _ = model.generate(**inputs, streamer=streamer, max_new_tokens=20)
An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven,
```


key: _node_content
string_value:
{"id_": "ea0c1d54-ca93-45ff-b3c5-0b0bb6201af7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e9f71c25-71f0-4b18-be11-c33b6fcecac6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "d7c025cad61c09f008bde35b4154c5dc0fe13b268c8411677421925cfff7b247", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "36010df9-d214-4b7e-afad-53fca156618a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "e1994a494fd36d1e3985199e500765c2bd611c1763e15c89cc8e90c7799f5ddb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "834a0f22-b00f-4367-97bf-b8735635bf4d", "node_type": "1", "metadata": {}, "hash": "462a8a57a2972a99d7d53232891503a1fa7270721ac4c29e50c0ec1c0d8a19b3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7432, "end_char_idx": 10054, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
", return_tensors="pt")
>>> outputs = model.generate(**inputs, generation_config=generation_config)
>>> print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
['Les fichiers de configuration sont faciles à utiliser!']
```

## Streaming

The `generate()` supports streaming, through its `streamer` input. The `streamer` input is compatible with any instance
from a class that has the following methods: `put()` and `end()`. Internally, `put()` is used to push new tokens and
`end()` is used to flag the end of text generation.

<Tip warning={true}>

The API for the streamer classes is still under development and may change in the future.

</Tip>

In practice, you can craft your own streaming class for all sorts of purposes! We also have basic streaming classes
ready for you to use. For example, you can use the [`TextStreamer`] class to stream the output of `generate()` into
your screen, one word at a time:

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer

>>> tok = AutoTokenizer.from_pretrained("openai-community/gpt2")
>>> model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2")
>>> inputs = tok(["An increasing sequence: one,"], return_tensors="pt")
>>> streamer = TextStreamer(tok)

>>> # Despite returning the usual output, the streamer will also print the generated text to stdout.
>>> _ = model.generate(**inputs, streamer=streamer, max_new_tokens=20)
An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven,
```


## Watermarking

The `generate()` supports watermarking the generated text by randomly marking a portion of tokens as "green".
When generating the "green" will have a small 'bias' value added to their logits, thus having a higher chance to be generated.
The watermarked text can be detected by calculating the proportion of "green" tokens in the text and estimating how likely it is
statistically to obtain that amount of "green" tokens for human-generated text. This watermarking strategy was proposed in the paper
["On the Reliability of Watermarks for Large Language Models"](https://arxiv.org/abs/2306.04634). For more information on
the inner functioning of watermarking, it is recommended to refer to the paper.

The watermarking can be used with any generative model in `tranformers` and does not require an extra classification model
to detect watermarked text. To trigger watermarking, pass in a [`WatermarkingConfig`] with needed arguments directly to the
`.generate()` method or add it to the [`GenerationConfig`]. Watermarked text can be later detected with a [`WatermarkDetector`].


key: _node_content
string_value:
{"id_": "834a0f22-b00f-4367-97bf-b8735635bf4d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e9f71c25-71f0-4b18-be11-c33b6fcecac6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "d7c025cad61c09f008bde35b4154c5dc0fe13b268c8411677421925cfff7b247", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ea0c1d54-ca93-45ff-b3c5-0b0bb6201af7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "38e09346f89313c0322b26636d55ec18bf71b428a42fc5fe22b058fa78ad12c2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2d883b63-128d-472e-b760-8fc82bd46e76", "node_type": "1", "metadata": {}, "hash": "c640f14398609ff3b93ffd30af14b8034baf2751233afb4bffb1765d7bf4a02e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8792, "end_char_idx": 12254, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> _ = model.generate(**inputs, streamer=streamer, max_new_tokens=20)
An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven,
```


## Watermarking

The `generate()` supports watermarking the generated text by randomly marking a portion of tokens as "green".
When generating the "green" will have a small 'bias' value added to their logits, thus having a higher chance to be generated.
The watermarked text can be detected by calculating the proportion of "green" tokens in the text and estimating how likely it is
statistically to obtain that amount of "green" tokens for human-generated text. This watermarking strategy was proposed in the paper
["On the Reliability of Watermarks for Large Language Models"](https://arxiv.org/abs/2306.04634). For more information on
the inner functioning of watermarking, it is recommended to refer to the paper.

The watermarking can be used with any generative model in `tranformers` and does not require an extra classification model
to detect watermarked text. To trigger watermarking, pass in a [`WatermarkingConfig`] with needed arguments directly to the
`.generate()` method or add it to the [`GenerationConfig`]. Watermarked text can be later detected with a [`WatermarkDetector`].


<Tip warning={true}>

The WatermarkDetector internally relies on the proportion of "green" tokens, and whether generated text follows the coloring pattern.
That is why it is recommended to strip off the prompt text, if it is much longer than the generated text.
This also can have an effect when one sequence in the batch is a lot longer causing other rows to be padded.
Additionally, the detector **must** be initiated with identical watermark configuration arguments used when generating.

</Tip>

Let's generate some text with watermarking. In the below code snippet, we set the bias to 2.5 which is a value that
will be added to "green" tokens' logits. After generating watermarked text, we can pass it directly to the `WatermarkDetector`
to check if the text is machine-generated (outputs `True` for machine-generated and `False` otherwise).

```python
>>> from transformers import AutoTokenizer, AutoModelForCausalLM, WatermarkDetector, WatermarkingConfig

>>> model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2")
>>> tok = AutoTokenizer.from_pretrained("openai-community/gpt2")
>>> tok.pad_token_id = tok.eos_token_id
>>> tok.padding_side = "left"

>>> inputs = tok(["This is the beginning of a long story", "Alice and Bob are"], padding=True, return_tensors="pt")
>>> input_len = inputs["input_ids"].shape[-1]

>>> watermarking_config = WatermarkingConfig(bias=2.5, seeding_scheme="selfhash")
>>> out = model.generate(**inputs, watermarking_config=watermarking_config, do_sample=False, max_length=20)

>>> detector = WatermarkDetector(model_config=model.config, device="cpu", watermarking_config=watermarking_config)
>>> detection_out = detector(out, return_dict=True)
>>> detection_out.prediction
array([True, True])
```


## Decoding strategies

Certain combinations of the `generate()` parameters, and ultimately `generation_config`, can be used to enable specific
decoding strategies. If you are new to this concept, we recommend reading
[this blog post that illustrates how common decoding strategies work](https://huggingface.co/blog/how-to-generate).

Here, we'll show some of the parameters that control the decoding strategies and illustrate how you can use them.


key: _node_content
string_value:
{"id_": "2d883b63-128d-472e-b760-8fc82bd46e76", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e9f71c25-71f0-4b18-be11-c33b6fcecac6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "d7c025cad61c09f008bde35b4154c5dc0fe13b268c8411677421925cfff7b247", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "834a0f22-b00f-4367-97bf-b8735635bf4d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "74d575c6ccadcbe76b05b23ce3e9447bb21e47873096d27d6d3da4ee6c328fb0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "86f1bc39-e22c-43a5-aac6-95a5955b467a", "node_type": "1", "metadata": {}, "hash": "f794ae1b220d767481bbd5832536252890ce56086c9389688f1b1dd2155d49b4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11806, "end_char_idx": 15329, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Decoding strategies

Certain combinations of the `generate()` parameters, and ultimately `generation_config`, can be used to enable specific
decoding strategies. If you are new to this concept, we recommend reading
[this blog post that illustrates how common decoding strategies work](https://huggingface.co/blog/how-to-generate).

Here, we'll show some of the parameters that control the decoding strategies and illustrate how you can use them.

<Tip>

Selecting a given decoding strategy is not the only way you can influence the outcome of `generate()` with your model.
The decoding strategies act based (mostly) on the logits, the distribution of probabilities for the next token, and
thus selecting a good logits manipulation strategy can go a long way! In other words, manipulating the logits is another
dimension you can act upon, in addition to selecting a decoding strategy. Popular logits manipulation strategies include
`top_p`, `min_p`, and `repetition_penalty` -- you can check the full list in the [`GenerationConfig`] class.

</Tip>

### Greedy Search

[`generate`] uses greedy search decoding by default so you don't have to pass any parameters to enable it. This means the parameters `num_beams` is set to 1 and `do_sample=False`.

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> prompt = "I look forward to"
>>> checkpoint = "distilbert/distilgpt2"

>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> inputs = tokenizer(prompt, return_tensors="pt")

>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)
>>> outputs = model.generate(**inputs)
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['I look forward to seeing you all again!\n\n\n\n\n\n\n\n\n\n\n']
```

### Contrastive search

The contrastive search decoding strategy was proposed in the 2022 paper [A Contrastive Framework for Neural Text Generation](https://arxiv.org/abs/2202.06417).
It demonstrates superior results for generating non-repetitive yet coherent long outputs. To learn how contrastive search
works, check out [this blog post](https://huggingface.co/blog/introducing-csearch).
The two main parameters that enable and control the behavior of contrastive search are `penalty_alpha` and `top_k`:

```python
>>> from transformers import AutoTokenizer, AutoModelForCausalLM

>>> checkpoint = "openai-community/gpt2-large"
>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)

>>> prompt = "Hugging Face Company is"
>>> inputs = tokenizer(prompt, return_tensors="pt")

>>> outputs = model.generate(**inputs, penalty_alpha=0.6, top_k=4, max_new_tokens=100)
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['Hugging Face Company is a family owned and operated business. We pride ourselves on being the best
in the business and our customer service is second to none.\n\nIf you have any questions about our
products or services, feel free to contact us at any time. We look forward to hearing from you!']
```

### Multinomial sampling

As opposed to greedy search that always chooses a token with the highest probability as the
next token, multinomial sampling (also called ancestral sampling) randomly selects the next token based on the probability distribution over the entire
vocabulary given by the model. Every token with a non-zero probability has a chance of being selected, thus reducing the
risk of repetition.

To enable multinomial sampling set `do_sample=True` and `num_beams=1`.


key: _node_content
string_value:
{"id_": "86f1bc39-e22c-43a5-aac6-95a5955b467a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e9f71c25-71f0-4b18-be11-c33b6fcecac6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "d7c025cad61c09f008bde35b4154c5dc0fe13b268c8411677421925cfff7b247", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2d883b63-128d-472e-b760-8fc82bd46e76", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "3cee82c08c54ce28a08249ba9a7c6bf937790a52a47048c424cad6918438893b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "387866e7-b911-4f96-8851-78607da9f745", "node_type": "1", "metadata": {}, "hash": "8210bcfb1f111f467e38e5c1420a1363697456b6c481598ec3ab4ad4e2b777d1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13737, "end_char_idx": 17265, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
It demonstrates superior results for generating non-repetitive yet coherent long outputs. To learn how contrastive search
works, check out [this blog post](https://huggingface.co/blog/introducing-csearch).
The two main parameters that enable and control the behavior of contrastive search are `penalty_alpha` and `top_k`:

```python
>>> from transformers import AutoTokenizer, AutoModelForCausalLM

>>> checkpoint = "openai-community/gpt2-large"
>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)

>>> prompt = "Hugging Face Company is"
>>> inputs = tokenizer(prompt, return_tensors="pt")

>>> outputs = model.generate(**inputs, penalty_alpha=0.6, top_k=4, max_new_tokens=100)
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['Hugging Face Company is a family owned and operated business. We pride ourselves on being the best
in the business and our customer service is second to none.\n\nIf you have any questions about our
products or services, feel free to contact us at any time. We look forward to hearing from you!']
```

### Multinomial sampling

As opposed to greedy search that always chooses a token with the highest probability as the
next token, multinomial sampling (also called ancestral sampling) randomly selects the next token based on the probability distribution over the entire
vocabulary given by the model. Every token with a non-zero probability has a chance of being selected, thus reducing the
risk of repetition.

To enable multinomial sampling set `do_sample=True` and `num_beams=1`.

```python
>>> from transformers import AutoTokenizer, AutoModelForCausalLM, set_seed
>>> set_seed(0)  # For reproducibility

>>> checkpoint = "openai-community/gpt2-large"
>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)

>>> prompt = "Today was an amazing day because"
>>> inputs = tokenizer(prompt, return_tensors="pt")

>>> outputs = model.generate(**inputs, do_sample=True, num_beams=1, max_new_tokens=100)
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
["Today was an amazing day because we received these wonderful items by the way of a gift shop. The box arrived on a Thursday and I opened it on Monday afternoon to receive the gifts. Both bags featured pieces from all the previous years!\n\nThe box had lots of surprises in it, including some sweet little mini chocolate chips! I don't think I'd eat all of these. This was definitely one of the most expensive presents I have ever got, I actually got most of them for free!\n\nThe first package came"]
```

### Beam-search decoding

Unlike greedy search, beam-search decoding keeps several hypotheses at each time step and eventually chooses
the hypothesis that has the overall highest probability for the entire sequence. This has the advantage of identifying high-probability
sequences that start with lower probability initial tokens and would've been ignored by the greedy search.

<a href="https://huggingface.co/spaces/m-ric/beam_search_visualizer" class="flex flex-col justify-center">
    <img style="max-width: 90%; margin: auto;" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/beam_search.png"/>
</a>

You can visualize how beam-search decoding works in [this interactive demo](https://huggingface.co/spaces/m-ric/beam_search_visualizer): type your input sentence, and play with the parameters to see how the decoding beams change.


key: _node_content
string_value:
{"id_": "387866e7-b911-4f96-8851-78607da9f745", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e9f71c25-71f0-4b18-be11-c33b6fcecac6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "d7c025cad61c09f008bde35b4154c5dc0fe13b268c8411677421925cfff7b247", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "86f1bc39-e22c-43a5-aac6-95a5955b467a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "fca82065c573ef10a05126571637cd3324102b2e58bfa457b789f1296a3fd6cd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1038a8d1-71df-474c-8bce-11350d33776e", "node_type": "1", "metadata": {}, "hash": "4190f9d3f54bf2362cfe9ac5587ed94c2728f22e72b460924e63af6275b16c81", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15970, "end_char_idx": 19350, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The box arrived on a Thursday and I opened it on Monday afternoon to receive the gifts. Both bags featured pieces from all the previous years!\n\nThe box had lots of surprises in it, including some sweet little mini chocolate chips! I don't think I'd eat all of these. This was definitely one of the most expensive presents I have ever got, I actually got most of them for free!\n\nThe first package came"]
```

### Beam-search decoding

Unlike greedy search, beam-search decoding keeps several hypotheses at each time step and eventually chooses
the hypothesis that has the overall highest probability for the entire sequence. This has the advantage of identifying high-probability
sequences that start with lower probability initial tokens and would've been ignored by the greedy search.

<a href="https://huggingface.co/spaces/m-ric/beam_search_visualizer" class="flex flex-col justify-center">
    <img style="max-width: 90%; margin: auto;" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/beam_search.png"/>
</a>

You can visualize how beam-search decoding works in [this interactive demo](https://huggingface.co/spaces/m-ric/beam_search_visualizer): type your input sentence, and play with the parameters to see how the decoding beams change.

To enable this decoding strategy, specify the `num_beams` (aka number of hypotheses to keep track of) that is greater than 1.

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> prompt = "It is astonishing how one can"
>>> checkpoint = "openai-community/gpt2-medium"

>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> inputs = tokenizer(prompt, return_tensors="pt")

>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)

>>> outputs = model.generate(**inputs, num_beams=5, max_new_tokens=50)
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['It is astonishing how one can have such a profound impact on the lives of so many people in such a short period of
time."\n\nHe added: "I am very proud of the work I have been able to do in the last few years.\n\n"I have']
```

### Beam-search multinomial sampling

As the name implies, this decoding strategy combines beam search with multinomial sampling. You need to specify
the `num_beams` greater than 1, and set `do_sample=True` to use this decoding strategy.

```python
>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, set_seed
>>> set_seed(0)  # For reproducibility

>>> prompt = "translate English to German: The house is wonderful."
>>> checkpoint = "google-t5/t5-small"

>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> inputs = tokenizer(prompt, return_tensors="pt")

>>> model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint)

>>> outputs = model.generate(**inputs, num_beams=5, do_sample=True)
>>> tokenizer.decode(outputs[0], skip_special_tokens=True)
'Das Haus ist wunderbar.'
```

### Diverse beam search decoding

The diverse beam search decoding strategy is an extension of the beam search strategy that allows for generating a more diverse
set of beam sequences to choose from. To learn how it works, refer to [Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models](https://arxiv.org/pdf/1610.02424.pdf).
This approach has three main parameters: `num_beams`, `num_beam_groups`, and `diversity_penalty`.


key: _node_content
string_value:
{"id_": "1038a8d1-71df-474c-8bce-11350d33776e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e9f71c25-71f0-4b18-be11-c33b6fcecac6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "d7c025cad61c09f008bde35b4154c5dc0fe13b268c8411677421925cfff7b247", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "387866e7-b911-4f96-8851-78607da9f745", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "c481a340155c1d5bbbdfbd6a84c27fc38397ced010bceebc61a482d75619d088", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2a75c261-9b2a-4b8d-9d9a-6ae46ddf4b22", "node_type": "1", "metadata": {}, "hash": "c50688cbc1c4a812083d356a11b1b9c4b2b46d1aedc7c403f4018f7d091dec5b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17992, "end_char_idx": 21470, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"\n\nHe added: "I am very proud of the work I have been able to do in the last few years.\n\n"I have']
```

### Beam-search multinomial sampling

As the name implies, this decoding strategy combines beam search with multinomial sampling. You need to specify
the `num_beams` greater than 1, and set `do_sample=True` to use this decoding strategy.

```python
>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, set_seed
>>> set_seed(0)  # For reproducibility

>>> prompt = "translate English to German: The house is wonderful."
>>> checkpoint = "google-t5/t5-small"

>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> inputs = tokenizer(prompt, return_tensors="pt")

>>> model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint)

>>> outputs = model.generate(**inputs, num_beams=5, do_sample=True)
>>> tokenizer.decode(outputs[0], skip_special_tokens=True)
'Das Haus ist wunderbar.'
```

### Diverse beam search decoding

The diverse beam search decoding strategy is an extension of the beam search strategy that allows for generating a more diverse
set of beam sequences to choose from. To learn how it works, refer to [Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models](https://arxiv.org/pdf/1610.02424.pdf).
This approach has three main parameters: `num_beams`, `num_beam_groups`, and `diversity_penalty`.
The diversity penalty ensures the outputs are distinct across groups, and beam search is used within each group.


```python
>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

>>> checkpoint = "google/pegasus-xsum"
>>> prompt = (
...     "The Permaculture Design Principles are a set of universal design principles "
...     "that can be applied to any location, climate and culture, and they allow us to design "
...     "the most efficient and sustainable human habitation and food production systems. "
...     "Permaculture is a design system that encompasses a wide variety of disciplines, such "
...     "as ecology, landscape design, environmental science and energy conservation, and the "
...     "Permaculture design principles are drawn from these various disciplines. Each individual "
...     "design principle itself embodies a complete conceptual framework based on sound "
...     "scientific principles. When we bring all these separate  principles together, we can "
...     "create a design system that both looks at whole systems, the parts that these systems "
...     "consist of, and how those parts interact with each other to create a complex, dynamic, "
...     "living system. Each design principle serves as a tool that allows us to integrate all "
...     "the separate parts of a design, referred to as elements, into a functional, synergistic, "
...     "whole system, where the elements harmoniously interact and work together in the most "
...     "efficient way possible."
... )

>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> inputs = tokenizer(prompt, return_tensors="pt")

>>> model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint)

>>> outputs = model.generate(**inputs, num_beams=5, num_beam_groups=5, max_new_tokens=30, diversity_penalty=1.0)
>>> tokenizer.decode(outputs[0], skip_special_tokens=True)
'The Design Principles are a set of universal design principles that can be applied to any location, climate and
culture, and they allow us to design the'
```

This guide illustrates the main parameters that enable various decoding strategies.


key: _node_content
string_value:
{"id_": "2a75c261-9b2a-4b8d-9d9a-6ae46ddf4b22", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e9f71c25-71f0-4b18-be11-c33b6fcecac6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "d7c025cad61c09f008bde35b4154c5dc0fe13b268c8411677421925cfff7b247", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1038a8d1-71df-474c-8bce-11350d33776e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "0ce449a0792b08c9693cc72227923b6367faacaa86e9bc0ca36d2e89929dbcc7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0ccfd617-bbe4-461f-b2b5-0ed978f399b9", "node_type": "1", "metadata": {}, "hash": "4ddefa557c7c4b62d69e8b25ba956e9f97039e80b2ca8f01654ee97892a2993b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19873, "end_char_idx": 23515, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
...     "Permaculture is a design system that encompasses a wide variety of disciplines, such "
...     "as ecology, landscape design, environmental science and energy conservation, and the "
...     "Permaculture design principles are drawn from these various disciplines. Each individual "
...     "design principle itself embodies a complete conceptual framework based on sound "
...     "scientific principles. When we bring all these separate  principles together, we can "
...     "create a design system that both looks at whole systems, the parts that these systems "
...     "consist of, and how those parts interact with each other to create a complex, dynamic, "
...     "living system. Each design principle serves as a tool that allows us to integrate all "
...     "the separate parts of a design, referred to as elements, into a functional, synergistic, "
...     "whole system, where the elements harmoniously interact and work together in the most "
...     "efficient way possible."
... )

>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> inputs = tokenizer(prompt, return_tensors="pt")

>>> model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint)

>>> outputs = model.generate(**inputs, num_beams=5, num_beam_groups=5, max_new_tokens=30, diversity_penalty=1.0)
>>> tokenizer.decode(outputs[0], skip_special_tokens=True)
'The Design Principles are a set of universal design principles that can be applied to any location, climate and
culture, and they allow us to design the'
```

This guide illustrates the main parameters that enable various decoding strategies. More advanced parameters exist for the
[`generate`] method, which gives you even further control over the [`generate`] method's behavior.
For the complete list of the available parameters, refer to the [API documentation](./main_classes/text_generation.md).

### Speculative Decoding

Speculative decoding (also known as assisted decoding) is a modification of the decoding strategies above, that uses an
assistant model (ideally a much smaller one) with the same tokenizer, to generate a few candidate tokens. The main
model then validates the candidate tokens in a single forward pass, which speeds up the decoding process. If
`do_sample=True`, then the token validation with resampling introduced in the
[speculative decoding paper](https://arxiv.org/pdf/2211.17192.pdf) is used.

Currently, only greedy search and sampling are supported with assisted decoding, and assisted decoding doesn't support batched inputs.
To learn more about assisted decoding, check [this blog post](https://huggingface.co/blog/assisted-generation).

To enable assisted decoding, set the `assistant_model` argument with a model.

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> prompt = "Alice and Bob"
>>> checkpoint = "EleutherAI/pythia-1.4b-deduped"
>>> assistant_checkpoint = "EleutherAI/pythia-160m-deduped"

>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> inputs = tokenizer(prompt, return_tensors="pt")

>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)
>>> assistant_model = AutoModelForCausalLM.from_pretrained(assistant_checkpoint)
>>> outputs = model.generate(**inputs, assistant_model=assistant_model)
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['Alice and Bob are sitting in a bar. Alice is drinking a beer and Bob is drinking a']
```

When using assisted decoding with sampling methods, you can use the `temperature` argument to control the randomness,
just like in multinomial sampling. However, in assisted decoding, reducing the temperature may help improve the latency.


key: _node_content
string_value:
{"id_": "0ccfd617-bbe4-461f-b2b5-0ed978f399b9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e9f71c25-71f0-4b18-be11-c33b6fcecac6", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "d7c025cad61c09f008bde35b4154c5dc0fe13b268c8411677421925cfff7b247", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2a75c261-9b2a-4b8d-9d9a-6ae46ddf4b22", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/generation_strategies", "title": "Text generation strategies", "tokens": 6637, "retrieve_doc": true, "source": "transformers"}, "hash": "4caeda4e903c1dd62dafe2cf5c943380f5be40400c299f81cb71ed0117b46910", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "25c9a4fb-136b-42a3-a9ed-821772ffb7b8", "node_type": "1", "metadata": {}, "hash": "64075a65916a5d71032f6056e99eb54ce7c52f2e5836e94ee8f437372eb50559", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21730, "end_char_idx": 25060, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Speculative Decoding

Speculative decoding (also known as assisted decoding) is a modification of the decoding strategies above, that uses an
assistant model (ideally a much smaller one) with the same tokenizer, to generate a few candidate tokens. The main
model then validates the candidate tokens in a single forward pass, which speeds up the decoding process. If
`do_sample=True`, then the token validation with resampling introduced in the
[speculative decoding paper](https://arxiv.org/pdf/2211.17192.pdf) is used.

Currently, only greedy search and sampling are supported with assisted decoding, and assisted decoding doesn't support batched inputs.
To learn more about assisted decoding, check [this blog post](https://huggingface.co/blog/assisted-generation).

To enable assisted decoding, set the `assistant_model` argument with a model.

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer

>>> prompt = "Alice and Bob"
>>> checkpoint = "EleutherAI/pythia-1.4b-deduped"
>>> assistant_checkpoint = "EleutherAI/pythia-160m-deduped"

>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> inputs = tokenizer(prompt, return_tensors="pt")

>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)
>>> assistant_model = AutoModelForCausalLM.from_pretrained(assistant_checkpoint)
>>> outputs = model.generate(**inputs, assistant_model=assistant_model)
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['Alice and Bob are sitting in a bar. Alice is drinking a beer and Bob is drinking a']
```

When using assisted decoding with sampling methods, you can use the `temperature` argument to control the randomness,
just like in multinomial sampling. However, in assisted decoding, reducing the temperature may help improve the latency.

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed
>>> set_seed(42)  # For reproducibility

>>> prompt = "Alice and Bob"
>>> checkpoint = "EleutherAI/pythia-1.4b-deduped"
>>> assistant_checkpoint = "EleutherAI/pythia-160m-deduped"

>>> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
>>> inputs = tokenizer(prompt, return_tensors="pt")

>>> model = AutoModelForCausalLM.from_pretrained(checkpoint)
>>> assistant_model = AutoModelForCausalLM.from_pretrained(assistant_checkpoint)
>>> outputs = model.generate(**inputs, assistant_model=assistant_model, do_sample=True, temperature=0.5)
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['Alice and Bob, a couple of friends of mine, who are both in the same office as']
```

Alternativelly, you can also set the `prompt_lookup_num_tokens` to trigger n-gram based assisted decoding, as opposed
to model based assisted decoding. You can read more about it [here](https://twitter.com/joao_gante/status/1747322413006643259).
### DoLa Decoding

**D**ecoding by C**o**ntrasting **La**yers (DoLa) is a contrastive decoding strategy to improve the factuality and reduce the
hallucinations of LLMs, as described in this paper of ICLR 2024 [DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models](https://arxiv.org/abs/2309.03883).

DoLa is achieved by contrasting the differences in logits obtained from final
layers versus earlier layers, thus amplify the factual knowledge localized to particular part of transformer layers.


