# embedding_metadata

key: _node_content
string_value:
{"id_": "1635f8f4-3fb7-4313-89da-97fd519809ea", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "hash": "3ed0d14d3467b190aad4d7f8c3c47f3f942dcbd36a3398099319a304d5bc33f6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7b900600-5651-4114-82bc-526c2c42e296", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "hash": "e7459a97d625e3e888d0b90fde07ac534fbf7e17179eb0b820eba35a196cff34", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "672fabf0-66c2-414a-ae6d-833360ba779a", "node_type": "1", "metadata": {}, "hash": "252a9171a10cb583b6e3fae6ac2f660fb6b8ea53fee1097430f10738bcc90211", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12063, "end_char_idx": 15210, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We're using the [`pytorchvideo.data.Ucf101()`](https://pytorchvideo.readthedocs.io/en/latest/api/data/data.html#pytorchvideo.data.Ucf101) function because it's tailored for the UCF-101 dataset. Under the hood, it returns a [`pytorchvideo.data.labeled_video_dataset.LabeledVideoDataset`](https://pytorchvideo.readthedocs.io/en/latest/api/data/data.html#pytorchvideo.data.LabeledVideoDataset) object. `LabeledVideoDataset` class is the base class for all things video in the PyTorchVideo dataset. So, if you want to use a custom dataset not supported off-the-shelf by PyTorchVideo, you can extend the `LabeledVideoDataset` class accordingly. Refer to the `data` API [documentation to](https://pytorchvideo.readthedocs.io/en/latest/api/data/data.html) learn more. Also, if your dataset follows a similar structure (as shown above), then using the `pytorchvideo.data.Ucf101()` should work just fine. 

You can access the `num_videos` argument to know the number of videos in the dataset.

```py
>>> print(train_dataset.num_videos, val_dataset.num_videos, test_dataset.num_videos)
# (300, 30, 75)
```

## Visualize the preprocessed video for better debugging 

```py 
>>> import imageio
>>> import numpy as np
>>> from IPython.display import Image

>>> def unnormalize_img(img):
...     """Un-normalizes the image pixels."""
...     img = (img * std) + mean
...     img = (img * 255).astype("uint8")
...     return img.clip(0, 255)

>>> def create_gif(video_tensor, filename="sample.gif"):
...     """Prepares a GIF from a video tensor.
...     
...     The video tensor is expected to have the following shape:
...     (num_frames, num_channels, height, width).
...     """
...     frames = []
...     for video_frame in video_tensor:
...         frame_unnormalized = unnormalize_img(video_frame.permute(1, 2, 0).numpy())
...         frames.append(frame_unnormalized)
...     kargs = {"duration": 0.25}
...     imageio.mimsave(filename, frames, "GIF", **kargs)
...     return filename

>>> def display_gif(video_tensor, gif_name="sample.gif"):
...     """Prepares and displays a GIF from a video tensor."""
...     video_tensor = video_tensor.permute(1, 0, 2, 3)
...     gif_filename = create_gif(video_tensor, gif_name)
...     return Image(filename=gif_filename)

>>> sample_video = next(iter(train_dataset))
>>> video_tensor = sample_video["video"]
>>> display_gif(video_tensor)
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/sample_gif.gif" alt="Person playing basketball"/>
</div>

## Train the model 

Leverage [`Trainer`](https://huggingface.co/docs/transformers/main_classes/trainer) from  🤗 Transformers for training the model. To instantiate a `Trainer`, you need to define the training configuration and an evaluation metric. The most important is the [`TrainingArguments`](https://huggingface.co/transformers/main_classes/trainer.html#transformers.TrainingArguments), which is a class that contains all the attributes to configure the training. It requires an output folder name, which will be used to save the checkpoints of the model.


key: _node_content
string_value:
{"id_": "672fabf0-66c2-414a-ae6d-833360ba779a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "hash": "3ed0d14d3467b190aad4d7f8c3c47f3f942dcbd36a3398099319a304d5bc33f6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1635f8f4-3fb7-4313-89da-97fd519809ea", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "hash": "af42fbaafbe2d9d7781e3a88518c512e39e3c6650872d16f5c957bcf7f1caf3f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fe895121-9907-4f64-b243-b99bb0b97c2b", "node_type": "1", "metadata": {}, "hash": "261cbc62abd204d47716f98d7d8cc4aa5c70275ca87ab7b1d02b41ed5749f22b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13595, "end_char_idx": 15793, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
...     
...     The video tensor is expected to have the following shape:
...     (num_frames, num_channels, height, width).
...     """
...     frames = []
...     for video_frame in video_tensor:
...         frame_unnormalized = unnormalize_img(video_frame.permute(1, 2, 0).numpy())
...         frames.append(frame_unnormalized)
...     kargs = {"duration": 0.25}
...     imageio.mimsave(filename, frames, "GIF", **kargs)
...     return filename

>>> def display_gif(video_tensor, gif_name="sample.gif"):
...     """Prepares and displays a GIF from a video tensor."""
...     video_tensor = video_tensor.permute(1, 0, 2, 3)
...     gif_filename = create_gif(video_tensor, gif_name)
...     return Image(filename=gif_filename)

>>> sample_video = next(iter(train_dataset))
>>> video_tensor = sample_video["video"]
>>> display_gif(video_tensor)
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/sample_gif.gif" alt="Person playing basketball"/>
</div>

## Train the model 

Leverage [`Trainer`](https://huggingface.co/docs/transformers/main_classes/trainer) from  🤗 Transformers for training the model. To instantiate a `Trainer`, you need to define the training configuration and an evaluation metric. The most important is the [`TrainingArguments`](https://huggingface.co/transformers/main_classes/trainer.html#transformers.TrainingArguments), which is a class that contains all the attributes to configure the training. It requires an output folder name, which will be used to save the checkpoints of the model. It also helps sync all the information in the model repository on 🤗 Hub.

Most of the training arguments are self-explanatory, but one that is quite important here is `remove_unused_columns=False`. This one will drop any features not used by the model's call function. By default it's `True` because usually it's ideal to drop unused feature columns, making it easier to unpack inputs into the model's call function. But, in this case, you need the unused features ('video' in particular) in order to create `pixel_values` (which is a mandatory key our model expects in its inputs).


key: _node_content
string_value:
{"id_": "fe895121-9907-4f64-b243-b99bb0b97c2b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "hash": "3ed0d14d3467b190aad4d7f8c3c47f3f942dcbd36a3398099319a304d5bc33f6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "672fabf0-66c2-414a-ae6d-833360ba779a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "hash": "a2c6aabfbe051f8cad3d8a3a35c8debb7ac42993fa85cf245e702f0cdadf4404", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "89ef923f-c142-4602-9140-3b4989b05bee", "node_type": "1", "metadata": {}, "hash": "66a10dfc39813344cb7a40227ed7c5059f2dd0fe19763a9c886a0709dc94bc6c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14166, "end_char_idx": 17483, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
...     video_tensor = video_tensor.permute(1, 0, 2, 3)
...     gif_filename = create_gif(video_tensor, gif_name)
...     return Image(filename=gif_filename)

>>> sample_video = next(iter(train_dataset))
>>> video_tensor = sample_video["video"]
>>> display_gif(video_tensor)
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/sample_gif.gif" alt="Person playing basketball"/>
</div>

## Train the model 

Leverage [`Trainer`](https://huggingface.co/docs/transformers/main_classes/trainer) from  🤗 Transformers for training the model. To instantiate a `Trainer`, you need to define the training configuration and an evaluation metric. The most important is the [`TrainingArguments`](https://huggingface.co/transformers/main_classes/trainer.html#transformers.TrainingArguments), which is a class that contains all the attributes to configure the training. It requires an output folder name, which will be used to save the checkpoints of the model. It also helps sync all the information in the model repository on 🤗 Hub.

Most of the training arguments are self-explanatory, but one that is quite important here is `remove_unused_columns=False`. This one will drop any features not used by the model's call function. By default it's `True` because usually it's ideal to drop unused feature columns, making it easier to unpack inputs into the model's call function. But, in this case, you need the unused features ('video' in particular) in order to create `pixel_values` (which is a mandatory key our model expects in its inputs).


```py 
>>> from transformers import TrainingArguments, Trainer

>>> model_name = model_ckpt.split("/")[-1]
>>> new_model_name = f"{model_name}-finetuned-ucf101-subset"
>>> num_epochs = 4

>>> args = TrainingArguments(
...     new_model_name,
...     remove_unused_columns=False,
...     eval_strategy="epoch",
...     save_strategy="epoch",
...     learning_rate=5e-5,
...     per_device_train_batch_size=batch_size,
...     per_device_eval_batch_size=batch_size,
...     warmup_ratio=0.1,
...     logging_steps=10,
...     load_best_model_at_end=True,
...     metric_for_best_model="accuracy",
...     push_to_hub=True,
...     max_steps=(train_dataset.num_videos // batch_size) * num_epochs,
... )
```

The dataset returned by `pytorchvideo.data.Ucf101()` doesn't implement the `__len__` method. As such, we must define `max_steps` when instantiating `TrainingArguments`. 

Next, you need to define a function to compute the metrics from the predictions, which will use the `metric` you'll load now. The only preprocessing you have to do is to take the argmax of our predicted logits:

```py
import evaluate

metric = evaluate.load("accuracy")


def compute_metrics(eval_pred):
    predictions = np.argmax(eval_pred.predictions, axis=1)
    return metric.compute(predictions=predictions, references=eval_pred.label_ids)
```

**A note on evaluation**:

In the [VideoMAE paper](https://arxiv.org/abs/2203.12602), the authors use the following evaluation strategy. They evaluate the model on several clips from test videos and apply different crops to those clips and report the aggregate score. However, in the interest of simplicity and brevity, we don't consider that in this tutorial.


key: _node_content
string_value:
{"id_": "89ef923f-c142-4602-9140-3b4989b05bee", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "hash": "3ed0d14d3467b190aad4d7f8c3c47f3f942dcbd36a3398099319a304d5bc33f6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fe895121-9907-4f64-b243-b99bb0b97c2b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "hash": "68949190028592aa342d9aa3c44142620efdc22b435085465ec9894637e43398", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "de68dfdb-5a9b-4372-bb67-e3fc77808157", "node_type": "1", "metadata": {}, "hash": "936d7c3bbcc39ed382a03c177501fef63f6af3c7d3bf51a0f5872c4fcdd0573b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16944, "end_char_idx": 19437, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
def compute_metrics(eval_pred):
    predictions = np.argmax(eval_pred.predictions, axis=1)
    return metric.compute(predictions=predictions, references=eval_pred.label_ids)
```

**A note on evaluation**:

In the [VideoMAE paper](https://arxiv.org/abs/2203.12602), the authors use the following evaluation strategy. They evaluate the model on several clips from test videos and apply different crops to those clips and report the aggregate score. However, in the interest of simplicity and brevity, we don't consider that in this tutorial.

Also, define a `collate_fn`, which will be used to batch examples together. Each batch consists of 2 keys, namely `pixel_values` and `labels`.

```py 
>>> def collate_fn(examples):
...     # permute to (num_frames, num_channels, height, width)
...     pixel_values = torch.stack(
...         [example["video"].permute(1, 0, 2, 3) for example in examples]
...     )
...     labels = torch.tensor([example["label"] for example in examples])
...     return {"pixel_values": pixel_values, "labels": labels}
```

Then you just pass all of this along with the datasets to `Trainer`:

```py 
>>> trainer = Trainer(
...     model,
...     args,
...     train_dataset=train_dataset,
...     eval_dataset=val_dataset,
...     tokenizer=image_processor,
...     compute_metrics=compute_metrics,
...     data_collator=collate_fn,
... )
```

You might wonder why you passed along the `image_processor` as a tokenizer when you preprocessed the data already. This is only to make sure the image processor configuration file (stored as JSON) will also be uploaded to the repo on the Hub.

Now fine-tune our model by calling the `train` method:

```py 
>>> train_results = trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```

## Inference

Great, now that you have fine-tuned a model, you can use it for inference!

Load a video for inference:

```py 
>>> sample_test_video = next(iter(test_dataset))
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/sample_gif_two.gif" alt="Teams playing basketball"/>
</div>

The simplest way to try out your fine-tuned model for inference is to use it in a [`pipeline`](https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#transformers.VideoClassificationPipeline).


key: _node_content
string_value:
{"id_": "de68dfdb-5a9b-4372-bb67-e3fc77808157", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "161d7d29-43e1-4306-bfa1-36cd04a2474d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "hash": "3ed0d14d3467b190aad4d7f8c3c47f3f942dcbd36a3398099319a304d5bc33f6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "89ef923f-c142-4602-9140-3b4989b05bee", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "transformers"}, "hash": "aed5cbdcc2cfcc5ca11dde644ba7ba7f18393946511dc1e9a8ec327403d2d321", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18429, "end_char_idx": 21219, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This is only to make sure the image processor configuration file (stored as JSON) will also be uploaded to the repo on the Hub.

Now fine-tune our model by calling the `train` method:

```py 
>>> train_results = trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```

## Inference

Great, now that you have fine-tuned a model, you can use it for inference!

Load a video for inference:

```py 
>>> sample_test_video = next(iter(test_dataset))
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/sample_gif_two.gif" alt="Teams playing basketball"/>
</div>

The simplest way to try out your fine-tuned model for inference is to use it in a [`pipeline`](https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#transformers.VideoClassificationPipeline). Instantiate a `pipeline` for video classification with your model, and pass your video to it:

```py
>>> from transformers import pipeline

>>> video_cls = pipeline(model="my_awesome_video_cls_model")
>>> video_cls("https://huggingface.co/datasets/sayakpaul/ucf101-subset/resolve/main/v_BasketballDunk_g14_c06.avi")
[{'score': 0.9272987842559814, 'label': 'BasketballDunk'},
 {'score': 0.017777055501937866, 'label': 'BabyCrawling'},
 {'score': 0.01663011871278286, 'label': 'BalanceBeam'},
 {'score': 0.009560945443809032, 'label': 'BandMarching'},
 {'score': 0.0068979403004050255, 'label': 'BaseballPitch'}]
```

You can also manually replicate the results of the `pipeline` if you'd like.


```py
>>> def run_inference(model, video):
...     # (num_frames, num_channels, height, width)
...     perumuted_sample_test_video = video.permute(1, 0, 2, 3)
...     inputs = {
...         "pixel_values": perumuted_sample_test_video.unsqueeze(0),
...         "labels": torch.tensor(
...             [sample_test_video["label"]]
...         ),  # this can be skipped if you don't have labels available.
...     }

...     device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
...     inputs = {k: v.to(device) for k, v in inputs.items()}
...     model = model.to(device)

...     # forward pass
...     with torch.no_grad():
...         outputs = model(**inputs)
...         logits = outputs.logits

...     return logits
```

Now, pass your input to the model and return the `logits`:

```py
>>> logits = run_inference(trained_model, sample_test_video["video"])
```

Decoding the `logits`, we get: 

```py 
>>> predicted_class_idx = logits.argmax(-1).item()
>>> print("Predicted class:", model.config.id2label[predicted_class_idx])
# Predicted class: BasketballDunk
```


key: _node_content
string_value:
{"id_": "c8496c81-0527-4a6a-8252-468e1f595e96", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "hash": "7fef769416a8a51ead2ae01a92eef274cf124d5c7c49a1c12621ab470154a9e4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "579e961f-454b-4147-bf6d-88941c373943", "node_type": "1", "metadata": {}, "hash": "8e2262c58b77ad0d91f432ac8815c9faac51f95c8265e1f89a64bb2e923e9168", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2936, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Text to speech

[[open-in-colab]]

Text-to-speech (TTS) is the task of creating natural-sounding speech from text, where the speech can be generated in multiple 
languages and for multiple speakers. Several text-to-speech models are currently available in 🤗 Transformers, such as 
[Bark](../model_doc/bark), [MMS](../model_doc/mms), [VITS](../model_doc/vits) and [SpeechT5](../model_doc/speecht5). 

You can easily generate audio using the `"text-to-audio"` pipeline (or its alias - `"text-to-speech"`). Some models, like Bark, 
can also be conditioned to generate non-verbal communications such as laughing, sighing and crying, or even add music.
Here's an example of how you would use the `"text-to-speech"` pipeline with Bark: 

```py
>>> from transformers import pipeline

>>> pipe = pipeline("text-to-speech", model="suno/bark-small")
>>> text = "[clears throat] This is a test ... and I just took a long pause."
>>> output = pipe(text)
```

Here's a code snippet you can use to listen to the resulting audio in a notebook: 

```python
>>> from IPython.display import Audio
>>> Audio(output["audio"], rate=output["sampling_rate"])
```

For more examples on what Bark and other pretrained TTS models can do, refer to our 
[Audio course](https://huggingface.co/learn/audio-course/chapter6/pre-trained_models). 

If you are looking to fine-tune a TTS model, the only text-to-speech models currently available in 🤗 Transformers 
are [SpeechT5](model_doc/speecht5) and [FastSpeech2Conformer](model_doc/fastspeech2_conformer), though more will be added in the future. SpeechT5 is pre-trained on a combination of speech-to-text and text-to-speech data, allowing it to learn a unified space of hidden representations shared by both text and speech. This means that the same pre-trained model can be fine-tuned for different tasks. Furthermore, SpeechT5 supports multiple speakers through x-vector speaker embeddings. 

The remainder of this guide illustrates how to:

1. Fine-tune [SpeechT5](../model_doc/speecht5) that was originally trained on English speech on the Dutch (`nl`) language subset of the [VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli) dataset.
2. Use your refined model for inference in one of two ways: using a pipeline or directly.

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install datasets soundfile speechbrain accelerate
```

Install 🤗Transformers from source as not all the SpeechT5 features have been merged into an official release yet:

```bash
pip install git+https://github.com/huggingface/transformers.git
```

<Tip>

To follow this guide you will need a GPU. If you're working in a notebook, run the following line to check if a GPU is available: 

```bash
!nvidia-smi
```

or alternatively for AMD GPUs:

```bash
!rocm-smi
```

</Tip>

We encourage you to log in to your Hugging Face account to upload and share your model with the community.


key: _node_content
string_value:
{"id_": "579e961f-454b-4147-bf6d-88941c373943", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "hash": "7fef769416a8a51ead2ae01a92eef274cf124d5c7c49a1c12621ab470154a9e4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c8496c81-0527-4a6a-8252-468e1f595e96", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "hash": "ec0e882a1f110d71111eff27b46fbc18f8fa6879102004bc3dd0186734422c34", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dfe03b92-2654-4108-a98b-5f4ca21e24fc", "node_type": "1", "metadata": {}, "hash": "45e60029c3e31b5a8cbc1c5d8dcc76ca82d1ab1a77f6ff782100bd7218221430", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1317, "end_char_idx": 4586, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If you are looking to fine-tune a TTS model, the only text-to-speech models currently available in 🤗 Transformers 
are [SpeechT5](model_doc/speecht5) and [FastSpeech2Conformer](model_doc/fastspeech2_conformer), though more will be added in the future. SpeechT5 is pre-trained on a combination of speech-to-text and text-to-speech data, allowing it to learn a unified space of hidden representations shared by both text and speech. This means that the same pre-trained model can be fine-tuned for different tasks. Furthermore, SpeechT5 supports multiple speakers through x-vector speaker embeddings. 

The remainder of this guide illustrates how to:

1. Fine-tune [SpeechT5](../model_doc/speecht5) that was originally trained on English speech on the Dutch (`nl`) language subset of the [VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli) dataset.
2. Use your refined model for inference in one of two ways: using a pipeline or directly.

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install datasets soundfile speechbrain accelerate
```

Install 🤗Transformers from source as not all the SpeechT5 features have been merged into an official release yet:

```bash
pip install git+https://github.com/huggingface/transformers.git
```

<Tip>

To follow this guide you will need a GPU. If you're working in a notebook, run the following line to check if a GPU is available: 

```bash
!nvidia-smi
```

or alternatively for AMD GPUs:

```bash
!rocm-smi
```

</Tip>

We encourage you to log in to your Hugging Face account to upload and share your model with the community. When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load the dataset

[VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli) is a large-scale multilingual speech corpus consisting of 
data sourced from 2009-2020 European Parliament event recordings. It contains labelled audio-transcription data for 15 
European languages. In this guide, we are using the Dutch language subset, feel free to pick another subset. 

Note that VoxPopuli or any other automated speech recognition (ASR) dataset may not be the most suitable 
option for training TTS models. The features that make it beneficial for ASR, such as excessive background noise, are 
typically undesirable in TTS. However, finding top-quality, multilingual, and multi-speaker TTS datasets can be quite 
challenging.

Let's load the data:

```py
>>> from datasets import load_dataset, Audio

>>> dataset = load_dataset("facebook/voxpopuli", "nl", split="train")
>>> len(dataset)
20968
```

20968 examples should be sufficient for fine-tuning. SpeechT5 expects audio data to have a sampling rate of 16 kHz, so 
make sure the examples in the dataset meet this requirement:

```py
dataset = dataset.cast_column("audio", Audio(sampling_rate=16000))
```

## Preprocess the data

Let's begin by defining the model checkpoint to use and loading the appropriate processor: 

```py
>>> from transformers import SpeechT5Processor

>>> checkpoint = "microsoft/speecht5_tts"
>>> processor = SpeechT5Processor.from_pretrained(checkpoint)
```

### Text cleanup for SpeechT5 tokenization 

Start by cleaning up the text data.


key: _node_content
string_value:
{"id_": "dfe03b92-2654-4108-a98b-5f4ca21e24fc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "hash": "7fef769416a8a51ead2ae01a92eef274cf124d5c7c49a1c12621ab470154a9e4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "579e961f-454b-4147-bf6d-88941c373943", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "hash": "687ca46cfa1332a77d430b8c7a6f2766eb42ca9da61e68bd619b8bde0b5e102e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cdca21eb-9442-497d-9d8e-5ab3537321b2", "node_type": "1", "metadata": {}, "hash": "6ebb7bbfb08bd865b19de6fe4749c8d678be5cacd0af1a65b8c1b696cbb225dd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2937, "end_char_idx": 6065, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load the dataset

[VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli) is a large-scale multilingual speech corpus consisting of 
data sourced from 2009-2020 European Parliament event recordings. It contains labelled audio-transcription data for 15 
European languages. In this guide, we are using the Dutch language subset, feel free to pick another subset. 

Note that VoxPopuli or any other automated speech recognition (ASR) dataset may not be the most suitable 
option for training TTS models. The features that make it beneficial for ASR, such as excessive background noise, are 
typically undesirable in TTS. However, finding top-quality, multilingual, and multi-speaker TTS datasets can be quite 
challenging.

Let's load the data:

```py
>>> from datasets import load_dataset, Audio

>>> dataset = load_dataset("facebook/voxpopuli", "nl", split="train")
>>> len(dataset)
20968
```

20968 examples should be sufficient for fine-tuning. SpeechT5 expects audio data to have a sampling rate of 16 kHz, so 
make sure the examples in the dataset meet this requirement:

```py
dataset = dataset.cast_column("audio", Audio(sampling_rate=16000))
```

## Preprocess the data

Let's begin by defining the model checkpoint to use and loading the appropriate processor: 

```py
>>> from transformers import SpeechT5Processor

>>> checkpoint = "microsoft/speecht5_tts"
>>> processor = SpeechT5Processor.from_pretrained(checkpoint)
```

### Text cleanup for SpeechT5 tokenization 

Start by cleaning up the text data. You'll need the tokenizer part of the processor to process the text:

```py
>>> tokenizer = processor.tokenizer
```

The dataset examples contain `raw_text` and `normalized_text` features. When deciding which feature to use as the text input, 
consider that the SpeechT5 tokenizer doesn't have any tokens for numbers. In `normalized_text` the numbers are written 
out as text. Thus, it is a better fit, and we recommend using    `normalized_text` as input text.

Because SpeechT5 was trained on the English language, it may not recognize certain characters in the Dutch dataset. If 
left as is, these characters will be converted to `<unk>` tokens. However, in Dutch, certain characters like `à` are 
used to stress syllables. In order to preserve the meaning of the text, we can replace this character with a regular `a`.

To identify unsupported tokens, extract all unique characters in the dataset using the `SpeechT5Tokenizer` which 
works with characters as tokens. To do this, write the `extract_all_chars` mapping function that concatenates 
the transcriptions from all examples into one string and converts it to a set of characters. 
Make sure to set `batched=True` and `batch_size=-1` in `dataset.map()` so that all transcriptions are available at once for 
the mapping function.

```py
>>> def extract_all_chars(batch):
...     all_text = " ".join(batch["normalized_text"])
...     vocab = list(set(all_text))
...     return {"vocab": [vocab], "all_text": [all_text]}


key: _node_content
string_value:
{"id_": "cdca21eb-9442-497d-9d8e-5ab3537321b2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "hash": "7fef769416a8a51ead2ae01a92eef274cf124d5c7c49a1c12621ab470154a9e4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dfe03b92-2654-4108-a98b-5f4ca21e24fc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "hash": "88428cf5e78b65c092bce9f6411e82d068d95008713be59396339d5d83efd785", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bad2910b-1e5c-4352-92ff-ff885cc5ccc4", "node_type": "1", "metadata": {}, "hash": "df5c17246e6d4b73f5930a815a076e38794c9cb847de788868e2055f2c964460", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4587, "end_char_idx": 7394, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You'll need the tokenizer part of the processor to process the text:

```py
>>> tokenizer = processor.tokenizer
```

The dataset examples contain `raw_text` and `normalized_text` features. When deciding which feature to use as the text input, 
consider that the SpeechT5 tokenizer doesn't have any tokens for numbers. In `normalized_text` the numbers are written 
out as text. Thus, it is a better fit, and we recommend using    `normalized_text` as input text.

Because SpeechT5 was trained on the English language, it may not recognize certain characters in the Dutch dataset. If 
left as is, these characters will be converted to `<unk>` tokens. However, in Dutch, certain characters like `à` are 
used to stress syllables. In order to preserve the meaning of the text, we can replace this character with a regular `a`.

To identify unsupported tokens, extract all unique characters in the dataset using the `SpeechT5Tokenizer` which 
works with characters as tokens. To do this, write the `extract_all_chars` mapping function that concatenates 
the transcriptions from all examples into one string and converts it to a set of characters. 
Make sure to set `batched=True` and `batch_size=-1` in `dataset.map()` so that all transcriptions are available at once for 
the mapping function.

```py
>>> def extract_all_chars(batch):
...     all_text = " ".join(batch["normalized_text"])
...     vocab = list(set(all_text))
...     return {"vocab": [vocab], "all_text": [all_text]}


>>> vocabs = dataset.map(
...     extract_all_chars,
...     batched=True,
...     batch_size=-1,
...     keep_in_memory=True,
...     remove_columns=dataset.column_names,
... )

>>> dataset_vocab = set(vocabs["vocab"][0])
>>> tokenizer_vocab = {k for k, _ in tokenizer.get_vocab().items()}
```

Now you have two sets of characters: one with the vocabulary from the dataset and one with the vocabulary from the tokenizer. 
To identify any unsupported characters in the dataset, you can take the difference between these two sets. The resulting 
set will contain the characters that are in the dataset but not in the tokenizer.

```py
>>> dataset_vocab - tokenizer_vocab
{' ', 'à', 'ç', 'è', 'ë', 'í', 'ï', 'ö', 'ü'}
```

To handle the unsupported characters identified in the previous step, define a function that maps these characters to 
valid tokens. Note that spaces are already replaced by `▁` in the tokenizer and don't need to be handled separately.

```py
>>> replacements = [
...     ("à", "a"),
...     ("ç", "c"),
...     ("è", "e"),
...     ("ë", "e"),
...     ("í", "i"),
...     ("ï", "i"),
...     ("ö", "o"),
...     ("ü", "u"),
... ]


>>> def cleanup_text(inputs):
...     for src, dst in replacements:
...         inputs["normalized_text"] = inputs["normalized_text"].replace(src, dst)
...     return inputs


key: _node_content
string_value:
{"id_": "bad2910b-1e5c-4352-92ff-ff885cc5ccc4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "hash": "7fef769416a8a51ead2ae01a92eef274cf124d5c7c49a1c12621ab470154a9e4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cdca21eb-9442-497d-9d8e-5ab3537321b2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "hash": "15c61a2b000ffa1ba426ff610012551e8bb868363f578af5f6d937bcf354cf3c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "83684c47-7456-4491-a4b6-ff64bb63559c", "node_type": "1", "metadata": {}, "hash": "b849a9bd0545b38b87205d48571d0ccd2ff607c4e57c73a19465e6a35ccf97d2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5878, "end_char_idx": 9018, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> def extract_all_chars(batch):
...     all_text = " ".join(batch["normalized_text"])
...     vocab = list(set(all_text))
...     return {"vocab": [vocab], "all_text": [all_text]}


>>> vocabs = dataset.map(
...     extract_all_chars,
...     batched=True,
...     batch_size=-1,
...     keep_in_memory=True,
...     remove_columns=dataset.column_names,
... )

>>> dataset_vocab = set(vocabs["vocab"][0])
>>> tokenizer_vocab = {k for k, _ in tokenizer.get_vocab().items()}
```

Now you have two sets of characters: one with the vocabulary from the dataset and one with the vocabulary from the tokenizer. 
To identify any unsupported characters in the dataset, you can take the difference between these two sets. The resulting 
set will contain the characters that are in the dataset but not in the tokenizer.

```py
>>> dataset_vocab - tokenizer_vocab
{' ', 'à', 'ç', 'è', 'ë', 'í', 'ï', 'ö', 'ü'}
```

To handle the unsupported characters identified in the previous step, define a function that maps these characters to 
valid tokens. Note that spaces are already replaced by `▁` in the tokenizer and don't need to be handled separately.

```py
>>> replacements = [
...     ("à", "a"),
...     ("ç", "c"),
...     ("è", "e"),
...     ("ë", "e"),
...     ("í", "i"),
...     ("ï", "i"),
...     ("ö", "o"),
...     ("ü", "u"),
... ]


>>> def cleanup_text(inputs):
...     for src, dst in replacements:
...         inputs["normalized_text"] = inputs["normalized_text"].replace(src, dst)
...     return inputs


>>> dataset = dataset.map(cleanup_text)
```

Now that you have dealt with special characters in the text, it's time to shift focus to the audio data.

### Speakers

The VoxPopuli dataset includes speech from multiple speakers, but how many speakers are represented in the dataset? To 
determine this, we can count the number of unique speakers and the number of examples each speaker contributes to the dataset. 
With a total of 20,968 examples in the dataset, this information will give us a better understanding of the distribution of 
speakers and examples in the data.

```py
>>> from collections import defaultdict

>>> speaker_counts = defaultdict(int)

>>> for speaker_id in dataset["speaker_id"]:
...     speaker_counts[speaker_id] += 1
```

By plotting a histogram you can get a sense of how much data there is for each speaker.

```py
>>> import matplotlib.pyplot as plt

>>> plt.figure()
>>> plt.hist(speaker_counts.values(), bins=20)
>>> plt.ylabel("Speakers")
>>> plt.xlabel("Examples")
>>> plt.show()
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/tts_speakers_histogram.png" alt="Speakers histogram"/>
</div>

The histogram reveals that approximately one-third of the speakers in the dataset have fewer than 100 examples, while 
around ten speakers have more than 500 examples. To improve training efficiency and balance the dataset, we can limit 
the data to speakers with between 100 and 400 examples. 

```py
>>> def select_speaker(speaker_id):
...     return 100 <= speaker_counts[speaker_id] <= 400


key: _node_content
string_value:
{"id_": "83684c47-7456-4491-a4b6-ff64bb63559c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "hash": "7fef769416a8a51ead2ae01a92eef274cf124d5c7c49a1c12621ab470154a9e4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bad2910b-1e5c-4352-92ff-ff885cc5ccc4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "hash": "3e9ec4f60e4c45577a53748098d3df766ad5a75a4d5e739f5233a3483466ab11", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cb8e7ec0-d00e-411b-83da-f2893736d467", "node_type": "1", "metadata": {}, "hash": "38a1167684bc6a98f1a6372a0d6d33d4f097430ab06153c627b7c2a6a4d3ac68", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7221, "end_char_idx": 10739, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> def cleanup_text(inputs):
...     for src, dst in replacements:
...         inputs["normalized_text"] = inputs["normalized_text"].replace(src, dst)
...     return inputs


>>> dataset = dataset.map(cleanup_text)
```

Now that you have dealt with special characters in the text, it's time to shift focus to the audio data.

### Speakers

The VoxPopuli dataset includes speech from multiple speakers, but how many speakers are represented in the dataset? To 
determine this, we can count the number of unique speakers and the number of examples each speaker contributes to the dataset. 
With a total of 20,968 examples in the dataset, this information will give us a better understanding of the distribution of 
speakers and examples in the data.

```py
>>> from collections import defaultdict

>>> speaker_counts = defaultdict(int)

>>> for speaker_id in dataset["speaker_id"]:
...     speaker_counts[speaker_id] += 1
```

By plotting a histogram you can get a sense of how much data there is for each speaker.

```py
>>> import matplotlib.pyplot as plt

>>> plt.figure()
>>> plt.hist(speaker_counts.values(), bins=20)
>>> plt.ylabel("Speakers")
>>> plt.xlabel("Examples")
>>> plt.show()
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/tts_speakers_histogram.png" alt="Speakers histogram"/>
</div>

The histogram reveals that approximately one-third of the speakers in the dataset have fewer than 100 examples, while 
around ten speakers have more than 500 examples. To improve training efficiency and balance the dataset, we can limit 
the data to speakers with between 100 and 400 examples. 

```py
>>> def select_speaker(speaker_id):
...     return 100 <= speaker_counts[speaker_id] <= 400


>>> dataset = dataset.filter(select_speaker, input_columns=["speaker_id"])
```

Let's check how many speakers remain: 

```py
>>> len(set(dataset["speaker_id"]))
42
```

Let's see how many examples are left: 

```py
>>> len(dataset)
9973
```

You are left with just under 10,000 examples from approximately 40 unique speakers, which should be sufficient.

Note that some speakers with few examples may actually have more audio available if the examples are long. However, 
determining the total amount of audio for each speaker requires scanning through the entire dataset, which is a 
time-consuming process that involves loading and decoding each audio file. As such, we have chosen to skip this step here.

### Speaker embeddings

To enable the TTS model to differentiate between multiple speakers, you'll need to create a speaker embedding for each example. 
The speaker embedding is an additional input into the model that captures a particular speaker's voice characteristics.
To generate these speaker embeddings, use the pre-trained [spkrec-xvect-voxceleb](https://huggingface.co/speechbrain/spkrec-xvect-voxceleb) 
model from SpeechBrain. 

Create a function `create_speaker_embedding()` that takes an input audio waveform and outputs a 512-element vector 
containing the corresponding speaker embedding.

```py
>>> import os
>>> import torch
>>> from speechbrain.inference.classifiers import EncoderClassifier

>>> spk_model_name = "speechbrain/spkrec-xvect-voxceleb"

>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> speaker_model = EncoderClassifier.from_hparams(
...     source=spk_model_name,
...     run_opts={"device": device},
...     savedir=os.path.join("/tmp", spk_model_name),
... )


key: _node_content
string_value:
{"id_": "cb8e7ec0-d00e-411b-83da-f2893736d467", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "hash": "7fef769416a8a51ead2ae01a92eef274cf124d5c7c49a1c12621ab470154a9e4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "83684c47-7456-4491-a4b6-ff64bb63559c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "hash": "e6b7cbcc51065520bc0c2063ca7a90ee4e6e73dc8bda40f686b05378a17fa495", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "79178a0a-9f15-46fa-8756-4e219353d2c2", "node_type": "1", "metadata": {}, "hash": "6d7add132ba9f351d722e730eb5f831fe0f00af5b91003ee0ddb261a63dc5dd2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9021, "end_char_idx": 14224, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> dataset = dataset.filter(select_speaker, input_columns=["speaker_id"])
```

Let's check how many speakers remain: 

```py
>>> len(set(dataset["speaker_id"]))
42
```

Let's see how many examples are left: 

```py
>>> len(dataset)
9973
```

You are left with just under 10,000 examples from approximately 40 unique speakers, which should be sufficient.

Note that some speakers with few examples may actually have more audio available if the examples are long. However, 
determining the total amount of audio for each speaker requires scanning through the entire dataset, which is a 
time-consuming process that involves loading and decoding each audio file. As such, we have chosen to skip this step here.

### Speaker embeddings

To enable the TTS model to differentiate between multiple speakers, you'll need to create a speaker embedding for each example. 
The speaker embedding is an additional input into the model that captures a particular speaker's voice characteristics.
To generate these speaker embeddings, use the pre-trained [spkrec-xvect-voxceleb](https://huggingface.co/speechbrain/spkrec-xvect-voxceleb) 
model from SpeechBrain. 

Create a function `create_speaker_embedding()` that takes an input audio waveform and outputs a 512-element vector 
containing the corresponding speaker embedding.

```py
>>> import os
>>> import torch
>>> from speechbrain.inference.classifiers import EncoderClassifier

>>> spk_model_name = "speechbrain/spkrec-xvect-voxceleb"

>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> speaker_model = EncoderClassifier.from_hparams(
...     source=spk_model_name,
...     run_opts={"device": device},
...     savedir=os.path.join("/tmp", spk_model_name),
... )


>>> def create_speaker_embedding(waveform):
...     with torch.no_grad():
...         speaker_embeddings = speaker_model.encode_batch(torch.tensor(waveform))
...         speaker_embeddings = torch.nn.functional.normalize(speaker_embeddings, dim=2)
...         speaker_embeddings = speaker_embeddings.squeeze().cpu().numpy()
...     return speaker_embeddings
```

It's important to note that the `speechbrain/spkrec-xvect-voxceleb` model was trained on English speech from the VoxCeleb 
dataset, whereas the training examples in this guide are in Dutch. While we believe that this model will still generate 
reasonable speaker embeddings for our Dutch dataset, this assumption may not hold true in all cases.

For optimal results, we recommend training an X-vector model on the target speech first. This will ensure that the model 
is better able to capture the unique voice characteristics present in the Dutch language.

### Processing the dataset

Finally, let's process the data into the format the model expects. Create a `prepare_dataset` function that takes in a 
single example and uses the `SpeechT5Processor` object to tokenize the input text and load the target audio into a log-mel spectrogram. 
It should also add the speaker embeddings as an additional input.

```py
>>> def prepare_dataset(example):
...     audio = example["audio"]

...     example = processor(
...         text=example["normalized_text"],
...         audio_target=audio["array"],
...         sampling_rate=audio["sampling_rate"],
...         return_attention_mask=False,
...     )

...     # strip off the batch dimension
...     example["labels"] = example["labels"][0]

...     # use SpeechBrain to obtain x-vector
...     example["speaker_embeddings"] = create_speaker_embedding(audio["array"])

...     return example
```

Verify the processing is correct by looking at a single example:

```py
>>> processed_example = prepare_dataset(dataset[0])
>>> list(processed_example.keys())
['input_ids', 'labels', 'stop_labels', 'speaker_embeddings']
```

Speaker embeddings should be a 512-element vector:

```py
>>> processed_example["speaker_embeddings"].shape
(512,)
```

The labels should be a log-mel spectrogram with 80 mel bins.

```py
>>> import matplotlib.pyplot as plt

>>> plt.figure()
>>> plt.imshow(processed_example["labels"].T)
>>> plt.show()
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/tts_logmelspectrogram_1.png" alt="Log-mel spectrogram with 80 mel bins"/>
</div>

Side note: If you find this spectrogram confusing, it may be due to your familiarity with the convention of placing low frequencies 
at the bottom and high frequencies at the top of a plot. However, when plotting spectrograms as an image using the matplotlib library, 
the y-axis is flipped and the spectrograms appear upside down.

Now apply the processing function to the entire dataset. This will take between 5 and 10 minutes.

```py
>>> dataset = dataset.map(prepare_dataset, remove_columns=dataset.column_names)
```

You'll see a warning saying that some examples in the dataset are longer than the maximum input length the model can handle (600 tokens). 
Remove those examples from the dataset. Here we go even further and to allow for larger batch sizes we remove anything over 200 tokens.

```py
>>> def is_not_too_long(input_ids):
...     input_length = len(input_ids)
...     return input_length < 200


key: _node_content
string_value:
{"id_": "79178a0a-9f15-46fa-8756-4e219353d2c2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "hash": "7fef769416a8a51ead2ae01a92eef274cf124d5c7c49a1c12621ab470154a9e4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cb8e7ec0-d00e-411b-83da-f2893736d467", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "hash": "6e9f00a82ee112b84c656d1c6a790f931b586c2f2d0fc186522eba9194411834", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ecf177ec-4e67-43dd-bd5e-6fae91e745fb", "node_type": "1", "metadata": {}, "hash": "c714b49e06cad6574270544a8f89ccec3835287262f9c07bdc2670db40de3f8e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14227, "end_char_idx": 17470, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> dataset = dataset.filter(is_not_too_long, input_columns=["input_ids"])
>>> len(dataset)
8259
```

Next, create a basic train/test split: 

```py
>>> dataset = dataset.train_test_split(test_size=0.1)
```

### Data collator

In order to combine multiple examples into a batch, you need to define a custom data collator. This collator will pad shorter sequences with padding 
tokens, ensuring that all examples have the same length. For the spectrogram labels, the padded portions are replaced with the special value `-100`. This special value 
instructs the model to ignore that part of the spectrogram when calculating the spectrogram loss.

```py
>>> from dataclasses import dataclass
>>> from typing import Any, Dict, List, Union


>>> @dataclass
... class TTSDataCollatorWithPadding:
...     processor: Any

...     def __call__(self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]:
...         input_ids = [{"input_ids": feature["input_ids"]} for feature in features]
...         label_features = [{"input_values": feature["labels"]} for feature in features]
...         speaker_features = [feature["speaker_embeddings"] for feature in features]

...         # collate the inputs and targets into a batch
...         batch = processor.pad(input_ids=input_ids, labels=label_features, return_tensors="pt")

...         # replace padding with -100 to ignore loss correctly
...         batch["labels"] = batch["labels"].masked_fill(batch.decoder_attention_mask.unsqueeze(-1).ne(1), -100)

...         # not used during fine-tuning
...         del batch["decoder_attention_mask"]

...         # round down target lengths to multiple of reduction factor
...         if model.config.reduction_factor > 1:
...             target_lengths = torch.tensor([len(feature["input_values"]) for feature in label_features])
...             target_lengths = target_lengths.new(
...                 [length - length % model.config.reduction_factor for length in target_lengths]
...             )
...             max_length = max(target_lengths)
...             batch["labels"] = batch["labels"][:, :max_length]

...         # also add in the speaker embeddings
...         batch["speaker_embeddings"] = torch.tensor(speaker_features)

...         return batch
```

In SpeechT5, the input to the decoder part of the model is reduced by a factor 2. In other words, it throws away every 
other timestep from the target sequence. The decoder then predicts a sequence that is twice as long. Since the original 
target sequence length may be odd, the data collator makes sure to round the maximum length of the batch down to be a 
multiple of 2.

```py 
>>> data_collator = TTSDataCollatorWithPadding(processor=processor)
```

## Train the model

Load the pre-trained model from the same checkpoint as you used for loading the processor: 

```py
>>> from transformers import SpeechT5ForTextToSpeech

>>> model = SpeechT5ForTextToSpeech.from_pretrained(checkpoint)
```

The `use_cache=True` option is incompatible with gradient checkpointing. Disable it for training.

```py 
>>> model.config.use_cache = False
```

Define the training arguments. Here we are not computing any evaluation metrics during the training process.


key: _node_content
string_value:
{"id_": "ecf177ec-4e67-43dd-bd5e-6fae91e745fb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "hash": "7fef769416a8a51ead2ae01a92eef274cf124d5c7c49a1c12621ab470154a9e4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "79178a0a-9f15-46fa-8756-4e219353d2c2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "hash": "218b91c79768474f2e20cedc693321cac6573e563880a83c2b520a085d4b7701", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "35786f47-44d4-4221-b79c-779d39eaeebe", "node_type": "1", "metadata": {}, "hash": "da446ccbb7397f67d7d8f81ed1881539faf70f049a624183443dc9fd16dcc40c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16600, "end_char_idx": 19737, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In other words, it throws away every 
other timestep from the target sequence. The decoder then predicts a sequence that is twice as long. Since the original 
target sequence length may be odd, the data collator makes sure to round the maximum length of the batch down to be a 
multiple of 2.

```py 
>>> data_collator = TTSDataCollatorWithPadding(processor=processor)
```

## Train the model

Load the pre-trained model from the same checkpoint as you used for loading the processor: 

```py
>>> from transformers import SpeechT5ForTextToSpeech

>>> model = SpeechT5ForTextToSpeech.from_pretrained(checkpoint)
```

The `use_cache=True` option is incompatible with gradient checkpointing. Disable it for training.

```py 
>>> model.config.use_cache = False
```

Define the training arguments. Here we are not computing any evaluation metrics during the training process. Instead, we'll 
only look at the loss:

```python
>>> from transformers import Seq2SeqTrainingArguments

>>> training_args = Seq2SeqTrainingArguments(
...     output_dir="speecht5_finetuned_voxpopuli_nl",  # change to a repo name of your choice
...     per_device_train_batch_size=4,
...     gradient_accumulation_steps=8,
...     learning_rate=1e-5,
...     warmup_steps=500,
...     max_steps=4000,
...     gradient_checkpointing=True,
...     fp16=True,
...     eval_strategy="steps",
...     per_device_eval_batch_size=2,
...     save_steps=1000,
...     eval_steps=1000,
...     logging_steps=25,
...     report_to=["tensorboard"],
...     load_best_model_at_end=True,
...     greater_is_better=False,
...     label_names=["labels"],
...     push_to_hub=True,
... )
```

Instantiate the `Trainer` object  and pass the model, dataset, and data collator to it.

```py
>>> from transformers import Seq2SeqTrainer

>>> trainer = Seq2SeqTrainer(
...     args=training_args,
...     model=model,
...     train_dataset=dataset["train"],
...     eval_dataset=dataset["test"],
...     data_collator=data_collator,
...     tokenizer=processor,
... )
```

And with that, you're ready to start training! Training will take several hours. Depending on your GPU, 
it is possible that you will encounter a CUDA "out-of-memory" error when you start training. In this case, you can reduce 
the `per_device_train_batch_size` incrementally by factors of 2 and increase `gradient_accumulation_steps` by 2x to compensate.

```py
>>> trainer.train()
```

To be able to use your checkpoint with a pipeline, make sure to save the processor with the checkpoint: 

```py
>>> processor.save_pretrained("YOUR_ACCOUNT_NAME/speecht5_finetuned_voxpopuli_nl")
```

Push the final model to the 🤗 Hub:

```py
>>> trainer.push_to_hub()
```

## Inference

### Inference with a pipeline

Great, now that you've fine-tuned a model, you can use it for inference!
First, let's see how you can use it with a corresponding pipeline. Let's create a `"text-to-speech"` pipeline with your 
checkpoint: 

```py
>>> from transformers import pipeline

>>> pipe = pipeline("text-to-speech", model="YOUR_ACCOUNT_NAME/speecht5_finetuned_voxpopuli_nl")
```

Pick a piece of text in Dutch you'd like narrated, e.g.


key: _node_content
string_value:
{"id_": "35786f47-44d4-4221-b79c-779d39eaeebe", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "hash": "7fef769416a8a51ead2ae01a92eef274cf124d5c7c49a1c12621ab470154a9e4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ecf177ec-4e67-43dd-bd5e-6fae91e745fb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "hash": "4eef41cc8002cbd1890e4aa417c64447ee5778220c2870fcaa9b4e221dbe5f7f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "440b5a6b-a4ca-4606-8ca0-42cf02ce7b24", "node_type": "1", "metadata": {}, "hash": "2da01f93b02a5634f12f7ec1a450faf61ee8d8c3bc9baa9f3cc8b1506004a58c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18336, "end_char_idx": 21232, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> from transformers import Seq2SeqTrainer

>>> trainer = Seq2SeqTrainer(
...     args=training_args,
...     model=model,
...     train_dataset=dataset["train"],
...     eval_dataset=dataset["test"],
...     data_collator=data_collator,
...     tokenizer=processor,
... )
```

And with that, you're ready to start training! Training will take several hours. Depending on your GPU, 
it is possible that you will encounter a CUDA "out-of-memory" error when you start training. In this case, you can reduce 
the `per_device_train_batch_size` incrementally by factors of 2 and increase `gradient_accumulation_steps` by 2x to compensate.

```py
>>> trainer.train()
```

To be able to use your checkpoint with a pipeline, make sure to save the processor with the checkpoint: 

```py
>>> processor.save_pretrained("YOUR_ACCOUNT_NAME/speecht5_finetuned_voxpopuli_nl")
```

Push the final model to the 🤗 Hub:

```py
>>> trainer.push_to_hub()
```

## Inference

### Inference with a pipeline

Great, now that you've fine-tuned a model, you can use it for inference!
First, let's see how you can use it with a corresponding pipeline. Let's create a `"text-to-speech"` pipeline with your 
checkpoint: 

```py
>>> from transformers import pipeline

>>> pipe = pipeline("text-to-speech", model="YOUR_ACCOUNT_NAME/speecht5_finetuned_voxpopuli_nl")
```

Pick a piece of text in Dutch you'd like narrated, e.g.:

```py
>>> text = "hallo allemaal, ik praat nederlands. groetjes aan iedereen!"
```

To use SpeechT5 with the pipeline, you'll need a speaker embedding. Let's get it from an example in the test dataset: 

```py
>>> example = dataset["test"][304]
>>> speaker_embeddings = torch.tensor(example["speaker_embeddings"]).unsqueeze(0)
```

Now you can pass the text and speaker embeddings to the pipeline, and it will take care of the rest: 

```py
>>> forward_params = {"speaker_embeddings": speaker_embeddings}
>>> output = pipe(text, forward_params=forward_params)
>>> output
{'audio': array([-6.82714235e-05, -4.26525949e-04,  1.06134125e-04, ...,
        -1.22392643e-03, -7.76011671e-04,  3.29112721e-04], dtype=float32),
 'sampling_rate': 16000}
```

You can then listen to the result:

```py
>>> from IPython.display import Audio
>>> Audio(output['audio'], rate=output['sampling_rate']) 
```

### Run inference manually

You can achieve the same inference results without using the pipeline, however, more steps will be required. 

Load the model from the 🤗 Hub: 

```py
>>> model = SpeechT5ForTextToSpeech.from_pretrained("YOUR_ACCOUNT/speecht5_finetuned_voxpopuli_nl")
```

Pick an example from the test dataset obtain a speaker embedding. 

```py 
>>> example = dataset["test"][304]
>>> speaker_embeddings = torch.tensor(example["speaker_embeddings"]).unsqueeze(0)
```

Define the input text and tokenize it.

```py 
>>> text = "hallo allemaal, ik praat nederlands. groetjes aan iedereen!"


key: _node_content
string_value:
{"id_": "440b5a6b-a4ca-4606-8ca0-42cf02ce7b24", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "60f2b7d1-bb2c-4be0-af8a-b1061d12cb6e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "hash": "7fef769416a8a51ead2ae01a92eef274cf124d5c7c49a1c12621ab470154a9e4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "35786f47-44d4-4221-b79c-779d39eaeebe", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "transformers"}, "hash": "dab92c96aabf87d48593079560229fbcb0180c167a65f1c6aa109126ef40da7c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19795, "end_char_idx": 23025, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
groetjes aan iedereen!"
```

To use SpeechT5 with the pipeline, you'll need a speaker embedding. Let's get it from an example in the test dataset: 

```py
>>> example = dataset["test"][304]
>>> speaker_embeddings = torch.tensor(example["speaker_embeddings"]).unsqueeze(0)
```

Now you can pass the text and speaker embeddings to the pipeline, and it will take care of the rest: 

```py
>>> forward_params = {"speaker_embeddings": speaker_embeddings}
>>> output = pipe(text, forward_params=forward_params)
>>> output
{'audio': array([-6.82714235e-05, -4.26525949e-04,  1.06134125e-04, ...,
        -1.22392643e-03, -7.76011671e-04,  3.29112721e-04], dtype=float32),
 'sampling_rate': 16000}
```

You can then listen to the result:

```py
>>> from IPython.display import Audio
>>> Audio(output['audio'], rate=output['sampling_rate']) 
```

### Run inference manually

You can achieve the same inference results without using the pipeline, however, more steps will be required. 

Load the model from the 🤗 Hub: 

```py
>>> model = SpeechT5ForTextToSpeech.from_pretrained("YOUR_ACCOUNT/speecht5_finetuned_voxpopuli_nl")
```

Pick an example from the test dataset obtain a speaker embedding. 

```py 
>>> example = dataset["test"][304]
>>> speaker_embeddings = torch.tensor(example["speaker_embeddings"]).unsqueeze(0)
```

Define the input text and tokenize it.

```py 
>>> text = "hallo allemaal, ik praat nederlands. groetjes aan iedereen!"
>>> inputs = processor(text=text, return_tensors="pt")
```

Create a spectrogram with your model: 

```py
>>> spectrogram = model.generate_speech(inputs["input_ids"], speaker_embeddings)
```

Visualize the spectrogram, if you'd like to: 

```py
>>> plt.figure()
>>> plt.imshow(spectrogram.T)
>>> plt.show()
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/tts_logmelspectrogram_2.png" alt="Generated log-mel spectrogram"/>
</div>

Finally, use the vocoder to turn the spectrogram into sound.

```py
>>> with torch.no_grad():
...     speech = vocoder(spectrogram)

>>> from IPython.display import Audio

>>> Audio(speech.numpy(), rate=16000)
```

In our experience, obtaining satisfactory results from this model can be challenging. The quality of the speaker 
embeddings appears to be a significant factor. Since SpeechT5 was pre-trained with English x-vectors, it performs best 
when using English speaker embeddings. If the synthesized speech sounds poor, try using a different speaker embedding.

Increasing the training duration is also likely to enhance the quality of the results. Even so, the speech clearly is Dutch instead of English, and it does 
capture the voice characteristics of the speaker (compare to the original audio in the example).
Another thing to experiment with is the model's configuration. For example, try using `config.reduction_factor = 1` to 
see if this improves the results.

Finally, it is essential to consider ethical considerations. Although TTS technology has numerous useful applications, it 
may also be used for malicious purposes, such as impersonating someone's voice without their knowledge or consent. Please 
use TTS judiciously and responsibly.


key: _node_content
string_value:
{"id_": "6a06cac1-23aa-4b68-9b70-68cf66ee9114", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers"}, "hash": "ba7d015d1f001e84999554903bbcaa4d8191bf47f199a4ab3f4bfff5b10daec4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3a895197-5dcd-4588-bddf-2fb54247d648", "node_type": "1", "metadata": {}, "hash": "e46b03a0acd26943c7b5ee99ae79944766ced380d41b1ea730447865ce28ebb6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3010, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Audio classification

[[open-in-colab]]

<Youtube id="KWwzcmG98Ds"/>

Audio classification - just like with text - assigns a class label output from the input data. The only difference is instead of text inputs, you have raw audio waveforms. Some practical applications of audio classification include identifying speaker intent, language classification, and even animal species by their sounds.

This guide will show you how to:

1. Finetune [Wav2Vec2](https://huggingface.co/facebook/wav2vec2-base) on the [MInDS-14](https://huggingface.co/datasets/PolyAI/minds14) dataset to classify speaker intent.
2. Use your finetuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/audio-classification)

</Tip>

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install transformers datasets evaluate
```

We encourage you to login to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to login:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load MInDS-14 dataset

Start by loading the MInDS-14 dataset from the 🤗 Datasets library:

```py
>>> from datasets import load_dataset, Audio

>>> minds = load_dataset("PolyAI/minds14", name="en-US", split="train")
```

Split the dataset's `train` split into a smaller train and test set with the [`~datasets.Dataset.train_test_split`] method. This'll give you a chance to experiment and make sure everything works before spending more time on the full dataset.

```py
>>> minds = minds.train_test_split(test_size=0.2)
```

Then take a look at the dataset:

```py
>>> minds
DatasetDict({
    train: Dataset({
        features: ['path', 'audio', 'transcription', 'english_transcription', 'intent_class', 'lang_id'],
        num_rows: 450
    })
    test: Dataset({
        features: ['path', 'audio', 'transcription', 'english_transcription', 'intent_class', 'lang_id'],
        num_rows: 113
    })
})
```

While the dataset contains a lot of useful information, like `lang_id` and `english_transcription`, you'll focus on the `audio` and `intent_class` in this guide. Remove the other columns with the [`~datasets.Dataset.remove_columns`] method:

```py
>>> minds = minds.remove_columns(["path", "transcription", "english_transcription", "lang_id"])
```

Take a look at an example now:

```py
>>> minds["train"][0]
{'audio': {'array': array([ 0.        ,  0.        ,  0.        , ..., -0.00048828,
         -0.00024414, -0.00024414], dtype=float32),
  'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602b9a5fbb1e6d0fbce91f52.wav',
  'sampling_rate': 8000},
 'intent_class': 2}
```

There are two fields:

- `audio`: a 1-dimensional `array` of the speech signal that must be called to load and resample the audio file.


key: _node_content
string_value:
{"id_": "3a895197-5dcd-4588-bddf-2fb54247d648", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers"}, "hash": "ba7d015d1f001e84999554903bbcaa4d8191bf47f199a4ab3f4bfff5b10daec4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6a06cac1-23aa-4b68-9b70-68cf66ee9114", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers"}, "hash": "9012919719e644386bfcdb103b81761976f5e0adf10467d55af8d79db92e51a3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3958c7bf-8966-4308-b289-19b53e5f06eb", "node_type": "1", "metadata": {}, "hash": "1d65f8041fce8b6124dfeb4287efad2a57fe5b8e88527f66f8d2509ca1d50682", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1658, "end_char_idx": 3078, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> minds = minds.train_test_split(test_size=0.2)
```

Then take a look at the dataset:

```py
>>> minds
DatasetDict({
    train: Dataset({
        features: ['path', 'audio', 'transcription', 'english_transcription', 'intent_class', 'lang_id'],
        num_rows: 450
    })
    test: Dataset({
        features: ['path', 'audio', 'transcription', 'english_transcription', 'intent_class', 'lang_id'],
        num_rows: 113
    })
})
```

While the dataset contains a lot of useful information, like `lang_id` and `english_transcription`, you'll focus on the `audio` and `intent_class` in this guide. Remove the other columns with the [`~datasets.Dataset.remove_columns`] method:

```py
>>> minds = minds.remove_columns(["path", "transcription", "english_transcription", "lang_id"])
```

Take a look at an example now:

```py
>>> minds["train"][0]
{'audio': {'array': array([ 0.        ,  0.        ,  0.        , ..., -0.00048828,
         -0.00024414, -0.00024414], dtype=float32),
  'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602b9a5fbb1e6d0fbce91f52.wav',
  'sampling_rate': 8000},
 'intent_class': 2}
```

There are two fields:

- `audio`: a 1-dimensional `array` of the speech signal that must be called to load and resample the audio file. 
- `intent_class`: represents the class id of the speaker's intent.


key: _node_content
string_value:
{"id_": "3958c7bf-8966-4308-b289-19b53e5f06eb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers"}, "hash": "ba7d015d1f001e84999554903bbcaa4d8191bf47f199a4ab3f4bfff5b10daec4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3a895197-5dcd-4588-bddf-2fb54247d648", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers"}, "hash": "a284fdbe04ef2afa46509f4815762a7e5dadd2ce37430838cf0df89de9a19a1f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2ce9b706-d38f-4b7f-961e-35493c694121", "node_type": "1", "metadata": {}, "hash": "232f4c0308f321a3cfb0f11a06934bda667361a969991ac8c85f04d165c5f287", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2264, "end_char_idx": 4766, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Remove the other columns with the [`~datasets.Dataset.remove_columns`] method:

```py
>>> minds = minds.remove_columns(["path", "transcription", "english_transcription", "lang_id"])
```

Take a look at an example now:

```py
>>> minds["train"][0]
{'audio': {'array': array([ 0.        ,  0.        ,  0.        , ..., -0.00048828,
         -0.00024414, -0.00024414], dtype=float32),
  'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602b9a5fbb1e6d0fbce91f52.wav',
  'sampling_rate': 8000},
 'intent_class': 2}
```

There are two fields:

- `audio`: a 1-dimensional `array` of the speech signal that must be called to load and resample the audio file. 
- `intent_class`: represents the class id of the speaker's intent. 

To make it easier for the model to get the label name from the label id, create a dictionary that maps the label name to an integer and vice versa:

```py
>>> labels = minds["train"].features["intent_class"].names
>>> label2id, id2label = dict(), dict()
>>> for i, label in enumerate(labels):
...     label2id[label] = str(i)
...     id2label[str(i)] = label
```

Now you can convert the label id to a label name:

```py
>>> id2label[str(2)]
'app_error'
```

## Preprocess

The next step is to load a Wav2Vec2 feature extractor to process the audio signal:

```py
>>> from transformers import AutoFeatureExtractor

>>> feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base")
```

The MInDS-14 dataset has a sampling rate of 8000khz (you can find this information in it's [dataset card](https://huggingface.co/datasets/PolyAI/minds14)), which means you'll need to resample the dataset to 16000kHz to use the pretrained Wav2Vec2 model:

```py
>>> minds = minds.cast_column("audio", Audio(sampling_rate=16_000))
>>> minds["train"][0]
{'audio': {'array': array([ 2.2098757e-05,  4.6582241e-05, -2.2803260e-05, ...,
         -2.8419291e-04, -2.3305941e-04, -1.1425107e-04], dtype=float32),
  'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602b9a5fbb1e6d0fbce91f52.wav',
  'sampling_rate': 16000},
 'intent_class': 2}
```

Now create a preprocessing function that:

1. Calls the `audio` column to load, and if necessary, resample the audio file.
2. Checks if the sampling rate of the audio file matches the sampling rate of the audio data a model was pretrained with.


key: _node_content
string_value:
{"id_": "2ce9b706-d38f-4b7f-961e-35493c694121", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers"}, "hash": "ba7d015d1f001e84999554903bbcaa4d8191bf47f199a4ab3f4bfff5b10daec4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3958c7bf-8966-4308-b289-19b53e5f06eb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers"}, "hash": "4b30dc3357cb80641981e368f2e7882333e621b4c48297bbeae7b550d6b88fc7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a2205ebb-3ca1-4161-a2f3-1af1bfabe826", "node_type": "1", "metadata": {}, "hash": "2363388941310bed7533ab41df23e616f83761ce5ee0c6254ac038e9a9852316", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4568, "end_char_idx": 6438, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Calls the `audio` column to load, and if necessary, resample the audio file.
2. Checks if the sampling rate of the audio file matches the sampling rate of the audio data a model was pretrained with. You can find this information in the Wav2Vec2 [model card](https://huggingface.co/facebook/wav2vec2-base).
3. Set a maximum input length to batch longer inputs without truncating them.

```py
>>> def preprocess_function(examples):
...     audio_arrays = [x["array"] for x in examples["audio"]]
...     inputs = feature_extractor(
...         audio_arrays, sampling_rate=feature_extractor.sampling_rate, max_length=16000, truncation=True
...     )
...     return inputs
```

To apply the preprocessing function over the entire dataset, use 🤗 Datasets [`~datasets.Dataset.map`] function. You can speed up `map` by setting `batched=True` to process multiple elements of the dataset at once. Remove the columns you don't need, and rename `intent_class` to `label` because that's the name the model expects:

```py
>>> encoded_minds = minds.map(preprocess_function, remove_columns="audio", batched=True)
>>> encoded_minds = encoded_minds.rename_column("intent_class", "label")
```

## Evaluate

Including a metric during training is often helpful for evaluating your model's performance. You can quickly load a evaluation method with the 🤗 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [accuracy](https://huggingface.co/spaces/evaluate-metric/accuracy) metric (see the 🤗 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric):

```py
>>> import evaluate

>>> accuracy = evaluate.load("accuracy")
```

Then create a function that passes your predictions and labels to [`~evaluate.EvaluationModule.compute`] to calculate the accuracy:

```py
>>> import numpy as np


key: _node_content
string_value:
{"id_": "a2205ebb-3ca1-4161-a2f3-1af1bfabe826", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers"}, "hash": "ba7d015d1f001e84999554903bbcaa4d8191bf47f199a4ab3f4bfff5b10daec4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2ce9b706-d38f-4b7f-961e-35493c694121", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers"}, "hash": "10c6d7f4673224e0508ab5d3d4132be2698cb18aee15d2b5a6e8de3622eae081", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "831257b6-668e-49ef-ba40-6bdc951e997a", "node_type": "1", "metadata": {}, "hash": "35f1521d6daf246591a96ca24d3cf27487e477f83566550ee1204ac88bd03e4b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4767, "end_char_idx": 8014, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can find this information in the Wav2Vec2 [model card](https://huggingface.co/facebook/wav2vec2-base).
3. Set a maximum input length to batch longer inputs without truncating them.

```py
>>> def preprocess_function(examples):
...     audio_arrays = [x["array"] for x in examples["audio"]]
...     inputs = feature_extractor(
...         audio_arrays, sampling_rate=feature_extractor.sampling_rate, max_length=16000, truncation=True
...     )
...     return inputs
```

To apply the preprocessing function over the entire dataset, use 🤗 Datasets [`~datasets.Dataset.map`] function. You can speed up `map` by setting `batched=True` to process multiple elements of the dataset at once. Remove the columns you don't need, and rename `intent_class` to `label` because that's the name the model expects:

```py
>>> encoded_minds = minds.map(preprocess_function, remove_columns="audio", batched=True)
>>> encoded_minds = encoded_minds.rename_column("intent_class", "label")
```

## Evaluate

Including a metric during training is often helpful for evaluating your model's performance. You can quickly load a evaluation method with the 🤗 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [accuracy](https://huggingface.co/spaces/evaluate-metric/accuracy) metric (see the 🤗 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric):

```py
>>> import evaluate

>>> accuracy = evaluate.load("accuracy")
```

Then create a function that passes your predictions and labels to [`~evaluate.EvaluationModule.compute`] to calculate the accuracy:

```py
>>> import numpy as np


>>> def compute_metrics(eval_pred):
...     predictions = np.argmax(eval_pred.predictions, axis=1)
...     return accuracy.compute(predictions=predictions, references=eval_pred.label_ids)
```

Your `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.

## Train

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load Wav2Vec2 with [`AutoModelForAudioClassification`] along with the number of expected labels, and the label mappings:

```py
>>> from transformers import AutoModelForAudioClassification, TrainingArguments, Trainer

>>> num_labels = len(id2label)
>>> model = AutoModelForAudioClassification.from_pretrained(
...     "facebook/wav2vec2-base", num_labels=num_labels, label2id=label2id, id2label=id2label
... )
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the accuracy and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.


key: _node_content
string_value:
{"id_": "831257b6-668e-49ef-ba40-6bdc951e997a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers"}, "hash": "ba7d015d1f001e84999554903bbcaa4d8191bf47f199a4ab3f4bfff5b10daec4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a2205ebb-3ca1-4161-a2f3-1af1bfabe826", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers"}, "hash": "71d58706f76f9abec53ec540ef2eaee4817d53aa54909893297f1c649acb0ccc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "717330e7-975d-4440-a583-1a9fcb725526", "node_type": "1", "metadata": {}, "hash": "1dd51449c610acb4d99403f94fdf1307011c0cce91d4c77c8cd14e6d63842e60", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6441, "end_char_idx": 9495, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> def compute_metrics(eval_pred):
...     predictions = np.argmax(eval_pred.predictions, axis=1)
...     return accuracy.compute(predictions=predictions, references=eval_pred.label_ids)
```

Your `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.

## Train

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load Wav2Vec2 with [`AutoModelForAudioClassification`] along with the number of expected labels, and the label mappings:

```py
>>> from transformers import AutoModelForAudioClassification, TrainingArguments, Trainer

>>> num_labels = len(id2label)
>>> model = AutoModelForAudioClassification.from_pretrained(
...     "facebook/wav2vec2-base", num_labels=num_labels, label2id=label2id, id2label=id2label
... )
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the accuracy and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.


```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_mind_model",
...     eval_strategy="epoch",
...     save_strategy="epoch",
...     learning_rate=3e-5,
...     per_device_train_batch_size=32,
...     gradient_accumulation_steps=4,
...     per_device_eval_batch_size=32,
...     num_train_epochs=10,
...     warmup_ratio=0.1,
...     logging_steps=10,
...     load_best_model_at_end=True,
...     metric_for_best_model="accuracy",
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=encoded_minds["train"],
...     eval_dataset=encoded_minds["test"],
...     tokenizer=feature_extractor,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for audio classification, take a look at the corresponding [PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/audio_classification.ipynb).

</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Load an audio file you'd like to run inference on. Remember to resample the sampling rate of the audio file to match the sampling rate of the model if you need to!


key: _node_content
string_value:
{"id_": "717330e7-975d-4440-a583-1a9fcb725526", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers"}, "hash": "ba7d015d1f001e84999554903bbcaa4d8191bf47f199a4ab3f4bfff5b10daec4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "831257b6-668e-49ef-ba40-6bdc951e997a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers"}, "hash": "4f67c6786fdf8c125355169c5a79e42b15d64254340495db87a7c114d5a04a2a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "aa9fe628-9d39-466e-b2c0-f93ef1896f91", "node_type": "1", "metadata": {}, "hash": "86aa687b5bf1bc2ef0f15b47aa01a8a2708715e386f54f2bf04fc5db8cc2ed4f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8017, "end_char_idx": 9898, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_mind_model",
...     eval_strategy="epoch",
...     save_strategy="epoch",
...     learning_rate=3e-5,
...     per_device_train_batch_size=32,
...     gradient_accumulation_steps=4,
...     per_device_eval_batch_size=32,
...     num_train_epochs=10,
...     warmup_ratio=0.1,
...     logging_steps=10,
...     load_best_model_at_end=True,
...     metric_for_best_model="accuracy",
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=encoded_minds["train"],
...     eval_dataset=encoded_minds["test"],
...     tokenizer=feature_extractor,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for audio classification, take a look at the corresponding [PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/audio_classification.ipynb).

</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Load an audio file you'd like to run inference on. Remember to resample the sampling rate of the audio file to match the sampling rate of the model if you need to!

```py
>>> from datasets import load_dataset, Audio

>>> dataset = load_dataset("PolyAI/minds14", name="en-US", split="train")
>>> dataset = dataset.cast_column("audio", Audio(sampling_rate=16000))
>>> sampling_rate = dataset.features["audio"].sampling_rate
>>> audio_file = dataset[0]["audio"]["path"]
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`].


key: _node_content
string_value:
{"id_": "aa9fe628-9d39-466e-b2c0-f93ef1896f91", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d9bf9e9b-6893-461b-bf0e-fc7387e1155c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers"}, "hash": "ba7d015d1f001e84999554903bbcaa4d8191bf47f199a4ab3f4bfff5b10daec4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "717330e7-975d-4440-a583-1a9fcb725526", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "transformers"}, "hash": "1a12ef90d0cdbd5d9ea2b8e03529877017739aa32b6e293ce97714e8ec5df484", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9237, "end_char_idx": 11536, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Load an audio file you'd like to run inference on. Remember to resample the sampling rate of the audio file to match the sampling rate of the model if you need to!

```py
>>> from datasets import load_dataset, Audio

>>> dataset = load_dataset("PolyAI/minds14", name="en-US", split="train")
>>> dataset = dataset.cast_column("audio", Audio(sampling_rate=16000))
>>> sampling_rate = dataset.features["audio"].sampling_rate
>>> audio_file = dataset[0]["audio"]["path"]
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for audio classification with your model, and pass your audio file to it:

```py
>>> from transformers import pipeline

>>> classifier = pipeline("audio-classification", model="stevhliu/my_awesome_minds_model")
>>> classifier(audio_file)
[
    {'score': 0.09766869246959686, 'label': 'cash_deposit'},
    {'score': 0.07998877018690109, 'label': 'app_error'},
    {'score': 0.0781070664525032, 'label': 'joint_account'},
    {'score': 0.07667109370231628, 'label': 'pay_bill'},
    {'score': 0.0755252093076706, 'label': 'balance'}
]
```

You can also manually replicate the results of the `pipeline` if you'd like:

<frameworkcontent>
<pt>
Load a feature extractor to preprocess the audio file and return the `input` as PyTorch tensors:

```py
>>> from transformers import AutoFeatureExtractor

>>> feature_extractor = AutoFeatureExtractor.from_pretrained("stevhliu/my_awesome_minds_model")
>>> inputs = feature_extractor(dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt")
```

Pass your inputs to the model and return the logits:

```py
>>> from transformers import AutoModelForAudioClassification

>>> model = AutoModelForAudioClassification.from_pretrained("stevhliu/my_awesome_minds_model")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
```

Get the class with the highest probability, and use the model's `id2label` mapping to convert it to a label:

```py
>>> import torch

>>> predicted_class_ids = torch.argmax(logits).item()
>>> predicted_label = model.config.id2label[predicted_class_ids]
>>> predicted_label
'cash_deposit'
```
</pt>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "bce545eb-d48e-4260-a7c9-dc6d9764590b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers"}, "hash": "9b81a527b304d685e80ec5838d0e7b1a16f807e377c4b561c33af2ae14d2511a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "614974d0-aa78-451f-a2fc-61d0a24d5eb0", "node_type": "1", "metadata": {}, "hash": "e7c1e3e61568da0728550fd1a57b7d0893c28f06f29b24dcf530460b4b9da5a1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2336, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Automatic speech recognition

[[open-in-colab]]

<Youtube id="TksaY_FDgnk"/>

Automatic speech recognition (ASR) converts a speech signal to text, mapping a sequence of audio inputs to text outputs. Virtual assistants like Siri and Alexa use ASR models to help users everyday, and there are many other useful user-facing applications like live captioning and note-taking during meetings.

This guide will show you how to:

1. Finetune [Wav2Vec2](https://huggingface.co/facebook/wav2vec2-base) on the [MInDS-14](https://huggingface.co/datasets/PolyAI/minds14) dataset to transcribe audio to text.
2. Use your finetuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/automatic-speech-recognition)

</Tip>

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install transformers datasets evaluate jiwer
```

We encourage you to login to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to login:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load MInDS-14 dataset

Start by loading a smaller subset of the [MInDS-14](https://huggingface.co/datasets/PolyAI/minds14) dataset from the 🤗 Datasets library. This'll give you a chance to experiment and make sure everything works before spending more time training on the full dataset.

```py
>>> from datasets import load_dataset, Audio

>>> minds = load_dataset("PolyAI/minds14", name="en-US", split="train[:100]")
```

Split the dataset's `train` split into a train and test set with the [`~Dataset.train_test_split`] method:

```py
>>> minds = minds.train_test_split(test_size=0.2)
```

Then take a look at the dataset:

```py
>>> minds
DatasetDict({
    train: Dataset({
        features: ['path', 'audio', 'transcription', 'english_transcription', 'intent_class', 'lang_id'],
        num_rows: 16
    })
    test: Dataset({
        features: ['path', 'audio', 'transcription', 'english_transcription', 'intent_class', 'lang_id'],
        num_rows: 4
    })
})
```

While the dataset contains a lot of useful information, like `lang_id` and `english_transcription`, you'll focus on the `audio` and `transcription` in this guide.


key: _node_content
string_value:
{"id_": "614974d0-aa78-451f-a2fc-61d0a24d5eb0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers"}, "hash": "9b81a527b304d685e80ec5838d0e7b1a16f807e377c4b561c33af2ae14d2511a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bce545eb-d48e-4260-a7c9-dc6d9764590b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers"}, "hash": "ec9e215b8493fcacc9feb652a4319b5e63dfeb86750f6892ce63d139c6f3dde8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b15dbf7c-83fc-4694-98bb-a5e9e9c3296a", "node_type": "1", "metadata": {}, "hash": "89a4d8fa95f76450a573f182b251964f92ba429790df045628917c517b962071", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1076, "end_char_idx": 3411, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
When prompted, enter your token to login:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load MInDS-14 dataset

Start by loading a smaller subset of the [MInDS-14](https://huggingface.co/datasets/PolyAI/minds14) dataset from the 🤗 Datasets library. This'll give you a chance to experiment and make sure everything works before spending more time training on the full dataset.

```py
>>> from datasets import load_dataset, Audio

>>> minds = load_dataset("PolyAI/minds14", name="en-US", split="train[:100]")
```

Split the dataset's `train` split into a train and test set with the [`~Dataset.train_test_split`] method:

```py
>>> minds = minds.train_test_split(test_size=0.2)
```

Then take a look at the dataset:

```py
>>> minds
DatasetDict({
    train: Dataset({
        features: ['path', 'audio', 'transcription', 'english_transcription', 'intent_class', 'lang_id'],
        num_rows: 16
    })
    test: Dataset({
        features: ['path', 'audio', 'transcription', 'english_transcription', 'intent_class', 'lang_id'],
        num_rows: 4
    })
})
```

While the dataset contains a lot of useful information, like `lang_id` and `english_transcription`, you'll focus on the `audio` and `transcription` in this guide. Remove the other columns with the [`~datasets.Dataset.remove_columns`] method:

```py
>>> minds = minds.remove_columns(["english_transcription", "intent_class", "lang_id"])
```

Take a look at the example again:

```py
>>> minds["train"][0]
{'audio': {'array': array([-0.00024414,  0.        ,  0.        , ...,  0.00024414,
          0.00024414,  0.00024414], dtype=float32),
  'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602ba9e2963e11ccd901cd4f.wav',
  'sampling_rate': 8000},
 'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602ba9e2963e11ccd901cd4f.wav',
 'transcription': "hi I'm trying to use the banking app on my phone and currently my checking and savings account balance is not refreshing"}
```

There are two fields:

- `audio`: a 1-dimensional `array` of the speech signal that must be called to load and resample the audio file.
- `transcription`: the target text.


key: _node_content
string_value:
{"id_": "b15dbf7c-83fc-4694-98bb-a5e9e9c3296a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers"}, "hash": "9b81a527b304d685e80ec5838d0e7b1a16f807e377c4b561c33af2ae14d2511a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "614974d0-aa78-451f-a2fc-61d0a24d5eb0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers"}, "hash": "8cef1a99ad4ce9e5ae4b54b9b9a2f22de9dc47f5028f43c49f491ddb334b3046", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e7f2aab3-4258-4e9c-92df-3be7bf2f7f61", "node_type": "1", "metadata": {}, "hash": "bf6f46d1a3e333cf23c787f0d8f171a04441907d7cdb51ca46ac1043a9043700", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2337, "end_char_idx": 4771, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Remove the other columns with the [`~datasets.Dataset.remove_columns`] method:

```py
>>> minds = minds.remove_columns(["english_transcription", "intent_class", "lang_id"])
```

Take a look at the example again:

```py
>>> minds["train"][0]
{'audio': {'array': array([-0.00024414,  0.        ,  0.        , ...,  0.00024414,
          0.00024414,  0.00024414], dtype=float32),
  'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602ba9e2963e11ccd901cd4f.wav',
  'sampling_rate': 8000},
 'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602ba9e2963e11ccd901cd4f.wav',
 'transcription': "hi I'm trying to use the banking app on my phone and currently my checking and savings account balance is not refreshing"}
```

There are two fields:

- `audio`: a 1-dimensional `array` of the speech signal that must be called to load and resample the audio file.
- `transcription`: the target text.

## Preprocess

The next step is to load a Wav2Vec2 processor to process the audio signal:

```py
>>> from transformers import AutoProcessor

>>> processor = AutoProcessor.from_pretrained("facebook/wav2vec2-base")
```

The MInDS-14 dataset has a sampling rate of 8000kHz (you can find this information in its [dataset card](https://huggingface.co/datasets/PolyAI/minds14)), which means you'll need to resample the dataset to 16000kHz to use the pretrained Wav2Vec2 model:

```py
>>> minds = minds.cast_column("audio", Audio(sampling_rate=16_000))
>>> minds["train"][0]
{'audio': {'array': array([-2.38064706e-04, -1.58618059e-04, -5.43987835e-06, ...,
          2.78103951e-04,  2.38446111e-04,  1.18740834e-04], dtype=float32),
  'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602ba9e2963e11ccd901cd4f.wav',
  'sampling_rate': 16000},
 'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602ba9e2963e11ccd901cd4f.wav',
 'transcription': "hi I'm trying to use the banking app on my phone and currently my checking and savings account balance is not refreshing"}
```

As you can see in the `transcription` above, the text contains a mix of upper and lowercase characters.


key: _node_content
string_value:
{"id_": "e7f2aab3-4258-4e9c-92df-3be7bf2f7f61", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers"}, "hash": "9b81a527b304d685e80ec5838d0e7b1a16f807e377c4b561c33af2ae14d2511a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b15dbf7c-83fc-4694-98bb-a5e9e9c3296a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers"}, "hash": "ff7ad6654f87432bd33b52401634650199b7a32738873cff083ee4f1a2de458b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "50dc72e3-1d06-433b-bb63-74c04b33487b", "node_type": "1", "metadata": {}, "hash": "8e0c90daed7ad60e4ab66ba8f62a516c42f87c277e79b7bb1c6c9e9e2de4e733", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4772, "end_char_idx": 6628, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The Wav2Vec2 tokenizer is only trained on uppercase characters so you'll need to make sure the text matches the tokenizer's vocabulary:

```py
>>> def uppercase(example):
...     return {"transcription": example["transcription"].upper()}


>>> minds = minds.map(uppercase)
```

Now create a preprocessing function that:

1. Calls the `audio` column to load and resample the audio file.
2. Extracts the `input_values` from the audio file and tokenize the `transcription` column with the processor.

```py
>>> def prepare_dataset(batch):
...     audio = batch["audio"]
...     batch = processor(audio["array"], sampling_rate=audio["sampling_rate"], text=batch["transcription"])
...     batch["input_length"] = len(batch["input_values"][0])
...     return batch
```

To apply the preprocessing function over the entire dataset, use 🤗 Datasets [`~datasets.Dataset.map`] function. You can speed up `map` by increasing the number of processes with the `num_proc` parameter. Remove the columns you don't need with the [`~datasets.Dataset.remove_columns`] method:

```py
>>> encoded_minds = minds.map(prepare_dataset, remove_columns=minds.column_names["train"], num_proc=4)
```

🤗 Transformers doesn't have a data collator for ASR, so you'll need to adapt the [`DataCollatorWithPadding`] to create a batch of examples. It'll also dynamically pad your text and labels to the length of the longest element in its batch (instead of the entire dataset) so they are a uniform length. While it is possible to pad your text in the `tokenizer` function by setting `padding=True`, dynamic padding is more efficient.

Unlike other data collators, this specific data collator needs to apply a different padding method to `input_values` and `labels`:

```py
>>> import torch

>>> from dataclasses import dataclass, field
>>> from typing import Any, Dict, List, Optional, Union


key: _node_content
string_value:
{"id_": "50dc72e3-1d06-433b-bb63-74c04b33487b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers"}, "hash": "9b81a527b304d685e80ec5838d0e7b1a16f807e377c4b561c33af2ae14d2511a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e7f2aab3-4258-4e9c-92df-3be7bf2f7f61", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers"}, "hash": "14b9337583825b7db8adf6d1e8da7197557cf78f1b9da28a6ad41397aa7c0660", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6ae0b745-d0ce-408e-b737-c41a820055f9", "node_type": "1", "metadata": {}, "hash": "37743a80c704723b1384deea748f086bf401348d80219d43ac4071072a678f90", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5012, "end_char_idx": 8483, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> minds = minds.map(uppercase)
```

Now create a preprocessing function that:

1. Calls the `audio` column to load and resample the audio file.
2. Extracts the `input_values` from the audio file and tokenize the `transcription` column with the processor.

```py
>>> def prepare_dataset(batch):
...     audio = batch["audio"]
...     batch = processor(audio["array"], sampling_rate=audio["sampling_rate"], text=batch["transcription"])
...     batch["input_length"] = len(batch["input_values"][0])
...     return batch
```

To apply the preprocessing function over the entire dataset, use 🤗 Datasets [`~datasets.Dataset.map`] function. You can speed up `map` by increasing the number of processes with the `num_proc` parameter. Remove the columns you don't need with the [`~datasets.Dataset.remove_columns`] method:

```py
>>> encoded_minds = minds.map(prepare_dataset, remove_columns=minds.column_names["train"], num_proc=4)
```

🤗 Transformers doesn't have a data collator for ASR, so you'll need to adapt the [`DataCollatorWithPadding`] to create a batch of examples. It'll also dynamically pad your text and labels to the length of the longest element in its batch (instead of the entire dataset) so they are a uniform length. While it is possible to pad your text in the `tokenizer` function by setting `padding=True`, dynamic padding is more efficient.

Unlike other data collators, this specific data collator needs to apply a different padding method to `input_values` and `labels`:

```py
>>> import torch

>>> from dataclasses import dataclass, field
>>> from typing import Any, Dict, List, Optional, Union


>>> @dataclass
... class DataCollatorCTCWithPadding:
...     processor: AutoProcessor
...     padding: Union[bool, str] = "longest"

...     def __call__(self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]:
...         # split inputs and labels since they have to be of different lengths and need
...         # different padding methods
...         input_features = [{"input_values": feature["input_values"][0]} for feature in features]
...         label_features = [{"input_ids": feature["labels"]} for feature in features]

...         batch = self.processor.pad(input_features, padding=self.padding, return_tensors="pt")

...         labels_batch = self.processor.pad(labels=label_features, padding=self.padding, return_tensors="pt")

...         # replace padding with -100 to ignore loss correctly
...         labels = labels_batch["input_ids"].masked_fill(labels_batch.attention_mask.ne(1), -100)

...         batch["labels"] = labels

...         return batch
```

Now instantiate your `DataCollatorForCTCWithPadding`:

```py
>>> data_collator = DataCollatorCTCWithPadding(processor=processor, padding="longest")
```

## Evaluate

Including a metric during training is often helpful for evaluating your model's performance. You can quickly load a evaluation method with the 🤗 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [word error rate](https://huggingface.co/spaces/evaluate-metric/wer) (WER) metric (see the 🤗 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric):

```py
>>> import evaluate

>>> wer = evaluate.load("wer")
```

Then create a function that passes your predictions and labels to [`~evaluate.EvaluationModule.compute`] to calculate the WER:

```py
>>> import numpy as np


key: _node_content
string_value:
{"id_": "6ae0b745-d0ce-408e-b737-c41a820055f9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers"}, "hash": "9b81a527b304d685e80ec5838d0e7b1a16f807e377c4b561c33af2ae14d2511a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "50dc72e3-1d06-433b-bb63-74c04b33487b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers"}, "hash": "fdfa0145076ffe84a12cc12156589ff20c46828cd2f172bdeb70ae22f6eb89c0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "07f8f36f-164a-4576-a13f-cb2989d6ecdc", "node_type": "1", "metadata": {}, "hash": "d68f279720a591c69093189a95e8d14ae4bba946208c081bb32e621ea114fa19", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8486, "end_char_idx": 10332, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> def compute_metrics(pred):
...     pred_logits = pred.predictions
...     pred_ids = np.argmax(pred_logits, axis=-1)

...     pred.label_ids[pred.label_ids == -100] = processor.tokenizer.pad_token_id

...     pred_str = processor.batch_decode(pred_ids)
...     label_str = processor.batch_decode(pred.label_ids, group_tokens=False)

...     wer = wer.compute(predictions=pred_str, references=label_str)

...     return {"wer": wer}
```

Your `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.

## Train

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load Wav2Vec2 with [`AutoModelForCTC`]. Specify the reduction to apply with the `ctc_loss_reduction` parameter. It is often better to use the average instead of the default summation:

```py
>>> from transformers import AutoModelForCTC, TrainingArguments, Trainer

>>> model = AutoModelForCTC.from_pretrained(
...     "facebook/wav2vec2-base",
...     ctc_loss_reduction="mean",
...     pad_token_id=processor.tokenizer.pad_token_id,
... )
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the WER and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.


key: _node_content
string_value:
{"id_": "07f8f36f-164a-4576-a13f-cb2989d6ecdc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers"}, "hash": "9b81a527b304d685e80ec5838d0e7b1a16f807e377c4b561c33af2ae14d2511a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6ae0b745-d0ce-408e-b737-c41a820055f9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers"}, "hash": "027e886a7a71540baee70e2e866de0336442677f8c5a05b2151e67369414c710", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d51960a2-2dc4-40f2-8fa1-bc70e5345a08", "node_type": "1", "metadata": {}, "hash": "340b985da6e92e10e3219dbb60d41f5c1cd778a5bb6c4cf97cb7df5a6a23df25", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9034, "end_char_idx": 12011, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Train

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load Wav2Vec2 with [`AutoModelForCTC`]. Specify the reduction to apply with the `ctc_loss_reduction` parameter. It is often better to use the average instead of the default summation:

```py
>>> from transformers import AutoModelForCTC, TrainingArguments, Trainer

>>> model = AutoModelForCTC.from_pretrained(
...     "facebook/wav2vec2-base",
...     ctc_loss_reduction="mean",
...     pad_token_id=processor.tokenizer.pad_token_id,
... )
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the WER and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_asr_mind_model",
...     per_device_train_batch_size=8,
...     gradient_accumulation_steps=2,
...     learning_rate=1e-5,
...     warmup_steps=500,
...     max_steps=2000,
...     gradient_checkpointing=True,
...     fp16=True,
...     group_by_length=True,
...     eval_strategy="steps",
...     per_device_eval_batch_size=8,
...     save_steps=1000,
...     eval_steps=1000,
...     logging_steps=25,
...     load_best_model_at_end=True,
...     metric_for_best_model="wer",
...     greater_is_better=False,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=encoded_minds["train"],
...     eval_dataset=encoded_minds["test"],
...     tokenizer=processor,
...     data_collator=data_collator,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for automatic speech recognition, take a look at this blog [post](https://huggingface.co/blog/fine-tune-wav2vec2-english) for English ASR and this [post](https://huggingface.co/blog/fine-tune-xlsr-wav2vec2) for multilingual ASR.

</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Load an audio file you'd like to run inference on. Remember to resample the sampling rate of the audio file to match the sampling rate of the model if you need to!


key: _node_content
string_value:
{"id_": "d51960a2-2dc4-40f2-8fa1-bc70e5345a08", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "703c7679-54d0-40de-ae8b-5eb2c5753369", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers"}, "hash": "9b81a527b304d685e80ec5838d0e7b1a16f807e377c4b561c33af2ae14d2511a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "07f8f36f-164a-4576-a13f-cb2989d6ecdc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "transformers"}, "hash": "ca21b2001796f10f7d99caf9f4fc627f6a4fab25468e5a761c67b517e6507a26", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11753, "end_char_idx": 13976, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Load an audio file you'd like to run inference on. Remember to resample the sampling rate of the audio file to match the sampling rate of the model if you need to!

```py
>>> from datasets import load_dataset, Audio

>>> dataset = load_dataset("PolyAI/minds14", "en-US", split="train")
>>> dataset = dataset.cast_column("audio", Audio(sampling_rate=16000))
>>> sampling_rate = dataset.features["audio"].sampling_rate
>>> audio_file = dataset[0]["audio"]["path"]
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for automatic speech recognition with your model, and pass your audio file to it:

```py
>>> from transformers import pipeline

>>> transcriber = pipeline("automatic-speech-recognition", model="stevhliu/my_awesome_asr_minds_model")
>>> transcriber(audio_file)
{'text': 'I WOUD LIKE O SET UP JOINT ACOUNT WTH Y PARTNER'}
```

<Tip>

The transcription is decent, but it could be better! Try finetuning your model on more examples to get even better results!

</Tip>

You can also manually replicate the results of the `pipeline` if you'd like:

<frameworkcontent>
<pt>
Load a processor to preprocess the audio file and transcription and return the `input` as PyTorch tensors:

```py
>>> from transformers import AutoProcessor

>>> processor = AutoProcessor.from_pretrained("stevhliu/my_awesome_asr_mind_model")
>>> inputs = processor(dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt")
```

Pass your inputs to the model and return the logits:

```py
>>> from transformers import AutoModelForCTC

>>> model = AutoModelForCTC.from_pretrained("stevhliu/my_awesome_asr_mind_model")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
```

Get the predicted `input_ids` with the highest probability, and use the processor to decode the predicted `input_ids` back into text:

```py
>>> import torch

>>> predicted_ids = torch.argmax(logits, dim=-1)
>>> transcription = processor.batch_decode(predicted_ids)
>>> transcription
['I WOUL LIKE O SET UP JOINT ACOUNT WTH Y PARTNER']
```
</pt>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "0a73ddb2-93e7-4a94-936d-7b8d4ea2fb4c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers"}, "hash": "326c9c538adbd0a701b7618bacf6e60d06fa9b09a486befaf8143c411e1560e8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1d2e96b5-1bca-40b2-84d0-feb7afd5e04c", "node_type": "1", "metadata": {}, "hash": "7733f907b8600bdc0ea7c332460b88a5192f4933d5b80079932f7055ba69b2e3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3113, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Causal language modeling

[[open-in-colab]]

There are two types of language modeling, causal and masked. This guide illustrates causal language modeling.
Causal language models are frequently used for text generation. You can use these models for creative applications like
choosing your own text adventure or an intelligent coding assistant like Copilot or CodeParrot.

<Youtube id="Vpjb1lu0MDk"/>

Causal language modeling predicts the next token in a sequence of tokens, and the model can only attend to tokens on
the left. This means the model cannot see future tokens. GPT-2 is an example of a causal language model.

This guide will show you how to:

1. Finetune [DistilGPT2](https://huggingface.co/distilbert/distilgpt2) on the [r/askscience](https://www.reddit.com/r/askscience/) subset of the [ELI5](https://huggingface.co/datasets/eli5) dataset.
2. Use your finetuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/text-generation)

</Tip>

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install transformers datasets evaluate
```

We encourage you to log in to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load ELI5 dataset

Start by loading the first 5000 examples from the [ELI5-Category](https://huggingface.co/datasets/eli5_category) dataset with the 🤗 Datasets library. This'll give you a chance to experiment and make sure everything works before spending more time training on the full dataset.

```py
>>> from datasets import load_dataset

>>> eli5 = load_dataset("eli5_category", split="train[:5000]")
```

Split the dataset's `train` split into a train and test set with the [`~datasets.Dataset.train_test_split`] method:

```py
>>> eli5 = eli5.train_test_split(test_size=0.2)
```

Then take a look at an example:

```py
>>> eli5["train"][0]
{'q_id': '7h191n',
 'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?',
 'selftext': '',
 'category': 'Economics',
 'subreddit': 'explainlikeimfive',
 'answers': {'a_id': ['dqnds8l', 'dqnd1jl', 'dqng3i1', 'dqnku5x'],
  'text': ["The tax bill is 500 pages long and there were a lot of changes still going on right to the end. It's not just an adjustment to the income tax brackets, it's a whole bunch of changes. As such there is no good answer to your question. The big take aways are: - Big reduction in corporate income tax rate will make large companies very happy. - Pass through rate change will make certain styles of business (law firms, hedge funds) extremely happy - Income tax changes are moderate, and are set to expire (though it's the kind of thing that might just always get re-applied without being made permanent) - People in high tax states (California, New York) lose out, and many of them will end up with their taxes raised.",
   'None yet.


key: _node_content
string_value:
{"id_": "1d2e96b5-1bca-40b2-84d0-feb7afd5e04c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers"}, "hash": "326c9c538adbd0a701b7618bacf6e60d06fa9b09a486befaf8143c411e1560e8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0a73ddb2-93e7-4a94-936d-7b8d4ea2fb4c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers"}, "hash": "b1b1634cf1ee551b3d7d985715eb4f4b1d78f64a412f6e10bd766fd494912d55", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "34390285-d521-4e02-b786-77e05e7cacb7", "node_type": "1", "metadata": {}, "hash": "8ee1073c72965596d03a49fb92a3bc923905d5c8f3b23a97e27b797a31d18236", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1615, "end_char_idx": 4536, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This'll give you a chance to experiment and make sure everything works before spending more time training on the full dataset.

```py
>>> from datasets import load_dataset

>>> eli5 = load_dataset("eli5_category", split="train[:5000]")
```

Split the dataset's `train` split into a train and test set with the [`~datasets.Dataset.train_test_split`] method:

```py
>>> eli5 = eli5.train_test_split(test_size=0.2)
```

Then take a look at an example:

```py
>>> eli5["train"][0]
{'q_id': '7h191n',
 'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?',
 'selftext': '',
 'category': 'Economics',
 'subreddit': 'explainlikeimfive',
 'answers': {'a_id': ['dqnds8l', 'dqnd1jl', 'dqng3i1', 'dqnku5x'],
  'text': ["The tax bill is 500 pages long and there were a lot of changes still going on right to the end. It's not just an adjustment to the income tax brackets, it's a whole bunch of changes. As such there is no good answer to your question. The big take aways are: - Big reduction in corporate income tax rate will make large companies very happy. - Pass through rate change will make certain styles of business (law firms, hedge funds) extremely happy - Income tax changes are moderate, and are set to expire (though it's the kind of thing that might just always get re-applied without being made permanent) - People in high tax states (California, New York) lose out, and many of them will end up with their taxes raised.",
   'None yet. It has to be reconciled with a vastly different house bill and then passed again.',
   'Also: does this apply to 2017 taxes? Or does it start with 2018 taxes?',
   'This article explains both the House and senate bills, including the proposed changes to your income taxes based on your income level. URL_0'],
  'score': [21, 19, 5, 3],
  'text_urls': [[],
   [],
   [],
   ['https://www.investopedia.com/news/trumps-tax-reform-what-can-be-done/']]},
 'title_urls': ['url'],
 'selftext_urls': ['url']}
```

While this may look like a lot, you're only really interested in the `text` field. What's cool about language modeling
tasks is you don't need labels (also known as an unsupervised task) because the next word *is* the label.

## Preprocess

<Youtube id="ma1TrR7gE7I"/>

The next step is to load a DistilGPT2 tokenizer to process the `text` subfield:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilgpt2")
```

You'll notice from the example above, the `text` field is actually nested inside `answers`. This means you'll need to
extract the `text` subfield from its nested structure with the [`flatten`](https://huggingface.co/docs/datasets/process#flatten) method:

```py
>>> eli5 = eli5.flatten()
>>> eli5["train"][0]
{'q_id': '7h191n',
 'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?


key: _node_content
string_value:
{"id_": "34390285-d521-4e02-b786-77e05e7cacb7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers"}, "hash": "326c9c538adbd0a701b7618bacf6e60d06fa9b09a486befaf8143c411e1560e8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1d2e96b5-1bca-40b2-84d0-feb7afd5e04c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers"}, "hash": "540d187167a880ff0ed7626520f912705aae7a9e53183d9f5e73fec53dc18946", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9b148aa9-9b52-45e4-9768-d4cb99cabbb9", "node_type": "1", "metadata": {}, "hash": "60ead5054e0da33219d7c8bed13b92301992a131cc258bfe42ccb382dfe9c2f5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3097, "end_char_idx": 5727, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
",
   'None yet. It has to be reconciled with a vastly different house bill and then passed again.',
   'Also: does this apply to 2017 taxes? Or does it start with 2018 taxes?',
   'This article explains both the House and senate bills, including the proposed changes to your income taxes based on your income level. URL_0'],
  'score': [21, 19, 5, 3],
  'text_urls': [[],
   [],
   [],
   ['https://www.investopedia.com/news/trumps-tax-reform-what-can-be-done/']]},
 'title_urls': ['url'],
 'selftext_urls': ['url']}
```

While this may look like a lot, you're only really interested in the `text` field. What's cool about language modeling
tasks is you don't need labels (also known as an unsupervised task) because the next word *is* the label.

## Preprocess

<Youtube id="ma1TrR7gE7I"/>

The next step is to load a DistilGPT2 tokenizer to process the `text` subfield:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilgpt2")
```

You'll notice from the example above, the `text` field is actually nested inside `answers`. This means you'll need to
extract the `text` subfield from its nested structure with the [`flatten`](https://huggingface.co/docs/datasets/process#flatten) method:

```py
>>> eli5 = eli5.flatten()
>>> eli5["train"][0]
{'q_id': '7h191n',
 'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?',
 'selftext': '',
 'category': 'Economics',
 'subreddit': 'explainlikeimfive',
 'answers.a_id': ['dqnds8l', 'dqnd1jl', 'dqng3i1', 'dqnku5x'],
 'answers.text': ["The tax bill is 500 pages long and there were a lot of changes still going on right to the end. It's not just an adjustment to the income tax brackets, it's a whole bunch of changes. As such there is no good answer to your question. The big take aways are: - Big reduction in corporate income tax rate will make large companies very happy. - Pass through rate change will make certain styles of business (law firms, hedge funds) extremely happy - Income tax changes are moderate, and are set to expire (though it's the kind of thing that might just always get re-applied without being made permanent) - People in high tax states (California, New York) lose out, and many of them will end up with their taxes raised.",
  'None yet. It has to be reconciled with a vastly different house bill and then passed again.',
  'Also: does this apply to 2017 taxes? Or does it start with 2018 taxes?',
  'This article explains both the House and senate bills, including the proposed changes to your income taxes based on your income level.


key: _node_content
string_value:
{"id_": "9b148aa9-9b52-45e4-9768-d4cb99cabbb9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers"}, "hash": "326c9c538adbd0a701b7618bacf6e60d06fa9b09a486befaf8143c411e1560e8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "34390285-d521-4e02-b786-77e05e7cacb7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers"}, "hash": "c93659434cb8ff9ddf926b3ed7536987c6dd827a5b470b6f3c9659d436f7b53a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9885de43-d86a-4b20-abcd-a29250cbfbc2", "node_type": "1", "metadata": {}, "hash": "1ea3a628a8690de0cc7fcf1cc4a76cf98dfe790e6ba0b2ce26bc629dac64bb5e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4189, "end_char_idx": 7298, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This means you'll need to
extract the `text` subfield from its nested structure with the [`flatten`](https://huggingface.co/docs/datasets/process#flatten) method:

```py
>>> eli5 = eli5.flatten()
>>> eli5["train"][0]
{'q_id': '7h191n',
 'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?',
 'selftext': '',
 'category': 'Economics',
 'subreddit': 'explainlikeimfive',
 'answers.a_id': ['dqnds8l', 'dqnd1jl', 'dqng3i1', 'dqnku5x'],
 'answers.text': ["The tax bill is 500 pages long and there were a lot of changes still going on right to the end. It's not just an adjustment to the income tax brackets, it's a whole bunch of changes. As such there is no good answer to your question. The big take aways are: - Big reduction in corporate income tax rate will make large companies very happy. - Pass through rate change will make certain styles of business (law firms, hedge funds) extremely happy - Income tax changes are moderate, and are set to expire (though it's the kind of thing that might just always get re-applied without being made permanent) - People in high tax states (California, New York) lose out, and many of them will end up with their taxes raised.",
  'None yet. It has to be reconciled with a vastly different house bill and then passed again.',
  'Also: does this apply to 2017 taxes? Or does it start with 2018 taxes?',
  'This article explains both the House and senate bills, including the proposed changes to your income taxes based on your income level. URL_0'],
 'answers.score': [21, 19, 5, 3],
 'answers.text_urls': [[],
  [],
  [],
  ['https://www.investopedia.com/news/trumps-tax-reform-what-can-be-done/']],
 'title_urls': ['url'],
 'selftext_urls': ['url']}
```

Each subfield is now a separate column as indicated by the `answers` prefix, and the `text` field is a list now. Instead
of tokenizing each sentence separately, convert the list to a string so you can jointly tokenize them.

Here is a first preprocessing function to join the list of strings for each example and tokenize the result:

```py
>>> def preprocess_function(examples):
...     return tokenizer([" ".join(x) for x in examples["answers.text"]])
```

To apply this preprocessing function over the entire dataset, use the 🤗 Datasets [`~datasets.Dataset.map`] method. You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once, and increasing the number of processes with `num_proc`. Remove any columns you don't need:

```py
>>> tokenized_eli5 = eli5.map(
...     preprocess_function,
...     batched=True,
...     num_proc=4,
...     remove_columns=eli5["train"].column_names,
... )
```

This dataset contains the token sequences, but some of these are longer than the maximum input length for the model.

You can now use a second preprocessing function to

- concatenate all the sequences
- split the concatenated sequences into shorter chunks defined by `block_size`, which should be both shorter than the maximum input length and short enough for your GPU RAM.

```py
>>> block_size = 128


key: _node_content
string_value:
{"id_": "9885de43-d86a-4b20-abcd-a29250cbfbc2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers"}, "hash": "326c9c538adbd0a701b7618bacf6e60d06fa9b09a486befaf8143c411e1560e8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9b148aa9-9b52-45e4-9768-d4cb99cabbb9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers"}, "hash": "6db1a56e39fd7830a7dd560fcd7f86aa717b42c748ae8c6c7cc8b885bd86d7e5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cf65ee5e-6aaf-411f-bae8-d803e275759b", "node_type": "1", "metadata": {}, "hash": "580433daf2582c45b8b8c46fda6daafb27cdf7b1d3f9d8a86d5f313535a0e3e1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5728, "end_char_idx": 9198, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
URL_0'],
 'answers.score': [21, 19, 5, 3],
 'answers.text_urls': [[],
  [],
  [],
  ['https://www.investopedia.com/news/trumps-tax-reform-what-can-be-done/']],
 'title_urls': ['url'],
 'selftext_urls': ['url']}
```

Each subfield is now a separate column as indicated by the `answers` prefix, and the `text` field is a list now. Instead
of tokenizing each sentence separately, convert the list to a string so you can jointly tokenize them.

Here is a first preprocessing function to join the list of strings for each example and tokenize the result:

```py
>>> def preprocess_function(examples):
...     return tokenizer([" ".join(x) for x in examples["answers.text"]])
```

To apply this preprocessing function over the entire dataset, use the 🤗 Datasets [`~datasets.Dataset.map`] method. You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once, and increasing the number of processes with `num_proc`. Remove any columns you don't need:

```py
>>> tokenized_eli5 = eli5.map(
...     preprocess_function,
...     batched=True,
...     num_proc=4,
...     remove_columns=eli5["train"].column_names,
... )
```

This dataset contains the token sequences, but some of these are longer than the maximum input length for the model.

You can now use a second preprocessing function to

- concatenate all the sequences
- split the concatenated sequences into shorter chunks defined by `block_size`, which should be both shorter than the maximum input length and short enough for your GPU RAM.

```py
>>> block_size = 128


>>> def group_texts(examples):
...     # Concatenate all texts.
...     concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
...     total_length = len(concatenated_examples[list(examples.keys())[0]])
...     # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can
...     # customize this part to your needs.
...     if total_length >= block_size:
...         total_length = (total_length // block_size) * block_size
...     # Split by chunks of block_size.
...     result = {
...         k: [t[i : i + block_size] for i in range(0, total_length, block_size)]
...         for k, t in concatenated_examples.items()
...     }
...     result["labels"] = result["input_ids"].copy()
...     return result
```

Apply the `group_texts` function over the entire dataset:

```py
>>> lm_dataset = tokenized_eli5.map(group_texts, batched=True, num_proc=4)
```

Now create a batch of examples using [`DataCollatorForLanguageModeling`]. It's more efficient to *dynamically pad* the
sentences to the longest length in a batch during collation, instead of padding the whole dataset to the maximum length.

<frameworkcontent>
<pt>
Use the end-of-sequence token as the padding token and set `mlm=False`. This will use the inputs as labels shifted to the right by one element:

```py
>>> from transformers import DataCollatorForLanguageModeling

>>> tokenizer.pad_token = tokenizer.eos_token
>>> data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)
```

</pt>
<tf>
Use the end-of-sequence token as the padding token and set `mlm=False`. This will use the inputs as labels shifted to the right by one element:

```py
>>> from transformers import DataCollatorForLanguageModeling

>>> data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False, return_tensors="tf")
```

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "cf65ee5e-6aaf-411f-bae8-d803e275759b", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers"}, "hash": "326c9c538adbd0a701b7618bacf6e60d06fa9b09a486befaf8143c411e1560e8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9885de43-d86a-4b20-abcd-a29250cbfbc2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers"}, "hash": "3b9731b020923dd41c73b8ec88f649673af7e91903bc4bb737261e1e25b6b9e6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b3d5def7-06ef-441a-bc5c-cc5b2e254203", "node_type": "1", "metadata": {}, "hash": "6e259563c812d45c702b4fbe8d10ec8160479a7f6a985dd4e36224fd4b845f2e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9201, "end_char_idx": 12221, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Train

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the [basic tutorial](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load DistilGPT2 with [`AutoModelForCausalLM`]:

```py
>>> from transformers import AutoModelForCausalLM, TrainingArguments, Trainer

>>> model = AutoModelForCausalLM.from_pretrained("distilbert/distilgpt2")
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model).
2. Pass the training arguments to [`Trainer`] along with the model, datasets, and data collator.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_eli5_clm-model",
...     eval_strategy="epoch",
...     learning_rate=2e-5,
...     weight_decay=0.01,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=lm_dataset["train"],
...     eval_dataset=lm_dataset["test"],
...     data_collator=data_collator,
... )

>>> trainer.train()
```

Once training is completed, use the [`~transformers.Trainer.evaluate`] method to evaluate your model and get its perplexity:

```py
>>> import math

>>> eval_results = trainer.evaluate()
>>> print(f"Perplexity: {math.exp(eval_results['eval_loss']):.2f}")
Perplexity: 49.61
```

Then share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the [basic tutorial](../training#train-a-tensorflow-model-with-keras)!

</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import create_optimizer, AdamWeightDecay

>>> optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)
```

Then you can load DistilGPT2 with [`TFAutoModelForCausalLM`]:

```py
>>> from transformers import TFAutoModelForCausalLM

>>> model = TFAutoModelForCausalLM.from_pretrained("distilbert/distilgpt2")
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> tf_train_set = model.prepare_tf_dataset(
...     lm_dataset["train"],
...     shuffle=True,
...     batch_size=16,
...     collate_fn=data_collator,
... )

>>> tf_test_set = model.prepare_tf_dataset(
...     lm_dataset["test"],
...     shuffle=False,
...     batch_size=16,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method).


key: _node_content
string_value:
{"id_": "b3d5def7-06ef-441a-bc5c-cc5b2e254203", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers"}, "hash": "326c9c538adbd0a701b7618bacf6e60d06fa9b09a486befaf8143c411e1560e8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cf65ee5e-6aaf-411f-bae8-d803e275759b", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers"}, "hash": "d4cfb4de79c09731d5529160527ad4bc247a696919c98bae5853386725baceb2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "792f192f-5f1c-4033-9d66-7d4c35c517a1", "node_type": "1", "metadata": {}, "hash": "5524ba32f3c72e3290214d6a1bdc7e1eb019fe9c6eea66c43cc1c26f769ddba0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11162, "end_char_idx": 14440, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import create_optimizer, AdamWeightDecay

>>> optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)
```

Then you can load DistilGPT2 with [`TFAutoModelForCausalLM`]:

```py
>>> from transformers import TFAutoModelForCausalLM

>>> model = TFAutoModelForCausalLM.from_pretrained("distilbert/distilgpt2")
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> tf_train_set = model.prepare_tf_dataset(
...     lm_dataset["train"],
...     shuffle=True,
...     batch_size=16,
...     collate_fn=data_collator,
... )

>>> tf_test_set = model.prepare_tf_dataset(
...     lm_dataset["test"],
...     shuffle=False,
...     batch_size=16,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method). Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> import tensorflow as tf

>>> model.compile(optimizer=optimizer)  # No loss argument!
```

This can be done by specifying where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:

```py
>>> from transformers.keras_callbacks import PushToHubCallback

>>> callback = PushToHubCallback(
...     output_dir="my_awesome_eli5_clm-model",
...     tokenizer=tokenizer,
... )
```

Finally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callback to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_test_set, epochs=3, callbacks=[callback])
```

Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
</tf>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for causal language modeling, take a look at the corresponding
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).

</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Come up with a prompt you'd like to generate text from:

```py
>>> prompt = "Somatic hypermutation allows the immune system to"
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for text generation with your model, and pass your text to it:

```py
>>> from transformers import pipeline

>>> generator = pipeline("text-generation", model="username/my_awesome_eli5_clm-model")
>>> generator(prompt)
[{'generated_text': "Somatic hypermutation allows the immune system to be able to effectively reverse the damage caused by an infection.\n\n\nThe damage caused by an infection is caused by the immune system's ability to perform its own self-correcting tasks."}]


key: _node_content
string_value:
{"id_": "792f192f-5f1c-4033-9d66-7d4c35c517a1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers"}, "hash": "326c9c538adbd0a701b7618bacf6e60d06fa9b09a486befaf8143c411e1560e8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b3d5def7-06ef-441a-bc5c-cc5b2e254203", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers"}, "hash": "4930bf51777760f5698a713f09aa331216982123b384361caabbbf02412f2f6f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "37fbec03-0428-487e-a268-657967d0b71e", "node_type": "1", "metadata": {}, "hash": "706cbd17ce0f9dc5d8ab0095a276d702b241e00b9088aac983b7600e5a70ae81", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12813, "end_char_idx": 15868, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callback to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_test_set, epochs=3, callbacks=[callback])
```

Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
</tf>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for causal language modeling, take a look at the corresponding
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).

</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Come up with a prompt you'd like to generate text from:

```py
>>> prompt = "Somatic hypermutation allows the immune system to"
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for text generation with your model, and pass your text to it:

```py
>>> from transformers import pipeline

>>> generator = pipeline("text-generation", model="username/my_awesome_eli5_clm-model")
>>> generator(prompt)
[{'generated_text': "Somatic hypermutation allows the immune system to be able to effectively reverse the damage caused by an infection.\n\n\nThe damage caused by an infection is caused by the immune system's ability to perform its own self-correcting tasks."}]
```

<frameworkcontent>
<pt>
Tokenize the text and return the `input_ids` as PyTorch tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("username/my_awesome_eli5_clm-model")
>>> inputs = tokenizer(prompt, return_tensors="pt").input_ids
```

Use the [`~generation.GenerationMixin.generate`] method to generate text.
For more details about the different text generation strategies and parameters for controlling generation, check out the [Text generation strategies](../generation_strategies) page.

```py
>>> from transformers import AutoModelForCausalLM

>>> model = AutoModelForCausalLM.from_pretrained("username/my_awesome_eli5_clm-model")
>>> outputs = model.generate(inputs, max_new_tokens=100, do_sample=True, top_k=50, top_p=0.95)
```

Decode the generated token ids back into text:

```py
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
["Somatic hypermutation allows the immune system to react to drugs with the ability to adapt to a different environmental situation. In other words, a system of 'hypermutation' can help the immune system to adapt to a different environmental situation or in some cases even a single life. In contrast, researchers at the University of Massachusetts-Boston have found that 'hypermutation' is much stronger in mice than in humans but can be found in humans, and that it's not completely unknown to the immune system.


key: _node_content
string_value:
{"id_": "37fbec03-0428-487e-a268-657967d0b71e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0ab020d2-9cfe-4bb1-a1c1-64f4a68a7d1d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers"}, "hash": "326c9c538adbd0a701b7618bacf6e60d06fa9b09a486befaf8143c411e1560e8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "792f192f-5f1c-4033-9d66-7d4c35c517a1", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "transformers"}, "hash": "b8466ba1126345346404343b8e9c471bdc41e3799f85941d8cc305d942e9fcc5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14441, "end_char_idx": 17489, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```

<frameworkcontent>
<pt>
Tokenize the text and return the `input_ids` as PyTorch tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("username/my_awesome_eli5_clm-model")
>>> inputs = tokenizer(prompt, return_tensors="pt").input_ids
```

Use the [`~generation.GenerationMixin.generate`] method to generate text.
For more details about the different text generation strategies and parameters for controlling generation, check out the [Text generation strategies](../generation_strategies) page.

```py
>>> from transformers import AutoModelForCausalLM

>>> model = AutoModelForCausalLM.from_pretrained("username/my_awesome_eli5_clm-model")
>>> outputs = model.generate(inputs, max_new_tokens=100, do_sample=True, top_k=50, top_p=0.95)
```

Decode the generated token ids back into text:

```py
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
["Somatic hypermutation allows the immune system to react to drugs with the ability to adapt to a different environmental situation. In other words, a system of 'hypermutation' can help the immune system to adapt to a different environmental situation or in some cases even a single life. In contrast, researchers at the University of Massachusetts-Boston have found that 'hypermutation' is much stronger in mice than in humans but can be found in humans, and that it's not completely unknown to the immune system. A study on how the immune system"]
```
</pt>
<tf>
Tokenize the text and return the `input_ids` as TensorFlow tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("username/my_awesome_eli5_clm-model")
>>> inputs = tokenizer(prompt, return_tensors="tf").input_ids
```

Use the [`~transformers.generation_tf_utils.TFGenerationMixin.generate`] method to create the summarization. For more details about the different text generation strategies and parameters for controlling generation, check out the [Text generation strategies](../generation_strategies) page.

```py
>>> from transformers import TFAutoModelForCausalLM

>>> model = TFAutoModelForCausalLM.from_pretrained("username/my_awesome_eli5_clm-model")
>>> outputs = model.generate(input_ids=inputs, max_new_tokens=100, do_sample=True, top_k=50, top_p=0.95)
```

Decode the generated token ids back into text:

```py
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['Somatic hypermutation allows the immune system to detect the presence of other viruses as they become more prevalent. Therefore, researchers have identified a high proportion of human viruses. The proportion of virus-associated viruses in our study increases with age. Therefore, we propose a simple algorithm to detect the presence of these new viruses in our samples as a sign of improved immunity. A first study based on this algorithm, which will be published in Science on Friday, aims to show that this finding could translate into the development of a better vaccine that is more effective for']
```
</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "17880d83-56f5-4b6d-ad62-a12837eee84f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "hash": "87094aa86af139e913445f44b4ef5b8586c1ce3c46f8f7ba6b5f094bd133cd43", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6779fcd0-f13a-403a-89c4-94628c393fc2", "node_type": "1", "metadata": {}, "hash": "656c0869a016a237b5020712d12db0967f1a86ebe21a4d51676f7326d4975deb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3069, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# LLM prompting guide

[[open-in-colab]]

Large Language Models such as Falcon, LLaMA, etc. are pretrained transformer models initially trained to predict the 
next token given some input text. They typically have billions of parameters and have been trained on trillions of 
tokens for an extended period of time. As a result, these models become quite powerful and versatile, and you can use 
them to solve multiple NLP tasks out of the box by instructing the models with natural language prompts.

Designing such prompts to ensure the optimal output is often called "prompt engineering". Prompt engineering is an 
iterative process that requires a fair amount of experimentation. Natural languages are much more flexible and expressive 
than programming languages, however, they can also introduce some ambiguity. At the same time, prompts in natural language 
are quite sensitive to changes. Even minor modifications in prompts can lead to wildly different outputs.

While there is no exact recipe for creating prompts to match all cases, researchers have worked out a number of best 
practices that help to achieve optimal results more consistently. 

This guide covers the prompt engineering best practices to help you craft better LLM prompts and solve various NLP tasks. 
You'll learn:

- [Basics of prompting](#basics-of-prompting)
- [Best practices of LLM prompting](#best-practices-of-llm-prompting)
- [Advanced prompting techniques: few-shot prompting and chain-of-thought](#advanced-prompting-techniques)
- [When to fine-tune instead of prompting](#prompting-vs-fine-tuning)

<Tip>

Prompt engineering is only a part of the LLM output optimization process. Another essential component is choosing the 
optimal text generation strategy. You can customize how your LLM selects each of the subsequent tokens when generating 
the text without modifying any of the trainable parameters. By tweaking the text generation parameters, you can reduce 
repetition in the generated text and make it more coherent and human-sounding. 
Text generation strategies and parameters are out of scope for this guide, but you can learn more about these topics in 
the following guides: 
 
* [Generation with LLMs](../llm_tutorial)
* [Text generation strategies](../generation_strategies)

</Tip>

## Basics of prompting

### Types of models 

The majority of modern LLMs are decoder-only transformers. Some examples include: [LLaMA](../model_doc/llama), 
[Llama2](../model_doc/llama2), [Falcon](../model_doc/falcon), [GPT2](../model_doc/gpt2). However, you may encounter
encoder-decoder transformer LLMs as well, for instance, [Flan-T5](../model_doc/flan-t5) and [BART](../model_doc/bart).

Encoder-decoder-style models are typically used in generative tasks where the output **heavily** relies on the input, for 
example, in translation and summarization. The decoder-only models are used for all other types of generative tasks.

When using a pipeline to generate text with an LLM, it's important to know what type of LLM you are using, because 
they use different pipelines.


key: _node_content
string_value:
{"id_": "6779fcd0-f13a-403a-89c4-94628c393fc2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "hash": "87094aa86af139e913445f44b4ef5b8586c1ce3c46f8f7ba6b5f094bd133cd43", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "17880d83-56f5-4b6d-ad62-a12837eee84f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "hash": "d65d612da17f442df4d403ead652e280ad2d3ce9651429e38f1623f95b7bfd94", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8fdf6f14-5d91-406a-929a-cb001f19d26d", "node_type": "1", "metadata": {}, "hash": "63581c22a359f713342e19976c753b25b474ebd67578855a8973c42e79663fde", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1670, "end_char_idx": 4865, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Another essential component is choosing the 
optimal text generation strategy. You can customize how your LLM selects each of the subsequent tokens when generating 
the text without modifying any of the trainable parameters. By tweaking the text generation parameters, you can reduce 
repetition in the generated text and make it more coherent and human-sounding. 
Text generation strategies and parameters are out of scope for this guide, but you can learn more about these topics in 
the following guides: 
 
* [Generation with LLMs](../llm_tutorial)
* [Text generation strategies](../generation_strategies)

</Tip>

## Basics of prompting

### Types of models 

The majority of modern LLMs are decoder-only transformers. Some examples include: [LLaMA](../model_doc/llama), 
[Llama2](../model_doc/llama2), [Falcon](../model_doc/falcon), [GPT2](../model_doc/gpt2). However, you may encounter
encoder-decoder transformer LLMs as well, for instance, [Flan-T5](../model_doc/flan-t5) and [BART](../model_doc/bart).

Encoder-decoder-style models are typically used in generative tasks where the output **heavily** relies on the input, for 
example, in translation and summarization. The decoder-only models are used for all other types of generative tasks.

When using a pipeline to generate text with an LLM, it's important to know what type of LLM you are using, because 
they use different pipelines. 

Run inference with decoder-only models with the `text-generation` pipeline:

```python
>>> from transformers import pipeline
>>> import torch

>>> torch.manual_seed(0) # doctest: +IGNORE_RESULT

>>> generator = pipeline('text-generation', model = 'openai-community/gpt2')
>>> prompt = "Hello, I'm a language model"

>>> generator(prompt, max_length = 30)
[{'generated_text': "Hello, I'm a language model programmer so you can use some of my stuff. But you also need some sort of a C program to run."}]
```

To run inference with an encoder-decoder, use the `text2text-generation` pipeline:

```python
>>> text2text_generator = pipeline("text2text-generation", model = 'google/flan-t5-base')
>>> prompt = "Translate from English to French: I'm very happy to see you"

>>> text2text_generator(prompt)
[{'generated_text': 'Je suis très heureuse de vous rencontrer.'}]
```

### Base vs instruct/chat models

Most of the recent LLM checkpoints available on 🤗 Hub come in two versions: base and instruct (or chat). For example, 
[`tiiuae/falcon-7b`](https://huggingface.co/tiiuae/falcon-7b) and [`tiiuae/falcon-7b-instruct`](https://huggingface.co/tiiuae/falcon-7b-instruct).

Base models are excellent at completing the text when given an initial prompt, however, they are not ideal for NLP tasks 
where they need to follow instructions, or for conversational use. This is where the instruct (chat) versions come in. 
These checkpoints are the result of further fine-tuning of the pre-trained base versions on instructions and conversational data. 
This additional fine-tuning makes them a better choice for many NLP tasks.  

Let's illustrate some simple prompts that you can use with [`tiiuae/falcon-7b-instruct`](https://huggingface.co/tiiuae/falcon-7b-instruct) 
to solve some common NLP tasks.


key: _node_content
string_value:
{"id_": "8fdf6f14-5d91-406a-929a-cb001f19d26d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "hash": "87094aa86af139e913445f44b4ef5b8586c1ce3c46f8f7ba6b5f094bd133cd43", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6779fcd0-f13a-403a-89c4-94628c393fc2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "hash": "a857a8d2bbb513bcd95c283fb6e009c883a4234dcdab3b3053e642075ce03b62", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dec1d932-6c77-49a5-87c7-08313ba1ccd7", "node_type": "1", "metadata": {}, "hash": "c3a31a8dd216c705a8499d4502e35785011d93f7a74ba4135ee68e9fee63196d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3520, "end_char_idx": 6568, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
But you also need some sort of a C program to run."}]
```

To run inference with an encoder-decoder, use the `text2text-generation` pipeline:

```python
>>> text2text_generator = pipeline("text2text-generation", model = 'google/flan-t5-base')
>>> prompt = "Translate from English to French: I'm very happy to see you"

>>> text2text_generator(prompt)
[{'generated_text': 'Je suis très heureuse de vous rencontrer.'}]
```

### Base vs instruct/chat models

Most of the recent LLM checkpoints available on 🤗 Hub come in two versions: base and instruct (or chat). For example, 
[`tiiuae/falcon-7b`](https://huggingface.co/tiiuae/falcon-7b) and [`tiiuae/falcon-7b-instruct`](https://huggingface.co/tiiuae/falcon-7b-instruct).

Base models are excellent at completing the text when given an initial prompt, however, they are not ideal for NLP tasks 
where they need to follow instructions, or for conversational use. This is where the instruct (chat) versions come in. 
These checkpoints are the result of further fine-tuning of the pre-trained base versions on instructions and conversational data. 
This additional fine-tuning makes them a better choice for many NLP tasks.  

Let's illustrate some simple prompts that you can use with [`tiiuae/falcon-7b-instruct`](https://huggingface.co/tiiuae/falcon-7b-instruct) 
to solve some common NLP tasks.

### NLP tasks 

First, let's set up the environment: 

```bash
pip install -q transformers accelerate
```

Next, let's load the model with the appropriate pipeline (`"text-generation"`): 

```python
>>> from transformers import pipeline, AutoTokenizer
>>> import torch

>>> torch.manual_seed(0) # doctest: +IGNORE_RESULT
>>> model = "tiiuae/falcon-7b-instruct"

>>> tokenizer = AutoTokenizer.from_pretrained(model)
>>> pipe = pipeline(
...     "text-generation",
...     model=model,
...     tokenizer=tokenizer,
...     torch_dtype=torch.bfloat16,
...     device_map="auto",
... )
```

<Tip>

Note that Falcon models were trained using the `bfloat16` datatype, so we recommend you use the same. This requires a recent 
version of CUDA and works best on modern cards.

</Tip>

Now that we have the model loaded via the pipeline, let's explore how you can use prompts to solve NLP tasks.

#### Text classification

One of the most common forms of text classification is sentiment analysis, which assigns a label like "positive", "negative", 
or "neutral" to a sequence of text. Let's write a prompt that instructs the model to classify a given text (a movie review). 
We'll start by giving the instruction, and then specifying the text to classify. Note that instead of leaving it at that, we're 
also adding the beginning of the response - `"Sentiment: "`:

```python
>>> torch.manual_seed(0) # doctest: +IGNORE_RESULT
>>> prompt = """Classify the text into neutral, negative or positive. 
... Text: This movie is definitely one of my favorite movies of its kind. The interaction between respectable and morally strong characters is an ode to chivalry and the honor code amongst thieves and policemen.


key: _node_content
string_value:
{"id_": "dec1d932-6c77-49a5-87c7-08313ba1ccd7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "hash": "87094aa86af139e913445f44b4ef5b8586c1ce3c46f8f7ba6b5f094bd133cd43", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8fdf6f14-5d91-406a-929a-cb001f19d26d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "hash": "834a333b360517b1a2f0a685352cfb0f17be893f33295b8797130ddf1ca536dc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "31da29e1-ba61-4572-8798-8d059cb5f036", "node_type": "1", "metadata": {}, "hash": "0767139c1b1a02a31e4eb8f57639e7a004174333d919b7509921baa7cac2659d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4867, "end_char_idx": 8037, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### NLP tasks 

First, let's set up the environment: 

```bash
pip install -q transformers accelerate
```

Next, let's load the model with the appropriate pipeline (`"text-generation"`): 

```python
>>> from transformers import pipeline, AutoTokenizer
>>> import torch

>>> torch.manual_seed(0) # doctest: +IGNORE_RESULT
>>> model = "tiiuae/falcon-7b-instruct"

>>> tokenizer = AutoTokenizer.from_pretrained(model)
>>> pipe = pipeline(
...     "text-generation",
...     model=model,
...     tokenizer=tokenizer,
...     torch_dtype=torch.bfloat16,
...     device_map="auto",
... )
```

<Tip>

Note that Falcon models were trained using the `bfloat16` datatype, so we recommend you use the same. This requires a recent 
version of CUDA and works best on modern cards.

</Tip>

Now that we have the model loaded via the pipeline, let's explore how you can use prompts to solve NLP tasks.

#### Text classification

One of the most common forms of text classification is sentiment analysis, which assigns a label like "positive", "negative", 
or "neutral" to a sequence of text. Let's write a prompt that instructs the model to classify a given text (a movie review). 
We'll start by giving the instruction, and then specifying the text to classify. Note that instead of leaving it at that, we're 
also adding the beginning of the response - `"Sentiment: "`:

```python
>>> torch.manual_seed(0) # doctest: +IGNORE_RESULT
>>> prompt = """Classify the text into neutral, negative or positive. 
... Text: This movie is definitely one of my favorite movies of its kind. The interaction between respectable and morally strong characters is an ode to chivalry and the honor code amongst thieves and policemen.
... Sentiment:
... """

>>> sequences = pipe(
...     prompt,
...     max_new_tokens=10,
... )

>>> for seq in sequences:
...     print(f"Result: {seq['generated_text']}")
Result: Classify the text into neutral, negative or positive. 
Text: This movie is definitely one of my favorite movies of its kind. The interaction between respectable and morally strong characters is an ode to chivalry and the honor code amongst thieves and policemen.
Sentiment:
Positive
```

As a result, the output contains a classification label from the list we have provided in the instructions, and it is a correct one!

<Tip>

You may notice that in addition to the prompt, we pass a `max_new_tokens` parameter. It controls the number of tokens the 
model shall generate, and it is one of the many text generation parameters that you can learn about 
in [Text generation strategies](../generation_strategies) guide.

</Tip>

#### Named Entity Recognition

Named Entity Recognition (NER) is a task of finding named entities in a piece of text, such as a person, location, or organization.
Let's modify the instructions in the prompt to make the LLM perform this task. Here, let's also set `return_full_text = False` 
so that output doesn't contain the prompt:

```python
>>> torch.manual_seed(1) # doctest: +IGNORE_RESULT
>>> prompt = """Return a list of named entities in the text.
... Text: The Golden State Warriors are an American professional basketball team based in San Francisco.


key: _node_content
string_value:
{"id_": "31da29e1-ba61-4572-8798-8d059cb5f036", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "hash": "87094aa86af139e913445f44b4ef5b8586c1ce3c46f8f7ba6b5f094bd133cd43", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dec1d932-6c77-49a5-87c7-08313ba1ccd7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "hash": "e6c8fb24860ffbfd8cb5f7f2e36ff24879b9c94cbfcc1435766318c2d5db766e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a1fae959-44d9-4531-b20b-ec51834eaa3f", "node_type": "1", "metadata": {}, "hash": "a48562b15cccd099d04321985244c4b7d302f5b2119c34af32d1d255531a06b8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6357, "end_char_idx": 9810, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
... Text: This movie is definitely one of my favorite movies of its kind. The interaction between respectable and morally strong characters is an ode to chivalry and the honor code amongst thieves and policemen.
... Sentiment:
... """

>>> sequences = pipe(
...     prompt,
...     max_new_tokens=10,
... )

>>> for seq in sequences:
...     print(f"Result: {seq['generated_text']}")
Result: Classify the text into neutral, negative or positive. 
Text: This movie is definitely one of my favorite movies of its kind. The interaction between respectable and morally strong characters is an ode to chivalry and the honor code amongst thieves and policemen.
Sentiment:
Positive
```

As a result, the output contains a classification label from the list we have provided in the instructions, and it is a correct one!

<Tip>

You may notice that in addition to the prompt, we pass a `max_new_tokens` parameter. It controls the number of tokens the 
model shall generate, and it is one of the many text generation parameters that you can learn about 
in [Text generation strategies](../generation_strategies) guide.

</Tip>

#### Named Entity Recognition

Named Entity Recognition (NER) is a task of finding named entities in a piece of text, such as a person, location, or organization.
Let's modify the instructions in the prompt to make the LLM perform this task. Here, let's also set `return_full_text = False` 
so that output doesn't contain the prompt:

```python
>>> torch.manual_seed(1) # doctest: +IGNORE_RESULT
>>> prompt = """Return a list of named entities in the text.
... Text: The Golden State Warriors are an American professional basketball team based in San Francisco.
... Named entities:
... """

>>> sequences = pipe(
...     prompt,
...     max_new_tokens=15,
...     return_full_text = False,    
... )

>>> for seq in sequences:
...     print(f"{seq['generated_text']}")
- Golden State Warriors
- San Francisco
```

As you can see, the model correctly identified two named entities from the given text.

#### Translation

Another task LLMs can perform is translation. You can choose to use encoder-decoder models for this task, however, here,
for the simplicity of the examples, we'll keep using Falcon-7b-instruct, which does a decent job. Once again, here's how 
you can write a basic prompt to instruct a model to translate a piece of text from English to Italian: 

```python
>>> torch.manual_seed(2) # doctest: +IGNORE_RESULT
>>> prompt = """Translate the English text to Italian.
... Text: Sometimes, I've believed as many as six impossible things before breakfast.
... Translation:
... """

>>> sequences = pipe(
...     prompt,
...     max_new_tokens=20,
...     do_sample=True,
...     top_k=10,
...     return_full_text = False,
... )

>>> for seq in sequences:
...     print(f"{seq['generated_text']}")
A volte, ho creduto a sei impossibili cose prima di colazione.
```

Here we've added a `do_sample=True` and `top_k=10` to allow the model to be a bit more flexible when generating output.

#### Text summarization

Similar to the translation, text summarization is another generative task where the output **heavily** relies on the input, 
and encoder-decoder models can be a better choice. However, decoder-style models can be used for this task as well.
Previously, we have placed the instructions at the very beginning of the prompt. However, the very end of the prompt can 
also be a suitable location for instructions.


key: _node_content
string_value:
{"id_": "a1fae959-44d9-4531-b20b-ec51834eaa3f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "hash": "87094aa86af139e913445f44b4ef5b8586c1ce3c46f8f7ba6b5f094bd133cd43", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "31da29e1-ba61-4572-8798-8d059cb5f036", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "hash": "35f6b7690dce355c700fb0184a5748ab05e332c2274615c356b58ac5e72355f3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6b3b6ea1-5d47-47b7-9f6d-fb4f09bc2f42", "node_type": "1", "metadata": {}, "hash": "da625d01de5300012b1895629a71e7dce3d00229a908353241e951748d7d2387", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8378, "end_char_idx": 11937, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
#### Translation

Another task LLMs can perform is translation. You can choose to use encoder-decoder models for this task, however, here,
for the simplicity of the examples, we'll keep using Falcon-7b-instruct, which does a decent job. Once again, here's how 
you can write a basic prompt to instruct a model to translate a piece of text from English to Italian: 

```python
>>> torch.manual_seed(2) # doctest: +IGNORE_RESULT
>>> prompt = """Translate the English text to Italian.
... Text: Sometimes, I've believed as many as six impossible things before breakfast.
... Translation:
... """

>>> sequences = pipe(
...     prompt,
...     max_new_tokens=20,
...     do_sample=True,
...     top_k=10,
...     return_full_text = False,
... )

>>> for seq in sequences:
...     print(f"{seq['generated_text']}")
A volte, ho creduto a sei impossibili cose prima di colazione.
```

Here we've added a `do_sample=True` and `top_k=10` to allow the model to be a bit more flexible when generating output.

#### Text summarization

Similar to the translation, text summarization is another generative task where the output **heavily** relies on the input, 
and encoder-decoder models can be a better choice. However, decoder-style models can be used for this task as well.
Previously, we have placed the instructions at the very beginning of the prompt. However, the very end of the prompt can 
also be a suitable location for instructions. Typically, it's better to place the instruction on one of the extreme ends.  

```python
>>> torch.manual_seed(3) # doctest: +IGNORE_RESULT
>>> prompt = """Permaculture is a design process mimicking the diversity, functionality and resilience of natural ecosystems. The principles and practices are drawn from traditional ecological knowledge of indigenous cultures combined with modern scientific understanding and technological innovations. Permaculture design provides a framework helping individuals and communities develop innovative, creative and effective strategies for meeting basic needs while preparing for and mitigating the projected impacts of climate change.
... Write a summary of the above text.
... Summary:
... """

>>> sequences = pipe(
...     prompt,
...     max_new_tokens=30,
...     do_sample=True,
...     top_k=10,
...     return_full_text = False,
... )

>>> for seq in sequences:
...     print(f"{seq['generated_text']}")
Permaculture is an ecological design mimicking natural ecosystems to meet basic needs and prepare for climate change. It is based on traditional knowledge and scientific understanding.
```

#### Question answering

For question answering task we can structure the prompt into the following logical components: instructions, context, question, and 
the leading word or phrase (`"Answer:"`) to nudge the model to start generating the answer:

```python
>>> torch.manual_seed(4) # doctest: +IGNORE_RESULT
>>> prompt = """Answer the question using the context below.
... Context: Gazpacho is a cold soup and drink made of raw, blended vegetables. Most gazpacho includes stale bread, tomato, cucumbers, onion, bell peppers, garlic, olive oil, wine vinegar, water, and salt. Northern recipes often include cumin and/or pimentón (smoked sweet paprika). Traditionally, gazpacho was made by pounding the vegetables in a mortar with a pestle; this more laborious method is still sometimes used as it helps keep the gazpacho cool and avoids the foam and silky consistency of smoothie versions made in blenders or food processors.
... Question: What modern tool is used to make gazpacho?


key: _node_content
string_value:
{"id_": "6b3b6ea1-5d47-47b7-9f6d-fb4f09bc2f42", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "hash": "87094aa86af139e913445f44b4ef5b8586c1ce3c46f8f7ba6b5f094bd133cd43", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a1fae959-44d9-4531-b20b-ec51834eaa3f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "hash": "4ca6fed20f5dc1192a37a97eaef919d33fbf61d094bf97f47f1b0cfb4d09795a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dbfd220e-e4f0-4327-8759-1679e6d108b2", "node_type": "1", "metadata": {}, "hash": "4965024648f8fdcedce6901e2caf15795f090780c617b1cb2d57af77a2df1b36", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10077, "end_char_idx": 13381, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The principles and practices are drawn from traditional ecological knowledge of indigenous cultures combined with modern scientific understanding and technological innovations. Permaculture design provides a framework helping individuals and communities develop innovative, creative and effective strategies for meeting basic needs while preparing for and mitigating the projected impacts of climate change.
... Write a summary of the above text.
... Summary:
... """

>>> sequences = pipe(
...     prompt,
...     max_new_tokens=30,
...     do_sample=True,
...     top_k=10,
...     return_full_text = False,
... )

>>> for seq in sequences:
...     print(f"{seq['generated_text']}")
Permaculture is an ecological design mimicking natural ecosystems to meet basic needs and prepare for climate change. It is based on traditional knowledge and scientific understanding.
```

#### Question answering

For question answering task we can structure the prompt into the following logical components: instructions, context, question, and 
the leading word or phrase (`"Answer:"`) to nudge the model to start generating the answer:

```python
>>> torch.manual_seed(4) # doctest: +IGNORE_RESULT
>>> prompt = """Answer the question using the context below.
... Context: Gazpacho is a cold soup and drink made of raw, blended vegetables. Most gazpacho includes stale bread, tomato, cucumbers, onion, bell peppers, garlic, olive oil, wine vinegar, water, and salt. Northern recipes often include cumin and/or pimentón (smoked sweet paprika). Traditionally, gazpacho was made by pounding the vegetables in a mortar with a pestle; this more laborious method is still sometimes used as it helps keep the gazpacho cool and avoids the foam and silky consistency of smoothie versions made in blenders or food processors.
... Question: What modern tool is used to make gazpacho?
... Answer:
... """

>>> sequences = pipe(
...     prompt,
...     max_new_tokens=10,
...     do_sample=True,
...     top_k=10,
...     return_full_text = False,
... )

>>> for seq in sequences:
...     print(f"Result: {seq['generated_text']}")
Result: Modern tools often used to make gazpacho include
```

#### Reasoning

Reasoning is one of the most difficult tasks for LLMs, and achieving good results often requires applying advanced prompting techniques, like 
[Chain-of-though](#chain-of-thought).

Let's try if we can make a model reason about a simple arithmetics task with a basic prompt: 

```python
>>> torch.manual_seed(5) # doctest: +IGNORE_RESULT
>>> prompt = """There are 5 groups of students in the class. Each group has 4 students. How many students are there in the class?"""

>>> sequences = pipe(
...     prompt,
...     max_new_tokens=30,
...     do_sample=True,
...     top_k=10,
...     return_full_text = False,
... )

>>> for seq in sequences:
...     print(f"Result: {seq['generated_text']}")
Result: 
There are a total of 5 groups, so there are 5 x 4=20 students in the class.
```

Correct! Let's increase the complexity a little and see if we can still get away with a basic prompt:

```python
>>> torch.manual_seed(6) # doctest: +IGNORE_RESULT
>>> prompt = """I baked 15 muffins. I ate 2 muffins and gave 5 muffins to a neighbor. My partner then bought 6 more muffins and ate 2. How many muffins do we now have?"""


key: _node_content
string_value:
{"id_": "dbfd220e-e4f0-4327-8759-1679e6d108b2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "hash": "87094aa86af139e913445f44b4ef5b8586c1ce3c46f8f7ba6b5f094bd133cd43", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6b3b6ea1-5d47-47b7-9f6d-fb4f09bc2f42", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "hash": "1beaf4910b255e2a1e20a0c80758fd8dfc22ed9395830a61a40f7c5d9b025dae", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f6db85fc-923e-4f8c-98a4-a3f66ae6de95", "node_type": "1", "metadata": {}, "hash": "abda1ecd7f6a252d38b6c6d758b288b1206f071d8922386db280b0f1ba6ace4b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11881, "end_char_idx": 15023, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
... Question: What modern tool is used to make gazpacho?
... Answer:
... """

>>> sequences = pipe(
...     prompt,
...     max_new_tokens=10,
...     do_sample=True,
...     top_k=10,
...     return_full_text = False,
... )

>>> for seq in sequences:
...     print(f"Result: {seq['generated_text']}")
Result: Modern tools often used to make gazpacho include
```

#### Reasoning

Reasoning is one of the most difficult tasks for LLMs, and achieving good results often requires applying advanced prompting techniques, like 
[Chain-of-though](#chain-of-thought).

Let's try if we can make a model reason about a simple arithmetics task with a basic prompt: 

```python
>>> torch.manual_seed(5) # doctest: +IGNORE_RESULT
>>> prompt = """There are 5 groups of students in the class. Each group has 4 students. How many students are there in the class?"""

>>> sequences = pipe(
...     prompt,
...     max_new_tokens=30,
...     do_sample=True,
...     top_k=10,
...     return_full_text = False,
... )

>>> for seq in sequences:
...     print(f"Result: {seq['generated_text']}")
Result: 
There are a total of 5 groups, so there are 5 x 4=20 students in the class.
```

Correct! Let's increase the complexity a little and see if we can still get away with a basic prompt:

```python
>>> torch.manual_seed(6) # doctest: +IGNORE_RESULT
>>> prompt = """I baked 15 muffins. I ate 2 muffins and gave 5 muffins to a neighbor. My partner then bought 6 more muffins and ate 2. How many muffins do we now have?"""

>>> sequences = pipe(
...     prompt,
...     max_new_tokens=10,
...     do_sample=True,
...     top_k=10,
...     return_full_text = False,
... )

>>> for seq in sequences:
...     print(f"Result: {seq['generated_text']}")
Result: 
The total number of muffins now is 21
```

This is a wrong answer, it should be 12. In this case, this can be due to the prompt being too basic, or due to the choice 
of model, after all we've picked the smallest version of Falcon. Reasoning is difficult for models of all sizes, but larger 
models are likely to perform better. 

## Best practices of LLM prompting

In this section of the guide we have compiled a list of best practices that tend to improve the prompt results:

* When choosing the model to work with, the latest and most capable models are likely to perform better. 
* Start with a simple and short prompt, and iterate from there.
* Put the instructions at the beginning of the prompt, or at the very end. When working with large context, models apply various optimizations to prevent Attention complexity from scaling quadratically. This may make a model more attentive to the beginning or end of a prompt than the middle.
* Clearly separate instructions from the text they apply to - more on this in the next section. 
* Be specific and descriptive about the task and the desired outcome - its format, length, style, language, etc.
* Avoid ambiguous descriptions and instructions.
* Favor instructions that say "what to do" instead of those that say "what not to do".
* "Lead" the output in the right direction by writing the first word (or even begin the first sentence for the model).


key: _node_content
string_value:
{"id_": "f6db85fc-923e-4f8c-98a4-a3f66ae6de95", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "hash": "87094aa86af139e913445f44b4ef5b8586c1ce3c46f8f7ba6b5f094bd133cd43", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dbfd220e-e4f0-4327-8759-1679e6d108b2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "hash": "9b7c6f45e6da5eaddd485c8beee12b3a2b2447f2ccd9906c27126869cbf96908", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "61c21fa6-b00f-475a-87e1-1c6f9e8dac0e", "node_type": "1", "metadata": {}, "hash": "8b3a606d274766b2746a230d1a90339abbdd921988772a75fd31113ad58271be", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13297, "end_char_idx": 16446, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
My partner then bought 6 more muffins and ate 2. How many muffins do we now have?"""

>>> sequences = pipe(
...     prompt,
...     max_new_tokens=10,
...     do_sample=True,
...     top_k=10,
...     return_full_text = False,
... )

>>> for seq in sequences:
...     print(f"Result: {seq['generated_text']}")
Result: 
The total number of muffins now is 21
```

This is a wrong answer, it should be 12. In this case, this can be due to the prompt being too basic, or due to the choice 
of model, after all we've picked the smallest version of Falcon. Reasoning is difficult for models of all sizes, but larger 
models are likely to perform better. 

## Best practices of LLM prompting

In this section of the guide we have compiled a list of best practices that tend to improve the prompt results:

* When choosing the model to work with, the latest and most capable models are likely to perform better. 
* Start with a simple and short prompt, and iterate from there.
* Put the instructions at the beginning of the prompt, or at the very end. When working with large context, models apply various optimizations to prevent Attention complexity from scaling quadratically. This may make a model more attentive to the beginning or end of a prompt than the middle.
* Clearly separate instructions from the text they apply to - more on this in the next section. 
* Be specific and descriptive about the task and the desired outcome - its format, length, style, language, etc.
* Avoid ambiguous descriptions and instructions.
* Favor instructions that say "what to do" instead of those that say "what not to do".
* "Lead" the output in the right direction by writing the first word (or even begin the first sentence for the model).
* Use advanced techniques like [Few-shot prompting](#few-shot-prompting) and [Chain-of-thought](#chain-of-thought)
* Test your prompts with different models to assess their robustness. 
* Version and track the performance of your prompts. 

## Advanced prompting techniques

### Few-shot prompting

The basic prompts in the sections above are the examples of "zero-shot" prompts, meaning, the model has been given 
instructions and context, but no examples with solutions. LLMs that have been fine-tuned on instruction datasets, generally 
perform well on such "zero-shot" tasks. However, you may find that your task has more complexity or nuance, and, perhaps, 
you have some requirements for the output that the model doesn't catch on just from the instructions. In this case, you can 
try the technique called few-shot prompting. 

In few-shot prompting, we provide examples in the prompt giving the model more context to improve the performance. 
The examples condition the model to generate the output following the patterns in the examples.

Here's an example: 

```python
>>> torch.manual_seed(0) # doctest: +IGNORE_RESULT
>>> prompt = """Text: The first human went into space and orbited the Earth on April 12, 1961.
... Date: 04/12/1961
... Text: The first-ever televised presidential debate in the United States took place on September 28, 1960, between presidential candidates John F. Kennedy and Richard Nixon.


key: _node_content
string_value:
{"id_": "61c21fa6-b00f-475a-87e1-1c6f9e8dac0e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "hash": "87094aa86af139e913445f44b4ef5b8586c1ce3c46f8f7ba6b5f094bd133cd43", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f6db85fc-923e-4f8c-98a4-a3f66ae6de95", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "hash": "53fe599b117017c912e3fa9aaa1a95cab1aef9c2ee2ca3228b6a3d31f8988c08", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a51ea7db-4125-47af-93d1-bae91071acc2", "node_type": "1", "metadata": {}, "hash": "c5a4439e54f05064446fc6b74544e1e4fcb7a049bdb97044326c2ee526b1998e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14656, "end_char_idx": 18090, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
* Be specific and descriptive about the task and the desired outcome - its format, length, style, language, etc.
* Avoid ambiguous descriptions and instructions.
* Favor instructions that say "what to do" instead of those that say "what not to do".
* "Lead" the output in the right direction by writing the first word (or even begin the first sentence for the model).
* Use advanced techniques like [Few-shot prompting](#few-shot-prompting) and [Chain-of-thought](#chain-of-thought)
* Test your prompts with different models to assess their robustness. 
* Version and track the performance of your prompts. 

## Advanced prompting techniques

### Few-shot prompting

The basic prompts in the sections above are the examples of "zero-shot" prompts, meaning, the model has been given 
instructions and context, but no examples with solutions. LLMs that have been fine-tuned on instruction datasets, generally 
perform well on such "zero-shot" tasks. However, you may find that your task has more complexity or nuance, and, perhaps, 
you have some requirements for the output that the model doesn't catch on just from the instructions. In this case, you can 
try the technique called few-shot prompting. 

In few-shot prompting, we provide examples in the prompt giving the model more context to improve the performance. 
The examples condition the model to generate the output following the patterns in the examples.

Here's an example: 

```python
>>> torch.manual_seed(0) # doctest: +IGNORE_RESULT
>>> prompt = """Text: The first human went into space and orbited the Earth on April 12, 1961.
... Date: 04/12/1961
... Text: The first-ever televised presidential debate in the United States took place on September 28, 1960, between presidential candidates John F. Kennedy and Richard Nixon. 
... Date:"""

>>> sequences = pipe(
...     prompt,
...     max_new_tokens=8,
...     do_sample=True,
...     top_k=10,
... )

>>> for seq in sequences:
...     print(f"Result: {seq['generated_text']}")
Result: Text: The first human went into space and orbited the Earth on April 12, 1961.
Date: 04/12/1961
Text: The first-ever televised presidential debate in the United States took place on September 28, 1960, between presidential candidates John F. Kennedy and Richard Nixon. 
Date: 09/28/1960
```

In the above code snippet we used a single example to demonstrate the desired output to the model, so this can be called a 
"one-shot" prompting. However, depending on the task complexity you may need to use more than one example. 

Limitations of the few-shot prompting technique: 
- While LLMs can pick up on the patterns in the examples, these technique doesn't work well on complex reasoning tasks
- Few-shot prompting requires creating lengthy prompts. Prompts with large number of tokens can increase computation and latency. There's also a limit to the length of the prompts.  
- Sometimes when given a number of examples, models can learn patterns that you didn't intend them to learn, e.g. that the third movie review is always negative.

### Chain-of-thought

Chain-of-thought (CoT) prompting is a technique that nudges a model to produce intermediate reasoning steps thus improving 
the results on complex reasoning tasks. 

There are two ways of steering a model to producing the reasoning steps:
- few-shot prompting by illustrating examples with detailed answers to questions, showing the model how to work through a problem.


key: _node_content
string_value:
{"id_": "a51ea7db-4125-47af-93d1-bae91071acc2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "hash": "87094aa86af139e913445f44b4ef5b8586c1ce3c46f8f7ba6b5f094bd133cd43", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "61c21fa6-b00f-475a-87e1-1c6f9e8dac0e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "hash": "0a7054e4652aeaf5821ccb87c14193d8e602117dbf19efd1fb3bd0014415f7de", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0df8790f-8999-4dac-b2a0-4384472d7e73", "node_type": "1", "metadata": {}, "hash": "c96370dedb093cf98777c9e0a2f9c993a97a7eceb89f76d041faa082788e1e0f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16448, "end_char_idx": 19751, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
... Date:"""

>>> sequences = pipe(
...     prompt,
...     max_new_tokens=8,
...     do_sample=True,
...     top_k=10,
... )

>>> for seq in sequences:
...     print(f"Result: {seq['generated_text']}")
Result: Text: The first human went into space and orbited the Earth on April 12, 1961.
Date: 04/12/1961
Text: The first-ever televised presidential debate in the United States took place on September 28, 1960, between presidential candidates John F. Kennedy and Richard Nixon. 
Date: 09/28/1960
```

In the above code snippet we used a single example to demonstrate the desired output to the model, so this can be called a 
"one-shot" prompting. However, depending on the task complexity you may need to use more than one example. 

Limitations of the few-shot prompting technique: 
- While LLMs can pick up on the patterns in the examples, these technique doesn't work well on complex reasoning tasks
- Few-shot prompting requires creating lengthy prompts. Prompts with large number of tokens can increase computation and latency. There's also a limit to the length of the prompts.  
- Sometimes when given a number of examples, models can learn patterns that you didn't intend them to learn, e.g. that the third movie review is always negative.

### Chain-of-thought

Chain-of-thought (CoT) prompting is a technique that nudges a model to produce intermediate reasoning steps thus improving 
the results on complex reasoning tasks. 

There are two ways of steering a model to producing the reasoning steps:
- few-shot prompting by illustrating examples with detailed answers to questions, showing the model how to work through a problem.
- by instructing the model to reason by adding phrases like "Let's think step by step" or "Take a deep breath and work through the problem step by step."

If we apply the CoT technique to the muffins example from the [reasoning section](#reasoning) and use a larger model, 
such as (`tiiuae/falcon-180B-chat`) which you can play with in the [HuggingChat](https://huggingface.co/chat/), 
we'll get a significant improvement on the reasoning result:

```text
Let's go through this step-by-step:
1. You start with 15 muffins.
2. You eat 2 muffins, leaving you with 13 muffins.
3. You give 5 muffins to your neighbor, leaving you with 8 muffins.
4. Your partner buys 6 more muffins, bringing the total number of muffins to 14.
5. Your partner eats 2 muffins, leaving you with 12 muffins.
Therefore, you now have 12 muffins.
```

## Prompting vs fine-tuning

You can achieve great results by optimizing your prompts, however, you may still ponder whether fine-tuning a model 
would work better for your case. Here are some scenarios when fine-tuning a smaller model may be a preferred option:

- Your domain is wildly different from what LLMs were pre-trained on and extensive prompt optimization did not yield sufficient results. 
- You need your model to work well in a low-resource language.
- You need the model to be trained on sensitive data that is under strict regulations. 
- You have to use a small model due to cost, privacy, infrastructure or other limitations. 

In all of the above examples, you will need to make sure that you either already have or can easily obtain a large enough 
domain-specific dataset at a reasonable cost to fine-tune a model.


key: _node_content
string_value:
{"id_": "0df8790f-8999-4dac-b2a0-4384472d7e73", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1de2f80f-f53c-43e8-b5f3-49b3b915c9f7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "hash": "87094aa86af139e913445f44b4ef5b8586c1ce3c46f8f7ba6b5f094bd133cd43", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a51ea7db-4125-47af-93d1-bae91071acc2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "transformers"}, "hash": "5945652a544f9922ead69b9aa9b6069f1a29b979865e6104bc159b0e7daa48bc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18246, "end_char_idx": 19928, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If we apply the CoT technique to the muffins example from the [reasoning section](#reasoning) and use a larger model, 
such as (`tiiuae/falcon-180B-chat`) which you can play with in the [HuggingChat](https://huggingface.co/chat/), 
we'll get a significant improvement on the reasoning result:

```text
Let's go through this step-by-step:
1. You start with 15 muffins.
2. You eat 2 muffins, leaving you with 13 muffins.
3. You give 5 muffins to your neighbor, leaving you with 8 muffins.
4. Your partner buys 6 more muffins, bringing the total number of muffins to 14.
5. Your partner eats 2 muffins, leaving you with 12 muffins.
Therefore, you now have 12 muffins.
```

## Prompting vs fine-tuning

You can achieve great results by optimizing your prompts, however, you may still ponder whether fine-tuning a model 
would work better for your case. Here are some scenarios when fine-tuning a smaller model may be a preferred option:

- Your domain is wildly different from what LLMs were pre-trained on and extensive prompt optimization did not yield sufficient results. 
- You need your model to work well in a low-resource language.
- You need the model to be trained on sensitive data that is under strict regulations. 
- You have to use a small model due to cost, privacy, infrastructure or other limitations. 

In all of the above examples, you will need to make sure that you either already have or can easily obtain a large enough 
domain-specific dataset at a reasonable cost to fine-tune a model. You will also need to have enough time and resources 
to fine-tune a model.

If the above examples are not the case for you, optimizing prompts can prove to be more beneficial.


key: _node_content
string_value:
{"id_": "6edf77b6-42cf-4c62-9046-09a7c8058dbc", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_feature_extraction", "title": "Image Feature Extraction", "tokens": 1193, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea7b80cd-6ab7-4a17-8572-9839fa0fd05c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_feature_extraction", "title": "Image Feature Extraction", "tokens": 1193, "retrieve_doc": true, "source": "transformers"}, "hash": "f3c15d4d2bf0ad340d04363c6ded3b1efccdb513f6513786ce9db9b215c5c609", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bba040d3-860b-45be-9a4d-6efd8480cbf5", "node_type": "1", "metadata": {}, "hash": "36a127afa0ec2632ec488a1ebb400bd26a02d56908eb287d635de86b6cfdbd84", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3190, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Image Feature Extraction

[[open-in-colab]]

Image feature extraction is the task of extracting semantically meaningful features given an image. This has many use cases, including image similarity and image retrieval. Moreover, most computer vision models can be used for image feature extraction, where one can remove the task-specific head (image classification, object detection etc) and get the features. These features are very useful on a higher level: edge detection, corner detection and so on. They may also contain information about the real world (e.g. what a cat looks like) depending on how deep the model is. Therefore, these outputs can be used to train new classifiers on a specific dataset.

In this guide, you will:

- Learn to build a simple image similarity system on top of the `image-feature-extraction` pipeline.
- Accomplish the same task with bare model inference.

## Image Similarity using `image-feature-extraction` Pipeline

We have two images of cats sitting on top of fish nets, one of them is generated. 

```python
from PIL import Image
import requests

img_urls = ["https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/cats.png", "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/cats.jpeg"]
image_real = Image.open(requests.get(img_urls[0], stream=True).raw).convert("RGB")
image_gen = Image.open(requests.get(img_urls[1], stream=True).raw).convert("RGB")
```

Let's see the pipeline in action. First, initialize the pipeline. If you don't pass any model to it, the pipeline will be automatically initialized with [google/vit-base-patch16-224](google/vit-base-patch16-224). If you'd like to calculate similarity, set `pool` to True.

```python
import torch
from transformers import pipeline

DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
pipe = pipeline(task="image-feature-extraction", model_name="google/vit-base-patch16-384", device=DEVICE, pool=True)
```

To infer with `pipe` pass both images to it.

```python
outputs = pipe([image_real, image_gen])
```

The output contains pooled embeddings of those two images.

```python
# get the length of a single output
print(len(outputs[0][0]))
# show outputs
print(outputs)

# 768
# [[[-0.03909236937761307, 0.43381670117378235, -0.06913255900144577,
```

To get the similarity score, we need to pass them to a similarity function. 

```python
from torch.nn.functional import cosine_similarity

similarity_score = cosine_similarity(torch.Tensor(outputs[0]),
                                     torch.Tensor(outputs[1]), dim=1)

print(similarity_score)

# tensor([0.6043])
```

If you want to get the last hidden states before pooling, avoid passing any value for the `pool` parameter, as it is set to `False` by default. These hidden states are useful for training new classifiers or models based on the features from the model.

```python
pipe = pipeline(task="image-feature-extraction", model_name="google/vit-base-patch16-224", device=DEVICE)
output = pipe(image_real)
```

Since the outputs are unpooled, we get the last hidden states where the first dimension is the batch size, and the last two are the embedding shape.


key: _node_content
string_value:
{"id_": "bba040d3-860b-45be-9a4d-6efd8480cbf5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_feature_extraction", "title": "Image Feature Extraction", "tokens": 1193, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ea7b80cd-6ab7-4a17-8572-9839fa0fd05c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_feature_extraction", "title": "Image Feature Extraction", "tokens": 1193, "retrieve_doc": true, "source": "transformers"}, "hash": "f3c15d4d2bf0ad340d04363c6ded3b1efccdb513f6513786ce9db9b215c5c609", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6edf77b6-42cf-4c62-9046-09a7c8058dbc", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_feature_extraction", "title": "Image Feature Extraction", "tokens": 1193, "retrieve_doc": true, "source": "transformers"}, "hash": "a5de3d07742147b3cb9f23467455361cafea61d21d1b5dc9e9ee31cddd8deba3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1665, "end_char_idx": 4439, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If you'd like to calculate similarity, set `pool` to True.

```python
import torch
from transformers import pipeline

DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
pipe = pipeline(task="image-feature-extraction", model_name="google/vit-base-patch16-384", device=DEVICE, pool=True)
```

To infer with `pipe` pass both images to it.

```python
outputs = pipe([image_real, image_gen])
```

The output contains pooled embeddings of those two images.

```python
# get the length of a single output
print(len(outputs[0][0]))
# show outputs
print(outputs)

# 768
# [[[-0.03909236937761307, 0.43381670117378235, -0.06913255900144577,
```

To get the similarity score, we need to pass them to a similarity function. 

```python
from torch.nn.functional import cosine_similarity

similarity_score = cosine_similarity(torch.Tensor(outputs[0]),
                                     torch.Tensor(outputs[1]), dim=1)

print(similarity_score)

# tensor([0.6043])
```

If you want to get the last hidden states before pooling, avoid passing any value for the `pool` parameter, as it is set to `False` by default. These hidden states are useful for training new classifiers or models based on the features from the model.

```python
pipe = pipeline(task="image-feature-extraction", model_name="google/vit-base-patch16-224", device=DEVICE)
output = pipe(image_real)
```

Since the outputs are unpooled, we get the last hidden states where the first dimension is the batch size, and the last two are the embedding shape.

```python
import numpy as np
print(np.array(outputs).shape)
# (1, 197, 768)
```

## Getting Features and Similarities using `AutoModel`

We can also use `AutoModel` class of transformers to get the features. `AutoModel` loads any transformers model with no task-specific head, and we can use this to get the features.

```python
from transformers import AutoImageProcessor, AutoModel

processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224")
model = AutoModel.from_pretrained("google/vit-base-patch16-224").to(DEVICE)
```

Let's write a simple function for inference. We will pass the inputs to the `processor` first and pass its outputs to the `model`.

```python
def infer(image):
  inputs = processor(image, return_tensors="pt").to(DEVICE)
  outputs = model(**inputs)
  return outputs.pooler_output
```

We can pass the images directly to this function and get the embeddings.

```python
embed_real = infer(image_real)
embed_gen = infer(image_gen)
```

We can get the similarity again over the embeddings.

```python
from torch.nn.functional import cosine_similarity

similarity_score = cosine_similarity(embed_real, embed_gen, dim=1)
print(similarity_score)

# tensor([0.6061], device='cuda:0', grad_fn=<SumBackward1>)
```


key: _node_content
string_value:
{"id_": "ec26ebd8-0874-431b-83ac-2174cd73007d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/timesformer", "title": "TimeSformer", "tokens": 618, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8a98df82-0cd5-45d8-ac75-c95fce631549", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/timesformer", "title": "TimeSformer", "tokens": 618, "retrieve_doc": true, "source": "transformers"}, "hash": "8a6d62b2b634dd0d8525b1f44755a2784b055d22d16f36afd0c670958b8c63dc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2264, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# TimeSformer

## Overview

The TimeSformer model was proposed in [TimeSformer: Is Space-Time Attention All You Need for Video Understanding?](https://arxiv.org/abs/2102.05095) by Facebook Research.
This work is a milestone in action-recognition field being the first video transformer. It inspired many transformer based video understanding and classification papers.

The abstract from the paper is the following:

*We present a convolution-free approach to video classification built exclusively on self-attention over space and time. Our method, named "TimeSformer," adapts the standard Transformer architecture to video by enabling spatiotemporal feature learning directly from a sequence of frame-level patches. Our experimental study compares different self-attention schemes and suggests that "divided attention," where temporal attention and spatial attention are separately applied within each block, leads to the best video classification accuracy among the design choices considered. Despite the radically new design, TimeSformer achieves state-of-the-art results on several action recognition benchmarks, including the best reported accuracy on Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks, our model is faster to train, it can achieve dramatically higher test efficiency (at a small drop in accuracy), and it can also be applied to much longer video clips (over one minute long). Code and models are available at: [this https URL](https://github.com/facebookresearch/TimeSformer).*

This model was contributed by [fcakyon](https://huggingface.co/fcakyon).
The original code can be found [here](https://github.com/facebookresearch/TimeSformer).

## Usage tips

There are many pretrained variants. Select your pretrained model based on the dataset it is trained on. Moreover,
the number of input frames per clip changes based on the model size so you should consider this parameter while selecting your pretrained model.

## Resources

- [Video classification task guide](../tasks/video_classification)

## TimesformerConfig

[[autodoc]] TimesformerConfig

## TimesformerModel

[[autodoc]] TimesformerModel
    - forward

## TimesformerForVideoClassification

[[autodoc]] TimesformerForVideoClassification
    - forward


key: _node_content
string_value:
{"id_": "f9a8ba69-b972-4790-9ac6-c84bde842107", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deberta-v2", "title": "DeBERTa-v2", "tokens": 1600, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "48d7be7f-faba-4d81-a3f6-9657d9866e78", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deberta-v2", "title": "DeBERTa-v2", "tokens": 1600, "retrieve_doc": true, "source": "transformers"}, "hash": "a2b0a3f7da958b59525f998fad765610dcae6d310d2e5b34c37ee1d9dd99673f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "308c0ac3-6452-458c-881c-c6aa986427f2", "node_type": "1", "metadata": {}, "hash": "5c0b5410f0f79766d3f04ee7b3176639182f584280148e40fb995f1e79f65b53", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3233, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# DeBERTa-v2

## Overview

The DeBERTa model was proposed in [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://arxiv.org/abs/2006.03654) by Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen It is based on Google's
BERT model released in 2018 and Facebook's RoBERTa model released in 2019.

It builds on RoBERTa with disentangled attention and enhanced mask decoder training with half of the data used in
RoBERTa.

The abstract from the paper is the following:

*Recent progress in pre-trained neural language models has significantly improved the performance of many natural
language processing (NLP) tasks. In this paper we propose a new model architecture DeBERTa (Decoding-enhanced BERT with
disentangled attention) that improves the BERT and RoBERTa models using two novel techniques. The first is the
disentangled attention mechanism, where each word is represented using two vectors that encode its content and
position, respectively, and the attention weights among words are computed using disentangled matrices on their
contents and relative positions. Second, an enhanced mask decoder is used to replace the output softmax layer to
predict the masked tokens for model pretraining. We show that these two techniques significantly improve the efficiency
of model pretraining and performance of downstream tasks. Compared to RoBERTa-Large, a DeBERTa model trained on half of
the training data performs consistently better on a wide range of NLP tasks, achieving improvements on MNLI by +0.9%
(90.2% vs. 91.1%), on SQuAD v2.0 by +2.3% (88.4% vs. 90.7%) and RACE by +3.6% (83.2% vs. 86.8%). The DeBERTa code and
pre-trained models will be made publicly available at https://github.com/microsoft/DeBERTa.*


The following information is visible directly on the [original implementation
repository](https://github.com/microsoft/DeBERTa). DeBERTa v2 is the second version of the DeBERTa model. It includes
the 1.5B model used for the SuperGLUE single-model submission and achieving 89.9, versus human baseline 89.8. You can
find more details about this submission in the authors'
[blog](https://www.microsoft.com/en-us/research/blog/microsoft-deberta-surpasses-human-performance-on-the-superglue-benchmark/)

New in v2:

- **Vocabulary** In v2 the tokenizer is changed to use a new vocabulary of size 128K built from the training data.
  Instead of a GPT2-based tokenizer, the tokenizer is now
  [sentencepiece-based](https://github.com/google/sentencepiece) tokenizer.
- **nGiE(nGram Induced Input Encoding)** The DeBERTa-v2 model uses an additional convolution layer aside with the first
  transformer layer to better learn the local dependency of input tokens.
- **Sharing position projection matrix with content projection matrix in attention layer** Based on previous
  experiments, this can save parameters without affecting the performance.
- **Apply bucket to encode relative positions** The DeBERTa-v2 model uses log bucket to encode relative positions
  similar to T5.
- **900M model & 1.5B model** Two additional model sizes are available: 900M and 1.5B, which significantly improves the
  performance of downstream tasks.

This model was contributed by [DeBERTa](https://huggingface.co/DeBERTa).


key: _node_content
string_value:
{"id_": "308c0ac3-6452-458c-881c-c6aa986427f2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deberta-v2", "title": "DeBERTa-v2", "tokens": 1600, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "48d7be7f-faba-4d81-a3f6-9657d9866e78", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deberta-v2", "title": "DeBERTa-v2", "tokens": 1600, "retrieve_doc": true, "source": "transformers"}, "hash": "a2b0a3f7da958b59525f998fad765610dcae6d310d2e5b34c37ee1d9dd99673f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f9a8ba69-b972-4790-9ac6-c84bde842107", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deberta-v2", "title": "DeBERTa-v2", "tokens": 1600, "retrieve_doc": true, "source": "transformers"}, "hash": "9dcf18ab7d28fa214fd4eb62252653e9e79c6043ff3affb1e9f902b33e09b1e3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2436728d-7440-46c3-a5e2-56c0177fb27c", "node_type": "1", "metadata": {}, "hash": "dd5f0f9b6e3dc3970365ecc33f1ef9975b636bd5c84316f667c8b3ce5db19402", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1736, "end_char_idx": 3409, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The following information is visible directly on the [original implementation
repository](https://github.com/microsoft/DeBERTa). DeBERTa v2 is the second version of the DeBERTa model. It includes
the 1.5B model used for the SuperGLUE single-model submission and achieving 89.9, versus human baseline 89.8. You can
find more details about this submission in the authors'
[blog](https://www.microsoft.com/en-us/research/blog/microsoft-deberta-surpasses-human-performance-on-the-superglue-benchmark/)

New in v2:

- **Vocabulary** In v2 the tokenizer is changed to use a new vocabulary of size 128K built from the training data.
  Instead of a GPT2-based tokenizer, the tokenizer is now
  [sentencepiece-based](https://github.com/google/sentencepiece) tokenizer.
- **nGiE(nGram Induced Input Encoding)** The DeBERTa-v2 model uses an additional convolution layer aside with the first
  transformer layer to better learn the local dependency of input tokens.
- **Sharing position projection matrix with content projection matrix in attention layer** Based on previous
  experiments, this can save parameters without affecting the performance.
- **Apply bucket to encode relative positions** The DeBERTa-v2 model uses log bucket to encode relative positions
  similar to T5.
- **900M model & 1.5B model** Two additional model sizes are available: 900M and 1.5B, which significantly improves the
  performance of downstream tasks.

This model was contributed by [DeBERTa](https://huggingface.co/DeBERTa). This model TF 2.0 implementation was
contributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/microsoft/DeBERTa).


key: _node_content
string_value:
{"id_": "2436728d-7440-46c3-a5e2-56c0177fb27c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deberta-v2", "title": "DeBERTa-v2", "tokens": 1600, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "48d7be7f-faba-4d81-a3f6-9657d9866e78", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deberta-v2", "title": "DeBERTa-v2", "tokens": 1600, "retrieve_doc": true, "source": "transformers"}, "hash": "a2b0a3f7da958b59525f998fad765610dcae6d310d2e5b34c37ee1d9dd99673f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "308c0ac3-6452-458c-881c-c6aa986427f2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/deberta-v2", "title": "DeBERTa-v2", "tokens": 1600, "retrieve_doc": true, "source": "transformers"}, "hash": "8199d0506116bfa5c5c9eb7073030c49fcb48bd014b8ad8524ac907b26abdaa5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1865, "end_char_idx": 5383, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
DeBERTa v2 is the second version of the DeBERTa model. It includes
the 1.5B model used for the SuperGLUE single-model submission and achieving 89.9, versus human baseline 89.8. You can
find more details about this submission in the authors'
[blog](https://www.microsoft.com/en-us/research/blog/microsoft-deberta-surpasses-human-performance-on-the-superglue-benchmark/)

New in v2:

- **Vocabulary** In v2 the tokenizer is changed to use a new vocabulary of size 128K built from the training data.
  Instead of a GPT2-based tokenizer, the tokenizer is now
  [sentencepiece-based](https://github.com/google/sentencepiece) tokenizer.
- **nGiE(nGram Induced Input Encoding)** The DeBERTa-v2 model uses an additional convolution layer aside with the first
  transformer layer to better learn the local dependency of input tokens.
- **Sharing position projection matrix with content projection matrix in attention layer** Based on previous
  experiments, this can save parameters without affecting the performance.
- **Apply bucket to encode relative positions** The DeBERTa-v2 model uses log bucket to encode relative positions
  similar to T5.
- **900M model & 1.5B model** Two additional model sizes are available: 900M and 1.5B, which significantly improves the
  performance of downstream tasks.

This model was contributed by [DeBERTa](https://huggingface.co/DeBERTa). This model TF 2.0 implementation was
contributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/microsoft/DeBERTa).

## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Masked language modeling task guide](../tasks/masked_language_modeling)
- [Multiple choice task guide](../tasks/multiple_choice)

## DebertaV2Config

[[autodoc]] DebertaV2Config

## DebertaV2Tokenizer

[[autodoc]] DebertaV2Tokenizer
    - build_inputs_with_special_tokens
    - get_special_tokens_mask
    - create_token_type_ids_from_sequences
    - save_vocabulary

## DebertaV2TokenizerFast

[[autodoc]] DebertaV2TokenizerFast
    - build_inputs_with_special_tokens
    - create_token_type_ids_from_sequences

<frameworkcontent>
<pt>

## DebertaV2Model

[[autodoc]] DebertaV2Model
    - forward

## DebertaV2PreTrainedModel

[[autodoc]] DebertaV2PreTrainedModel
    - forward

## DebertaV2ForMaskedLM

[[autodoc]] DebertaV2ForMaskedLM
    - forward

## DebertaV2ForSequenceClassification

[[autodoc]] DebertaV2ForSequenceClassification
    - forward

## DebertaV2ForTokenClassification

[[autodoc]] DebertaV2ForTokenClassification
    - forward

## DebertaV2ForQuestionAnswering

[[autodoc]] DebertaV2ForQuestionAnswering
    - forward

## DebertaV2ForMultipleChoice

[[autodoc]] DebertaV2ForMultipleChoice
    - forward

</pt>
<tf>

## TFDebertaV2Model

[[autodoc]] TFDebertaV2Model
    - call

## TFDebertaV2PreTrainedModel

[[autodoc]] TFDebertaV2PreTrainedModel
    - call

## TFDebertaV2ForMaskedLM

[[autodoc]] TFDebertaV2ForMaskedLM
    - call

## TFDebertaV2ForSequenceClassification

[[autodoc]] TFDebertaV2ForSequenceClassification
    - call

## TFDebertaV2ForTokenClassification

[[autodoc]] TFDebertaV2ForTokenClassification
    - call

## TFDebertaV2ForQuestionAnswering

[[autodoc]] TFDebertaV2ForQuestionAnswering
    - call

## TFDebertaV2ForMultipleChoice

[[autodoc]] TFDebertaV2ForMultipleChoice
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "b8bedfe5-feb1-4e0b-bcc0-c3fe71775f06", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/chinese_clip", "title": "Chinese-CLIP", "tokens": 1360, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cf55ed01-ba10-4c53-a99f-402b16d63c90", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/chinese_clip", "title": "Chinese-CLIP", "tokens": 1360, "retrieve_doc": true, "source": "transformers"}, "hash": "7be88278ac429548bc0d58c1994b3bb78b3b6e719e977a7a3b2651371ee5e6c1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "97d81c10-e8a3-422a-8708-47bdbbd91c99", "node_type": "1", "metadata": {}, "hash": "17f2bd829d8c2ac5492787b81b2441a879e5e6c1cbde4e13e95adfa66ac5bb1f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1854, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Chinese-CLIP

## Overview

The Chinese-CLIP model was proposed in [Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese](https://arxiv.org/abs/2211.01335) by An Yang, Junshu Pan, Junyang Lin, Rui Men, Yichang Zhang, Jingren Zhou, Chang Zhou.
Chinese-CLIP is an implementation of CLIP (Radford et al., 2021) on a large-scale dataset of Chinese image-text pairs. It is capable of performing cross-modal retrieval and also playing as a vision backbone for vision tasks like zero-shot image classification, open-domain object detection, etc. The original Chinese-CLIP code is released [at this link](https://github.com/OFA-Sys/Chinese-CLIP).

The abstract from the paper is the following:

*The tremendous success of CLIP (Radford et al., 2021) has promoted the research and application of contrastive learning for vision-language pretraining. In this work, we construct a large-scale dataset of image-text pairs in Chinese, where most data are retrieved from publicly available datasets, and we pretrain Chinese CLIP models on the new dataset. We develop 5 Chinese CLIP models of multiple sizes, spanning from 77 to 958 million parameters. Furthermore, we propose a two-stage pretraining method, where the model is first trained with the image encoder frozen and then trained with all parameters being optimized, to achieve enhanced model performance. Our comprehensive experiments demonstrate that Chinese CLIP can achieve the state-of-the-art performance on MUGE, Flickr30K-CN, and COCO-CN in the setups of zero-shot learning and finetuning, and it is able to achieve competitive performance in zero-shot image classification based on the evaluation on the ELEVATER benchmark (Li et al., 2022). Our codes, pretrained models, and demos have been released.*

The Chinese-CLIP model was contributed by [OFA-Sys](https://huggingface.co/OFA-Sys).


key: _node_content
string_value:
{"id_": "97d81c10-e8a3-422a-8708-47bdbbd91c99", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/chinese_clip", "title": "Chinese-CLIP", "tokens": 1360, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cf55ed01-ba10-4c53-a99f-402b16d63c90", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/chinese_clip", "title": "Chinese-CLIP", "tokens": 1360, "retrieve_doc": true, "source": "transformers"}, "hash": "7be88278ac429548bc0d58c1994b3bb78b3b6e719e977a7a3b2651371ee5e6c1", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b8bedfe5-feb1-4e0b-bcc0-c3fe71775f06", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/chinese_clip", "title": "Chinese-CLIP", "tokens": 1360, "retrieve_doc": true, "source": "transformers"}, "hash": "241507b6ecfee96fb26dd0b1c9eb6f233998d68861ed27921da9e12ae5633bd1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 256, "end_char_idx": 4559, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Chinese-CLIP is an implementation of CLIP (Radford et al., 2021) on a large-scale dataset of Chinese image-text pairs. It is capable of performing cross-modal retrieval and also playing as a vision backbone for vision tasks like zero-shot image classification, open-domain object detection, etc. The original Chinese-CLIP code is released [at this link](https://github.com/OFA-Sys/Chinese-CLIP).

The abstract from the paper is the following:

*The tremendous success of CLIP (Radford et al., 2021) has promoted the research and application of contrastive learning for vision-language pretraining. In this work, we construct a large-scale dataset of image-text pairs in Chinese, where most data are retrieved from publicly available datasets, and we pretrain Chinese CLIP models on the new dataset. We develop 5 Chinese CLIP models of multiple sizes, spanning from 77 to 958 million parameters. Furthermore, we propose a two-stage pretraining method, where the model is first trained with the image encoder frozen and then trained with all parameters being optimized, to achieve enhanced model performance. Our comprehensive experiments demonstrate that Chinese CLIP can achieve the state-of-the-art performance on MUGE, Flickr30K-CN, and COCO-CN in the setups of zero-shot learning and finetuning, and it is able to achieve competitive performance in zero-shot image classification based on the evaluation on the ELEVATER benchmark (Li et al., 2022). Our codes, pretrained models, and demos have been released.*

The Chinese-CLIP model was contributed by [OFA-Sys](https://huggingface.co/OFA-Sys).

## Usage example

The code snippet below shows how to compute image & text features and similarities:

```python
>>> from PIL import Image
>>> import requests
>>> from transformers import ChineseCLIPProcessor, ChineseCLIPModel

>>> model = ChineseCLIPModel.from_pretrained("OFA-Sys/chinese-clip-vit-base-patch16")
>>> processor = ChineseCLIPProcessor.from_pretrained("OFA-Sys/chinese-clip-vit-base-patch16")

>>> url = "https://clip-cn-beijing.oss-cn-beijing.aliyuncs.com/pokemon.jpeg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> # Squirtle, Bulbasaur, Charmander, Pikachu in English
>>> texts = ["杰尼龟", "妙蛙种子", "小火龙", "皮卡丘"]

>>> # compute image feature
>>> inputs = processor(images=image, return_tensors="pt")
>>> image_features = model.get_image_features(**inputs)
>>> image_features = image_features / image_features.norm(p=2, dim=-1, keepdim=True)  # normalize

>>> # compute text features
>>> inputs = processor(text=texts, padding=True, return_tensors="pt")
>>> text_features = model.get_text_features(**inputs)
>>> text_features = text_features / text_features.norm(p=2, dim=-1, keepdim=True)  # normalize

>>> # compute image-text similarity scores
>>> inputs = processor(text=texts, images=image, return_tensors="pt", padding=True)
>>> outputs = model(**inputs)
>>> logits_per_image = outputs.logits_per_image  # this is the image-text similarity score
>>> probs = logits_per_image.softmax(dim=1)  # probs: [[1.2686e-03, 5.4499e-02, 6.7968e-04, 9.4355e-01]]
```

Currently, following scales of pretrained Chinese-CLIP models are available on 🤗 Hub:

- [OFA-Sys/chinese-clip-vit-base-patch16](https://huggingface.co/OFA-Sys/chinese-clip-vit-base-patch16)
- [OFA-Sys/chinese-clip-vit-large-patch14](https://huggingface.co/OFA-Sys/chinese-clip-vit-large-patch14)
- [OFA-Sys/chinese-clip-vit-large-patch14-336px](https://huggingface.co/OFA-Sys/chinese-clip-vit-large-patch14-336px)
- [OFA-Sys/chinese-clip-vit-huge-patch14](https://huggingface.co/OFA-Sys/chinese-clip-vit-huge-patch14)

## ChineseCLIPConfig

[[autodoc]] ChineseCLIPConfig
    - from_text_vision_configs

## ChineseCLIPTextConfig

[[autodoc]] ChineseCLIPTextConfig

## ChineseCLIPVisionConfig

[[autodoc]] ChineseCLIPVisionConfig

## ChineseCLIPImageProcessor

[[autodoc]] ChineseCLIPImageProcessor
    - preprocess

## ChineseCLIPFeatureExtractor

[[autodoc]] ChineseCLIPFeatureExtractor

## ChineseCLIPProcessor

[[autodoc]] ChineseCLIPProcessor

## ChineseCLIPModel

[[autodoc]] ChineseCLIPModel
    - forward
    - get_text_features
    - get_image_features

## ChineseCLIPTextModel

[[autodoc]] ChineseCLIPTextModel
    - forward

## ChineseCLIPVisionModel

[[autodoc]] ChineseCLIPVisionModel
    - forward


key: _node_content
string_value:
{"id_": "50215eac-4d5e-4ebe-b4fe-5c9cd7201c47", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/phobert", "title": "PhoBERT", "tokens": 619, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "49b99832-036d-48ce-b095-a41b4e92e377", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/phobert", "title": "PhoBERT", "tokens": 619, "retrieve_doc": true, "source": "transformers"}, "hash": "2ecd4a04b3b65c3cbb24f9a6baf9ba35c728b34ef27c378d76ee1e69a7d6ebf2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1819, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# PhoBERT

## Overview

The PhoBERT model was proposed in [PhoBERT: Pre-trained language models for Vietnamese](https://www.aclweb.org/anthology/2020.findings-emnlp.92.pdf) by Dat Quoc Nguyen, Anh Tuan Nguyen.

The abstract from the paper is the following:

*We present PhoBERT with two versions, PhoBERT-base and PhoBERT-large, the first public large-scale monolingual
language models pre-trained for Vietnamese. Experimental results show that PhoBERT consistently outperforms the recent
best pre-trained multilingual model XLM-R (Conneau et al., 2020) and improves the state-of-the-art in multiple
Vietnamese-specific NLP tasks including Part-of-speech tagging, Dependency parsing, Named-entity recognition and
Natural language inference.*

This model was contributed by [dqnguyen](https://huggingface.co/dqnguyen). The original code can be found [here](https://github.com/VinAIResearch/PhoBERT).

## Usage example

```python
>>> import torch
>>> from transformers import AutoModel, AutoTokenizer

>>> phobert = AutoModel.from_pretrained("vinai/phobert-base")
>>> tokenizer = AutoTokenizer.from_pretrained("vinai/phobert-base")

>>> # INPUT TEXT MUST BE ALREADY WORD-SEGMENTED!
>>> line = "Tôi là sinh_viên trường đại_học Công_nghệ ."

>>> input_ids = torch.tensor([tokenizer.encode(line)])

>>> with torch.no_grad():
...     features = phobert(input_ids)  # Models outputs are now tuples

>>> # With TensorFlow 2.0+:
>>> # from transformers import TFAutoModel
>>> # phobert = TFAutoModel.from_pretrained("vinai/phobert-base")
```

<Tip> 

PhoBERT implementation is the same as BERT, except for tokenization. Refer to [EART documentation](bert) for information on 
configuration classes and their parameters. PhoBERT-specific tokenizer is documented below.  

</Tip>

## PhobertTokenizer

[[autodoc]] PhobertTokenizer


key: _node_content
string_value:
{"id_": "701fc691-e684-477d-9484-d7e70a03c357", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/fuyu", "title": "Fuyu", "tokens": 1342, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a4cd5900-b582-42bb-ab60-af2099564227", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/fuyu", "title": "Fuyu", "tokens": 1342, "retrieve_doc": true, "source": "transformers"}, "hash": "9f69fb8ba7f94fc215cb82cfaf869c3e94d5935701b4afce105e805f35f7140c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c0d2b0dd-fbc2-4abc-b147-225727d934f1", "node_type": "1", "metadata": {}, "hash": "df19df25a4103fd5170846116d2543e7497198f30e0a85086934b9c093a04e62", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1802, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Fuyu

## Overview

The Fuyu model was created by [ADEPT](https://www.adept.ai/blog/fuyu-8b), and authored by Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, Sağnak Taşırlar. 

The authors introduced Fuyu-8B, a decoder-only multimodal model based on the classic transformers architecture, with query and key normalization. A linear encoder is added to create multimodal embeddings from image inputs. 

By treating image tokens like text tokens and using a special image-newline character, the model knows when an image line ends. Image positional embeddings are removed. This avoids the need for different training phases for various image resolutions. With 8 billion parameters and licensed under CC-BY-NC, Fuyu-8B is notable for its ability to handle both text and images, its impressive context size of 16K, and its overall performance.

<Tip warning={true}>

The `Fuyu` models were trained using `bfloat16`, but the original inference uses `float16` The checkpoints uploaded on the hub use `torch_dtype = 'float16'` which will be
used by the `AutoModel` API to cast the checkpoints from `torch.float32` to `torch.float16`. 

The `dtype` of the online weights is mostly irrelevant, unless you are using `torch_dtype="auto"` when initializing a model using `model = AutoModelForCausalLM.from_pretrained("path", torch_dtype = "auto")`. The reason is that the model will first be downloaded ( using the `dtype` of the checkpoints online) then it will be cast to the default `dtype` of `torch` (becomes `torch.float32`). Users should specify the `torch_dtype` they want, and if they don't it will be `torch.float32`.

Finetuning the model in `float16` is not recommended and known to produce `nan`, as such the model should be fine-tuned in `bfloat16`.

</Tip>


key: _node_content
string_value:
{"id_": "c0d2b0dd-fbc2-4abc-b147-225727d934f1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/fuyu", "title": "Fuyu", "tokens": 1342, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a4cd5900-b582-42bb-ab60-af2099564227", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/fuyu", "title": "Fuyu", "tokens": 1342, "retrieve_doc": true, "source": "transformers"}, "hash": "9f69fb8ba7f94fc215cb82cfaf869c3e94d5935701b4afce105e805f35f7140c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "701fc691-e684-477d-9484-d7e70a03c357", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/fuyu", "title": "Fuyu", "tokens": 1342, "retrieve_doc": true, "source": "transformers"}, "hash": "08d439a23bae5a23b4c95547af01066a1222851ac9681ddb64fb35e72215b56c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1805, "end_char_idx": 4448, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Tips:

- To convert the model, you need to clone the original repository using `git clone https://github.com/persimmon-ai-labs/adept-inference`, then get the checkpoints:

```bash
git clone https://github.com/persimmon-ai-labs/adept-inference
wget path/to/fuyu-8b-model-weights.tar
tar -xvf fuyu-8b-model-weights.tar
python src/transformers/models/fuyu/convert_fuyu_weights_to_hf.py  --input_dir /path/to/downloaded/fuyu/weights/ --output_dir /output/path \
    --pt_model_path /path/to/fuyu_8b_release/iter_0001251/mp_rank_00/model_optim_rng.pt
    --ada_lib_path /path/to/adept-inference
```

For the chat model:
```bash
wget https://axtkn4xl5cip.objectstorage.us-phoenix-1.oci.customer-oci.com/n/axtkn4xl5cip/b/adept-public-data/o/8b_chat_model_release.tar
tar -xvf 8b_base_model_release.tar
```
Then, model can be loaded via:

```py 
from transformers import FuyuConfig, FuyuForCausalLM
model_config = FuyuConfig()
model = FuyuForCausalLM(model_config).from_pretrained('/output/path')
```

Inputs need to be passed through a specific Processor to have the correct formats.
A processor requires an image_processor and a tokenizer. Hence, inputs can be loaded via:

```py
from PIL import Image
from transformers import AutoTokenizer
from transformers.models.fuyu.processing_fuyu import FuyuProcessor
from transformers.models.fuyu.image_processing_fuyu import FuyuImageProcessor


tokenizer = AutoTokenizer.from_pretrained('adept-hf-collab/fuyu-8b')
image_processor = FuyuImageProcessor()


processor = FuyuProcessor(image_processor=image_processor, tokenizer=tokenizer)
text_prompt = "Generate a coco-style caption.\\n"

bus_image_url = "https://huggingface.co/datasets/hf-internal-testing/fixtures-captioning/resolve/main/bus.png"
bus_image_pil = Image.open(io.BytesIO(requests.get(bus_image_url).content))
inputs_to_model = processor(text=text_prompt, images=bus_image_pil)


```

This model was contributed by [Molbap](https://huggingface.co/Molbap).
The original code can be found [here](https://github.com/persimmon-ai-labs/adept-inference).

- Fuyu uses a `sentencepiece` based tokenizer, with a `Unigram` model. It supports bytefallback, which is only available in `tokenizers==0.14.0` for the fast tokenizer.
The `LlamaTokenizer` is used as it is a standard wrapper around sentencepiece. 

- The authors suggest to use the following prompt for image captioning: `f"Generate a coco-style caption.\\n"`


## FuyuConfig

[[autodoc]] FuyuConfig

## FuyuForCausalLM

[[autodoc]] FuyuForCausalLM
    - forward

## FuyuImageProcessor

[[autodoc]] FuyuImageProcessor
    - __call__

## FuyuProcessor

[[autodoc]] FuyuProcessor
    - __call__


key: _node_content
string_value:
{"id_": "8df1f6bb-6f05-403c-8ba9-5f42a587bd50", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder", "title": "Vision Encoder Decoder Models", "tokens": 1947, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86adba1a-edb0-487f-8d4e-64df592443ac", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder", "title": "Vision Encoder Decoder Models", "tokens": 1947, "retrieve_doc": true, "source": "transformers"}, "hash": "10a79f49aa80ba46749be3a4aa3a3c92ca07fef4f06196be9609dc10569a1691", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3b08b938-9444-4e5a-924c-b7bdd2e05222", "node_type": "1", "metadata": {}, "hash": "f8b4d53639728d9d0b0b5af771050b8218b32ab0696b56d5d4cfb8a5b0d7f134", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3285, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Vision Encoder Decoder Models

## Overview

The [`VisionEncoderDecoderModel`] can be used to initialize an image-to-text model with any
pretrained Transformer-based vision model as the encoder (*e.g.* [ViT](vit), [BEiT](beit), [DeiT](deit), [Swin](swin))
and any pretrained language model as the decoder (*e.g.* [RoBERTa](roberta), [GPT2](gpt2), [BERT](bert), [DistilBERT](distilbert)).

The effectiveness of initializing image-to-text-sequence models with pretrained checkpoints has been shown in (for
example) [TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models](https://arxiv.org/abs/2109.10282) by Minghao Li, Tengchao Lv, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang,
Zhoujun Li, Furu Wei.

After such a [`VisionEncoderDecoderModel`] has been trained/fine-tuned, it can be saved/loaded just like any other models (see the examples below
for more information).

An example application is image captioning, in which the encoder is used to encode the image, after which an autoregressive language model generates
the caption. Another example is optical character recognition. Refer to [TrOCR](trocr), which is an instance of [`VisionEncoderDecoderModel`].

## Randomly initializing `VisionEncoderDecoderModel` from model configurations.

[`VisionEncoderDecoderModel`] can be randomly initialized from an encoder and a decoder config. In the following example, we show how to do this using the default [`ViTModel`] configuration for the encoder
and the default [`BertForCausalLM`] configuration for the decoder.

```python
>>> from transformers import BertConfig, ViTConfig, VisionEncoderDecoderConfig, VisionEncoderDecoderModel

>>> config_encoder = ViTConfig()
>>> config_decoder = BertConfig()

>>> config = VisionEncoderDecoderConfig.from_encoder_decoder_configs(config_encoder, config_decoder)
>>> model = VisionEncoderDecoderModel(config=config)
```

## Initialising `VisionEncoderDecoderModel` from a pretrained encoder and a pretrained decoder.

[`VisionEncoderDecoderModel`] can be initialized from a pretrained encoder checkpoint and a pretrained decoder checkpoint. Note that any pretrained Transformer-based vision model, *e.g.* [Swin](swin), can serve as the encoder and both pretrained auto-encoding models, *e.g.* BERT, pretrained causal language models, *e.g.* GPT2, as well as the pretrained decoder part of sequence-to-sequence models, *e.g.* decoder of BART, can be used as the decoder.
Depending on which architecture you choose as the decoder, the cross-attention layers might be randomly initialized.
Initializing [`VisionEncoderDecoderModel`] from a pretrained encoder and decoder checkpoint requires the model to be fine-tuned on a downstream task, as has been shown in [the *Warm-starting-encoder-decoder blog post*](https://huggingface.co/blog/warm-starting-encoder-decoder).
To do so, the `VisionEncoderDecoderModel` class provides a [`VisionEncoderDecoderModel.from_encoder_decoder_pretrained`] method.

```python
>>> from transformers import VisionEncoderDecoderModel

>>> model = VisionEncoderDecoderModel.from_encoder_decoder_pretrained(
...     "microsoft/swin-base-patch4-window7-224-in22k", "google-bert/bert-base-uncased"
... )
```

## Loading an existing `VisionEncoderDecoderModel` checkpoint and perform inference.


key: _node_content
string_value:
{"id_": "3b08b938-9444-4e5a-924c-b7bdd2e05222", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder", "title": "Vision Encoder Decoder Models", "tokens": 1947, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86adba1a-edb0-487f-8d4e-64df592443ac", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder", "title": "Vision Encoder Decoder Models", "tokens": 1947, "retrieve_doc": true, "source": "transformers"}, "hash": "10a79f49aa80ba46749be3a4aa3a3c92ca07fef4f06196be9609dc10569a1691", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8df1f6bb-6f05-403c-8ba9-5f42a587bd50", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder", "title": "Vision Encoder Decoder Models", "tokens": 1947, "retrieve_doc": true, "source": "transformers"}, "hash": "5e5c4a2f6789647c09e90e846df3a33a64648af144c2ef8f3c57ff409aa02fd0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a1813132-d2ed-4628-a93c-178519631d23", "node_type": "1", "metadata": {}, "hash": "ae11813f277c9263e608bb4adf35038e84abe053db2eb15028784c20896a79dc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1546, "end_char_idx": 5022, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
>>> from transformers import BertConfig, ViTConfig, VisionEncoderDecoderConfig, VisionEncoderDecoderModel

>>> config_encoder = ViTConfig()
>>> config_decoder = BertConfig()

>>> config = VisionEncoderDecoderConfig.from_encoder_decoder_configs(config_encoder, config_decoder)
>>> model = VisionEncoderDecoderModel(config=config)
```

## Initialising `VisionEncoderDecoderModel` from a pretrained encoder and a pretrained decoder.

[`VisionEncoderDecoderModel`] can be initialized from a pretrained encoder checkpoint and a pretrained decoder checkpoint. Note that any pretrained Transformer-based vision model, *e.g.* [Swin](swin), can serve as the encoder and both pretrained auto-encoding models, *e.g.* BERT, pretrained causal language models, *e.g.* GPT2, as well as the pretrained decoder part of sequence-to-sequence models, *e.g.* decoder of BART, can be used as the decoder.
Depending on which architecture you choose as the decoder, the cross-attention layers might be randomly initialized.
Initializing [`VisionEncoderDecoderModel`] from a pretrained encoder and decoder checkpoint requires the model to be fine-tuned on a downstream task, as has been shown in [the *Warm-starting-encoder-decoder blog post*](https://huggingface.co/blog/warm-starting-encoder-decoder).
To do so, the `VisionEncoderDecoderModel` class provides a [`VisionEncoderDecoderModel.from_encoder_decoder_pretrained`] method.

```python
>>> from transformers import VisionEncoderDecoderModel

>>> model = VisionEncoderDecoderModel.from_encoder_decoder_pretrained(
...     "microsoft/swin-base-patch4-window7-224-in22k", "google-bert/bert-base-uncased"
... )
```

## Loading an existing `VisionEncoderDecoderModel` checkpoint and perform inference.

To load fine-tuned checkpoints of the `VisionEncoderDecoderModel` class, [`VisionEncoderDecoderModel`] provides the `from_pretrained(...)` method just like any other model architecture in Transformers.

To perform inference, one uses the [`generate`] method, which allows to autoregressively generate text. This method supports various forms of decoding, such as greedy, beam search and multinomial sampling.

```python
>>> import requests
>>> from PIL import Image

>>> from transformers import GPT2TokenizerFast, ViTImageProcessor, VisionEncoderDecoderModel

>>> # load a fine-tuned image captioning model and corresponding tokenizer and image processor
>>> model = VisionEncoderDecoderModel.from_pretrained("nlpconnect/vit-gpt2-image-captioning")
>>> tokenizer = GPT2TokenizerFast.from_pretrained("nlpconnect/vit-gpt2-image-captioning")
>>> image_processor = ViTImageProcessor.from_pretrained("nlpconnect/vit-gpt2-image-captioning")

>>> # let's perform inference on an image
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> pixel_values = image_processor(image, return_tensors="pt").pixel_values

>>> # autoregressively generate caption (uses greedy decoding by default)
>>> generated_ids = model.generate(pixel_values)
>>> generated_text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
>>> print(generated_text)
a cat laying on a blanket next to a cat laying on a bed
```

## Loading a PyTorch checkpoint into `TFVisionEncoderDecoderModel`.

[`TFVisionEncoderDecoderModel.from_pretrained`] currently doesn't support initializing the model from a
PyTorch checkpoint. Passing `from_pt=True` to this method will throw an exception.


key: _node_content
string_value:
{"id_": "a1813132-d2ed-4628-a93c-178519631d23", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder", "title": "Vision Encoder Decoder Models", "tokens": 1947, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86adba1a-edb0-487f-8d4e-64df592443ac", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder", "title": "Vision Encoder Decoder Models", "tokens": 1947, "retrieve_doc": true, "source": "transformers"}, "hash": "10a79f49aa80ba46749be3a4aa3a3c92ca07fef4f06196be9609dc10569a1691", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3b08b938-9444-4e5a-924c-b7bdd2e05222", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder", "title": "Vision Encoder Decoder Models", "tokens": 1947, "retrieve_doc": true, "source": "transformers"}, "hash": "f42acd7c56a246d61d9e6cba3b14c8d230269b97cf3f0d42f7eb096816c56e1f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1ec962fb-6f3b-4320-835d-805269e488ba", "node_type": "1", "metadata": {}, "hash": "f9c69b04f585b9f72c6c09ebeeba2531c7e37a4b5ac1766d4cddc3c8de2fd2d2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3287, "end_char_idx": 6038, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
To load fine-tuned checkpoints of the `VisionEncoderDecoderModel` class, [`VisionEncoderDecoderModel`] provides the `from_pretrained(...)` method just like any other model architecture in Transformers.

To perform inference, one uses the [`generate`] method, which allows to autoregressively generate text. This method supports various forms of decoding, such as greedy, beam search and multinomial sampling.

```python
>>> import requests
>>> from PIL import Image

>>> from transformers import GPT2TokenizerFast, ViTImageProcessor, VisionEncoderDecoderModel

>>> # load a fine-tuned image captioning model and corresponding tokenizer and image processor
>>> model = VisionEncoderDecoderModel.from_pretrained("nlpconnect/vit-gpt2-image-captioning")
>>> tokenizer = GPT2TokenizerFast.from_pretrained("nlpconnect/vit-gpt2-image-captioning")
>>> image_processor = ViTImageProcessor.from_pretrained("nlpconnect/vit-gpt2-image-captioning")

>>> # let's perform inference on an image
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> pixel_values = image_processor(image, return_tensors="pt").pixel_values

>>> # autoregressively generate caption (uses greedy decoding by default)
>>> generated_ids = model.generate(pixel_values)
>>> generated_text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
>>> print(generated_text)
a cat laying on a blanket next to a cat laying on a bed
```

## Loading a PyTorch checkpoint into `TFVisionEncoderDecoderModel`.

[`TFVisionEncoderDecoderModel.from_pretrained`] currently doesn't support initializing the model from a
PyTorch checkpoint. Passing `from_pt=True` to this method will throw an exception. If there are only PyTorch
checkpoints for a particular vision encoder-decoder model, a workaround is:

```python
>>> from transformers import VisionEncoderDecoderModel, TFVisionEncoderDecoderModel

>>> _model = VisionEncoderDecoderModel.from_pretrained("nlpconnect/vit-gpt2-image-captioning")

>>> _model.encoder.save_pretrained("./encoder")
>>> _model.decoder.save_pretrained("./decoder")

>>> model = TFVisionEncoderDecoderModel.from_encoder_decoder_pretrained(
...     "./encoder", "./decoder", encoder_from_pt=True, decoder_from_pt=True
... )
>>> # This is only for copying some specific attributes of this particular model.
>>> model.config = _model.config
```

## Training

Once the model is created, it can be fine-tuned similar to BART, T5 or any other encoder-decoder model on a dataset of (image, text) pairs.
As you can see, only 2 inputs are required for the model in order to compute a loss: `pixel_values` (which are the
images) and `labels` (which are the `input_ids` of the encoded target sequence).


key: _node_content
string_value:
{"id_": "1ec962fb-6f3b-4320-835d-805269e488ba", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder", "title": "Vision Encoder Decoder Models", "tokens": 1947, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "86adba1a-edb0-487f-8d4e-64df592443ac", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder", "title": "Vision Encoder Decoder Models", "tokens": 1947, "retrieve_doc": true, "source": "transformers"}, "hash": "10a79f49aa80ba46749be3a4aa3a3c92ca07fef4f06196be9609dc10569a1691", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a1813132-d2ed-4628-a93c-178519631d23", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder", "title": "Vision Encoder Decoder Models", "tokens": 1947, "retrieve_doc": true, "source": "transformers"}, "hash": "b70e2e65288c4fbf1ea87d67d6c6fcb20c6d037a9a9a6dfacc1524756b976c1b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4836, "end_char_idx": 7759, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
[`TFVisionEncoderDecoderModel.from_pretrained`] currently doesn't support initializing the model from a
PyTorch checkpoint. Passing `from_pt=True` to this method will throw an exception. If there are only PyTorch
checkpoints for a particular vision encoder-decoder model, a workaround is:

```python
>>> from transformers import VisionEncoderDecoderModel, TFVisionEncoderDecoderModel

>>> _model = VisionEncoderDecoderModel.from_pretrained("nlpconnect/vit-gpt2-image-captioning")

>>> _model.encoder.save_pretrained("./encoder")
>>> _model.decoder.save_pretrained("./decoder")

>>> model = TFVisionEncoderDecoderModel.from_encoder_decoder_pretrained(
...     "./encoder", "./decoder", encoder_from_pt=True, decoder_from_pt=True
... )
>>> # This is only for copying some specific attributes of this particular model.
>>> model.config = _model.config
```

## Training

Once the model is created, it can be fine-tuned similar to BART, T5 or any other encoder-decoder model on a dataset of (image, text) pairs.
As you can see, only 2 inputs are required for the model in order to compute a loss: `pixel_values` (which are the
images) and `labels` (which are the `input_ids` of the encoded target sequence).

```python
>>> from transformers import ViTImageProcessor, BertTokenizer, VisionEncoderDecoderModel
>>> from datasets import load_dataset

>>> image_processor = ViTImageProcessor.from_pretrained("google/vit-base-patch16-224-in21k")
>>> tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-uncased")
>>> model = VisionEncoderDecoderModel.from_encoder_decoder_pretrained(
...     "google/vit-base-patch16-224-in21k", "google-bert/bert-base-uncased"
... )

>>> model.config.decoder_start_token_id = tokenizer.cls_token_id
>>> model.config.pad_token_id = tokenizer.pad_token_id

>>> dataset = load_dataset("huggingface/cats-image")
>>> image = dataset["test"]["image"][0]
>>> pixel_values = image_processor(image, return_tensors="pt").pixel_values

>>> labels = tokenizer(
...     "an image of two cats chilling on a couch",
...     return_tensors="pt",
... ).input_ids

>>> # the forward function automatically creates the correct decoder_input_ids
>>> loss = model(pixel_values=pixel_values, labels=labels).loss
```

This model was contributed by [nielsr](https://github.com/nielsrogge). This model's TensorFlow and Flax versions
were contributed by [ydshieh](https://github.com/ydshieh).

## VisionEncoderDecoderConfig

[[autodoc]] VisionEncoderDecoderConfig

<frameworkcontent>
<pt>

## VisionEncoderDecoderModel

[[autodoc]] VisionEncoderDecoderModel
    - forward
    - from_encoder_decoder_pretrained

</pt>
<tf>

## TFVisionEncoderDecoderModel

[[autodoc]] TFVisionEncoderDecoderModel
    - call
    - from_encoder_decoder_pretrained

</tf>
<jax>

## FlaxVisionEncoderDecoderModel

[[autodoc]] FlaxVisionEncoderDecoderModel
    - __call__
    - from_encoder_decoder_pretrained

</jax>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "7535e774-a3f3-4b8b-897b-52cd75ca4160", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/matcha", "title": "MatCha", "tokens": 1174, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "28c876dc-4d86-4b38-b9d6-0d71ef9a6300", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/matcha", "title": "MatCha", "tokens": 1174, "retrieve_doc": true, "source": "transformers"}, "hash": "5cd6600845e500561381e36e302e0fefa62bd424dc7fd719db21ee6e1601b1ff", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e218cf9f-13d8-4752-a2f0-14544d92e1d1", "node_type": "1", "metadata": {}, "hash": "7ed83f2b8d269d3d272ee86167eb0f689b68a6c19bd30c9f985f9637e6483d36", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3243, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# MatCha

## Overview

MatCha has been proposed in the paper [MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering](https://arxiv.org/abs/2212.09662), from Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, Julian Martin Eisenschlos.

The abstract of the paper states the following:

*Visual language data such as plots, charts, and infographics are ubiquitous in the human world. However, state-of-the-art vision-language models do not perform well on these data. We propose MatCha (Math reasoning and Chart derendering pretraining) to enhance visual language models' capabilities in jointly modeling charts/plots and language data. Specifically, we propose several pretraining tasks that cover plot deconstruction and numerical reasoning which are the key capabilities in visual language modeling. We perform the MatCha pretraining starting from Pix2Struct, a recently proposed image-to-text visual language model. On standard benchmarks such as PlotQA and ChartQA, the MatCha model outperforms state-of-the-art methods by as much as nearly 20%. We also examine how well MatCha pretraining transfers to domains such as screenshots, textbook diagrams, and document figures and observe overall improvement, verifying the usefulness of MatCha pretraining on broader visual language tasks.*

## Model description

MatCha is a model that is trained using `Pix2Struct` architecture. You can find more information about `Pix2Struct` in the [Pix2Struct documentation](https://huggingface.co/docs/transformers/main/en/model_doc/pix2struct).
MatCha is a Visual Question Answering subset of `Pix2Struct` architecture. It renders the input question on the image and predicts the answer.

## Usage

Currently 6 checkpoints are available for MatCha:

- `google/matcha`: the base MatCha model, used to fine-tune MatCha on downstream tasks
- `google/matcha-chartqa`: MatCha model fine-tuned on ChartQA dataset. It can be used to answer questions about charts.
- `google/matcha-plotqa-v1`: MatCha model fine-tuned on PlotQA dataset. It can be used to answer questions about plots.
- `google/matcha-plotqa-v2`: MatCha model fine-tuned on PlotQA dataset. It can be used to answer questions about plots.
- `google/matcha-chart2text-statista`: MatCha model fine-tuned on Statista dataset. 
- `google/matcha-chart2text-pew`: MatCha model fine-tuned on Pew dataset.

The models finetuned on `chart2text-pew` and `chart2text-statista` are more suited for summarization, whereas the models finetuned on `plotqa` and `chartqa` are more suited for question answering.

You can use these models as follows (example on a ChatQA dataset):

```python
from transformers import AutoProcessor, Pix2StructForConditionalGeneration
import requests
from PIL import Image

model = Pix2StructForConditionalGeneration.from_pretrained("google/matcha-chartqa").to(0)
processor = AutoProcessor.from_pretrained("google/matcha-chartqa")
url = "https://raw.githubusercontent.com/vis-nlp/ChartQA/main/ChartQA%20Dataset/val/png/20294671002019.png"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(images=image, text="Is the sum of all 4 places greater than Laos?


key: _node_content
string_value:
{"id_": "e218cf9f-13d8-4752-a2f0-14544d92e1d1", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/matcha", "title": "MatCha", "tokens": 1174, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "28c876dc-4d86-4b38-b9d6-0d71ef9a6300", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/matcha", "title": "MatCha", "tokens": 1174, "retrieve_doc": true, "source": "transformers"}, "hash": "5cd6600845e500561381e36e302e0fefa62bd424dc7fd719db21ee6e1601b1ff", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7535e774-a3f3-4b8b-897b-52cd75ca4160", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/matcha", "title": "MatCha", "tokens": 1174, "retrieve_doc": true, "source": "transformers"}, "hash": "0397b63917c7aba7dc89b92dbddf37113e28355906cd0925690e57fa6f4929a8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1711, "end_char_idx": 4282, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
It renders the input question on the image and predicts the answer.

## Usage

Currently 6 checkpoints are available for MatCha:

- `google/matcha`: the base MatCha model, used to fine-tune MatCha on downstream tasks
- `google/matcha-chartqa`: MatCha model fine-tuned on ChartQA dataset. It can be used to answer questions about charts.
- `google/matcha-plotqa-v1`: MatCha model fine-tuned on PlotQA dataset. It can be used to answer questions about plots.
- `google/matcha-plotqa-v2`: MatCha model fine-tuned on PlotQA dataset. It can be used to answer questions about plots.
- `google/matcha-chart2text-statista`: MatCha model fine-tuned on Statista dataset. 
- `google/matcha-chart2text-pew`: MatCha model fine-tuned on Pew dataset.

The models finetuned on `chart2text-pew` and `chart2text-statista` are more suited for summarization, whereas the models finetuned on `plotqa` and `chartqa` are more suited for question answering.

You can use these models as follows (example on a ChatQA dataset):

```python
from transformers import AutoProcessor, Pix2StructForConditionalGeneration
import requests
from PIL import Image

model = Pix2StructForConditionalGeneration.from_pretrained("google/matcha-chartqa").to(0)
processor = AutoProcessor.from_pretrained("google/matcha-chartqa")
url = "https://raw.githubusercontent.com/vis-nlp/ChartQA/main/ChartQA%20Dataset/val/png/20294671002019.png"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(images=image, text="Is the sum of all 4 places greater than Laos?", return_tensors="pt").to(0)
predictions = model.generate(**inputs, max_new_tokens=512)
print(processor.decode(predictions[0], skip_special_tokens=True))
```

## Fine-tuning

To fine-tune MatCha, refer to the pix2struct [fine-tuning notebook](https://github.com/huggingface/notebooks/blob/main/examples/image_captioning_pix2struct.ipynb). For `Pix2Struct` models, we have found out that fine-tuning the model with Adafactor and cosine learning rate scheduler leads to faste convergence:
```python
from transformers.optimization import Adafactor, get_cosine_schedule_with_warmup

optimizer = Adafactor(self.parameters(), scale_parameter=False, relative_step=False, lr=0.01, weight_decay=1e-05)
scheduler = get_cosine_schedule_with_warmup(optimizer, num_warmup_steps=1000, num_training_steps=40000)
```

<Tip>

MatCha is a model that is trained using `Pix2Struct` architecture. You can find more information about `Pix2Struct` in the [Pix2Struct documentation](https://huggingface.co/docs/transformers/main/en/model_doc/pix2struct).

</Tip>


key: _node_content
string_value:
{"id_": "275f1d91-e8b1-4565-acbd-756d76343ce9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/switch_transformers", "title": "SwitchTransformers", "tokens": 918, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "37d53b56-2eee-4f39-9a56-f910fc14c4ca", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/switch_transformers", "title": "SwitchTransformers", "tokens": 918, "retrieve_doc": true, "source": "transformers"}, "hash": "42acf944716d237fecd5ac44ed291b6908dd33a616bd78436a9bf3847d13fde7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3455, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# SwitchTransformers

## Overview

The SwitchTransformers model was proposed in [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://arxiv.org/abs/2101.03961) by William Fedus, Barret Zoph, Noam Shazeer.

The Switch Transformer model uses a sparse T5 encoder-decoder architecture, where the MLP are replaced by a Mixture of Experts (MoE). A routing mechanism (top 1 in this case) associates each token to one of the expert, where each expert is a dense MLP. While switch transformers have a lot more weights than their equivalent dense models, the sparsity allows better scaling and better finetuning performance at scale.
During a forward pass, only a fraction of the weights are used. The routing mechanism allows the model to select relevant weights on the fly which increases the model capacity without increasing the number of operations.

The abstract from the paper is the following:

*In deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) defies this and instead selects different parameters for each incoming example. The result is a sparsely-activated model -- with outrageous numbers of parameters -- but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs and training instability -- we address these with the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques help wrangle the instabilities and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the "Colossal Clean Crawled Corpus" and achieve a 4x speedup over the T5-XXL model.*

This model was contributed by [Younes Belkada](https://huggingface.co/ybelkada) and [Arthur Zucker](https://huggingface.co/ArthurZ).
The original code can be found [here](https://github.com/google/flaxformer/tree/main/flaxformer/architectures/moe).

## Usage tips

- SwitchTransformers uses the [`T5Tokenizer`], which can be loaded directly from each model's repository.
- The released weights are pretrained on English [Masked Language Modeling](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19323/en/glossary#general-terms) task, and should be finetuned.

## Resources

- [Translation task guide](../tasks/translation)
- [Summarization task guide](../tasks/summarization)

## SwitchTransformersConfig

[[autodoc]] SwitchTransformersConfig

## SwitchTransformersTop1Router

[[autodoc]] SwitchTransformersTop1Router
    - _compute_router_probabilities
    - forward

## SwitchTransformersSparseMLP

[[autodoc]] SwitchTransformersSparseMLP
    - forward

## SwitchTransformersModel

[[autodoc]] SwitchTransformersModel
    - forward

## SwitchTransformersForConditionalGeneration

[[autodoc]] SwitchTransformersForConditionalGeneration
    - forward

## SwitchTransformersEncoderModel

[[autodoc]] SwitchTransformersEncoderModel
    - forward


key: _node_content
string_value:
{"id_": "40325ce9-c0a2-40bc-b885-3277b2e0f444", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vits", "title": "VITS", "tokens": 1929, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5a66721e-a9fd-4a58-9340-0a951ce67890", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vits", "title": "VITS", "tokens": 1929, "retrieve_doc": true, "source": "transformers"}, "hash": "fb1e1f5c47b860ac03acf8942d6c42928d6862d23da65e755d6c29cceea6d92d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "553e4d45-33df-455b-8184-ac89a6fa486a", "node_type": "1", "metadata": {}, "hash": "bcd05965cef9eeee86eb9d277262d3d4f84e1373e2d404e59e035f89057cbf80", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3450, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# VITS

## Overview

The VITS model was proposed in [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://arxiv.org/abs/2106.06103) by Jaehyeon Kim, Jungil Kong, Juhee Son.

VITS (**V**ariational **I**nference with adversarial learning for end-to-end **T**ext-to-**S**peech) is an end-to-end 
speech synthesis model that predicts a speech waveform conditional on an input text sequence. It is a conditional variational 
autoencoder (VAE) comprised of a posterior encoder, decoder, and conditional prior.

A set of spectrogram-based acoustic features are predicted by the flow-based module, which is formed of a Transformer-based
text encoder and multiple coupling layers. The spectrogram is decoded using a stack of transposed convolutional layers,
much in the same style as the HiFi-GAN vocoder. Motivated by the one-to-many nature of the TTS problem, where the same text 
input can be spoken in multiple ways, the model also includes a stochastic duration predictor, which allows the model to 
synthesise speech with different rhythms from the same input text. 

The model is trained end-to-end with a combination of losses derived from variational lower bound and adversarial training. 
To improve the expressiveness of the model, normalizing flows are applied to the conditional prior distribution. During 
inference, the text encodings are up-sampled based on the duration prediction module, and then mapped into the 
waveform using a cascade of the flow module and HiFi-GAN decoder. Due to the stochastic nature of the duration predictor,
the model is non-deterministic, and thus requires a fixed seed to generate the same speech waveform.

The abstract from the paper is the following:

*Several recent end-to-end text-to-speech (TTS) models enabling single-stage training and parallel sampling have been proposed, but their sample quality does not match that of two-stage TTS systems. In this work, we present a parallel end-to-end TTS method that generates more natural sounding audio than current two-stage models. Our method adopts variational inference augmented with normalizing flows and an adversarial training process, which improves the expressive power of generative modeling. We also propose a stochastic duration predictor to synthesize speech with diverse rhythms from input text. With the uncertainty modeling over latent variables and the stochastic duration predictor, our method expresses the natural one-to-many relationship in which a text input can be spoken in multiple ways with different pitches and rhythms. A subjective human evaluation (mean opinion score, or MOS) on the LJ Speech, a single speaker dataset, shows that our method outperforms the best publicly available TTS systems and achieves a MOS comparable to ground truth.*

This model can also be used with TTS checkpoints from [Massively Multilingual Speech (MMS)](https://arxiv.org/abs/2305.13516) 
as these checkpoints use the same architecture and a slightly modified tokenizer.

This model was contributed by [Matthijs](https://huggingface.co/Matthijs) and [sanchit-gandhi](https://huggingface.co/sanchit-gandhi). The original code can be found [here](https://github.com/jaywalnut310/vits).

## Usage examples

Both the VITS and MMS-TTS checkpoints can be used with the same API. Since the flow-based model is non-deterministic, it 
is good practice to set a seed to ensure reproducibility of the outputs.


key: _node_content
string_value:
{"id_": "553e4d45-33df-455b-8184-ac89a6fa486a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vits", "title": "VITS", "tokens": 1929, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5a66721e-a9fd-4a58-9340-0a951ce67890", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vits", "title": "VITS", "tokens": 1929, "retrieve_doc": true, "source": "transformers"}, "hash": "fb1e1f5c47b860ac03acf8942d6c42928d6862d23da65e755d6c29cceea6d92d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "40325ce9-c0a2-40bc-b885-3277b2e0f444", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vits", "title": "VITS", "tokens": 1929, "retrieve_doc": true, "source": "transformers"}, "hash": "cfe07cf9a305c264cf0edca895f79bff9a07fbc270789a83182a87501aadc99c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0a0a8694-df45-4027-94a8-306191644aa4", "node_type": "1", "metadata": {}, "hash": "b951b1690f54bc9eff057db02ffd2edfff11702be6c65f7ef7e2578a98d35702", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1695, "end_char_idx": 4609, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The abstract from the paper is the following:

*Several recent end-to-end text-to-speech (TTS) models enabling single-stage training and parallel sampling have been proposed, but their sample quality does not match that of two-stage TTS systems. In this work, we present a parallel end-to-end TTS method that generates more natural sounding audio than current two-stage models. Our method adopts variational inference augmented with normalizing flows and an adversarial training process, which improves the expressive power of generative modeling. We also propose a stochastic duration predictor to synthesize speech with diverse rhythms from input text. With the uncertainty modeling over latent variables and the stochastic duration predictor, our method expresses the natural one-to-many relationship in which a text input can be spoken in multiple ways with different pitches and rhythms. A subjective human evaluation (mean opinion score, or MOS) on the LJ Speech, a single speaker dataset, shows that our method outperforms the best publicly available TTS systems and achieves a MOS comparable to ground truth.*

This model can also be used with TTS checkpoints from [Massively Multilingual Speech (MMS)](https://arxiv.org/abs/2305.13516) 
as these checkpoints use the same architecture and a slightly modified tokenizer.

This model was contributed by [Matthijs](https://huggingface.co/Matthijs) and [sanchit-gandhi](https://huggingface.co/sanchit-gandhi). The original code can be found [here](https://github.com/jaywalnut310/vits).

## Usage examples

Both the VITS and MMS-TTS checkpoints can be used with the same API. Since the flow-based model is non-deterministic, it 
is good practice to set a seed to ensure reproducibility of the outputs. For languages with a Roman alphabet, 
such as English or French, the tokenizer can be used directly to pre-process the text inputs. The following code example 
runs a forward pass using the MMS-TTS English checkpoint:

```python
import torch
from transformers import VitsTokenizer, VitsModel, set_seed

tokenizer = VitsTokenizer.from_pretrained("facebook/mms-tts-eng")
model = VitsModel.from_pretrained("facebook/mms-tts-eng")

inputs = tokenizer(text="Hello - my dog is cute", return_tensors="pt")

set_seed(555)  # make deterministic

with torch.no_grad():
   outputs = model(**inputs)

waveform = outputs.waveform[0]
```

The resulting waveform can be saved as a `.wav` file:

```python
import scipy

scipy.io.wavfile.write("techno.wav", rate=model.config.sampling_rate, data=waveform)
```

Or displayed in a Jupyter Notebook / Google Colab:

```python
from IPython.display import Audio

Audio(waveform, rate=model.config.sampling_rate)
```

For certain languages with a non-Roman alphabet, such as Arabic, Mandarin or Hindi, the [`uroman`](https://github.com/isi-nlp/uroman) 
perl package is required to pre-process the text inputs to the Roman alphabet.


key: _node_content
string_value:
{"id_": "0a0a8694-df45-4027-94a8-306191644aa4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vits", "title": "VITS", "tokens": 1929, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5a66721e-a9fd-4a58-9340-0a951ce67890", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vits", "title": "VITS", "tokens": 1929, "retrieve_doc": true, "source": "transformers"}, "hash": "fb1e1f5c47b860ac03acf8942d6c42928d6862d23da65e755d6c29cceea6d92d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "553e4d45-33df-455b-8184-ac89a6fa486a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vits", "title": "VITS", "tokens": 1929, "retrieve_doc": true, "source": "transformers"}, "hash": "fbd3597fdf801e06827197e3bc78941f571fe96f5a295edb8e1b4f47ee660f28", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8a883442-ec7d-4ec6-851b-13976eb2aee0", "node_type": "1", "metadata": {}, "hash": "9de18149105f5d90d9ac4978bbcb04883328a392a0a49efc35ea4ab4a5dc0b46", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3024, "end_char_idx": 6000, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This model was contributed by [Matthijs](https://huggingface.co/Matthijs) and [sanchit-gandhi](https://huggingface.co/sanchit-gandhi). The original code can be found [here](https://github.com/jaywalnut310/vits).

## Usage examples

Both the VITS and MMS-TTS checkpoints can be used with the same API. Since the flow-based model is non-deterministic, it 
is good practice to set a seed to ensure reproducibility of the outputs. For languages with a Roman alphabet, 
such as English or French, the tokenizer can be used directly to pre-process the text inputs. The following code example 
runs a forward pass using the MMS-TTS English checkpoint:

```python
import torch
from transformers import VitsTokenizer, VitsModel, set_seed

tokenizer = VitsTokenizer.from_pretrained("facebook/mms-tts-eng")
model = VitsModel.from_pretrained("facebook/mms-tts-eng")

inputs = tokenizer(text="Hello - my dog is cute", return_tensors="pt")

set_seed(555)  # make deterministic

with torch.no_grad():
   outputs = model(**inputs)

waveform = outputs.waveform[0]
```

The resulting waveform can be saved as a `.wav` file:

```python
import scipy

scipy.io.wavfile.write("techno.wav", rate=model.config.sampling_rate, data=waveform)
```

Or displayed in a Jupyter Notebook / Google Colab:

```python
from IPython.display import Audio

Audio(waveform, rate=model.config.sampling_rate)
```

For certain languages with a non-Roman alphabet, such as Arabic, Mandarin or Hindi, the [`uroman`](https://github.com/isi-nlp/uroman) 
perl package is required to pre-process the text inputs to the Roman alphabet.

You can check whether you require the `uroman` package for your language by inspecting the `is_uroman` attribute of 
the pre-trained `tokenizer`:

```python
from transformers import VitsTokenizer

tokenizer = VitsTokenizer.from_pretrained("facebook/mms-tts-eng")
print(tokenizer.is_uroman)
```
If the is_uroman attribute is `True`, the tokenizer will automatically apply the `uroman` package to your text inputs, but you need to install uroman if not already installed using:  
```
pip install --upgrade uroman
```
Note: Python version required to use `uroman` as python package should be >= `3.10`. 
You can use the tokenizer as usual without any additional preprocessing steps:
```python
import torch
from transformers import VitsTokenizer, VitsModel, set_seed
import os
import subprocess

tokenizer = VitsTokenizer.from_pretrained("facebook/mms-tts-kor")
model = VitsModel.from_pretrained("facebook/mms-tts-kor")
text = "이봐 무슨 일이야"
inputs = tokenizer(text=text, return_tensors="pt")

set_seed(555)  # make deterministic
with torch.no_grad():
   outputs = model(inputs["input_ids"])

waveform = outputs.waveform[0]
```
If you don't want to upgrade to python >= `3.10`, then you can use the `uroman` perl package to pre-process the text inputs to the Roman alphabet.
To do this, first clone the uroman repository to your local machine and set the bash variable `UROMAN` to the local path:


key: _node_content
string_value:
{"id_": "8a883442-ec7d-4ec6-851b-13976eb2aee0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vits", "title": "VITS", "tokens": 1929, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5a66721e-a9fd-4a58-9340-0a951ce67890", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vits", "title": "VITS", "tokens": 1929, "retrieve_doc": true, "source": "transformers"}, "hash": "fb1e1f5c47b860ac03acf8942d6c42928d6862d23da65e755d6c29cceea6d92d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0a0a8694-df45-4027-94a8-306191644aa4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/vits", "title": "VITS", "tokens": 1929, "retrieve_doc": true, "source": "transformers"}, "hash": "a02ead86d1fae59615b1763b1347ef0e976c88bb4865be1c84ebe2a9553f5bb0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4611, "end_char_idx": 7688, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can check whether you require the `uroman` package for your language by inspecting the `is_uroman` attribute of 
the pre-trained `tokenizer`:

```python
from transformers import VitsTokenizer

tokenizer = VitsTokenizer.from_pretrained("facebook/mms-tts-eng")
print(tokenizer.is_uroman)
```
If the is_uroman attribute is `True`, the tokenizer will automatically apply the `uroman` package to your text inputs, but you need to install uroman if not already installed using:  
```
pip install --upgrade uroman
```
Note: Python version required to use `uroman` as python package should be >= `3.10`. 
You can use the tokenizer as usual without any additional preprocessing steps:
```python
import torch
from transformers import VitsTokenizer, VitsModel, set_seed
import os
import subprocess

tokenizer = VitsTokenizer.from_pretrained("facebook/mms-tts-kor")
model = VitsModel.from_pretrained("facebook/mms-tts-kor")
text = "이봐 무슨 일이야"
inputs = tokenizer(text=text, return_tensors="pt")

set_seed(555)  # make deterministic
with torch.no_grad():
   outputs = model(inputs["input_ids"])

waveform = outputs.waveform[0]
```
If you don't want to upgrade to python >= `3.10`, then you can use the `uroman` perl package to pre-process the text inputs to the Roman alphabet.
To do this, first clone the uroman repository to your local machine and set the bash variable `UROMAN` to the local path:


```bash
git clone https://github.com/isi-nlp/uroman.git
cd uroman
export UROMAN=$(pwd)
```

You can then pre-process the text input using the following code snippet. You can either rely on using the bash variable 
`UROMAN` to point to the uroman repository, or you can pass the uroman directory as an argument to the `uromaize` function:

```python
import torch
from transformers import VitsTokenizer, VitsModel, set_seed
import os
import subprocess

tokenizer = VitsTokenizer.from_pretrained("facebook/mms-tts-kor")
model = VitsModel.from_pretrained("facebook/mms-tts-kor")

def uromanize(input_string, uroman_path):
    """Convert non-Roman strings to Roman using the `uroman` perl package."""
    script_path = os.path.join(uroman_path, "bin", "uroman.pl")

    command = ["perl", script_path]

    process = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    # Execute the perl command
    stdout, stderr = process.communicate(input=input_string.encode())

    if process.returncode != 0:
        raise ValueError(f"Error {process.returncode}: {stderr.decode()}")

    # Return the output as a string and skip the new-line character at the end
    return stdout.decode()[:-1]

text = "이봐 무슨 일이야"
uromaized_text = uromanize(text, uroman_path=os.environ["UROMAN"])

inputs = tokenizer(text=uromaized_text, return_tensors="pt")

set_seed(555)  # make deterministic
with torch.no_grad():
   outputs = model(inputs["input_ids"])

waveform = outputs.waveform[0]
```

## VitsConfig

[[autodoc]] VitsConfig

## VitsTokenizer

[[autodoc]] VitsTokenizer
    - __call__
    - save_vocabulary

## VitsModel

[[autodoc]] VitsModel
    - forward


key: _node_content
string_value:
{"id_": "a4ace232-6b2d-4170-b71a-0a73fba78aa0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rag", "title": "RAG", "tokens": 1095, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "de0da16a-d32a-4b15-82d2-c76c347c4c7c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rag", "title": "RAG", "tokens": 1095, "retrieve_doc": true, "source": "transformers"}, "hash": "66f0f489a31b9270f6d92a744e267e2988c3f5e8881ccd9ca441948a02a4ba7b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8fd33163-53ed-4823-a5e7-180f54d12e16", "node_type": "1", "metadata": {}, "hash": "8ee0b0f41e322443a262654be367d4e29f922eeefaefd2a25fa267005dd11929", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3075, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# RAG

<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter=rag">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-rag-blueviolet">
</a>
</div>

## Overview

Retrieval-augmented generation ("RAG") models combine the powers of pretrained dense retrieval (DPR) and
sequence-to-sequence models. RAG models retrieve documents, pass them to a seq2seq model, then marginalize to generate
outputs. The retriever and seq2seq modules are initialized from pretrained models, and fine-tuned jointly, allowing
both retrieval and generation to adapt to downstream tasks.

It is based on the paper [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401) by Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir
Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela.

The abstract from the paper is the following:

*Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve
state-of-the-art results when fine-tuned on downstream NLP tasks. However, their ability to access and precisely
manipulate knowledge is still limited, and hence on knowledge-intensive tasks, their performance lags behind
task-specific architectures. Additionally, providing provenance for their decisions and updating their world knowledge
remain open research problems. Pre-trained models with a differentiable access mechanism to explicit nonparametric
memory can overcome this issue, but have so far been only investigated for extractive downstream tasks. We explore a
general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) — models which combine pre-trained
parametric and non-parametric memory for language generation. We introduce RAG models where the parametric memory is a
pre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a
pre-trained neural retriever. We compare two RAG formulations, one which conditions on the same retrieved passages
across the whole generated sequence, the other can use different passages per token. We fine-tune and evaluate our
models on a wide range of knowledge-intensive NLP tasks and set the state-of-the-art on three open domain QA tasks,
outperforming parametric seq2seq models and task-specific retrieve-and-extract architectures. For language generation
tasks, we find that RAG models generate more specific, diverse and factual language than a state-of-the-art
parametric-only seq2seq baseline.*

This model was contributed by [ola13](https://huggingface.co/ola13).

## Usage tips

Retrieval-augmented generation ("RAG") models combine the powers of pretrained dense retrieval (DPR) and Seq2Seq models. 
RAG models retrieve docs, pass them to a seq2seq model, then marginalize to generate outputs. The retriever and seq2seq 
modules are initialized from pretrained models, and fine-tuned jointly, allowing both retrieval and generation to adapt 
to downstream tasks.


key: _node_content
string_value:
{"id_": "8fd33163-53ed-4823-a5e7-180f54d12e16", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rag", "title": "RAG", "tokens": 1095, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "de0da16a-d32a-4b15-82d2-c76c347c4c7c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rag", "title": "RAG", "tokens": 1095, "retrieve_doc": true, "source": "transformers"}, "hash": "66f0f489a31b9270f6d92a744e267e2988c3f5e8881ccd9ca441948a02a4ba7b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a4ace232-6b2d-4170-b71a-0a73fba78aa0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/rag", "title": "RAG", "tokens": 1095, "retrieve_doc": true, "source": "transformers"}, "hash": "534d5ba6ad357a6229b61dafca3abb32685881a6464d92272e2c887ca3f63e55", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1336, "end_char_idx": 3878, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Additionally, providing provenance for their decisions and updating their world knowledge
remain open research problems. Pre-trained models with a differentiable access mechanism to explicit nonparametric
memory can overcome this issue, but have so far been only investigated for extractive downstream tasks. We explore a
general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) — models which combine pre-trained
parametric and non-parametric memory for language generation. We introduce RAG models where the parametric memory is a
pre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a
pre-trained neural retriever. We compare two RAG formulations, one which conditions on the same retrieved passages
across the whole generated sequence, the other can use different passages per token. We fine-tune and evaluate our
models on a wide range of knowledge-intensive NLP tasks and set the state-of-the-art on three open domain QA tasks,
outperforming parametric seq2seq models and task-specific retrieve-and-extract architectures. For language generation
tasks, we find that RAG models generate more specific, diverse and factual language than a state-of-the-art
parametric-only seq2seq baseline.*

This model was contributed by [ola13](https://huggingface.co/ola13).

## Usage tips

Retrieval-augmented generation ("RAG") models combine the powers of pretrained dense retrieval (DPR) and Seq2Seq models. 
RAG models retrieve docs, pass them to a seq2seq model, then marginalize to generate outputs. The retriever and seq2seq 
modules are initialized from pretrained models, and fine-tuned jointly, allowing both retrieval and generation to adapt 
to downstream tasks.

## RagConfig

[[autodoc]] RagConfig

## RagTokenizer

[[autodoc]] RagTokenizer

## Rag specific outputs

[[autodoc]] models.rag.modeling_rag.RetrievAugLMMarginOutput

[[autodoc]] models.rag.modeling_rag.RetrievAugLMOutput

## RagRetriever

[[autodoc]] RagRetriever

<frameworkcontent>
<pt>

## RagModel

[[autodoc]] RagModel
    - forward

## RagSequenceForGeneration

[[autodoc]] RagSequenceForGeneration
    - forward
    - generate

## RagTokenForGeneration

[[autodoc]] RagTokenForGeneration
    - forward
    - generate

</pt>
<tf>

## TFRagModel

[[autodoc]] TFRagModel
    - call

## TFRagSequenceForGeneration

[[autodoc]] TFRagSequenceForGeneration
    - call
    - generate

## TFRagTokenForGeneration

[[autodoc]] TFRagTokenForGeneration
    - call
    - generate

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "5fcf9e32-a53a-4d40-88d2-67dc4f3d3ce4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mobilebert", "title": "MobileBERT", "tokens": 1329, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "892b8804-de82-4145-9038-00aed791ff74", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mobilebert", "title": "MobileBERT", "tokens": 1329, "retrieve_doc": true, "source": "transformers"}, "hash": "d30515b3b1df27173a8f460e680315b37a9574b5299a9d3662909a89f6dceb62", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "541552d9-b8fb-46a9-9dbd-85f79792c51d", "node_type": "1", "metadata": {}, "hash": "e1fd1110ad3fe4e69c25bebca773101e3267a7868d3f71c12ac046126fb0a88c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2410, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# MobileBERT

## Overview

The MobileBERT model was proposed in [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://arxiv.org/abs/2004.02984) by Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny
Zhou. It's a bidirectional transformer based on the BERT model, which is compressed and accelerated using several
approaches.

The abstract from the paper is the following:

*Natural Language Processing (NLP) has recently achieved great success by using huge pre-trained models with hundreds
of millions of parameters. However, these models suffer from heavy model sizes and high latency such that they cannot
be deployed to resource-limited mobile devices. In this paper, we propose MobileBERT for compressing and accelerating
the popular BERT model. Like the original BERT, MobileBERT is task-agnostic, that is, it can be generically applied to
various downstream NLP tasks via simple fine-tuning. Basically, MobileBERT is a thin version of BERT_LARGE, while
equipped with bottleneck structures and a carefully designed balance between self-attentions and feed-forward networks.
To train MobileBERT, we first train a specially designed teacher model, an inverted-bottleneck incorporated BERT_LARGE
model. Then, we conduct knowledge transfer from this teacher to MobileBERT. Empirical studies show that MobileBERT is
4.3x smaller and 5.5x faster than BERT_BASE while achieving competitive results on well-known benchmarks. On the
natural language inference tasks of GLUE, MobileBERT achieves a GLUEscore o 77.7 (0.6 lower than BERT_BASE), and 62 ms
latency on a Pixel 4 phone. On the SQuAD v1.1/v2.0 question answering task, MobileBERT achieves a dev F1 score of
90.0/79.2 (1.5/2.1 higher than BERT_BASE).*

This model was contributed by [vshampor](https://huggingface.co/vshampor). The original code can be found [here](https://github.com/google-research/google-research/tree/master/mobilebert).

## Usage tips

- MobileBERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather
  than the left.
- MobileBERT is similar to BERT and therefore relies on the masked language modeling (MLM) objective. It is therefore
  efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation. Models trained
  with a causal language modeling (CLM) objective are better in that regard.


key: _node_content
string_value:
{"id_": "541552d9-b8fb-46a9-9dbd-85f79792c51d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mobilebert", "title": "MobileBERT", "tokens": 1329, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "892b8804-de82-4145-9038-00aed791ff74", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mobilebert", "title": "MobileBERT", "tokens": 1329, "retrieve_doc": true, "source": "transformers"}, "hash": "d30515b3b1df27173a8f460e680315b37a9574b5299a9d3662909a89f6dceb62", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5fcf9e32-a53a-4d40-88d2-67dc4f3d3ce4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/mobilebert", "title": "MobileBERT", "tokens": 1329, "retrieve_doc": true, "source": "transformers"}, "hash": "a063e637a21df6ec785bac6c22eac6c70d75cf2235bf6c0a241f7a169adc310a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2413, "end_char_idx": 4607, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Resources

- [Text classification task guide](../tasks/sequence_classification)
- [Token classification task guide](../tasks/token_classification)
- [Question answering task guide](../tasks/question_answering)
- [Masked language modeling task guide](../tasks/masked_language_modeling)
- [Multiple choice task guide](../tasks/multiple_choice)

## MobileBertConfig

[[autodoc]] MobileBertConfig

## MobileBertTokenizer

[[autodoc]] MobileBertTokenizer

## MobileBertTokenizerFast

[[autodoc]] MobileBertTokenizerFast

## MobileBert specific outputs

[[autodoc]] models.mobilebert.modeling_mobilebert.MobileBertForPreTrainingOutput

[[autodoc]] models.mobilebert.modeling_tf_mobilebert.TFMobileBertForPreTrainingOutput

<frameworkcontent>
<pt>

## MobileBertModel

[[autodoc]] MobileBertModel
    - forward

## MobileBertForPreTraining

[[autodoc]] MobileBertForPreTraining
    - forward

## MobileBertForMaskedLM

[[autodoc]] MobileBertForMaskedLM
    - forward

## MobileBertForNextSentencePrediction

[[autodoc]] MobileBertForNextSentencePrediction
    - forward

## MobileBertForSequenceClassification

[[autodoc]] MobileBertForSequenceClassification
    - forward

## MobileBertForMultipleChoice

[[autodoc]] MobileBertForMultipleChoice
    - forward

## MobileBertForTokenClassification

[[autodoc]] MobileBertForTokenClassification
    - forward

## MobileBertForQuestionAnswering

[[autodoc]] MobileBertForQuestionAnswering
    - forward

</pt>
<tf>

## TFMobileBertModel

[[autodoc]] TFMobileBertModel
    - call

## TFMobileBertForPreTraining

[[autodoc]] TFMobileBertForPreTraining
    - call

## TFMobileBertForMaskedLM

[[autodoc]] TFMobileBertForMaskedLM
    - call

## TFMobileBertForNextSentencePrediction

[[autodoc]] TFMobileBertForNextSentencePrediction
    - call

## TFMobileBertForSequenceClassification

[[autodoc]] TFMobileBertForSequenceClassification
    - call

## TFMobileBertForMultipleChoice

[[autodoc]] TFMobileBertForMultipleChoice
    - call

## TFMobileBertForTokenClassification

[[autodoc]] TFMobileBertForTokenClassification
    - call

## TFMobileBertForQuestionAnswering

[[autodoc]] TFMobileBertForQuestionAnswering
    - call

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "6f1c4b8a-77c1-4c72-b15c-176e95e003d6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pvt_v2", "title": "Pyramid Vision Transformer V2 (PVTv2)", "tokens": 2114, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "728fdd47-4068-4135-8b1d-c196ab97e3f1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pvt_v2", "title": "Pyramid Vision Transformer V2 (PVTv2)", "tokens": 2114, "retrieve_doc": true, "source": "transformers"}, "hash": "07cb9bb28b2efb5f14eeda98fb8e5660f2894f9e2bef7ead8f771c91e753ad88", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6bab012c-d81b-46fb-98a8-f57b69730be2", "node_type": "1", "metadata": {}, "hash": "26f5fa39308f72010911f4168a92ec9b3b44fb5a444268cb288b26b62c28445d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3435, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Pyramid Vision Transformer V2 (PVTv2)

## Overview

The PVTv2 model was proposed in
[PVT v2: Improved Baselines with Pyramid Vision Transformer](https://arxiv.org/abs/2106.13797) by Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. As an improved variant of PVT, it eschews position embeddings, relying instead on positional information encoded through zero-padding and overlapping patch embeddings. This lack of reliance on position embeddings simplifies the architecture, and enables running inference at any resolution without needing to interpolate them.

The PVTv2 encoder structure has been successfully deployed to achieve state-of-the-art scores in [Segformer](https://arxiv.org/abs/2105.15203) for semantic segmentation, [GLPN](https://arxiv.org/abs/2201.07436) for monocular depth, and [Panoptic Segformer](https://arxiv.org/abs/2109.03814) for panoptic segmentation.

PVTv2 belongs to a family of models called [hierarchical transformers](https://natecibik.medium.com/the-rise-of-vision-transformers-f623c980419f) , which make adaptations to transformer layers in order to generate multi-scale feature maps. Unlike the columnal structure of Vision Transformer ([ViT](https://arxiv.org/abs/2010.11929)) which loses fine-grained detail, multi-scale feature maps are known preserve this detail and aid performance in dense prediction tasks. In the case of PVTv2, this is achieved by generating image patch tokens using 2D convolution with overlapping kernels in each encoder layer.

The multi-scale features of hierarchical transformers allow them to be easily swapped in for traditional workhorse computer vision backbone models like ResNet in larger architectures. Both Segformer and Panoptic Segformer demonstrated that configurations using PVTv2 for a backbone consistently outperformed those with similarly sized ResNet backbones. 

Another powerful feature of the PVTv2 is the complexity reduction in the self-attention layers called Spatial Reduction Attention (SRA), which uses 2D convolution layers to project hidden states to a smaller resolution before attending to them with the queries, improving the $O(n^2)$ complexity of self-attention to $O(n^2/R)$, with $R$ being the spatial reduction ratio (`sr_ratio`, aka kernel size and stride in the 2D convolution).

SRA was introduced in PVT, and is the default attention complexity reduction method used in PVTv2. However, PVTv2 also introduced the option of using a self-attention mechanism with linear complexity related to image size, which they called "Linear SRA". This method uses average pooling to reduce the hidden states to a fixed size that is invariant to their original resolution (although this is inherently more lossy than regular SRA). This option can be enabled by setting `linear_attention` to `True` in the PVTv2Config.

### Abstract from the paper:

*Transformer recently has presented encouraging progress in computer vision. In this work, we present new baselines by improving the original Pyramid Vision Transformer (PVT v1) by adding three designs, including (1) linear complexity attention layer, (2) overlapping patch embedding, and (3) convolutional feed-forward network. With these modifications, PVT v2 reduces the computational complexity of PVT v1 to linear and achieves significant improvements on fundamental vision tasks such as classification, detection, and segmentation.


key: _node_content
string_value:
{"id_": "6bab012c-d81b-46fb-98a8-f57b69730be2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pvt_v2", "title": "Pyramid Vision Transformer V2 (PVTv2)", "tokens": 2114, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "728fdd47-4068-4135-8b1d-c196ab97e3f1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pvt_v2", "title": "Pyramid Vision Transformer V2 (PVTv2)", "tokens": 2114, "retrieve_doc": true, "source": "transformers"}, "hash": "07cb9bb28b2efb5f14eeda98fb8e5660f2894f9e2bef7ead8f771c91e753ad88", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6f1c4b8a-77c1-4c72-b15c-176e95e003d6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pvt_v2", "title": "Pyramid Vision Transformer V2 (PVTv2)", "tokens": 2114, "retrieve_doc": true, "source": "transformers"}, "hash": "c16956542091149944d0183cc460212f3f9228083f54f59418e79e3f1435dbff", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "37d04b24-1029-4c2a-b557-fd5ac196ae78", "node_type": "1", "metadata": {}, "hash": "2ad32c3bfe2e7b8e141b1e79a96f963d361ad02e739937f2a85577344cd21f04", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1550, "end_char_idx": 5124, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The multi-scale features of hierarchical transformers allow them to be easily swapped in for traditional workhorse computer vision backbone models like ResNet in larger architectures. Both Segformer and Panoptic Segformer demonstrated that configurations using PVTv2 for a backbone consistently outperformed those with similarly sized ResNet backbones. 

Another powerful feature of the PVTv2 is the complexity reduction in the self-attention layers called Spatial Reduction Attention (SRA), which uses 2D convolution layers to project hidden states to a smaller resolution before attending to them with the queries, improving the $O(n^2)$ complexity of self-attention to $O(n^2/R)$, with $R$ being the spatial reduction ratio (`sr_ratio`, aka kernel size and stride in the 2D convolution).

SRA was introduced in PVT, and is the default attention complexity reduction method used in PVTv2. However, PVTv2 also introduced the option of using a self-attention mechanism with linear complexity related to image size, which they called "Linear SRA". This method uses average pooling to reduce the hidden states to a fixed size that is invariant to their original resolution (although this is inherently more lossy than regular SRA). This option can be enabled by setting `linear_attention` to `True` in the PVTv2Config.

### Abstract from the paper:

*Transformer recently has presented encouraging progress in computer vision. In this work, we present new baselines by improving the original Pyramid Vision Transformer (PVT v1) by adding three designs, including (1) linear complexity attention layer, (2) overlapping patch embedding, and (3) convolutional feed-forward network. With these modifications, PVT v2 reduces the computational complexity of PVT v1 to linear and achieves significant improvements on fundamental vision tasks such as classification, detection, and segmentation. Notably, the proposed PVT v2 achieves comparable or better performances than recent works such as Swin Transformer. We hope this work will facilitate state-of-the-art Transformer researches in computer vision. Code is available at https://github.com/whai362/PVT.*

This model was contributed by [FoamoftheSea](https://huggingface.co/FoamoftheSea). The original code can be found [here](https://github.com/whai362/PVT).

## Usage tips

- [PVTv2](https://arxiv.org/abs/2106.13797) is a hierarchical transformer model which has demonstrated powerful performance in image classification and multiple other tasks, used as a backbone for semantic segmentation in [Segformer](https://arxiv.org/abs/2105.15203), monocular depth estimation in [GLPN](https://arxiv.org/abs/2201.07436), and panoptic segmentation in [Panoptic Segformer](https://arxiv.org/abs/2109.03814), consistently showing higher performance than similar ResNet configurations.
- Hierarchical transformers like PVTv2 achieve superior data and parameter efficiency on image data compared with pure transformer architectures by incorporating design elements of convolutional neural networks (CNNs) into their encoders. This creates a best-of-both-worlds architecture that infuses the useful inductive biases of CNNs like translation equivariance and locality into the network while still enjoying the benefits of dynamic data response and global relationship modeling provided by the self-attention mechanism of [transformers](https://arxiv.org/abs/1706.03762).
- PVTv2 uses overlapping patch embeddings to create multi-scale feature maps, which are infused with location information using zero-padding and depth-wise convolutions.


key: _node_content
string_value:
{"id_": "37d04b24-1029-4c2a-b557-fd5ac196ae78", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pvt_v2", "title": "Pyramid Vision Transformer V2 (PVTv2)", "tokens": 2114, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "728fdd47-4068-4135-8b1d-c196ab97e3f1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pvt_v2", "title": "Pyramid Vision Transformer V2 (PVTv2)", "tokens": 2114, "retrieve_doc": true, "source": "transformers"}, "hash": "07cb9bb28b2efb5f14eeda98fb8e5660f2894f9e2bef7ead8f771c91e753ad88", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6bab012c-d81b-46fb-98a8-f57b69730be2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pvt_v2", "title": "Pyramid Vision Transformer V2 (PVTv2)", "tokens": 2114, "retrieve_doc": true, "source": "transformers"}, "hash": "10e45edd72eab713eeeb4cab3a26f1fa1f089c2977e93e934b8811f854bc6e08", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cf133075-f8be-4266-bd73-714fb88870eb", "node_type": "1", "metadata": {}, "hash": "5e500083fc5272d9381c2e888bbee8a0d9d14a1caf55e12e02a27747269df45f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3436, "end_char_idx": 6012, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Notably, the proposed PVT v2 achieves comparable or better performances than recent works such as Swin Transformer. We hope this work will facilitate state-of-the-art Transformer researches in computer vision. Code is available at https://github.com/whai362/PVT.*

This model was contributed by [FoamoftheSea](https://huggingface.co/FoamoftheSea). The original code can be found [here](https://github.com/whai362/PVT).

## Usage tips

- [PVTv2](https://arxiv.org/abs/2106.13797) is a hierarchical transformer model which has demonstrated powerful performance in image classification and multiple other tasks, used as a backbone for semantic segmentation in [Segformer](https://arxiv.org/abs/2105.15203), monocular depth estimation in [GLPN](https://arxiv.org/abs/2201.07436), and panoptic segmentation in [Panoptic Segformer](https://arxiv.org/abs/2109.03814), consistently showing higher performance than similar ResNet configurations.
- Hierarchical transformers like PVTv2 achieve superior data and parameter efficiency on image data compared with pure transformer architectures by incorporating design elements of convolutional neural networks (CNNs) into their encoders. This creates a best-of-both-worlds architecture that infuses the useful inductive biases of CNNs like translation equivariance and locality into the network while still enjoying the benefits of dynamic data response and global relationship modeling provided by the self-attention mechanism of [transformers](https://arxiv.org/abs/1706.03762).
- PVTv2 uses overlapping patch embeddings to create multi-scale feature maps, which are infused with location information using zero-padding and depth-wise convolutions.
- To reduce the complexity in the attention layers, PVTv2 performs a spatial reduction on the hidden states using either strided 2D convolution (SRA) or fixed-size average pooling (Linear SRA). Although inherently more lossy, Linear SRA provides impressive performance with a linear complexity with respect to image size. To use Linear SRA in the self-attention layers, set `linear_attention=True` in the `PvtV2Config`.
- [`PvtV2Model`] is the hierarchical transformer encoder (which is also often referred to as Mix Transformer or MiT in the literature). [`PvtV2ForImageClassification`] adds a simple classifier head on top to perform Image Classification. [`PvtV2Backbone`] can be used with the [`AutoBackbone`] system in larger architectures like Deformable DETR.
- ImageNet pretrained weights for all model sizes can be found on the [hub](https://huggingface.co/models?other=pvt_v2).


key: _node_content
string_value:
{"id_": "cf133075-f8be-4266-bd73-714fb88870eb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pvt_v2", "title": "Pyramid Vision Transformer V2 (PVTv2)", "tokens": 2114, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "728fdd47-4068-4135-8b1d-c196ab97e3f1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pvt_v2", "title": "Pyramid Vision Transformer V2 (PVTv2)", "tokens": 2114, "retrieve_doc": true, "source": "transformers"}, "hash": "07cb9bb28b2efb5f14eeda98fb8e5660f2894f9e2bef7ead8f771c91e753ad88", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "37d04b24-1029-4c2a-b557-fd5ac196ae78", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pvt_v2", "title": "Pyramid Vision Transformer V2 (PVTv2)", "tokens": 2114, "retrieve_doc": true, "source": "transformers"}, "hash": "b99dfc8b3773d4937b2d931a773f61bc0f20075bdb8195151e99634ddb8e6378", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ab79e1c1-914b-4b20-8bac-007f8e76621e", "node_type": "1", "metadata": {}, "hash": "8a43301674d73bce30f4473587d937e21ad82c330b447e4e58f2f62dd70aedef", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4373, "end_char_idx": 8115, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- Hierarchical transformers like PVTv2 achieve superior data and parameter efficiency on image data compared with pure transformer architectures by incorporating design elements of convolutional neural networks (CNNs) into their encoders. This creates a best-of-both-worlds architecture that infuses the useful inductive biases of CNNs like translation equivariance and locality into the network while still enjoying the benefits of dynamic data response and global relationship modeling provided by the self-attention mechanism of [transformers](https://arxiv.org/abs/1706.03762).
- PVTv2 uses overlapping patch embeddings to create multi-scale feature maps, which are infused with location information using zero-padding and depth-wise convolutions.
- To reduce the complexity in the attention layers, PVTv2 performs a spatial reduction on the hidden states using either strided 2D convolution (SRA) or fixed-size average pooling (Linear SRA). Although inherently more lossy, Linear SRA provides impressive performance with a linear complexity with respect to image size. To use Linear SRA in the self-attention layers, set `linear_attention=True` in the `PvtV2Config`.
- [`PvtV2Model`] is the hierarchical transformer encoder (which is also often referred to as Mix Transformer or MiT in the literature). [`PvtV2ForImageClassification`] adds a simple classifier head on top to perform Image Classification. [`PvtV2Backbone`] can be used with the [`AutoBackbone`] system in larger architectures like Deformable DETR.
- ImageNet pretrained weights for all model sizes can be found on the [hub](https://huggingface.co/models?other=pvt_v2).

 The best way to get started with the PVTv2 is to load the pretrained checkpoint with the size of your choosing using `AutoModelForImageClassification`:
```python
import requests
import torch

from transformers import AutoModelForImageClassification, AutoImageProcessor
from PIL import Image

model = AutoModelForImageClassification.from_pretrained("OpenGVLab/pvt_v2_b0")
image_processor = AutoImageProcessor.from_pretrained("OpenGVLab/pvt_v2_b0")
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
processed = image_processor(image)
outputs = model(torch.tensor(processed["pixel_values"]))
```

To use the PVTv2 as a backbone for more complex architectures like DeformableDETR, you can use AutoBackbone (this model would need fine-tuning as you're replacing the backbone in the pretrained model):

```python
import requests
import torch

from transformers import AutoConfig, AutoModelForObjectDetection, AutoImageProcessor
from PIL import Image

model = AutoModelForObjectDetection.from_config(
    config=AutoConfig.from_pretrained(
        "SenseTime/deformable-detr",
        backbone_config=AutoConfig.from_pretrained("OpenGVLab/pvt_v2_b5"),
        use_timm_backbone=False
    ),
)

image_processor = AutoImageProcessor.from_pretrained("SenseTime/deformable-detr")
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
processed = image_processor(image)
outputs = model(torch.tensor(processed["pixel_values"]))
```

[PVTv2](https://github.com/whai362/PVT/tree/v2) performance on ImageNet-1K by model size (B0-B5):

| Method           | Size | Acc@1 | #Params (M) |
|------------------|:----:|:-----:|:-----------:|
| PVT-V2-B0        |  224 |  70.5 |     3.7     |
| PVT-V2-B1        |  224 |  78.7 |     14.0    |
| PVT-V2-B2-Linear |  224 |  82.1 |     22.6    |
| PVT-V2-B2        |  224 |  82.0 |     25.4    |
| PVT-V2-B3        |  224 |  83.1 |     45.2    |
| PVT-V2-B4        |  224 |  83.6 |     62.6    |
| PVT-V2-B5        |  224 |  83.8 |     82.0    |


key: _node_content
string_value:
{"id_": "ab79e1c1-914b-4b20-8bac-007f8e76621e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pvt_v2", "title": "Pyramid Vision Transformer V2 (PVTv2)", "tokens": 2114, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "728fdd47-4068-4135-8b1d-c196ab97e3f1", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pvt_v2", "title": "Pyramid Vision Transformer V2 (PVTv2)", "tokens": 2114, "retrieve_doc": true, "source": "transformers"}, "hash": "07cb9bb28b2efb5f14eeda98fb8e5660f2894f9e2bef7ead8f771c91e753ad88", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cf133075-f8be-4266-bd73-714fb88870eb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/pvt_v2", "title": "Pyramid Vision Transformer V2 (PVTv2)", "tokens": 2114, "retrieve_doc": true, "source": "transformers"}, "hash": "9b99656f03df57fa4e38d125a226e553f3f93c30ffd0464a16bdcea889d7b301", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8118, "end_char_idx": 8293, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## PvtV2Config

[[autodoc]] PvtV2Config

## PvtForImageClassification

[[autodoc]] PvtV2ForImageClassification
    - forward

## PvtModel

[[autodoc]] PvtV2Model
    - forward


key: _node_content
string_value:
{"id_": "f4fe8ca5-5ea6-4a4b-ae22-01fdba996700", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/trocr", "title": "TrOCR", "tokens": 1704, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "427fb293-6de9-476b-87c6-d5f1169e3090", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/trocr", "title": "TrOCR", "tokens": 1704, "retrieve_doc": true, "source": "transformers"}, "hash": "7b32a4b8847a1905f6b356bb3801af7070f92f55e4c9dd38e20fe57baf8cfd09", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ae6551ad-b71f-4f4e-9211-af26186407c5", "node_type": "1", "metadata": {}, "hash": "3083a32de0faa1e89f9510ce3035b357ea3c230a6c4a61f64b3d42ffc880c1c6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3221, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# TrOCR

## Overview

The TrOCR model was proposed in [TrOCR: Transformer-based Optical Character Recognition with Pre-trained
Models](https://arxiv.org/abs/2109.10282) by Minghao Li, Tengchao Lv, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang,
Zhoujun Li, Furu Wei. TrOCR consists of an image Transformer encoder and an autoregressive text Transformer decoder to
perform [optical character recognition (OCR)](https://en.wikipedia.org/wiki/Optical_character_recognition).

The abstract from the paper is the following:

*Text recognition is a long-standing research problem for document digitalization. Existing approaches for text recognition
are usually built based on CNN for image understanding and RNN for char-level text generation. In addition, another language
model is usually needed to improve the overall accuracy as a post-processing step. In this paper, we propose an end-to-end
text recognition approach with pre-trained image Transformer and text Transformer models, namely TrOCR, which leverages the
Transformer architecture for both image understanding and wordpiece-level text generation. The TrOCR model is simple but
effective, and can be pre-trained with large-scale synthetic data and fine-tuned with human-labeled datasets. Experiments
show that the TrOCR model outperforms the current state-of-the-art models on both printed and handwritten text recognition
tasks.*

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/trocr_architecture.jpg"
alt="drawing" width="600"/>

<small> TrOCR architecture. Taken from the <a href="https://arxiv.org/abs/2109.10282">original paper</a>. </small>

Please refer to the [`VisionEncoderDecoder`] class on how to use this model.

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found
[here](https://github.com/microsoft/unilm/tree/6f60612e7cc86a2a1ae85c47231507a587ab4e01/trocr).

## Usage tips

- The quickest way to get started with TrOCR is by checking the [tutorial
  notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/TrOCR), which show how to use the model
  at inference time as well as fine-tuning on custom data.
- TrOCR is pre-trained in 2 stages before being fine-tuned on downstream datasets. It achieves state-of-the-art results
  on both printed (e.g. the [SROIE dataset](https://paperswithcode.com/dataset/sroie) and handwritten (e.g. the [IAM
  Handwriting dataset](https://fki.tic.heia-fr.ch/databases/iam-handwriting-database>) text recognition tasks. For more
  information, see the [official models](https://huggingface.co/models?other=trocr>).
- TrOCR is always used within the [VisionEncoderDecoder](vision-encoder-decoder) framework.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with TrOCR. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

<PipelineTag pipeline="text-classification"/>

- A blog post on [Accelerating Document AI](https://huggingface.co/blog/document-ai) with TrOCR.


key: _node_content
string_value:
{"id_": "ae6551ad-b71f-4f4e-9211-af26186407c5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/trocr", "title": "TrOCR", "tokens": 1704, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "427fb293-6de9-476b-87c6-d5f1169e3090", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/trocr", "title": "TrOCR", "tokens": 1704, "retrieve_doc": true, "source": "transformers"}, "hash": "7b32a4b8847a1905f6b356bb3801af7070f92f55e4c9dd38e20fe57baf8cfd09", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f4fe8ca5-5ea6-4a4b-ae22-01fdba996700", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/trocr", "title": "TrOCR", "tokens": 1704, "retrieve_doc": true, "source": "transformers"}, "hash": "7fdf91138c5ded548f5e9a1247f45872352d7a8dce81823cd31dae612e2e57e7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f0312663-e793-492d-92ed-e73e4947e9df", "node_type": "1", "metadata": {}, "hash": "3aa8757042d4916f50e8470731435e2ba83e3964fec1c8ac69118c98cc40f045", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1726, "end_char_idx": 4527, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found
[here](https://github.com/microsoft/unilm/tree/6f60612e7cc86a2a1ae85c47231507a587ab4e01/trocr).

## Usage tips

- The quickest way to get started with TrOCR is by checking the [tutorial
  notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/TrOCR), which show how to use the model
  at inference time as well as fine-tuning on custom data.
- TrOCR is pre-trained in 2 stages before being fine-tuned on downstream datasets. It achieves state-of-the-art results
  on both printed (e.g. the [SROIE dataset](https://paperswithcode.com/dataset/sroie) and handwritten (e.g. the [IAM
  Handwriting dataset](https://fki.tic.heia-fr.ch/databases/iam-handwriting-database>) text recognition tasks. For more
  information, see the [official models](https://huggingface.co/models?other=trocr>).
- TrOCR is always used within the [VisionEncoderDecoder](vision-encoder-decoder) framework.

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with TrOCR. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

<PipelineTag pipeline="text-classification"/>

- A blog post on [Accelerating Document AI](https://huggingface.co/blog/document-ai) with TrOCR.
- A blog post on how to [Document AI](https://github.com/philschmid/document-ai-transformers) with TrOCR.
- A notebook on how to [finetune TrOCR on IAM Handwriting Database using Seq2SeqTrainer](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Fine_tune_TrOCR_on_IAM_Handwriting_Database_using_Seq2SeqTrainer.ipynb).
- A notebook on [inference with TrOCR](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Inference_with_TrOCR_%2B_Gradio_demo.ipynb) and Gradio demo.
- A notebook on [finetune TrOCR on the IAM Handwriting Database](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Fine_tune_TrOCR_on_IAM_Handwriting_Database_using_native_PyTorch.ipynb) using native PyTorch.
- A notebook on [evaluating TrOCR on the IAM test set](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Evaluating_TrOCR_base_handwritten_on_the_IAM_test_set.ipynb).

<PipelineTag pipeline="text-generation"/>

- [Casual language modeling](https://huggingface.co/docs/transformers/tasks/language_modeling) task guide.

⚡️ Inference

- An interactive-demo on [TrOCR handwritten character recognition](https://huggingface.co/spaces/nielsr/TrOCR-handwritten).


key: _node_content
string_value:
{"id_": "f0312663-e793-492d-92ed-e73e4947e9df", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/trocr", "title": "TrOCR", "tokens": 1704, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "427fb293-6de9-476b-87c6-d5f1169e3090", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/trocr", "title": "TrOCR", "tokens": 1704, "retrieve_doc": true, "source": "transformers"}, "hash": "7b32a4b8847a1905f6b356bb3801af7070f92f55e4c9dd38e20fe57baf8cfd09", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ae6551ad-b71f-4f4e-9211-af26186407c5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/trocr", "title": "TrOCR", "tokens": 1704, "retrieve_doc": true, "source": "transformers"}, "hash": "bfeabe17678bff387bfe52e3d23e80453157c3043c206e45cb46f287b196d183", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "88e54cc4-b21d-4592-9d43-1ec78e571242", "node_type": "1", "metadata": {}, "hash": "56b4c8050daf5f434ed000a8e2dfa18bd1e8a134ad0053ddcad8c5908ecc1e21", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3078, "end_char_idx": 5979, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<PipelineTag pipeline="text-classification"/>

- A blog post on [Accelerating Document AI](https://huggingface.co/blog/document-ai) with TrOCR.
- A blog post on how to [Document AI](https://github.com/philschmid/document-ai-transformers) with TrOCR.
- A notebook on how to [finetune TrOCR on IAM Handwriting Database using Seq2SeqTrainer](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Fine_tune_TrOCR_on_IAM_Handwriting_Database_using_Seq2SeqTrainer.ipynb).
- A notebook on [inference with TrOCR](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Inference_with_TrOCR_%2B_Gradio_demo.ipynb) and Gradio demo.
- A notebook on [finetune TrOCR on the IAM Handwriting Database](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Fine_tune_TrOCR_on_IAM_Handwriting_Database_using_native_PyTorch.ipynb) using native PyTorch.
- A notebook on [evaluating TrOCR on the IAM test set](https://colab.research.google.com/github/NielsRogge/Transformers-Tutorials/blob/master/TrOCR/Evaluating_TrOCR_base_handwritten_on_the_IAM_test_set.ipynb).

<PipelineTag pipeline="text-generation"/>

- [Casual language modeling](https://huggingface.co/docs/transformers/tasks/language_modeling) task guide.

⚡️ Inference

- An interactive-demo on [TrOCR handwritten character recognition](https://huggingface.co/spaces/nielsr/TrOCR-handwritten).

## Inference

TrOCR's [`VisionEncoderDecoder`] model accepts images as input and makes use of
[`~generation.GenerationMixin.generate`] to autoregressively generate text given the input image.

The [`ViTImageProcessor`/`DeiTImageProcessor`] class is responsible for preprocessing the input image and
[`RobertaTokenizer`/`XLMRobertaTokenizer`] decodes the generated target tokens to the target string. The
[`TrOCRProcessor`] wraps [`ViTImageProcessor`/`DeiTImageProcessor`] and [`RobertaTokenizer`/`XLMRobertaTokenizer`]
into a single instance to both extract the input features and decode the predicted token ids.

- Step-by-step Optical Character Recognition (OCR)

``` py
>>> from transformers import TrOCRProcessor, VisionEncoderDecoderModel
>>> import requests
>>> from PIL import Image

>>> processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-handwritten")
>>> model = VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-base-handwritten")

>>> # load image from the IAM dataset
>>> url = "https://fki.tic.heia-fr.ch/static/img/a01-122-02.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw).convert("RGB")

>>> pixel_values = processor(image, return_tensors="pt").pixel_values
>>> generated_ids = model.generate(pixel_values)

>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
```

See the [model hub](https://huggingface.co/models?filter=trocr) to look for TrOCR checkpoints.


key: _node_content
string_value:
{"id_": "88e54cc4-b21d-4592-9d43-1ec78e571242", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/trocr", "title": "TrOCR", "tokens": 1704, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "427fb293-6de9-476b-87c6-d5f1169e3090", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/trocr", "title": "TrOCR", "tokens": 1704, "retrieve_doc": true, "source": "transformers"}, "hash": "7b32a4b8847a1905f6b356bb3801af7070f92f55e4c9dd38e20fe57baf8cfd09", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f0312663-e793-492d-92ed-e73e4947e9df", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/trocr", "title": "TrOCR", "tokens": 1704, "retrieve_doc": true, "source": "transformers"}, "hash": "020469f650433c83c362977e0d992af0b84457414f1593189c6f3c6d9416fd2d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4390, "end_char_idx": 6224, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
⚡️ Inference

- An interactive-demo on [TrOCR handwritten character recognition](https://huggingface.co/spaces/nielsr/TrOCR-handwritten).

## Inference

TrOCR's [`VisionEncoderDecoder`] model accepts images as input and makes use of
[`~generation.GenerationMixin.generate`] to autoregressively generate text given the input image.

The [`ViTImageProcessor`/`DeiTImageProcessor`] class is responsible for preprocessing the input image and
[`RobertaTokenizer`/`XLMRobertaTokenizer`] decodes the generated target tokens to the target string. The
[`TrOCRProcessor`] wraps [`ViTImageProcessor`/`DeiTImageProcessor`] and [`RobertaTokenizer`/`XLMRobertaTokenizer`]
into a single instance to both extract the input features and decode the predicted token ids.

- Step-by-step Optical Character Recognition (OCR)

``` py
>>> from transformers import TrOCRProcessor, VisionEncoderDecoderModel
>>> import requests
>>> from PIL import Image

>>> processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-handwritten")
>>> model = VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-base-handwritten")

>>> # load image from the IAM dataset
>>> url = "https://fki.tic.heia-fr.ch/static/img/a01-122-02.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw).convert("RGB")

>>> pixel_values = processor(image, return_tensors="pt").pixel_values
>>> generated_ids = model.generate(pixel_values)

>>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
```

See the [model hub](https://huggingface.co/models?filter=trocr) to look for TrOCR checkpoints.

## TrOCRConfig

[[autodoc]] TrOCRConfig

## TrOCRProcessor

[[autodoc]] TrOCRProcessor
    - __call__
    - from_pretrained
    - save_pretrained
    - batch_decode
    - decode

## TrOCRForCausalLM

[[autodoc]] TrOCRForCausalLM
     - forward


key: _node_content
string_value:
{"id_": "c3070402-29a8-46f4-bfff-ac14b2ecd6b5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bartpho", "title": "BARTpho", "tokens": 959, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b3978b25-b1ec-4d9d-aab2-23195938b8e3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bartpho", "title": "BARTpho", "tokens": 959, "retrieve_doc": true, "source": "transformers"}, "hash": "8c44a25c732b5d6ef1588fb60f7c6dbadf4be3d9a29198625ce69077015d75ff", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "12062b97-3455-4c2b-a7ba-00f9856945bb", "node_type": "1", "metadata": {}, "hash": "659ef2213e8b23c6080b70df8143e0c7bbcd6c6fc793770b0453b383b7db4997", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2891, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# BARTpho

## Overview

The BARTpho model was proposed in [BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese](https://arxiv.org/abs/2109.09701) by Nguyen Luong Tran, Duong Minh Le and Dat Quoc Nguyen.

The abstract from the paper is the following:

*We present BARTpho with two versions -- BARTpho_word and BARTpho_syllable -- the first public large-scale monolingual
sequence-to-sequence models pre-trained for Vietnamese. Our BARTpho uses the "large" architecture and pre-training
scheme of the sequence-to-sequence denoising model BART, thus especially suitable for generative NLP tasks. Experiments
on a downstream task of Vietnamese text summarization show that in both automatic and human evaluations, our BARTpho
outperforms the strong baseline mBART and improves the state-of-the-art. We release BARTpho to facilitate future
research and applications of generative Vietnamese NLP tasks.*

This model was contributed by [dqnguyen](https://huggingface.co/dqnguyen). The original code can be found [here](https://github.com/VinAIResearch/BARTpho).

## Usage example

```python
>>> import torch
>>> from transformers import AutoModel, AutoTokenizer

>>> bartpho = AutoModel.from_pretrained("vinai/bartpho-syllable")

>>> tokenizer = AutoTokenizer.from_pretrained("vinai/bartpho-syllable")

>>> line = "Chúng tôi là những nghiên cứu viên."

>>> input_ids = tokenizer(line, return_tensors="pt")

>>> with torch.no_grad():
...     features = bartpho(**input_ids)  # Models outputs are now tuples

>>> # With TensorFlow 2.0+:
>>> from transformers import TFAutoModel

>>> bartpho = TFAutoModel.from_pretrained("vinai/bartpho-syllable")
>>> input_ids = tokenizer(line, return_tensors="tf")
>>> features = bartpho(**input_ids)
```

## Usage tips

- Following mBART, BARTpho uses the "large" architecture of BART with an additional layer-normalization layer on top of
  both the encoder and decoder. Thus, usage examples in the [documentation of BART](bart), when adapting to use
  with BARTpho, should be adjusted by replacing the BART-specialized classes with the mBART-specialized counterparts.
  For example:

```python
>>> from transformers import MBartForConditionalGeneration

>>> bartpho = MBartForConditionalGeneration.from_pretrained("vinai/bartpho-syllable")
>>> TXT = "Chúng tôi là <mask> nghiên cứu viên."
>>> input_ids = tokenizer([TXT], return_tensors="pt")["input_ids"]
>>> logits = bartpho(input_ids).logits
>>> masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item()
>>> probs = logits[0, masked_index].softmax(dim=0)
>>> values, predictions = probs.topk(5)
>>> print(tokenizer.decode(predictions).split())
```

- This implementation is only for tokenization: "monolingual_vocab_file" consists of Vietnamese-specialized types
  extracted from the pre-trained SentencePiece model "vocab_file" that is available from the multilingual XLM-RoBERTa.


key: _node_content
string_value:
{"id_": "12062b97-3455-4c2b-a7ba-00f9856945bb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bartpho", "title": "BARTpho", "tokens": 959, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b3978b25-b1ec-4d9d-aab2-23195938b8e3", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bartpho", "title": "BARTpho", "tokens": 959, "retrieve_doc": true, "source": "transformers"}, "hash": "8c44a25c732b5d6ef1588fb60f7c6dbadf4be3d9a29198625ce69077015d75ff", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c3070402-29a8-46f4-bfff-ac14b2ecd6b5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/bartpho", "title": "BARTpho", "tokens": 959, "retrieve_doc": true, "source": "transformers"}, "hash": "1373c9e34caffc244c05b548585112bd12ce3e81c8f70cbc2849ecd3aa1bc36f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1358, "end_char_idx": 3154, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> input_ids = tokenizer(line, return_tensors="pt")

>>> with torch.no_grad():
...     features = bartpho(**input_ids)  # Models outputs are now tuples

>>> # With TensorFlow 2.0+:
>>> from transformers import TFAutoModel

>>> bartpho = TFAutoModel.from_pretrained("vinai/bartpho-syllable")
>>> input_ids = tokenizer(line, return_tensors="tf")
>>> features = bartpho(**input_ids)
```

## Usage tips

- Following mBART, BARTpho uses the "large" architecture of BART with an additional layer-normalization layer on top of
  both the encoder and decoder. Thus, usage examples in the [documentation of BART](bart), when adapting to use
  with BARTpho, should be adjusted by replacing the BART-specialized classes with the mBART-specialized counterparts.
  For example:

```python
>>> from transformers import MBartForConditionalGeneration

>>> bartpho = MBartForConditionalGeneration.from_pretrained("vinai/bartpho-syllable")
>>> TXT = "Chúng tôi là <mask> nghiên cứu viên."
>>> input_ids = tokenizer([TXT], return_tensors="pt")["input_ids"]
>>> logits = bartpho(input_ids).logits
>>> masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item()
>>> probs = logits[0, masked_index].softmax(dim=0)
>>> values, predictions = probs.topk(5)
>>> print(tokenizer.decode(predictions).split())
```

- This implementation is only for tokenization: "monolingual_vocab_file" consists of Vietnamese-specialized types
  extracted from the pre-trained SentencePiece model "vocab_file" that is available from the multilingual XLM-RoBERTa.
  Other languages, if employing this pre-trained multilingual SentencePiece model "vocab_file" for subword
  segmentation, can reuse BartphoTokenizer with their own language-specialized "monolingual_vocab_file".

## BartphoTokenizer

[[autodoc]] BartphoTokenizer


key: _node_content
string_value:
{"id_": "e82f5414-3d5e-4e1f-b683-160d44af8842", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/biogpt", "title": "BioGPT", "tokens": 949, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "55a266c2-ced3-4c87-97a2-941a1a2582f8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/biogpt", "title": "BioGPT", "tokens": 949, "retrieve_doc": true, "source": "transformers"}, "hash": "1f0e9175b8bf15f63a0372f9f8c8b5083f311f6022d15799137b2d8d03dbad48", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "302801b5-1d2b-4b0c-b7d2-de1fa9b086cd", "node_type": "1", "metadata": {}, "hash": "222cf8c3c392d7965a3fc46ada93147f1de257bfc3f673339839c8b4d0b998a0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3231, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# BioGPT

## Overview

The BioGPT model was proposed in [BioGPT: generative pre-trained transformer for biomedical text generation and mining](https://academic.oup.com/bib/advance-article/doi/10.1093/bib/bbac409/6713511?guestAccessKey=a66d9b5d-4f83-4017-bb52-405815c907b9) by Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon and Tie-Yan Liu. BioGPT is a domain-specific generative pre-trained Transformer language model for biomedical text generation and mining. BioGPT follows the Transformer language model backbone, and is pre-trained on 15M PubMed abstracts from scratch.

The abstract from the paper is the following:

*Pre-trained language models have attracted increasing attention in the biomedical domain, inspired by their great success in the general natural language domain. Among the two main branches of pre-trained language models in the general language domain, i.e. BERT (and its variants) and GPT (and its variants), the first one has been extensively studied in the biomedical domain, such as BioBERT and PubMedBERT. While they have achieved great success on a variety of discriminative downstream biomedical tasks, the lack of generation ability constrains their application scope. In this paper, we propose BioGPT, a domain-specific generative Transformer language model pre-trained on large-scale biomedical literature. We evaluate BioGPT on six biomedical natural language processing tasks and demonstrate that our model outperforms previous models on most tasks. Especially, we get 44.98%, 38.42% and 40.76% F1 score on BC5CDR, KD-DTI and DDI end-to-end relation extraction tasks, respectively, and 78.2% accuracy on PubMedQA, creating a new record. Our case study on text generation further demonstrates the advantage of BioGPT on biomedical literature to generate fluent descriptions for biomedical terms.*

This model was contributed by [kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/microsoft/BioGPT).

## Usage tips

- BioGPT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than the left.
- BioGPT was trained with a causal language modeling (CLM) objective and is therefore powerful at predicting the next token in a sequence. Leveraging this feature allows BioGPT to generate syntactically coherent text as it can be observed in the run_generation.py example script.
- The model can take the `past_key_values` (for PyTorch) as input, which is the previously computed key/value attention pairs. Using this (past_key_values or past) value prevents the model from re-computing pre-computed values in the context of text generation. For PyTorch, see past_key_values argument of the BioGptForCausalLM.forward() method for more information on its usage.

## Resources

- [Causal language modeling task guide](../tasks/language_modeling)

## BioGptConfig

[[autodoc]] BioGptConfig


## BioGptTokenizer

[[autodoc]] BioGptTokenizer
    - save_vocabulary


## BioGptModel

[[autodoc]] BioGptModel
    - forward


## BioGptForCausalLM

[[autodoc]] BioGptForCausalLM
    - forward

    
## BioGptForTokenClassification

[[autodoc]] BioGptForTokenClassification
    - forward


key: _node_content
string_value:
{"id_": "302801b5-1d2b-4b0c-b7d2-de1fa9b086cd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/biogpt", "title": "BioGPT", "tokens": 949, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "55a266c2-ced3-4c87-97a2-941a1a2582f8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/biogpt", "title": "BioGPT", "tokens": 949, "retrieve_doc": true, "source": "transformers"}, "hash": "1f0e9175b8bf15f63a0372f9f8c8b5083f311f6022d15799137b2d8d03dbad48", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e82f5414-3d5e-4e1f-b683-160d44af8842", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/biogpt", "title": "BioGPT", "tokens": 949, "retrieve_doc": true, "source": "transformers"}, "hash": "5310debde05321e72c201fe5a229804509bd9e5be3e9c3053951708c553b0b9f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2944, "end_char_idx": 3327, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## BioGptTokenizer

[[autodoc]] BioGptTokenizer
    - save_vocabulary


## BioGptModel

[[autodoc]] BioGptModel
    - forward


## BioGptForCausalLM

[[autodoc]] BioGptForCausalLM
    - forward

    
## BioGptForTokenClassification

[[autodoc]] BioGptForTokenClassification
    - forward


## BioGptForSequenceClassification

[[autodoc]] BioGptForSequenceClassification
    - forward


key: _node_content
string_value:
{"id_": "1c04c8d1-21cb-4604-877c-7b81c1e644c2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/speecht5", "title": "SpeechT5", "tokens": 845, "retrieve_doc": true, "source": "transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d873360f-586a-4f20-8f68-2d238dfc9dbd", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/model_doc/speecht5", "title": "SpeechT5", "tokens": 845, "retrieve_doc": true, "source": "transformers"}, "hash": "5ab9b289ae152568516619e94dd97d860fa184aed6ffcf730e5f1035a65dffce", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2738, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


