# embedding_metadata

key: chroma:document
string_value:
```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_food_model",
...     remove_unused_columns=False,
...     eval_strategy="epoch",
...     save_strategy="epoch",
...     learning_rate=5e-5,
...     per_device_train_batch_size=16,
...     gradient_accumulation_steps=4,
...     per_device_eval_batch_size=16,
...     num_train_epochs=3,
...     warmup_ratio=0.1,
...     logging_steps=10,
...     load_best_model_at_end=True,
...     metric_for_best_model="accuracy",
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     data_collator=data_collator,
...     train_dataset=food["train"],
...     eval_dataset=food["test"],
...     tokenizer=image_processor,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
</frameworkcontent>

<frameworkcontent>
<tf>

<Tip>

If you are unfamiliar with fine-tuning a model with Keras, check out the [basic tutorial](./training#train-a-tensorflow-model-with-keras) first!

</Tip>

To fine-tune a model in TensorFlow, follow these steps:
1. Define the training hyperparameters, and set up an optimizer and a learning rate schedule.
2. Instantiate a pre-trained model.
3. Convert a 🤗 Dataset to a `tf.data.Dataset`.
4. Compile your model.
5. Add callbacks and use the `fit()` method to run the training.
6. Upload your model to 🤗 Hub to share with the community.

Start by defining the hyperparameters, optimizer and learning rate schedule:

```py
>>> from transformers import create_optimizer

>>> batch_size = 16
>>> num_epochs = 5
>>> num_train_steps = len(food["train"]) * num_epochs
>>> learning_rate = 3e-5
>>> weight_decay_rate = 0.01

>>> optimizer, lr_schedule = create_optimizer(
...     init_lr=learning_rate,
...     num_train_steps=num_train_steps,
...     weight_decay_rate=weight_decay_rate,
...     num_warmup_steps=0,
... )
```

Then, load ViT with [`TFAutoModelForImageClassification`] along with the label mappings:

```py
>>> from transformers import TFAutoModelForImageClassification

>>> model = TFAutoModelForImageClassification.from_pretrained(
...     checkpoint,
...     id2label=id2label,
...     label2id=label2id,
... )
```

Convert your datasets to the `tf.data.Dataset` format using the [`~datasets.Dataset.to_tf_dataset`] and your `data_collator`:

```py
>>> # converting our train dataset to tf.data.Dataset
>>> tf_train_dataset = food["train"].to_tf_dataset(
...     columns="pixel_values", label_cols="label", shuffle=True, batch_size=batch_size, collate_fn=data_collator
... )

>>> # converting our test dataset to tf.data.Dataset
>>> tf_eval_dataset = food["test"].to_tf_dataset(
...     columns="pixel_values", label_cols="label", shuffle=True, batch_size=batch_size, collate_fn=data_collator
... )
```

Configure the model for training with `compile()`:

```py
>>> from tensorflow.keras.losses import SparseCategoricalCrossentropy

>>> loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
>>> model.compile(optimizer=optimizer, loss=loss)
```

To compute the accuracy from the predictions and push your model to the 🤗 Hub, use [Keras callbacks](../main_classes/keras_callbacks).


key: _node_content
string_value:
{"id_": "1e4d49ef-b9e8-492c-8d6f-54e2b93d1f12", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification/", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fed03d05-36e5-474c-8f31-0f2312703df2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification/", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a547781279590d9269f442b1369dabcd2d365305a8c52accb25f3fdff75b8970", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "96d4f8df-d57c-410c-b023-cc8a120a1d01", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification/", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "782a9115cf363442bb737c48d1c22f58641c2db4f07757f54d69046dc6f5738b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3cee1911-842f-40b8-9ef4-2a8cba0b725e", "node_type": "1", "metadata": {}, "hash": "79a30dbd43256465653a5ee56e274196f9a0b4a59e3c0a6aacc274ab0e8c9ae5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13355, "end_char_idx": 15022, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Pass your `compute_metrics` function to [KerasMetricCallback](../main_classes/keras_callbacks#transformers.KerasMetricCallback),
and use the [PushToHubCallback](../main_classes/keras_callbacks#transformers.PushToHubCallback) to upload the model:

```py
>>> from transformers.keras_callbacks import KerasMetricCallback, PushToHubCallback

>>> metric_callback = KerasMetricCallback(metric_fn=compute_metrics, eval_dataset=tf_eval_dataset)
>>> push_to_hub_callback = PushToHubCallback(
...     output_dir="food_classifier",
...     tokenizer=image_processor,
...     save_strategy="no",
... )
>>> callbacks = [metric_callback, push_to_hub_callback]
```

Finally, you are ready to train your model! Call `fit()` with your training and validation datasets, the number of epochs,
and your callbacks to fine-tune the model:

```py
>>> model.fit(tf_train_dataset, validation_data=tf_eval_dataset, epochs=num_epochs, callbacks=callbacks)
Epoch 1/5
250/250 [==============================] - 313s 1s/step - loss: 2.5623 - val_loss: 1.4161 - accuracy: 0.9290
Epoch 2/5
250/250 [==============================] - 265s 1s/step - loss: 0.9181 - val_loss: 0.6808 - accuracy: 0.9690
Epoch 3/5
250/250 [==============================] - 252s 1s/step - loss: 0.3910 - val_loss: 0.4303 - accuracy: 0.9820
Epoch 4/5
250/250 [==============================] - 251s 1s/step - loss: 0.2028 - val_loss: 0.3191 - accuracy: 0.9900
Epoch 5/5
250/250 [==============================] - 238s 949ms/step - loss: 0.1232 - val_loss: 0.3259 - accuracy: 0.9890
```

Congratulations! You have fine-tuned your model and shared it on the 🤗 Hub. You can now use it for inference!
</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "3cee1911-842f-40b8-9ef4-2a8cba0b725e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification/", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fed03d05-36e5-474c-8f31-0f2312703df2", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification/", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a547781279590d9269f442b1369dabcd2d365305a8c52accb25f3fdff75b8970", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1e4d49ef-b9e8-492c-8d6f-54e2b93d1f12", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_classification/", "title": "Image classification", "tokens": 4527, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b62b66a7afa99ff608e6c43e243f9b3a110f21f5504b6634947b60c474a4fd04", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14050, "end_char_idx": 18122, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Call `fit()` with your training and validation datasets, the number of epochs,
and your callbacks to fine-tune the model:

```py
>>> model.fit(tf_train_dataset, validation_data=tf_eval_dataset, epochs=num_epochs, callbacks=callbacks)
Epoch 1/5
250/250 [==============================] - 313s 1s/step - loss: 2.5623 - val_loss: 1.4161 - accuracy: 0.9290
Epoch 2/5
250/250 [==============================] - 265s 1s/step - loss: 0.9181 - val_loss: 0.6808 - accuracy: 0.9690
Epoch 3/5
250/250 [==============================] - 252s 1s/step - loss: 0.3910 - val_loss: 0.4303 - accuracy: 0.9820
Epoch 4/5
250/250 [==============================] - 251s 1s/step - loss: 0.2028 - val_loss: 0.3191 - accuracy: 0.9900
Epoch 5/5
250/250 [==============================] - 238s 949ms/step - loss: 0.1232 - val_loss: 0.3259 - accuracy: 0.9890
```

Congratulations! You have fine-tuned your model and shared it on the 🤗 Hub. You can now use it for inference!
</tf>
</frameworkcontent>


<Tip>

For a more in-depth example of how to finetune a model for image classification, take a look at the corresponding [PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).

</Tip>

## Inference

Great, now that you've fine-tuned a model, you can use it for inference!

Load an image you'd like to run inference on:

```py
>>> ds = load_dataset("food101", split="validation[:10]")
>>> image = ds["image"][0]
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png" alt="image of beignets"/>
</div>

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for image classification with your model, and pass your image to it:

```py
>>> from transformers import pipeline

>>> classifier = pipeline("image-classification", model="my_awesome_food_model")
>>> classifier(image)
[{'score': 0.31856709718704224, 'label': 'beignets'},
 {'score': 0.015232225880026817, 'label': 'bruschetta'},
 {'score': 0.01519392803311348, 'label': 'chicken_wings'},
 {'score': 0.013022331520915031, 'label': 'pork_chop'},
 {'score': 0.012728818692266941, 'label': 'prime_rib'}]
```

You can also manually replicate the results of the `pipeline` if you'd like:

<frameworkcontent>
<pt>
Load an image processor to preprocess the image and return the `input` as PyTorch tensors:

```py
>>> from transformers import AutoImageProcessor
>>> import torch

>>> image_processor = AutoImageProcessor.from_pretrained("my_awesome_food_model")
>>> inputs = image_processor(image, return_tensors="pt")
```

Pass your inputs to the model and return the logits:

```py
>>> from transformers import AutoModelForImageClassification

>>> model = AutoModelForImageClassification.from_pretrained("my_awesome_food_model")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
```

Get the predicted label with the highest probability, and use the model's `id2label` mapping to convert it to a label:

```py
>>> predicted_label = logits.argmax(-1).item()
>>> model.config.id2label[predicted_label]
'beignets'
```
</pt>
</frameworkcontent>

<frameworkcontent>
<tf>
Load an image processor to preprocess the image and return the `input` as TensorFlow tensors:

```py
>>> from transformers import AutoImageProcessor

>>> image_processor = AutoImageProcessor.from_pretrained("MariaK/food_classifier")
>>> inputs = image_processor(image, return_tensors="tf")
```

Pass your inputs to the model and return the logits:

```py
>>> from transformers import TFAutoModelForImageClassification

>>> model = TFAutoModelForImageClassification.from_pretrained("MariaK/food_classifier")
>>> logits = model(**inputs).logits
```

Get the predicted label with the highest probability, and use the model's `id2label` mapping to convert it to a label:

```py
>>> predicted_class_id = int(tf.math.argmax(logits, axis=-1)[0])
>>> model.config.id2label[predicted_class_id]
'beignets'
```

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "889b2d8a-02c0-49e9-afa8-3158f72a7b4a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/monocular_depth_estimation/", "title": "Monocular depth estimation", "tokens": 2170, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e64ecfae-46f3-4464-aa1c-109f8dadf6b7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/monocular_depth_estimation/", "title": "Monocular depth estimation", "tokens": 2170, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3899928f408e594c3803b6742880304b912ad4b49cd2b7d4fe3d5825add77d96", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ccd2ed01-8d81-4e42-8945-311ebaba186a", "node_type": "1", "metadata": {}, "hash": "f43c9d37b11f915c9ea0a8ff53076de7693fb192863f789e66f4c4b65ed225e8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3286, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Monocular depth estimation

Monocular depth estimation is a computer vision task that involves predicting the depth information of a scene from a
single image. In other words, it is the process of estimating the distance of objects in a scene from
a single camera viewpoint.

Monocular depth estimation has various applications, including 3D reconstruction, augmented reality, autonomous driving,
and robotics. It is a challenging task as it requires the model to understand the complex relationships between objects
in the scene and the corresponding depth information, which can be affected by factors such as lighting conditions,
occlusion, and texture. 

There are two main depth estimation categories:

- **Absolute depth estimation**: This task variant aims to provide exact depth measurements from the camera. The term is used interchangeably with metric depth estimation, where depth is provided in precise measurements in meters or feet. Absolute depth estimation models output depth maps with numerical values that represent real-world distances.

- **Relative depth estimation**: Relative depth estimation aims to predict the depth order of objects or points in a scene without providing the precise measurements. These models output a depth map that indicates which parts of the scene are closer or farther relative to each other without the actual distances to A and B.

In this guide, we will see how to infer with [Depth Anything V2](https://huggingface.co/depth-anything/Depth-Anything-V2-Large), a state-of-the-art zero-shot relative depth estimation model, and [ZoeDepth](https://huggingface.co/docs/transformers/main/en/model_doc/zoedepth), an absolute depth estimation model.

<Tip>

Check the [Depth Estimation](https://huggingface.co/tasks/depth-estimation) task page to view all compatible architectures and checkpoints.

</Tip>

Before we begin, we need to install the latest version of Transformers:

```bash
pip install -q -U transformers
```

## Depth estimation pipeline

The simplest way to try out inference with a model supporting depth estimation is to use the corresponding [`pipeline`].
Instantiate a pipeline from a [checkpoint on the Hugging Face Hub](https://huggingface.co/models?pipeline_tag=depth-estimation&sort=downloads):

```py
>>> from transformers import pipeline
>>> import torch

>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> checkpoint = "depth-anything/Depth-Anything-V2-base-hf"
>>> pipe = pipeline("depth-estimation", model=checkpoint, device=device)
```

Next, choose an image to analyze:

```py
>>> from PIL import Image
>>> import requests

>>> url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> image
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg" alt="Photo of a bee"/>
</div>

Pass the image to the pipeline.

```py
>>> predictions = pipe(image)
```

The pipeline returns a dictionary with two entries. The first one, called `predicted_depth`, is a tensor with the values
being the depth expressed in meters for each pixel.
The second one, `depth`, is a PIL image that visualizes the depth estimation result.


key: _node_content
string_value:
{"id_": "ccd2ed01-8d81-4e42-8945-311ebaba186a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/monocular_depth_estimation/", "title": "Monocular depth estimation", "tokens": 2170, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e64ecfae-46f3-4464-aa1c-109f8dadf6b7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/monocular_depth_estimation/", "title": "Monocular depth estimation", "tokens": 2170, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3899928f408e594c3803b6742880304b912ad4b49cd2b7d4fe3d5825add77d96", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "889b2d8a-02c0-49e9-afa8-3158f72a7b4a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/monocular_depth_estimation/", "title": "Monocular depth estimation", "tokens": 2170, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c739dca9dee5723a18239390a855569ab0f42363ce1be191414a72dba38c72f4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fa28b761-66aa-4bf4-b422-5bde7c04ddbe", "node_type": "1", "metadata": {}, "hash": "60b0ddf4ee66370f637617ab88d293ceaf59c5681e880f8810b1cf490e13453b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1699, "end_char_idx": 4875, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<Tip>

Check the [Depth Estimation](https://huggingface.co/tasks/depth-estimation) task page to view all compatible architectures and checkpoints.

</Tip>

Before we begin, we need to install the latest version of Transformers:

```bash
pip install -q -U transformers
```

## Depth estimation pipeline

The simplest way to try out inference with a model supporting depth estimation is to use the corresponding [`pipeline`].
Instantiate a pipeline from a [checkpoint on the Hugging Face Hub](https://huggingface.co/models?pipeline_tag=depth-estimation&sort=downloads):

```py
>>> from transformers import pipeline
>>> import torch

>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> checkpoint = "depth-anything/Depth-Anything-V2-base-hf"
>>> pipe = pipeline("depth-estimation", model=checkpoint, device=device)
```

Next, choose an image to analyze:

```py
>>> from PIL import Image
>>> import requests

>>> url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> image
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg" alt="Photo of a bee"/>
</div>

Pass the image to the pipeline.

```py
>>> predictions = pipe(image)
```

The pipeline returns a dictionary with two entries. The first one, called `predicted_depth`, is a tensor with the values
being the depth expressed in meters for each pixel.
The second one, `depth`, is a PIL image that visualizes the depth estimation result.

Let's take a look at the visualized result:

```py
>>> predictions["depth"]
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/depth-visualization.png" alt="Depth estimation visualization"/>
</div>

## Depth estimation inference by hand

Now that you've seen how to use the depth estimation pipeline, let's see how we can replicate the same result by hand.

Start by loading the model and associated processor from a [checkpoint on the Hugging Face Hub](https://huggingface.co/models?pipeline_tag=depth-estimation&sort=downloads).
Here we'll use the same checkpoint as before:

```py
>>> from transformers import AutoImageProcessor, AutoModelForDepthEstimation

>>> checkpoint = "Intel/zoedepth-nyu-kitti"

>>> image_processor = AutoImageProcessor.from_pretrained(checkpoint)
>>> model = AutoModelForDepthEstimation.from_pretrained(checkpoint).to(device)
```

Prepare the image input for the model using the `image_processor` that will take care of the necessary image transformations
such as resizing and normalization:

```py
>>> pixel_values = image_processor(image, return_tensors="pt").pixel_values.to(device)
```

Pass the prepared inputs through the model:

```py
>>> import torch

>>> with torch.no_grad():
...     outputs = model(pixel_values)
```

Let's post-process and visualize the results. 

We need to pad and then resize the outputs so that predicted depth map has the same dimension as the original image. After resizing we will remove the padded regions from the depth.


key: _node_content
string_value:
{"id_": "fa28b761-66aa-4bf4-b422-5bde7c04ddbe", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/monocular_depth_estimation/", "title": "Monocular depth estimation", "tokens": 2170, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e64ecfae-46f3-4464-aa1c-109f8dadf6b7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/monocular_depth_estimation/", "title": "Monocular depth estimation", "tokens": 2170, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3899928f408e594c3803b6742880304b912ad4b49cd2b7d4fe3d5825add77d96", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ccd2ed01-8d81-4e42-8945-311ebaba186a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/monocular_depth_estimation/", "title": "Monocular depth estimation", "tokens": 2170, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "ece6705f615e6ec3ae8729258b86a73eff8f22898e9ac50dbb5b9a1cc90e9126", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1a6e950e-30f2-4550-a321-667f7417ef06", "node_type": "1", "metadata": {}, "hash": "6611cb9536c1c338f1a29f73e6cb075f2cb8c09d8101cad937a6385494e23de0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3081, "end_char_idx": 6301, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The first one, called `predicted_depth`, is a tensor with the values
being the depth expressed in meters for each pixel.
The second one, `depth`, is a PIL image that visualizes the depth estimation result.

Let's take a look at the visualized result:

```py
>>> predictions["depth"]
```

<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/depth-visualization.png" alt="Depth estimation visualization"/>
</div>

## Depth estimation inference by hand

Now that you've seen how to use the depth estimation pipeline, let's see how we can replicate the same result by hand.

Start by loading the model and associated processor from a [checkpoint on the Hugging Face Hub](https://huggingface.co/models?pipeline_tag=depth-estimation&sort=downloads).
Here we'll use the same checkpoint as before:

```py
>>> from transformers import AutoImageProcessor, AutoModelForDepthEstimation

>>> checkpoint = "Intel/zoedepth-nyu-kitti"

>>> image_processor = AutoImageProcessor.from_pretrained(checkpoint)
>>> model = AutoModelForDepthEstimation.from_pretrained(checkpoint).to(device)
```

Prepare the image input for the model using the `image_processor` that will take care of the necessary image transformations
such as resizing and normalization:

```py
>>> pixel_values = image_processor(image, return_tensors="pt").pixel_values.to(device)
```

Pass the prepared inputs through the model:

```py
>>> import torch

>>> with torch.no_grad():
...     outputs = model(pixel_values)
```

Let's post-process and visualize the results. 

We need to pad and then resize the outputs so that predicted depth map has the same dimension as the original image. After resizing we will remove the padded regions from the depth. 

```py
>>> import numpy as np
>>> import torch.nn.functional as F

>>> predicted_depth = outputs.predicted_depth.unsqueeze(dim=1)
>>> height, width = pixel_values.shape[2:]

>>> height_padding_factor = width_padding_factor = 3
>>> pad_h = int(np.sqrt(height/2) * height_padding_factor)
>>> pad_w = int(np.sqrt(width/2) * width_padding_factor)

>>> if predicted_depth.shape[-2:] != pixel_values.shape[-2:]:
>>>    predicted_depth = F.interpolate(predicted_depth, size= (height, width), mode='bicubic', align_corners=False)

>>> if pad_h > 0:
     predicted_depth = predicted_depth[:, :, pad_h:-pad_h,:]
>>> if pad_w > 0:
     predicted_depth = predicted_depth[:, :, :, pad_w:-pad_w]
```

We can now visualize the results (the function below is taken from the [GaussianObject](https://github.com/GaussianObject/GaussianObject/blob/ad6629efadb57902d5f8bc0fa562258029a4bdf1/pred_monodepth.py#L11) framework).

```py
import matplotlib

def colorize(value, vmin=None, vmax=None, cmap='gray_r', invalid_val=-99, invalid_mask=None, background_color=(128, 128, 128, 255), gamma_corrected=False, value_transform=None):
    """Converts a depth map to a color image.

    Args:
        value (torch.Tensor, numpy.ndarry): Input depth map. Shape: (H, W) or (1, H, W) or (1, 1, H, W). All singular dimensions are squeezed
        vmin (float, optional): vmin-valued entries are mapped to start color of cmap. If None, value.min() is used.


key: _node_content
string_value:
{"id_": "1a6e950e-30f2-4550-a321-667f7417ef06", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/monocular_depth_estimation/", "title": "Monocular depth estimation", "tokens": 2170, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e64ecfae-46f3-4464-aa1c-109f8dadf6b7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/monocular_depth_estimation/", "title": "Monocular depth estimation", "tokens": 2170, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3899928f408e594c3803b6742880304b912ad4b49cd2b7d4fe3d5825add77d96", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fa28b761-66aa-4bf4-b422-5bde7c04ddbe", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/monocular_depth_estimation/", "title": "Monocular depth estimation", "tokens": 2170, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "ab9505128d0c850d7b4514002234009d0fa81a182eee022eadf43a793c903f5d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a35cb620-bfc6-432d-aaca-691a63745dfd", "node_type": "1", "metadata": {}, "hash": "20999b3db30623ebf30659893e85941c9a515b278ada537140a5baf9b5d985e4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4811, "end_char_idx": 7739, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
After resizing we will remove the padded regions from the depth. 

```py
>>> import numpy as np
>>> import torch.nn.functional as F

>>> predicted_depth = outputs.predicted_depth.unsqueeze(dim=1)
>>> height, width = pixel_values.shape[2:]

>>> height_padding_factor = width_padding_factor = 3
>>> pad_h = int(np.sqrt(height/2) * height_padding_factor)
>>> pad_w = int(np.sqrt(width/2) * width_padding_factor)

>>> if predicted_depth.shape[-2:] != pixel_values.shape[-2:]:
>>>    predicted_depth = F.interpolate(predicted_depth, size= (height, width), mode='bicubic', align_corners=False)

>>> if pad_h > 0:
     predicted_depth = predicted_depth[:, :, pad_h:-pad_h,:]
>>> if pad_w > 0:
     predicted_depth = predicted_depth[:, :, :, pad_w:-pad_w]
```

We can now visualize the results (the function below is taken from the [GaussianObject](https://github.com/GaussianObject/GaussianObject/blob/ad6629efadb57902d5f8bc0fa562258029a4bdf1/pred_monodepth.py#L11) framework).

```py
import matplotlib

def colorize(value, vmin=None, vmax=None, cmap='gray_r', invalid_val=-99, invalid_mask=None, background_color=(128, 128, 128, 255), gamma_corrected=False, value_transform=None):
    """Converts a depth map to a color image.

    Args:
        value (torch.Tensor, numpy.ndarry): Input depth map. Shape: (H, W) or (1, H, W) or (1, 1, H, W). All singular dimensions are squeezed
        vmin (float, optional): vmin-valued entries are mapped to start color of cmap. If None, value.min() is used. Defaults to None.
        vmax (float, optional):  vmax-valued entries are mapped to end color of cmap. If None, value.max() is used. Defaults to None.
        cmap (str, optional): matplotlib colormap to use. Defaults to 'magma_r'.
        invalid_val (int, optional): Specifies value of invalid pixels that should be colored as 'background_color'. Defaults to -99.
        invalid_mask (numpy.ndarray, optional): Boolean mask for invalid regions. Defaults to None.
        background_color (tuple[int], optional): 4-tuple RGB color to give to invalid pixels. Defaults to (128, 128, 128, 255).
        gamma_corrected (bool, optional): Apply gamma correction to colored image. Defaults to False.
        value_transform (Callable, optional): Apply transform function to valid pixels before coloring. Defaults to None.

    Returns:
        numpy.ndarray, dtype - uint8: Colored depth map. Shape: (H, W, 4)
    """
    if isinstance(value, torch.Tensor):
        value = value.detach().cpu().numpy()

    value = value.squeeze()
    if invalid_mask is None:
        invalid_mask = value == invalid_val
    mask = np.logical_not(invalid_mask)

    # normalize
    vmin = np.percentile(value[mask],2) if vmin is None else vmin
    vmax = np.percentile(value[mask],85) if vmax is None else vmax
    if vmin != vmax:
        value = (value - vmin) / (vmax - vmin)  # vmin..vmax
    else:
        # Avoid 0-division
        value = value * 0.


key: _node_content
string_value:
{"id_": "a35cb620-bfc6-432d-aaca-691a63745dfd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/monocular_depth_estimation/", "title": "Monocular depth estimation", "tokens": 2170, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e64ecfae-46f3-4464-aa1c-109f8dadf6b7", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/monocular_depth_estimation/", "title": "Monocular depth estimation", "tokens": 2170, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3899928f408e594c3803b6742880304b912ad4b49cd2b7d4fe3d5825add77d96", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1a6e950e-30f2-4550-a321-667f7417ef06", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/monocular_depth_estimation/", "title": "Monocular depth estimation", "tokens": 2170, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3aa18d581e69fee970c5df2694b2a80eb31b264ca830bd8c1b46424e9b2bc417", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6148, "end_char_idx": 8680, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
All singular dimensions are squeezed
        vmin (float, optional): vmin-valued entries are mapped to start color of cmap. If None, value.min() is used. Defaults to None.
        vmax (float, optional):  vmax-valued entries are mapped to end color of cmap. If None, value.max() is used. Defaults to None.
        cmap (str, optional): matplotlib colormap to use. Defaults to 'magma_r'.
        invalid_val (int, optional): Specifies value of invalid pixels that should be colored as 'background_color'. Defaults to -99.
        invalid_mask (numpy.ndarray, optional): Boolean mask for invalid regions. Defaults to None.
        background_color (tuple[int], optional): 4-tuple RGB color to give to invalid pixels. Defaults to (128, 128, 128, 255).
        gamma_corrected (bool, optional): Apply gamma correction to colored image. Defaults to False.
        value_transform (Callable, optional): Apply transform function to valid pixels before coloring. Defaults to None.

    Returns:
        numpy.ndarray, dtype - uint8: Colored depth map. Shape: (H, W, 4)
    """
    if isinstance(value, torch.Tensor):
        value = value.detach().cpu().numpy()

    value = value.squeeze()
    if invalid_mask is None:
        invalid_mask = value == invalid_val
    mask = np.logical_not(invalid_mask)

    # normalize
    vmin = np.percentile(value[mask],2) if vmin is None else vmin
    vmax = np.percentile(value[mask],85) if vmax is None else vmax
    if vmin != vmax:
        value = (value - vmin) / (vmax - vmin)  # vmin..vmax
    else:
        # Avoid 0-division
        value = value * 0.

    # squeeze last dim if it exists
    # grey out the invalid values

    value[invalid_mask] = np.nan
    cmapper = matplotlib.colormaps.get_cmap(cmap)
    if value_transform:
        value = value_transform(value)
        # value = value / value.max()
    value = cmapper(value, bytes=True)  # (nxmx4)

    # img = value[:, :, :]
    img = value[...]
    img[invalid_mask] = background_color

    #     return img.transpose((2, 0, 1))
    if gamma_corrected:
        # gamma correction
        img = img / 255
        img = np.power(img, 2.2)
        img = img * 255
        img = img.astype(np.uint8)
    return img

>>> result = colorize(predicted_depth.cpu().squeeze().numpy())
>>> Image.fromarray(result)
```



<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/depth-visualization-zoe.png" alt="Depth estimation visualization"/>
</div>


key: _node_content
string_value:
{"id_": "2e817e50-1ad2-421f-9441-9c13753c8931", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice/", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7e8925bb-c911-4b48-b274-0207791e500e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice/", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8826953801c051c045aa6f1510cd5f98b9b125f896066d12f0fedb4bef3a1900", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2a3236c1-161b-41d7-a1bd-8cf18fefde58", "node_type": "1", "metadata": {}, "hash": "ec76c30556c9af596d7e0c3e842f8117db2d30443eac67efe33c949ddbab7bcd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2858, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Multiple choice

[[open-in-colab]]

A multiple choice task is similar to question answering, except several candidate answers are provided along with a context and the model is trained to select the correct answer.

This guide will show you how to:

1. Finetune [BERT](https://huggingface.co/google-bert/bert-base-uncased) on the `regular` configuration of the [SWAG](https://huggingface.co/datasets/swag) dataset to select the best answer given multiple options and some context.
2. Use your finetuned model for inference.

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install transformers datasets evaluate
```

We encourage you to login to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to login:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load SWAG dataset

Start by loading the `regular` configuration of the SWAG dataset from the 🤗 Datasets library:

```py
>>> from datasets import load_dataset

>>> swag = load_dataset("swag", "regular")
```

Then take a look at an example:

```py
>>> swag["train"][0]
{'ending0': 'passes by walking down the street playing their instruments.',
 'ending1': 'has heard approaching them.',
 'ending2': "arrives and they're outside dancing and asleep.",
 'ending3': 'turns the lead singer watches the performance.',
 'fold-ind': '3416',
 'gold-source': 'gold',
 'label': 0,
 'sent1': 'Members of the procession walk down the street holding small horn brass instruments.',
 'sent2': 'A drum line',
 'startphrase': 'Members of the procession walk down the street holding small horn brass instruments. A drum line',
 'video-id': 'anetv_jkn6uvmqwh4'}
```

While it looks like there are a lot of fields here, it is actually pretty straightforward:

- `sent1` and `sent2`: these fields show how a sentence starts, and if you put the two together, you get the `startphrase` field.
- `ending`: suggests a possible ending for how a sentence can end, but only one of them is correct.
- `label`: identifies the correct sentence ending.

## Preprocess

The next step is to load a BERT tokenizer to process the sentence starts and the four possible endings:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")
```

The preprocessing function you want to create needs to:

1. Make four copies of the `sent1` field and combine each of them with `sent2` to recreate how a sentence starts.
2. Combine `sent2` with each of the four possible sentence endings.
3. Flatten these two lists so you can tokenize them, and then unflatten them afterward so each example has a corresponding `input_ids`, `attention_mask`, and `labels` field.

```py
>>> ending_names = ["ending0", "ending1", "ending2", "ending3"]


key: _node_content
string_value:
{"id_": "2a3236c1-161b-41d7-a1bd-8cf18fefde58", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice/", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7e8925bb-c911-4b48-b274-0207791e500e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice/", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8826953801c051c045aa6f1510cd5f98b9b125f896066d12f0fedb4bef3a1900", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2e817e50-1ad2-421f-9441-9c13753c8931", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice/", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3ff8b7bdc5516c66376fca29136a4eb18600b8e786147e6370d948e21a72093f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d4cf148c-40ba-4501-8474-e8f83bee24d4", "node_type": "1", "metadata": {}, "hash": "a11acb0ccf274d2237be12b7aa9bf4519a33b6b4e53e4ee8993507253230d449", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2861, "end_char_idx": 5965, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> def preprocess_function(examples):
...     first_sentences = [[context] * 4 for context in examples["sent1"]]
...     question_headers = examples["sent2"]
...     second_sentences = [
...         [f"{header} {examples[end][i]}" for end in ending_names] for i, header in enumerate(question_headers)
...     ]

...     first_sentences = sum(first_sentences, [])
...     second_sentences = sum(second_sentences, [])

...     tokenized_examples = tokenizer(first_sentences, second_sentences, truncation=True)
...     return {k: [v[i : i + 4] for i in range(0, len(v), 4)] for k, v in tokenized_examples.items()}
```

To apply the preprocessing function over the entire dataset, use 🤗 Datasets [`~datasets.Dataset.map`] method. You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once:

```py
tokenized_swag = swag.map(preprocess_function, batched=True)
```

🤗 Transformers doesn't have a data collator for multiple choice, so you'll need to adapt the [`DataCollatorWithPadding`] to create a batch of examples. It's more efficient to *dynamically pad* the sentences to the longest length in a batch during collation, instead of padding the whole dataset to the maximum length.

`DataCollatorForMultipleChoice` flattens all the model inputs, applies padding, and then unflattens the results:

<frameworkcontent>
<pt>
```py
>>> from dataclasses import dataclass
>>> from transformers.tokenization_utils_base import PreTrainedTokenizerBase, PaddingStrategy
>>> from typing import Optional, Union
>>> import torch


>>> @dataclass
... class DataCollatorForMultipleChoice:
...     """
...     Data collator that will dynamically pad the inputs for multiple choice received.
...     """

...     tokenizer: PreTrainedTokenizerBase
...     padding: Union[bool, str, PaddingStrategy] = True
...     max_length: Optional[int] = None
...     pad_to_multiple_of: Optional[int] = None

...     def __call__(self, features):
...         label_name = "label" if "label" in features[0].keys() else "labels"
...         labels = [feature.pop(label_name) for feature in features]
...         batch_size = len(features)
...         num_choices = len(features[0]["input_ids"])
...         flattened_features = [
...             [{k: v[i] for k, v in feature.items()} for i in range(num_choices)] for feature in features
...         ]
...         flattened_features = sum(flattened_features, [])

...         batch = self.tokenizer.pad(
...             flattened_features,
...             padding=self.padding,
...             max_length=self.max_length,
...             pad_to_multiple_of=self.pad_to_multiple_of,
...             return_tensors="pt",
...         )

...         batch = {k: v.view(batch_size, num_choices, -1) for k, v in batch.items()}
...         batch["labels"] = torch.tensor(labels, dtype=torch.int64)
...         return batch
```
</pt>
<tf>
```py
>>> from dataclasses import dataclass
>>> from transformers.tokenization_utils_base import PreTrainedTokenizerBase, PaddingStrategy
>>> from typing import Optional, Union
>>> import tensorflow as tf


key: _node_content
string_value:
{"id_": "d4cf148c-40ba-4501-8474-e8f83bee24d4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice/", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7e8925bb-c911-4b48-b274-0207791e500e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice/", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8826953801c051c045aa6f1510cd5f98b9b125f896066d12f0fedb4bef3a1900", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2a3236c1-161b-41d7-a1bd-8cf18fefde58", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice/", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a879b2b5054284099eb11ffc24431eb569ebfb3029c793d1d0294e31674d1b5e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3cfeb1c7-9771-4fa3-935e-80d089f13ad4", "node_type": "1", "metadata": {}, "hash": "e9d978765075db1b22f1d74c74b718ef61d94a6a8bc634d06b440c420a259fdf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4429, "end_char_idx": 8024, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> @dataclass
... class DataCollatorForMultipleChoice:
...     """
...     Data collator that will dynamically pad the inputs for multiple choice received.
...     """

...     tokenizer: PreTrainedTokenizerBase
...     padding: Union[bool, str, PaddingStrategy] = True
...     max_length: Optional[int] = None
...     pad_to_multiple_of: Optional[int] = None

...     def __call__(self, features):
...         label_name = "label" if "label" in features[0].keys() else "labels"
...         labels = [feature.pop(label_name) for feature in features]
...         batch_size = len(features)
...         num_choices = len(features[0]["input_ids"])
...         flattened_features = [
...             [{k: v[i] for k, v in feature.items()} for i in range(num_choices)] for feature in features
...         ]
...         flattened_features = sum(flattened_features, [])

...         batch = self.tokenizer.pad(
...             flattened_features,
...             padding=self.padding,
...             max_length=self.max_length,
...             pad_to_multiple_of=self.pad_to_multiple_of,
...             return_tensors="pt",
...         )

...         batch = {k: v.view(batch_size, num_choices, -1) for k, v in batch.items()}
...         batch["labels"] = torch.tensor(labels, dtype=torch.int64)
...         return batch
```
</pt>
<tf>
```py
>>> from dataclasses import dataclass
>>> from transformers.tokenization_utils_base import PreTrainedTokenizerBase, PaddingStrategy
>>> from typing import Optional, Union
>>> import tensorflow as tf


>>> @dataclass
... class DataCollatorForMultipleChoice:
...     """
...     Data collator that will dynamically pad the inputs for multiple choice received.
...     """

...     tokenizer: PreTrainedTokenizerBase
...     padding: Union[bool, str, PaddingStrategy] = True
...     max_length: Optional[int] = None
...     pad_to_multiple_of: Optional[int] = None

...     def __call__(self, features):
...         label_name = "label" if "label" in features[0].keys() else "labels"
...         labels = [feature.pop(label_name) for feature in features]
...         batch_size = len(features)
...         num_choices = len(features[0]["input_ids"])
...         flattened_features = [
...             [{k: v[i] for k, v in feature.items()} for i in range(num_choices)] for feature in features
...         ]
...         flattened_features = sum(flattened_features, [])

...         batch = self.tokenizer.pad(
...             flattened_features,
...             padding=self.padding,
...             max_length=self.max_length,
...             pad_to_multiple_of=self.pad_to_multiple_of,
...             return_tensors="tf",
...         )

...         batch = {k: tf.reshape(v, (batch_size, num_choices, -1)) for k, v in batch.items()}
...         batch["labels"] = tf.convert_to_tensor(labels, dtype=tf.int64)
...         return batch
```
</tf>
</frameworkcontent>

## Evaluate

Including a metric during training is often helpful for evaluating your model's performance. You can quickly load a evaluation method with the 🤗 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [accuracy](https://huggingface.co/spaces/evaluate-metric/accuracy) metric (see the 🤗 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric):

```py
>>> import evaluate

>>> accuracy = evaluate.load("accuracy")
```

Then create a function that passes your predictions and labels to [`~evaluate.EvaluationModule.compute`] to calculate the accuracy:

```py
>>> import numpy as np


key: _node_content
string_value:
{"id_": "3cfeb1c7-9771-4fa3-935e-80d089f13ad4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice/", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7e8925bb-c911-4b48-b274-0207791e500e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice/", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8826953801c051c045aa6f1510cd5f98b9b125f896066d12f0fedb4bef3a1900", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d4cf148c-40ba-4501-8474-e8f83bee24d4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice/", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4c804414e88c5940a2b05058dcce800f895fae3a170cb5f531bf1246b8f8155e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e0f1b73d-3061-49ff-994d-4b33408e0db9", "node_type": "1", "metadata": {}, "hash": "66596a8766692767c9b6d27923e0c7ded80324e4ccd5e9b64dfd8b0773a749e1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8027, "end_char_idx": 10511, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> def compute_metrics(eval_pred):
...     predictions, labels = eval_pred
...     predictions = np.argmax(predictions, axis=1)
...     return accuracy.compute(predictions=predictions, references=labels)
```

Your `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.

## Train

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load BERT with [`AutoModelForMultipleChoice`]:

```py
>>> from transformers import AutoModelForMultipleChoice, TrainingArguments, Trainer

>>> model = AutoModelForMultipleChoice.from_pretrained("google-bert/bert-base-uncased")
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the accuracy and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_swag_model",
...     eval_strategy="epoch",
...     save_strategy="epoch",
...     load_best_model_at_end=True,
...     learning_rate=5e-5,
...     per_device_train_batch_size=16,
...     per_device_eval_batch_size=16,
...     num_train_epochs=3,
...     weight_decay=0.01,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=tokenized_swag["train"],
...     eval_dataset=tokenized_swag["validation"],
...     tokenizer=tokenizer,
...     data_collator=DataCollatorForMultipleChoice(tokenizer=tokenizer),
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!


key: _node_content
string_value:
{"id_": "e0f1b73d-3061-49ff-994d-4b33408e0db9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice/", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7e8925bb-c911-4b48-b274-0207791e500e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice/", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8826953801c051c045aa6f1510cd5f98b9b125f896066d12f0fedb4bef3a1900", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3cfeb1c7-9771-4fa3-935e-80d089f13ad4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice/", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "7d3feead265ff95c5022b6c9695ae2df2af9d57e40d0ade644e5b523290ba717", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "838ce74a-18cf-4557-9784-cd4f6822d7bd", "node_type": "1", "metadata": {}, "hash": "236a52b799fa6cd29198d206d0ef024845498ba2a8f422b42b1eb05ba9c54b04", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8931, "end_char_idx": 12049, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the accuracy and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_swag_model",
...     eval_strategy="epoch",
...     save_strategy="epoch",
...     load_best_model_at_end=True,
...     learning_rate=5e-5,
...     per_device_train_batch_size=16,
...     per_device_eval_batch_size=16,
...     num_train_epochs=3,
...     weight_decay=0.01,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=tokenized_swag["train"],
...     eval_dataset=tokenized_swag["validation"],
...     tokenizer=tokenizer,
...     data_collator=DataCollatorForMultipleChoice(tokenizer=tokenizer),
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!

</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import create_optimizer

>>> batch_size = 16
>>> num_train_epochs = 2
>>> total_train_steps = (len(tokenized_swag["train"]) // batch_size) * num_train_epochs
>>> optimizer, schedule = create_optimizer(init_lr=5e-5, num_warmup_steps=0, num_train_steps=total_train_steps)
```

Then you can load BERT with [`TFAutoModelForMultipleChoice`]:

```py
>>> from transformers import TFAutoModelForMultipleChoice

>>> model = TFAutoModelForMultipleChoice.from_pretrained("google-bert/bert-base-uncased")
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> data_collator = DataCollatorForMultipleChoice(tokenizer=tokenizer)
>>> tf_train_set = model.prepare_tf_dataset(
...     tokenized_swag["train"],
...     shuffle=True,
...     batch_size=batch_size,
...     collate_fn=data_collator,
... )

>>> tf_validation_set = model.prepare_tf_dataset(
...     tokenized_swag["validation"],
...     shuffle=False,
...     batch_size=batch_size,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method). Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> model.compile(optimizer=optimizer)  # No loss argument!


key: _node_content
string_value:
{"id_": "838ce74a-18cf-4557-9784-cd4f6822d7bd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice/", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7e8925bb-c911-4b48-b274-0207791e500e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice/", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8826953801c051c045aa6f1510cd5f98b9b125f896066d12f0fedb4bef3a1900", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e0f1b73d-3061-49ff-994d-4b33408e0db9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice/", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "451037cc8dfada71f7c7d5fc09945b90584735368f5cb56caf5f6d638f40a534", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d9ba3680-208f-4e21-a0b7-34abcc5a403c", "node_type": "1", "metadata": {}, "hash": "124d71ddebd8ec2fb2cb468b4020e45de97ffb9f4d0e01e488b0ea2775438c48", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10513, "end_char_idx": 13401, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import create_optimizer

>>> batch_size = 16
>>> num_train_epochs = 2
>>> total_train_steps = (len(tokenized_swag["train"]) // batch_size) * num_train_epochs
>>> optimizer, schedule = create_optimizer(init_lr=5e-5, num_warmup_steps=0, num_train_steps=total_train_steps)
```

Then you can load BERT with [`TFAutoModelForMultipleChoice`]:

```py
>>> from transformers import TFAutoModelForMultipleChoice

>>> model = TFAutoModelForMultipleChoice.from_pretrained("google-bert/bert-base-uncased")
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> data_collator = DataCollatorForMultipleChoice(tokenizer=tokenizer)
>>> tf_train_set = model.prepare_tf_dataset(
...     tokenized_swag["train"],
...     shuffle=True,
...     batch_size=batch_size,
...     collate_fn=data_collator,
... )

>>> tf_validation_set = model.prepare_tf_dataset(
...     tokenized_swag["validation"],
...     shuffle=False,
...     batch_size=batch_size,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method). Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> model.compile(optimizer=optimizer)  # No loss argument!
```

The last two things to setup before you start training is to compute the accuracy from the predictions, and provide a way to push your model to the Hub. Both are done by using [Keras callbacks](../main_classes/keras_callbacks).

Pass your `compute_metrics` function to [`~transformers.KerasMetricCallback`]:

```py
>>> from transformers.keras_callbacks import KerasMetricCallback

>>> metric_callback = KerasMetricCallback(metric_fn=compute_metrics, eval_dataset=tf_validation_set)
```

Specify where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:

```py
>>> from transformers.keras_callbacks import PushToHubCallback

>>> push_to_hub_callback = PushToHubCallback(
...     output_dir="my_awesome_model",
...     tokenizer=tokenizer,
... )
```

Then bundle your callbacks together:

```py
>>> callbacks = [metric_callback, push_to_hub_callback]
```

Finally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callbacks to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_validation_set, epochs=2, callbacks=callbacks)
```

Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "d9ba3680-208f-4e21-a0b7-34abcc5a403c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice/", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7e8925bb-c911-4b48-b274-0207791e500e", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice/", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8826953801c051c045aa6f1510cd5f98b9b125f896066d12f0fedb4bef3a1900", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "838ce74a-18cf-4557-9784-cd4f6822d7bd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/multiple_choice/", "title": "Multiple choice", "tokens": 3891, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6e26adf5154f1b63c638f3801fb54f9e83c6cc956154542fb0629e19c0b612c5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11852, "end_char_idx": 15851, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> model.compile(optimizer=optimizer)  # No loss argument!
```

The last two things to setup before you start training is to compute the accuracy from the predictions, and provide a way to push your model to the Hub. Both are done by using [Keras callbacks](../main_classes/keras_callbacks).

Pass your `compute_metrics` function to [`~transformers.KerasMetricCallback`]:

```py
>>> from transformers.keras_callbacks import KerasMetricCallback

>>> metric_callback = KerasMetricCallback(metric_fn=compute_metrics, eval_dataset=tf_validation_set)
```

Specify where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:

```py
>>> from transformers.keras_callbacks import PushToHubCallback

>>> push_to_hub_callback = PushToHubCallback(
...     output_dir="my_awesome_model",
...     tokenizer=tokenizer,
... )
```

Then bundle your callbacks together:

```py
>>> callbacks = [metric_callback, push_to_hub_callback]
```

Finally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callbacks to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_validation_set, epochs=2, callbacks=callbacks)
```

Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
</tf>
</frameworkcontent>


<Tip>

For a more in-depth example of how to finetune a model for multiple choice, take a look at the corresponding
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice-tf.ipynb).

</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Come up with some text and two candidate answers:

```py
>>> prompt = "France has a bread law, Le Décret Pain, with strict rules on what is allowed in a traditional baguette."
>>> candidate1 = "The law does not apply to croissants and brioche."
>>> candidate2 = "The law applies to baguettes."
```

<frameworkcontent>
<pt>
Tokenize each prompt and candidate answer pair and return PyTorch tensors. You should also create some `labels`:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("my_awesome_swag_model")
>>> inputs = tokenizer([[prompt, candidate1], [prompt, candidate2]], return_tensors="pt", padding=True)
>>> labels = torch.tensor(0).unsqueeze(0)
```

Pass your inputs and labels to the model and return the `logits`:

```py
>>> from transformers import AutoModelForMultipleChoice

>>> model = AutoModelForMultipleChoice.from_pretrained("my_awesome_swag_model")
>>> outputs = model(**{k: v.unsqueeze(0) for k, v in inputs.items()}, labels=labels)
>>> logits = outputs.logits
```

Get the class with the highest probability:

```py
>>> predicted_class = logits.argmax().item()
>>> predicted_class
'0'
```
</pt>
<tf>
Tokenize each prompt and candidate answer pair and return TensorFlow tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("my_awesome_swag_model")
>>> inputs = tokenizer([[prompt, candidate1], [prompt, candidate2]], return_tensors="tf", padding=True)
```

Pass your inputs to the model and return the `logits`:

```py
>>> from transformers import TFAutoModelForMultipleChoice

>>> model = TFAutoModelForMultipleChoice.from_pretrained("my_awesome_swag_model")
>>> inputs = {k: tf.expand_dims(v, 0) for k, v in inputs.items()}
>>> outputs = model(inputs)
>>> logits = outputs.logits
```

Get the class with the highest probability:

```py
>>> predicted_class = int(tf.math.argmax(logits, axis=-1)[0])
>>> predicted_class
'0'
```
</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "987f6f45-59d2-4bbd-b1bf-7d918a06e6a9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering/", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c70192d3-4b92-4784-859e-548ff0cab78c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering/", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c180d34c2b7401a8a601160b10da47255b29a8f5e3f44b00fe2612b37577ddb9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f7b49135-76e2-4547-8757-a46b24ad1666", "node_type": "1", "metadata": {}, "hash": "e04c506eb29a179de6d43eac4643a755c26e7f0cc743519bb09c2845305b7e39", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3063, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Question answering

[[open-in-colab]]

<Youtube id="ajPx5LwJD-I"/>

Question answering tasks return an answer given a question. If you've ever asked a virtual assistant like Alexa, Siri or Google what the weather is, then you've used a question answering model before. There are two common types of question answering tasks:

- Extractive: extract the answer from the given context.
- Abstractive: generate an answer from the context that correctly answers the question.

This guide will show you how to:

1. Finetune [DistilBERT](https://huggingface.co/distilbert/distilbert-base-uncased) on the [SQuAD](https://huggingface.co/datasets/squad) dataset for extractive question answering.
2. Use your finetuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/question-answering)

</Tip>

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install transformers datasets evaluate
```

We encourage you to login to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to login:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load SQuAD dataset

Start by loading a smaller subset of the SQuAD dataset from the 🤗 Datasets library. This'll give you a chance to experiment and make sure everything works before spending more time training on the full dataset.

```py
>>> from datasets import load_dataset

>>> squad = load_dataset("squad", split="train[:5000]")
```

Split the dataset's `train` split into a train and test set with the [`~datasets.Dataset.train_test_split`] method:

```py
>>> squad = squad.train_test_split(test_size=0.2)
```

Then take a look at an example:

```py
>>> squad["train"][0]
{'answers': {'answer_start': [515], 'text': ['Saint Bernadette Soubirous']},
 'context': 'Architecturally, the school has a Catholic character. Atop the Main Building\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend "Venite Ad Me Omnes". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.',
 'id': '5733be284776f41900661182',
 'question': 'To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?',
 'title': 'University_of_Notre_Dame'
}
```

There are several important fields here:

- `answers`: the starting location of the answer token and the answer text.
- `context`: background information from which the model needs to extract the answer.
- `question`: the question a model should answer.


key: _node_content
string_value:
{"id_": "f7b49135-76e2-4547-8757-a46b24ad1666", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering/", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c70192d3-4b92-4784-859e-548ff0cab78c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering/", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c180d34c2b7401a8a601160b10da47255b29a8f5e3f44b00fe2612b37577ddb9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "987f6f45-59d2-4bbd-b1bf-7d918a06e6a9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering/", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0867fd22b3c8cb2660137a44f6b765aad16ffd413b5a1056598539c7dd5c79d8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b02bb726-dc50-4566-832b-3ffb2018700a", "node_type": "1", "metadata": {}, "hash": "3d5c739a1ce353b979ff2d97ff8deb958eb9426bc5c6320b404f8d709ae0369c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1509, "end_char_idx": 4025, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> from datasets import load_dataset

>>> squad = load_dataset("squad", split="train[:5000]")
```

Split the dataset's `train` split into a train and test set with the [`~datasets.Dataset.train_test_split`] method:

```py
>>> squad = squad.train_test_split(test_size=0.2)
```

Then take a look at an example:

```py
>>> squad["train"][0]
{'answers': {'answer_start': [515], 'text': ['Saint Bernadette Soubirous']},
 'context': 'Architecturally, the school has a Catholic character. Atop the Main Building\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend "Venite Ad Me Omnes". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.',
 'id': '5733be284776f41900661182',
 'question': 'To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?',
 'title': 'University_of_Notre_Dame'
}
```

There are several important fields here:

- `answers`: the starting location of the answer token and the answer text.
- `context`: background information from which the model needs to extract the answer.
- `question`: the question a model should answer.

## Preprocess

<Youtube id="qgaM0weJHpA"/>

The next step is to load a DistilBERT tokenizer to process the `question` and `context` fields:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
```

There are a few preprocessing steps particular to question answering tasks you should be aware of:

1. Some examples in a dataset may have a very long `context` that exceeds the maximum input length of the model. To deal with longer sequences, truncate only the `context` by setting `truncation="only_second"`.
2. Next, map the start and end positions of the answer to the original `context` by setting
   `return_offset_mapping=True`.
3. With the mapping in hand, now you can find the start and end tokens of the answer. Use the [`~tokenizers.Encoding.sequence_ids`] method to
   find which part of the offset corresponds to the `question` and which corresponds to the `context`.


key: _node_content
string_value:
{"id_": "b02bb726-dc50-4566-832b-3ffb2018700a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering/", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c70192d3-4b92-4784-859e-548ff0cab78c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering/", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c180d34c2b7401a8a601160b10da47255b29a8f5e3f44b00fe2612b37577ddb9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f7b49135-76e2-4547-8757-a46b24ad1666", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering/", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "549658c1cfa9959db7fcc0e8a3a85bf4eeff9da53652570c6458c14a4be3c8e6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "63f491f0-9ed5-4852-8e6b-7aa3943aa1d4", "node_type": "1", "metadata": {}, "hash": "2f3b76818a8c4b90d8ef8b0e99271d348b4623d0126a1366d5c7318719e5f997", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2367, "end_char_idx": 6117, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.',
 'id': '5733be284776f41900661182',
 'question': 'To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?',
 'title': 'University_of_Notre_Dame'
}
```

There are several important fields here:

- `answers`: the starting location of the answer token and the answer text.
- `context`: background information from which the model needs to extract the answer.
- `question`: the question a model should answer.

## Preprocess

<Youtube id="qgaM0weJHpA"/>

The next step is to load a DistilBERT tokenizer to process the `question` and `context` fields:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
```

There are a few preprocessing steps particular to question answering tasks you should be aware of:

1. Some examples in a dataset may have a very long `context` that exceeds the maximum input length of the model. To deal with longer sequences, truncate only the `context` by setting `truncation="only_second"`.
2. Next, map the start and end positions of the answer to the original `context` by setting
   `return_offset_mapping=True`.
3. With the mapping in hand, now you can find the start and end tokens of the answer. Use the [`~tokenizers.Encoding.sequence_ids`] method to
   find which part of the offset corresponds to the `question` and which corresponds to the `context`.

Here is how you can create a function to truncate and map the start and end tokens of the `answer` to the `context`:

```py
>>> def preprocess_function(examples):
...     questions = [q.strip() for q in examples["question"]]
...     inputs = tokenizer(
...         questions,
...         examples["context"],
...         max_length=384,
...         truncation="only_second",
...         return_offsets_mapping=True,
...         padding="max_length",
...     )

...     offset_mapping = inputs.pop("offset_mapping")
...     answers = examples["answers"]
...     start_positions = []
...     end_positions = []

...     for i, offset in enumerate(offset_mapping):
...         answer = answers[i]
...         start_char = answer["answer_start"][0]
...         end_char = answer["answer_start"][0] + len(answer["text"][0])
...         sequence_ids = inputs.sequence_ids(i)

...         # Find the start and end of the context
...         idx = 0
...         while sequence_ids[idx] != 1:
...             idx += 1
...         context_start = idx
...         while sequence_ids[idx] == 1:
...             idx += 1
...         context_end = idx - 1

...         # If the answer is not fully inside the context, label it (0, 0)
...         if offset[context_start][0] > end_char or offset[context_end][1] < start_char:
...             start_positions.append(0)
...             end_positions.append(0)
...         else:
...             # Otherwise it's the start and end token positions
...             idx = context_start
...             while idx <= context_end and offset[idx][0] <= start_char:
...                 idx += 1
...             start_positions.append(idx - 1)

...             idx = context_end
...             while idx >= context_start and offset[idx][1] >= end_char:
...                 idx -= 1
...             end_positions.append(idx + 1)

...     inputs["start_positions"] = start_positions
...     inputs["end_positions"] = end_positions
...     return inputs
```

To apply the preprocessing function over the entire dataset, use 🤗 Datasets [`~datasets.Dataset.map`] function.


key: _node_content
string_value:
{"id_": "63f491f0-9ed5-4852-8e6b-7aa3943aa1d4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering/", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c70192d3-4b92-4784-859e-548ff0cab78c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering/", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c180d34c2b7401a8a601160b10da47255b29a8f5e3f44b00fe2612b37577ddb9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b02bb726-dc50-4566-832b-3ffb2018700a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering/", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5530a57464c52ccc4394c74b98cdb6e7f0e0640b3ca791015901a53c3c4e490d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5ca5352f-dcf9-4b28-b43c-1e3e975193b5", "node_type": "1", "metadata": {}, "hash": "f2632235133c1214f724dcec6853072d824dcae3146ced97f687cdc77f0981e9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6118, "end_char_idx": 8779, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once. Remove any columns you don't need:

```py
>>> tokenized_squad = squad.map(preprocess_function, batched=True, remove_columns=squad["train"].column_names)
```

Now create a batch of examples using [`DefaultDataCollator`]. Unlike other data collators in 🤗 Transformers, the [`DefaultDataCollator`] does not apply any additional preprocessing such as padding.

<frameworkcontent>
<pt>
```py
>>> from transformers import DefaultDataCollator

>>> data_collator = DefaultDataCollator()
```
</pt>
<tf>
```py
>>> from transformers import DefaultDataCollator

>>> data_collator = DefaultDataCollator(return_tensors="tf")
```
</tf>
</frameworkcontent>

## Train

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load DistilBERT with [`AutoModelForQuestionAnswering`]:

```py
>>> from transformers import AutoModelForQuestionAnswering, TrainingArguments, Trainer

>>> model = AutoModelForQuestionAnswering.from_pretrained("distilbert/distilbert-base-uncased")
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model).
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, and data collator.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_qa_model",
...     eval_strategy="epoch",
...     learning_rate=2e-5,
...     per_device_train_batch_size=16,
...     per_device_eval_batch_size=16,
...     num_train_epochs=3,
...     weight_decay=0.01,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=tokenized_squad["train"],
...     eval_dataset=tokenized_squad["test"],
...     tokenizer=tokenizer,
...     data_collator=data_collator,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!


key: _node_content
string_value:
{"id_": "5ca5352f-dcf9-4b28-b43c-1e3e975193b5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering/", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c70192d3-4b92-4784-859e-548ff0cab78c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering/", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c180d34c2b7401a8a601160b10da47255b29a8f5e3f44b00fe2612b37577ddb9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "63f491f0-9ed5-4852-8e6b-7aa3943aa1d4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering/", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "79fa6961c007d6a1d2101b430a1b1843c8588b6fd5aae0deb48ad253ffcff78f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ad54fc64-8443-43bf-b8be-aa9e1a3709ca", "node_type": "1", "metadata": {}, "hash": "a03f9849ba499f8975613becf42236d30b92d2585f0381d7cc7662419648d4ce", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7419, "end_char_idx": 10249, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model).
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, and data collator.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_qa_model",
...     eval_strategy="epoch",
...     learning_rate=2e-5,
...     per_device_train_batch_size=16,
...     per_device_eval_batch_size=16,
...     num_train_epochs=3,
...     weight_decay=0.01,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=tokenized_squad["train"],
...     eval_dataset=tokenized_squad["test"],
...     tokenizer=tokenizer,
...     data_collator=data_collator,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!

</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import create_optimizer

>>> batch_size = 16
>>> num_epochs = 2
>>> total_train_steps = (len(tokenized_squad["train"]) // batch_size) * num_epochs
>>> optimizer, schedule = create_optimizer(
...     init_lr=2e-5,
...     num_warmup_steps=0,
...     num_train_steps=total_train_steps,
... )
```

Then you can load DistilBERT with [`TFAutoModelForQuestionAnswering`]:

```py
>>> from transformers import TFAutoModelForQuestionAnswering

>>> model = TFAutoModelForQuestionAnswering.from_pretrained("distilbert/distilbert-base-uncased")
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> tf_train_set = model.prepare_tf_dataset(
...     tokenized_squad["train"],
...     shuffle=True,
...     batch_size=16,
...     collate_fn=data_collator,
... )

>>> tf_validation_set = model.prepare_tf_dataset(
...     tokenized_squad["test"],
...     shuffle=False,
...     batch_size=16,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method):

```py
>>> import tensorflow as tf

>>> model.compile(optimizer=optimizer)
```

The last thing to setup before you start training is to provide a way to push your model to the Hub.


key: _node_content
string_value:
{"id_": "ad54fc64-8443-43bf-b8be-aa9e1a3709ca", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering/", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c70192d3-4b92-4784-859e-548ff0cab78c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering/", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c180d34c2b7401a8a601160b10da47255b29a8f5e3f44b00fe2612b37577ddb9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5ca5352f-dcf9-4b28-b43c-1e3e975193b5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering/", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "f10f8e2108cd2cee6815097bb4d149fcdd79ef3fa83969b780973f3b9dee6f40", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "09265177-3d73-4a98-b93f-7973eec9f087", "node_type": "1", "metadata": {}, "hash": "5633c5115d0f00fe317080a6833cd9457e3c13cb02e8e80f4f639cb8966e7f99", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8781, "end_char_idx": 12050, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import create_optimizer

>>> batch_size = 16
>>> num_epochs = 2
>>> total_train_steps = (len(tokenized_squad["train"]) // batch_size) * num_epochs
>>> optimizer, schedule = create_optimizer(
...     init_lr=2e-5,
...     num_warmup_steps=0,
...     num_train_steps=total_train_steps,
... )
```

Then you can load DistilBERT with [`TFAutoModelForQuestionAnswering`]:

```py
>>> from transformers import TFAutoModelForQuestionAnswering

>>> model = TFAutoModelForQuestionAnswering.from_pretrained("distilbert/distilbert-base-uncased")
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> tf_train_set = model.prepare_tf_dataset(
...     tokenized_squad["train"],
...     shuffle=True,
...     batch_size=16,
...     collate_fn=data_collator,
... )

>>> tf_validation_set = model.prepare_tf_dataset(
...     tokenized_squad["test"],
...     shuffle=False,
...     batch_size=16,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method):

```py
>>> import tensorflow as tf

>>> model.compile(optimizer=optimizer)
```

The last thing to setup before you start training is to provide a way to push your model to the Hub. This can be done by specifying where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:

```py
>>> from transformers.keras_callbacks import PushToHubCallback

>>> callback = PushToHubCallback(
...     output_dir="my_awesome_qa_model",
...     tokenizer=tokenizer,
... )
```

Finally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callback to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_validation_set, epochs=3, callbacks=[callback])
```
Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
</tf>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for question answering, take a look at the corresponding
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).

</Tip>

## Evaluate

Evaluation for question answering requires a significant amount of postprocessing. To avoid taking up too much of your time, this guide skips the evaluation step. The [`Trainer`] still calculates the evaluation loss during training so you're not completely in the dark about your model's performance.

If have more time and you're interested in how to evaluate your model for question answering, take a look at the [Question answering](https://huggingface.co/course/chapter7/7?fw=pt#post-processing) chapter from the 🤗 Hugging Face Course!

## Inference

Great, now that you've finetuned a model, you can use it for inference!


key: _node_content
string_value:
{"id_": "09265177-3d73-4a98-b93f-7973eec9f087", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering/", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c70192d3-4b92-4784-859e-548ff0cab78c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering/", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c180d34c2b7401a8a601160b10da47255b29a8f5e3f44b00fe2612b37577ddb9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ad54fc64-8443-43bf-b8be-aa9e1a3709ca", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering/", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "24d3e9d9560c4acd50f06cc59debbb66dc9cf4a8fdd316c11d4191f85c411fcd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a3ba8785-8bc5-4833-b7c3-5a07434b7e35", "node_type": "1", "metadata": {}, "hash": "e746484cd609606399329f9f411df3abdbc236165789816fc3041c0b37e58bfe", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10603, "end_char_idx": 12435, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callback to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_validation_set, epochs=3, callbacks=[callback])
```
Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
</tf>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for question answering, take a look at the corresponding
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).

</Tip>

## Evaluate

Evaluation for question answering requires a significant amount of postprocessing. To avoid taking up too much of your time, this guide skips the evaluation step. The [`Trainer`] still calculates the evaluation loss during training so you're not completely in the dark about your model's performance.

If have more time and you're interested in how to evaluate your model for question answering, take a look at the [Question answering](https://huggingface.co/course/chapter7/7?fw=pt#post-processing) chapter from the 🤗 Hugging Face Course!

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Come up with a question and some context you'd like the model to predict:

```py
>>> question = "How many programming languages does BLOOM support?"
>>> context = "BLOOM has 176 billion parameters and can generate text in 46 languages natural languages and 13 programming languages."
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`].


key: _node_content
string_value:
{"id_": "a3ba8785-8bc5-4833-b7c3-5a07434b7e35", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering/", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c70192d3-4b92-4784-859e-548ff0cab78c", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering/", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c180d34c2b7401a8a601160b10da47255b29a8f5e3f44b00fe2612b37577ddb9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "09265177-3d73-4a98-b93f-7973eec9f087", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/question_answering/", "title": "Question answering", "tokens": 3690, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "926cafffa4842a08f2563fa751ce0e4894a62bd0604c92d79ce5a8e03c31dfc7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10992, "end_char_idx": 15021, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</tf>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for question answering, take a look at the corresponding
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/question_answering-tf.ipynb).

</Tip>

## Evaluate

Evaluation for question answering requires a significant amount of postprocessing. To avoid taking up too much of your time, this guide skips the evaluation step. The [`Trainer`] still calculates the evaluation loss during training so you're not completely in the dark about your model's performance.

If have more time and you're interested in how to evaluate your model for question answering, take a look at the [Question answering](https://huggingface.co/course/chapter7/7?fw=pt#post-processing) chapter from the 🤗 Hugging Face Course!

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Come up with a question and some context you'd like the model to predict:

```py
>>> question = "How many programming languages does BLOOM support?"
>>> context = "BLOOM has 176 billion parameters and can generate text in 46 languages natural languages and 13 programming languages."
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for question answering with your model, and pass your text to it:

```py
>>> from transformers import pipeline

>>> question_answerer = pipeline("question-answering", model="my_awesome_qa_model")
>>> question_answerer(question=question, context=context)
{'score': 0.2058267742395401,
 'start': 10,
 'end': 95,
 'answer': '176 billion parameters and can generate text in 46 languages natural languages and 13'}
```

You can also manually replicate the results of the `pipeline` if you'd like:

<frameworkcontent>
<pt>
Tokenize the text and return PyTorch tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("my_awesome_qa_model")
>>> inputs = tokenizer(question, context, return_tensors="pt")
```

Pass your inputs to the model and return the `logits`:

```py
>>> import torch
>>> from transformers import AutoModelForQuestionAnswering

>>> model = AutoModelForQuestionAnswering.from_pretrained("my_awesome_qa_model")
>>> with torch.no_grad():
...     outputs = model(**inputs)
```

Get the highest probability from the model output for the start and end positions:

```py
>>> answer_start_index = outputs.start_logits.argmax()
>>> answer_end_index = outputs.end_logits.argmax()
```

Decode the predicted tokens to get the answer:

```py
>>> predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1]
>>> tokenizer.decode(predict_answer_tokens)
'176 billion parameters and can generate text in 46 languages natural languages and 13'
```
</pt>
<tf>
Tokenize the text and return TensorFlow tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("my_awesome_qa_model")
>>> inputs = tokenizer(question, text, return_tensors="tf")
```

Pass your inputs to the model and return the `logits`:

```py
>>> from transformers import TFAutoModelForQuestionAnswering

>>> model = TFAutoModelForQuestionAnswering.from_pretrained("my_awesome_qa_model")
>>> outputs = model(**inputs)
```

Get the highest probability from the model output for the start and end positions:

```py
>>> answer_start_index = int(tf.math.argmax(outputs.start_logits, axis=-1)[0])
>>> answer_end_index = int(tf.math.argmax(outputs.end_logits, axis=-1)[0])
```

Decode the predicted tokens to get the answer:

```py
>>> predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1]
>>> tokenizer.decode(predict_answer_tokens)
'176 billion parameters and can generate text in 46 languages natural languages and 13'
```
</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "8e116ca4-2b42-486c-b9bc-c5327232f9d5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "155d909d-c5c6-4d9d-8ea6-427c691a3667", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6d0c4cc4285f21d2ead2f5e9e19cec64128d484f92afb841b373b4fe147dcdd0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c1734a8d-d6fd-4929-98b2-bb7816df4f58", "node_type": "1", "metadata": {}, "hash": "0ac9190c67250cdc30b5cc5d5a1399a7b21978528e02bf201b37f751e5f7a59c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3043, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Masked language modeling

[[open-in-colab]]

<Youtube id="mqElG5QJWUg"/>

Masked language modeling predicts a masked token in a sequence, and the model can attend to tokens bidirectionally. This
means the model has full access to the tokens on the left and right. Masked language modeling is great for tasks that
require a good contextual understanding of an entire sequence. BERT is an example of a masked language model.

This guide will show you how to:

1. Finetune [DistilRoBERTa](https://huggingface.co/distilbert/distilroberta-base) on the [r/askscience](https://www.reddit.com/r/askscience/) subset of the [ELI5](https://huggingface.co/datasets/eli5) dataset.
2. Use your finetuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/fill-mask)

</Tip>

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install transformers datasets evaluate
```

We encourage you to log in to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load ELI5 dataset

Start by loading the first 5000 examples from the [ELI5-Category](https://huggingface.co/datasets/eli5_category) dataset with the 🤗 Datasets library. This'll give you a chance to experiment and make sure everything works before spending more time training on the full dataset.

```py
>>> from datasets import load_dataset

>>> eli5 = load_dataset("eli5_category", split="train[:5000]")
```

Split the dataset's `train` split into a train and test set with the [`~datasets.Dataset.train_test_split`] method:

```py
>>> eli5 = eli5.train_test_split(test_size=0.2)
```

Then take a look at an example:

```py
>>> eli5["train"][0]
{'q_id': '7h191n',
 'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?',
 'selftext': '',
 'category': 'Economics',
 'subreddit': 'explainlikeimfive',
 'answers': {'a_id': ['dqnds8l', 'dqnd1jl', 'dqng3i1', 'dqnku5x'],
  'text': ["The tax bill is 500 pages long and there were a lot of changes still going on right to the end. It's not just an adjustment to the income tax brackets, it's a whole bunch of changes. As such there is no good answer to your question. The big take aways are: - Big reduction in corporate income tax rate will make large companies very happy. - Pass through rate change will make certain styles of business (law firms, hedge funds) extremely happy - Income tax changes are moderate, and are set to expire (though it's the kind of thing that might just always get re-applied without being made permanent) - People in high tax states (California, New York) lose out, and many of them will end up with their taxes raised.",
   'None yet. It has to be reconciled with a vastly different house bill and then passed again.',
   'Also: does this apply to 2017 taxes?


key: _node_content
string_value:
{"id_": "c1734a8d-d6fd-4929-98b2-bb7816df4f58", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "155d909d-c5c6-4d9d-8ea6-427c691a3667", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6d0c4cc4285f21d2ead2f5e9e19cec64128d484f92afb841b373b4fe147dcdd0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8e116ca4-2b42-486c-b9bc-c5327232f9d5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c31bbcdda143f523d35718427e9cd15aaa101760c7c2d6632a42cd9a6d0d1f37", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "aeb2a501-3f7a-4c11-a11f-713397c93c9c", "node_type": "1", "metadata": {}, "hash": "10a18a09b2703e1a9121b2c6ca4798b2e4fe00f3f60f54a87fdb508fc1cbce54", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1548, "end_char_idx": 4382, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> from datasets import load_dataset

>>> eli5 = load_dataset("eli5_category", split="train[:5000]")
```

Split the dataset's `train` split into a train and test set with the [`~datasets.Dataset.train_test_split`] method:

```py
>>> eli5 = eli5.train_test_split(test_size=0.2)
```

Then take a look at an example:

```py
>>> eli5["train"][0]
{'q_id': '7h191n',
 'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?',
 'selftext': '',
 'category': 'Economics',
 'subreddit': 'explainlikeimfive',
 'answers': {'a_id': ['dqnds8l', 'dqnd1jl', 'dqng3i1', 'dqnku5x'],
  'text': ["The tax bill is 500 pages long and there were a lot of changes still going on right to the end. It's not just an adjustment to the income tax brackets, it's a whole bunch of changes. As such there is no good answer to your question. The big take aways are: - Big reduction in corporate income tax rate will make large companies very happy. - Pass through rate change will make certain styles of business (law firms, hedge funds) extremely happy - Income tax changes are moderate, and are set to expire (though it's the kind of thing that might just always get re-applied without being made permanent) - People in high tax states (California, New York) lose out, and many of them will end up with their taxes raised.",
   'None yet. It has to be reconciled with a vastly different house bill and then passed again.',
   'Also: does this apply to 2017 taxes? Or does it start with 2018 taxes?',
   'This article explains both the House and senate bills, including the proposed changes to your income taxes based on your income level. URL_0'],
  'score': [21, 19, 5, 3],
  'text_urls': [[],
   [],
   [],
   ['https://www.investopedia.com/news/trumps-tax-reform-what-can-be-done/']]},
 'title_urls': ['url'],
 'selftext_urls': ['url']}
```

While this may look like a lot, you're only really interested in the `text` field. What's cool about language modeling tasks is you don't need labels (also known as an unsupervised task) because the next word *is* the label.

## Preprocess

<Youtube id="8PmhEIXhBvI"/>

For masked language modeling, the next step is to load a DistilRoBERTa tokenizer to process the `text` subfield:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilroberta-base")
```

You'll notice from the example above, the `text` field is actually nested inside `answers`. This means you'll need to extract the `text` subfield from its nested structure with the [`flatten`](https://huggingface.co/docs/datasets/process#flatten) method:

```py
>>> eli5 = eli5.flatten()
>>> eli5["train"][0]
{'q_id': '7h191n',
 'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?


key: _node_content
string_value:
{"id_": "aeb2a501-3f7a-4c11-a11f-713397c93c9c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "155d909d-c5c6-4d9d-8ea6-427c691a3667", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6d0c4cc4285f21d2ead2f5e9e19cec64128d484f92afb841b373b4fe147dcdd0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c1734a8d-d6fd-4929-98b2-bb7816df4f58", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "03d5866f7aef28f84d2c46b505a5f852fb9f2621e650e86306fd4910ef444371", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8bd46bb2-0a07-4acc-9cd7-a7bb049fb7e0", "node_type": "1", "metadata": {}, "hash": "3b189f9220d471a20826a314776184945e20c49a023966579d47f2a8dcd5ebf5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2902, "end_char_idx": 5573, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
",
   'None yet. It has to be reconciled with a vastly different house bill and then passed again.',
   'Also: does this apply to 2017 taxes? Or does it start with 2018 taxes?',
   'This article explains both the House and senate bills, including the proposed changes to your income taxes based on your income level. URL_0'],
  'score': [21, 19, 5, 3],
  'text_urls': [[],
   [],
   [],
   ['https://www.investopedia.com/news/trumps-tax-reform-what-can-be-done/']]},
 'title_urls': ['url'],
 'selftext_urls': ['url']}
```

While this may look like a lot, you're only really interested in the `text` field. What's cool about language modeling tasks is you don't need labels (also known as an unsupervised task) because the next word *is* the label.

## Preprocess

<Youtube id="8PmhEIXhBvI"/>

For masked language modeling, the next step is to load a DistilRoBERTa tokenizer to process the `text` subfield:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilroberta-base")
```

You'll notice from the example above, the `text` field is actually nested inside `answers`. This means you'll need to extract the `text` subfield from its nested structure with the [`flatten`](https://huggingface.co/docs/datasets/process#flatten) method:

```py
>>> eli5 = eli5.flatten()
>>> eli5["train"][0]
{'q_id': '7h191n',
 'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?',
 'selftext': '',
 'category': 'Economics',
 'subreddit': 'explainlikeimfive',
 'answers.a_id': ['dqnds8l', 'dqnd1jl', 'dqng3i1', 'dqnku5x'],
 'answers.text': ["The tax bill is 500 pages long and there were a lot of changes still going on right to the end. It's not just an adjustment to the income tax brackets, it's a whole bunch of changes. As such there is no good answer to your question. The big take aways are: - Big reduction in corporate income tax rate will make large companies very happy. - Pass through rate change will make certain styles of business (law firms, hedge funds) extremely happy - Income tax changes are moderate, and are set to expire (though it's the kind of thing that might just always get re-applied without being made permanent) - People in high tax states (California, New York) lose out, and many of them will end up with their taxes raised.",
  'None yet. It has to be reconciled with a vastly different house bill and then passed again.',
  'Also: does this apply to 2017 taxes? Or does it start with 2018 taxes?',
  'This article explains both the House and senate bills, including the proposed changes to your income taxes based on your income level.


key: _node_content
string_value:
{"id_": "8bd46bb2-0a07-4acc-9cd7-a7bb049fb7e0", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "155d909d-c5c6-4d9d-8ea6-427c691a3667", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6d0c4cc4285f21d2ead2f5e9e19cec64128d484f92afb841b373b4fe147dcdd0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "aeb2a501-3f7a-4c11-a11f-713397c93c9c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c3caddf9d896ab1e46e12bf8e450bd8ca9f63774f9443008f6254f05e148c4b7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8a8818bb-cbe7-44b6-85f6-5f405a99fed9", "node_type": "1", "metadata": {}, "hash": "f2c7825bbc3354e9bdf15018838139cfc69145eb4f4742220c3be48b9727b100", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4035, "end_char_idx": 7144, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This means you'll need to extract the `text` subfield from its nested structure with the [`flatten`](https://huggingface.co/docs/datasets/process#flatten) method:

```py
>>> eli5 = eli5.flatten()
>>> eli5["train"][0]
{'q_id': '7h191n',
 'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?',
 'selftext': '',
 'category': 'Economics',
 'subreddit': 'explainlikeimfive',
 'answers.a_id': ['dqnds8l', 'dqnd1jl', 'dqng3i1', 'dqnku5x'],
 'answers.text': ["The tax bill is 500 pages long and there were a lot of changes still going on right to the end. It's not just an adjustment to the income tax brackets, it's a whole bunch of changes. As such there is no good answer to your question. The big take aways are: - Big reduction in corporate income tax rate will make large companies very happy. - Pass through rate change will make certain styles of business (law firms, hedge funds) extremely happy - Income tax changes are moderate, and are set to expire (though it's the kind of thing that might just always get re-applied without being made permanent) - People in high tax states (California, New York) lose out, and many of them will end up with their taxes raised.",
  'None yet. It has to be reconciled with a vastly different house bill and then passed again.',
  'Also: does this apply to 2017 taxes? Or does it start with 2018 taxes?',
  'This article explains both the House and senate bills, including the proposed changes to your income taxes based on your income level. URL_0'],
 'answers.score': [21, 19, 5, 3],
 'answers.text_urls': [[],
  [],
  [],
  ['https://www.investopedia.com/news/trumps-tax-reform-what-can-be-done/']],
 'title_urls': ['url'],
 'selftext_urls': ['url']}
```

Each subfield is now a separate column as indicated by the `answers` prefix, and the `text` field is a list now. Instead
of tokenizing each sentence separately, convert the list to a string so you can jointly tokenize them.

Here is a first preprocessing function to join the list of strings for each example and tokenize the result:

```py
>>> def preprocess_function(examples):
...     return tokenizer([" ".join(x) for x in examples["answers.text"]])
```

To apply this preprocessing function over the entire dataset, use the 🤗 Datasets [`~datasets.Dataset.map`] method. You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once, and increasing the number of processes with `num_proc`. Remove any columns you don't need:

```py
>>> tokenized_eli5 = eli5.map(
...     preprocess_function,
...     batched=True,
...     num_proc=4,
...     remove_columns=eli5["train"].column_names,
... )
```

This dataset contains the token sequences, but some of these are longer than the maximum input length for the model.

You can now use a second preprocessing function to
- concatenate all the sequences
- split the concatenated sequences into shorter chunks defined by `block_size`, which should be both shorter than the maximum input length and short enough for your GPU RAM. 

```py
>>> block_size = 128


key: _node_content
string_value:
{"id_": "8a8818bb-cbe7-44b6-85f6-5f405a99fed9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "155d909d-c5c6-4d9d-8ea6-427c691a3667", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6d0c4cc4285f21d2ead2f5e9e19cec64128d484f92afb841b373b4fe147dcdd0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8bd46bb2-0a07-4acc-9cd7-a7bb049fb7e0", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8fc1e767d75e2659b88d6075c06e587a14cd2d8b1e6bf0220c52c70abcbd1b70", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "717dbebc-f284-4472-8aed-cb3a2a60d574", "node_type": "1", "metadata": {}, "hash": "6a9cabdbe3e33e8a8f20cd48f0ccf71c1285c81f74d759abd3802f14c3700765", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5574, "end_char_idx": 8257, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
URL_0'],
 'answers.score': [21, 19, 5, 3],
 'answers.text_urls': [[],
  [],
  [],
  ['https://www.investopedia.com/news/trumps-tax-reform-what-can-be-done/']],
 'title_urls': ['url'],
 'selftext_urls': ['url']}
```

Each subfield is now a separate column as indicated by the `answers` prefix, and the `text` field is a list now. Instead
of tokenizing each sentence separately, convert the list to a string so you can jointly tokenize them.

Here is a first preprocessing function to join the list of strings for each example and tokenize the result:

```py
>>> def preprocess_function(examples):
...     return tokenizer([" ".join(x) for x in examples["answers.text"]])
```

To apply this preprocessing function over the entire dataset, use the 🤗 Datasets [`~datasets.Dataset.map`] method. You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once, and increasing the number of processes with `num_proc`. Remove any columns you don't need:

```py
>>> tokenized_eli5 = eli5.map(
...     preprocess_function,
...     batched=True,
...     num_proc=4,
...     remove_columns=eli5["train"].column_names,
... )
```

This dataset contains the token sequences, but some of these are longer than the maximum input length for the model.

You can now use a second preprocessing function to
- concatenate all the sequences
- split the concatenated sequences into shorter chunks defined by `block_size`, which should be both shorter than the maximum input length and short enough for your GPU RAM. 

```py
>>> block_size = 128


>>> def group_texts(examples):
...     # Concatenate all texts.
...     concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
...     total_length = len(concatenated_examples[list(examples.keys())[0]])
...     # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can
...     # customize this part to your needs.
...     if total_length >= block_size:
...         total_length = (total_length // block_size) * block_size
...     # Split by chunks of block_size.
...     result = {
...         k: [t[i : i + block_size] for i in range(0, total_length, block_size)]
...         for k, t in concatenated_examples.items()
...     }
...     return result
```

Apply the `group_texts` function over the entire dataset:

```py
>>> lm_dataset = tokenized_eli5.map(group_texts, batched=True, num_proc=4)
```

Now create a batch of examples using [`DataCollatorForLanguageModeling`]. It's more efficient to *dynamically pad* the sentences to the longest length in a batch during collation, instead of padding the whole dataset to the maximum length.


key: _node_content
string_value:
{"id_": "717dbebc-f284-4472-8aed-cb3a2a60d574", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "155d909d-c5c6-4d9d-8ea6-427c691a3667", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6d0c4cc4285f21d2ead2f5e9e19cec64128d484f92afb841b373b4fe147dcdd0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8a8818bb-cbe7-44b6-85f6-5f405a99fed9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8c07616fe18e100d2229ec441d89e6aa7e00aadd5b49d47cb0348ff9ce7dfdbd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5311e44f-fca4-4c09-82ad-d27a7b1f230c", "node_type": "1", "metadata": {}, "hash": "3b9316d453b73d8fa9c755f39910f9cfd0a69a6510c912b4e155ff7851703518", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6859, "end_char_idx": 9922, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can now use a second preprocessing function to
- concatenate all the sequences
- split the concatenated sequences into shorter chunks defined by `block_size`, which should be both shorter than the maximum input length and short enough for your GPU RAM. 

```py
>>> block_size = 128


>>> def group_texts(examples):
...     # Concatenate all texts.
...     concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
...     total_length = len(concatenated_examples[list(examples.keys())[0]])
...     # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can
...     # customize this part to your needs.
...     if total_length >= block_size:
...         total_length = (total_length // block_size) * block_size
...     # Split by chunks of block_size.
...     result = {
...         k: [t[i : i + block_size] for i in range(0, total_length, block_size)]
...         for k, t in concatenated_examples.items()
...     }
...     return result
```

Apply the `group_texts` function over the entire dataset:

```py
>>> lm_dataset = tokenized_eli5.map(group_texts, batched=True, num_proc=4)
```

Now create a batch of examples using [`DataCollatorForLanguageModeling`]. It's more efficient to *dynamically pad* the sentences to the longest length in a batch during collation, instead of padding the whole dataset to the maximum length.

<frameworkcontent>
<pt>

Use the end-of-sequence token as the padding token and specify `mlm_probability` to randomly mask tokens each time you iterate over the data:

```py
>>> from transformers import DataCollatorForLanguageModeling

>>> tokenizer.pad_token = tokenizer.eos_token
>>> data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm_probability=0.15)
```
</pt>
<tf>

Use the end-of-sequence token as the padding token and specify `mlm_probability` to randomly mask tokens each time you iterate over the data:

```py
>>> from transformers import DataCollatorForLanguageModeling

>>> data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm_probability=0.15, return_tensors="tf")
```
</tf>
</frameworkcontent>

## Train

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load DistilRoBERTa with [`AutoModelForMaskedLM`]:

```py
>>> from transformers import AutoModelForMaskedLM

>>> model = AutoModelForMaskedLM.from_pretrained("distilbert/distilroberta-base")
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model).
2. Pass the training arguments to [`Trainer`] along with the model, datasets, and data collator.
3. Call [`~Trainer.train`] to finetune your model.


key: _node_content
string_value:
{"id_": "5311e44f-fca4-4c09-82ad-d27a7b1f230c", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "155d909d-c5c6-4d9d-8ea6-427c691a3667", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6d0c4cc4285f21d2ead2f5e9e19cec64128d484f92afb841b373b4fe147dcdd0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "717dbebc-f284-4472-8aed-cb3a2a60d574", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "999ba8183b7069dc1e4d0eceec199a61ea240e9300a6cd31a7a4e92f3b488ae8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b6e53a2d-66e3-4d09-ba0b-6f4724e11c95", "node_type": "1", "metadata": {}, "hash": "d9ad6c526f3cdbb490f3b38e09507f8418d81ae632ff0e2baf0144a461ca17c2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9203, "end_char_idx": 12061, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

You're ready to start training your model now! Load DistilRoBERTa with [`AutoModelForMaskedLM`]:

```py
>>> from transformers import AutoModelForMaskedLM

>>> model = AutoModelForMaskedLM.from_pretrained("distilbert/distilroberta-base")
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model).
2. Pass the training arguments to [`Trainer`] along with the model, datasets, and data collator.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_eli5_mlm_model",
...     eval_strategy="epoch",
...     learning_rate=2e-5,
...     num_train_epochs=3,
...     weight_decay=0.01,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=lm_dataset["train"],
...     eval_dataset=lm_dataset["test"],
...     data_collator=data_collator,
... )

>>> trainer.train()
```

Once training is completed, use the [`~transformers.Trainer.evaluate`] method to evaluate your model and get its perplexity:

```py
>>> import math

>>> eval_results = trainer.evaluate()
>>> print(f"Perplexity: {math.exp(eval_results['eval_loss']):.2f}")
Perplexity: 8.76
```

Then share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the basic tutorial [here](../training#train-a-tensorflow-model-with-keras)!

</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import create_optimizer, AdamWeightDecay

>>> optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)
```

Then you can load DistilRoBERTa with [`TFAutoModelForMaskedLM`]:

```py
>>> from transformers import TFAutoModelForMaskedLM

>>> model = TFAutoModelForMaskedLM.from_pretrained("distilbert/distilroberta-base")
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> tf_train_set = model.prepare_tf_dataset(
...     lm_dataset["train"],
...     shuffle=True,
...     batch_size=16,
...     collate_fn=data_collator,
... )

>>> tf_test_set = model.prepare_tf_dataset(
...     lm_dataset["test"],
...     shuffle=False,
...     batch_size=16,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method).


key: _node_content
string_value:
{"id_": "b6e53a2d-66e3-4d09-ba0b-6f4724e11c95", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "155d909d-c5c6-4d9d-8ea6-427c691a3667", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6d0c4cc4285f21d2ead2f5e9e19cec64128d484f92afb841b373b4fe147dcdd0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5311e44f-fca4-4c09-82ad-d27a7b1f230c", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4e27a2a5cce3301c82da2e8e08f53680575750fabf94ddd371761071ec410de7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "873229a8-e310-4905-9364-3cc3fa7f0832", "node_type": "1", "metadata": {}, "hash": "b693c2d971025a507c713a496013b00da3022f285966af65e4247c161de42532", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10991, "end_char_idx": 13911, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import create_optimizer, AdamWeightDecay

>>> optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)
```

Then you can load DistilRoBERTa with [`TFAutoModelForMaskedLM`]:

```py
>>> from transformers import TFAutoModelForMaskedLM

>>> model = TFAutoModelForMaskedLM.from_pretrained("distilbert/distilroberta-base")
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> tf_train_set = model.prepare_tf_dataset(
...     lm_dataset["train"],
...     shuffle=True,
...     batch_size=16,
...     collate_fn=data_collator,
... )

>>> tf_test_set = model.prepare_tf_dataset(
...     lm_dataset["test"],
...     shuffle=False,
...     batch_size=16,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method). Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> import tensorflow as tf

>>> model.compile(optimizer=optimizer)  # No loss argument!
```

This can be done by specifying where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:

```py
>>> from transformers.keras_callbacks import PushToHubCallback

>>> callback = PushToHubCallback(
...     output_dir="my_awesome_eli5_mlm_model",
...     tokenizer=tokenizer,
... )
```

Finally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callback to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_test_set, epochs=3, callbacks=[callback])
```

Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
</tf>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for masked language modeling, take a look at the corresponding
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).

</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Come up with some text you'd like the model to fill in the blank with, and use the special `<mask>` token to indicate the blank:

```py
>>> text = "The Milky Way is a <mask> galaxy."
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for fill-mask with your model, and pass your text to it.


key: _node_content
string_value:
{"id_": "873229a8-e310-4905-9364-3cc3fa7f0832", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "155d909d-c5c6-4d9d-8ea6-427c691a3667", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6d0c4cc4285f21d2ead2f5e9e19cec64128d484f92afb841b373b4fe147dcdd0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b6e53a2d-66e3-4d09-ba0b-6f4724e11c95", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "2b383ca8ba7d41ea5e24520f0529e3678a16aea6d6b2aea845ae621dd38da9c8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "816740ad-277a-4fb7-997f-5a32f514f33d", "node_type": "1", "metadata": {}, "hash": "5b3617d5ea882657a699cfcbadd744546e96406d02d90e3b5947fb7a308b50c6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12289, "end_char_idx": 14630, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```

This can be done by specifying where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:

```py
>>> from transformers.keras_callbacks import PushToHubCallback

>>> callback = PushToHubCallback(
...     output_dir="my_awesome_eli5_mlm_model",
...     tokenizer=tokenizer,
... )
```

Finally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callback to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_test_set, epochs=3, callbacks=[callback])
```

Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
</tf>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for masked language modeling, take a look at the corresponding
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).

</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Come up with some text you'd like the model to fill in the blank with, and use the special `<mask>` token to indicate the blank:

```py
>>> text = "The Milky Way is a <mask> galaxy."
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for fill-mask with your model, and pass your text to it. If you like, you can use the `top_k` parameter to specify how many predictions to return:

```py
>>> from transformers import pipeline

>>> mask_filler = pipeline("fill-mask", "username/my_awesome_eli5_mlm_model")
>>> mask_filler(text, top_k=3)
[{'score': 0.5150994658470154,
  'token': 21300,
  'token_str': ' spiral',
  'sequence': 'The Milky Way is a spiral galaxy.'},
 {'score': 0.07087188959121704,
  'token': 2232,
  'token_str': ' massive',
  'sequence': 'The Milky Way is a massive galaxy.'},
 {'score': 0.06434620916843414,
  'token': 650,
  'token_str': ' small',
  'sequence': 'The Milky Way is a small galaxy.'}]
```

<frameworkcontent>
<pt>
Tokenize the text and return the `input_ids` as PyTorch tensors.


key: _node_content
string_value:
{"id_": "816740ad-277a-4fb7-997f-5a32f514f33d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "155d909d-c5c6-4d9d-8ea6-427c691a3667", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6d0c4cc4285f21d2ead2f5e9e19cec64128d484f92afb841b373b4fe147dcdd0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "873229a8-e310-4905-9364-3cc3fa7f0832", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "37e7019c6c709e199d7db1fd67a8dbd3d8bf8fcc9a123ef3a9325704a1ab96dd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3fa3233a-063a-4098-9205-28f0a700886e", "node_type": "1", "metadata": {}, "hash": "829a88a74a4b39ff287a2f9e7093138bfaf6a7ff78359e5bf2132e5653f0584d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13452, "end_char_idx": 15751, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Come up with some text you'd like the model to fill in the blank with, and use the special `<mask>` token to indicate the blank:

```py
>>> text = "The Milky Way is a <mask> galaxy."
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for fill-mask with your model, and pass your text to it. If you like, you can use the `top_k` parameter to specify how many predictions to return:

```py
>>> from transformers import pipeline

>>> mask_filler = pipeline("fill-mask", "username/my_awesome_eli5_mlm_model")
>>> mask_filler(text, top_k=3)
[{'score': 0.5150994658470154,
  'token': 21300,
  'token_str': ' spiral',
  'sequence': 'The Milky Way is a spiral galaxy.'},
 {'score': 0.07087188959121704,
  'token': 2232,
  'token_str': ' massive',
  'sequence': 'The Milky Way is a massive galaxy.'},
 {'score': 0.06434620916843414,
  'token': 650,
  'token_str': ' small',
  'sequence': 'The Milky Way is a small galaxy.'}]
```

<frameworkcontent>
<pt>
Tokenize the text and return the `input_ids` as PyTorch tensors. You'll also need to specify the position of the `<mask>` token:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("username/my_awesome_eli5_mlm_model")
>>> inputs = tokenizer(text, return_tensors="pt")
>>> mask_token_index = torch.where(inputs["input_ids"] == tokenizer.mask_token_id)[1]
```

Pass your inputs to the model and return the `logits` of the masked token:

```py
>>> from transformers import AutoModelForMaskedLM

>>> model = AutoModelForMaskedLM.from_pretrained("username/my_awesome_eli5_mlm_model")
>>> logits = model(**inputs).logits
>>> mask_token_logits = logits[0, mask_token_index, :]
```

Then return the three masked tokens with the highest probability and print them out:

```py
>>> top_3_tokens = torch.topk(mask_token_logits, 3, dim=1).indices[0].tolist()

>>> for token in top_3_tokens:
...     print(text.replace(tokenizer.mask_token, tokenizer.decode([token])))
The Milky Way is a spiral galaxy.
The Milky Way is a massive galaxy.
The Milky Way is a small galaxy.
```
</pt>
<tf>
Tokenize the text and return the `input_ids` as TensorFlow tensors.


key: _node_content
string_value:
{"id_": "3fa3233a-063a-4098-9205-28f0a700886e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "155d909d-c5c6-4d9d-8ea6-427c691a3667", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6d0c4cc4285f21d2ead2f5e9e19cec64128d484f92afb841b373b4fe147dcdd0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "816740ad-277a-4fb7-997f-5a32f514f33d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/masked_language_modeling/", "title": "Masked language modeling", "tokens": 4346, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "72d047960e9a3155deb26683ae941493e3753080f66f7011d6c89e9aef6f3bd6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14280, "end_char_idx": 16815, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
'},
 {'score': 0.07087188959121704,
  'token': 2232,
  'token_str': ' massive',
  'sequence': 'The Milky Way is a massive galaxy.'},
 {'score': 0.06434620916843414,
  'token': 650,
  'token_str': ' small',
  'sequence': 'The Milky Way is a small galaxy.'}]
```

<frameworkcontent>
<pt>
Tokenize the text and return the `input_ids` as PyTorch tensors. You'll also need to specify the position of the `<mask>` token:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("username/my_awesome_eli5_mlm_model")
>>> inputs = tokenizer(text, return_tensors="pt")
>>> mask_token_index = torch.where(inputs["input_ids"] == tokenizer.mask_token_id)[1]
```

Pass your inputs to the model and return the `logits` of the masked token:

```py
>>> from transformers import AutoModelForMaskedLM

>>> model = AutoModelForMaskedLM.from_pretrained("username/my_awesome_eli5_mlm_model")
>>> logits = model(**inputs).logits
>>> mask_token_logits = logits[0, mask_token_index, :]
```

Then return the three masked tokens with the highest probability and print them out:

```py
>>> top_3_tokens = torch.topk(mask_token_logits, 3, dim=1).indices[0].tolist()

>>> for token in top_3_tokens:
...     print(text.replace(tokenizer.mask_token, tokenizer.decode([token])))
The Milky Way is a spiral galaxy.
The Milky Way is a massive galaxy.
The Milky Way is a small galaxy.
```
</pt>
<tf>
Tokenize the text and return the `input_ids` as TensorFlow tensors. You'll also need to specify the position of the `<mask>` token:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("username/my_awesome_eli5_mlm_model")
>>> inputs = tokenizer(text, return_tensors="tf")
>>> mask_token_index = tf.where(inputs["input_ids"] == tokenizer.mask_token_id)[0, 1]
```

Pass your inputs to the model and return the `logits` of the masked token:

```py
>>> from transformers import TFAutoModelForMaskedLM

>>> model = TFAutoModelForMaskedLM.from_pretrained("username/my_awesome_eli5_mlm_model")
>>> logits = model(**inputs).logits
>>> mask_token_logits = logits[0, mask_token_index, :]
```

Then return the three masked tokens with the highest probability and print them out:

```py
>>> top_3_tokens = tf.math.top_k(mask_token_logits, 3).indices.numpy()

>>> for token in top_3_tokens:
...     print(text.replace(tokenizer.mask_token, tokenizer.decode([token])))
The Milky Way is a spiral galaxy.
The Milky Way is a massive galaxy.
The Milky Way is a small galaxy.
```
</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "e0e9d572-c738-43e7-a413-8a0fac21aabb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_to_image/", "title": "Image-to-Image Task Guide", "tokens": 1389, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "29f78d9c-dce6-4af2-9c0b-572c2499cdbe", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_to_image/", "title": "Image-to-Image Task Guide", "tokens": 1389, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "aa82570a48f66630f622702664dbe8e39b9a37c7ea55acad962689bc45b61234", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e919b25d-f8f6-4cc9-bbe9-f2dbe766adf4", "node_type": "1", "metadata": {}, "hash": "1ff1cef968511d48382d6cbe7ee3c4b063282962845720e40a5569c0e026c6ed", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2778, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Image-to-Image Task Guide

[[open-in-colab]]

Image-to-Image task is the task where an application receives an image and outputs another image. This has various subtasks, including image enhancement (super resolution, low light enhancement, deraining and so on), image inpainting, and more. 

This guide will show you how to:
- Use an image-to-image pipeline for super resolution task,
- Run image-to-image models for same task without a pipeline.

Note that as of the time this guide is released, `image-to-image` pipeline only supports super resolution task.

Let's begin by installing the necessary libraries.

```bash
pip install transformers
```

We can now initialize the pipeline with a [Swin2SR model](https://huggingface.co/caidas/swin2SR-lightweight-x2-64). We can then infer with the pipeline by calling it with an image. As of now, only [Swin2SR models](https://huggingface.co/models?sort=trending&search=swin2sr) are supported in this pipeline. 

```python
from transformers import pipeline

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
pipe = pipeline(task="image-to-image", model="caidas/swin2SR-lightweight-x2-64", device=device)
```

Now, let's load an image.

```python
from PIL import Image
import requests

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/cat.jpg"
image = Image.open(requests.get(url, stream=True).raw)

print(image.size)
```
```bash
# (532, 432)
```
<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/cat.jpg" alt="Photo of a cat"/>
</div>

We can now do inference with the pipeline. We will get an upscaled version of the cat image. 

```python
upscaled = pipe(image)
print(upscaled.size)
```
```bash
# (1072, 880)
```

If you wish to do inference yourself with no pipeline, you can use the `Swin2SRForImageSuperResolution` and `Swin2SRImageProcessor` classes of transformers. We will use the same model checkpoint for this. Let's initialize the model and the processor.

```python
from transformers import Swin2SRForImageSuperResolution, Swin2SRImageProcessor 

model = Swin2SRForImageSuperResolution.from_pretrained("caidas/swin2SR-lightweight-x2-64").to(device)
processor = Swin2SRImageProcessor("caidas/swin2SR-lightweight-x2-64")
```

`pipeline` abstracts away the preprocessing and postprocessing steps that we have to do ourselves, so let's preprocess the image. We will pass the image to the processor and then move the pixel values to GPU. 

```python
pixel_values = processor(image, return_tensors="pt").pixel_values
print(pixel_values.shape)

pixel_values = pixel_values.to(device)
```

We can now infer the image by passing pixel values to the model.


key: _node_content
string_value:
{"id_": "e919b25d-f8f6-4cc9-bbe9-f2dbe766adf4", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_to_image/", "title": "Image-to-Image Task Guide", "tokens": 1389, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "29f78d9c-dce6-4af2-9c0b-572c2499cdbe", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_to_image/", "title": "Image-to-Image Task Guide", "tokens": 1389, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "aa82570a48f66630f622702664dbe8e39b9a37c7ea55acad962689bc45b61234", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e0e9d572-c738-43e7-a413-8a0fac21aabb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_to_image/", "title": "Image-to-Image Task Guide", "tokens": 1389, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "9631a7365e234ac8879209734ea47da2c15f3ae0fe187c60cd07bab399439c05", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e4012c4b-b999-4492-85f8-df5947d8a8e3", "node_type": "1", "metadata": {}, "hash": "c28a6859c39eb1a4971ec2bd86ffee8bb3aa9bd5d60ec072f553d0346e80214a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1205, "end_char_idx": 3746, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from PIL import Image
import requests

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/cat.jpg"
image = Image.open(requests.get(url, stream=True).raw)

print(image.size)
```
```bash
# (532, 432)
```
<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/cat.jpg" alt="Photo of a cat"/>
</div>

We can now do inference with the pipeline. We will get an upscaled version of the cat image. 

```python
upscaled = pipe(image)
print(upscaled.size)
```
```bash
# (1072, 880)
```

If you wish to do inference yourself with no pipeline, you can use the `Swin2SRForImageSuperResolution` and `Swin2SRImageProcessor` classes of transformers. We will use the same model checkpoint for this. Let's initialize the model and the processor.

```python
from transformers import Swin2SRForImageSuperResolution, Swin2SRImageProcessor 

model = Swin2SRForImageSuperResolution.from_pretrained("caidas/swin2SR-lightweight-x2-64").to(device)
processor = Swin2SRImageProcessor("caidas/swin2SR-lightweight-x2-64")
```

`pipeline` abstracts away the preprocessing and postprocessing steps that we have to do ourselves, so let's preprocess the image. We will pass the image to the processor and then move the pixel values to GPU. 

```python
pixel_values = processor(image, return_tensors="pt").pixel_values
print(pixel_values.shape)

pixel_values = pixel_values.to(device)
```

We can now infer the image by passing pixel values to the model.

```python
import torch

with torch.no_grad():
  outputs = model(pixel_values)
```
Output is an object of type `ImageSuperResolutionOutput` that looks like below 👇 

```
(loss=None, reconstruction=tensor([[[[0.8270, 0.8269, 0.8275,  ..., 0.7463, 0.7446, 0.7453],
          [0.8287, 0.8278, 0.8283,  ..., 0.7451, 0.7448, 0.7457],
          [0.8280, 0.8273, 0.8269,  ..., 0.7447, 0.7446, 0.7452],
          ...,
          [0.5923, 0.5933, 0.5924,  ..., 0.0697, 0.0695, 0.0706],
          [0.5926, 0.5932, 0.5926,  ..., 0.0673, 0.0687, 0.0705],
          [0.5927, 0.5914, 0.5922,  ..., 0.0664, 0.0694, 0.0718]]]],
       device='cuda:0'), hidden_states=None, attentions=None)
```
We need to get the `reconstruction` and post-process it for visualization. Let's see how it looks like.

```python
outputs.reconstruction.data.shape
# torch.Size([1, 3, 880, 1072])
```

We need to squeeze the output and get rid of axis 0, clip the values, then convert it to be numpy float.


key: _node_content
string_value:
{"id_": "e4012c4b-b999-4492-85f8-df5947d8a8e3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_to_image/", "title": "Image-to-Image Task Guide", "tokens": 1389, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "29f78d9c-dce6-4af2-9c0b-572c2499cdbe", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_to_image/", "title": "Image-to-Image Task Guide", "tokens": 1389, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "aa82570a48f66630f622702664dbe8e39b9a37c7ea55acad962689bc45b61234", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e919b25d-f8f6-4cc9-bbe9-f2dbe766adf4", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/image_to_image/", "title": "Image-to-Image Task Guide", "tokens": 1389, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b6a081ad9f7ace9ed4884ad098c445f555cad736d4eb78963a2e0bac1a755626", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2780, "end_char_idx": 4405, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import torch

with torch.no_grad():
  outputs = model(pixel_values)
```
Output is an object of type `ImageSuperResolutionOutput` that looks like below 👇 

```
(loss=None, reconstruction=tensor([[[[0.8270, 0.8269, 0.8275,  ..., 0.7463, 0.7446, 0.7453],
          [0.8287, 0.8278, 0.8283,  ..., 0.7451, 0.7448, 0.7457],
          [0.8280, 0.8273, 0.8269,  ..., 0.7447, 0.7446, 0.7452],
          ...,
          [0.5923, 0.5933, 0.5924,  ..., 0.0697, 0.0695, 0.0706],
          [0.5926, 0.5932, 0.5926,  ..., 0.0673, 0.0687, 0.0705],
          [0.5927, 0.5914, 0.5922,  ..., 0.0664, 0.0694, 0.0718]]]],
       device='cuda:0'), hidden_states=None, attentions=None)
```
We need to get the `reconstruction` and post-process it for visualization. Let's see how it looks like.

```python
outputs.reconstruction.data.shape
# torch.Size([1, 3, 880, 1072])
```

We need to squeeze the output and get rid of axis 0, clip the values, then convert it to be numpy float. Then we will arrange axes to have the shape [1072, 880], and finally, bring the output back to range [0, 255].

```python
import numpy as np

# squeeze, take to CPU and clip the values
output = outputs.reconstruction.data.squeeze().cpu().clamp_(0, 1).numpy()
# rearrange the axes
output = np.moveaxis(output, source=0, destination=-1)
# bring values back to pixel values range
output = (output * 255.0).round().astype(np.uint8)
Image.fromarray(output)
```
<div class="flex justify-center">
     <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/cat_upscaled.png" alt="Upscaled photo of a cat"/>
</div>


key: _node_content
string_value:
{"id_": "ef6101e5-bc12-4518-bdfd-a5d04ab1fe90", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6fccebea-b409-4fe0-b80c-1016cf4fdea8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b1fa69e298e5fb1d81175c275d4e7c914226df9ad6fa6eda7fb82d015f01d7bb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f64c2620-a05d-4539-8f7c-5fb7ae01c3c2", "node_type": "1", "metadata": {}, "hash": "243206efbd7b313c37e7fbee6b3f6512683901eef66261c089b61a728730135d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2872, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Video classification

[[open-in-colab]]

Video classification is the task of assigning a label or class to an entire video. Videos are expected to have only one class for each video. Video classification models take a video as input and return a prediction about which class the video belongs to. These models can be used to categorize what a video is all about. A real-world application of video classification is action / activity recognition, which is useful for fitness applications. It is also helpful for vision-impaired individuals, especially when they are commuting.

This guide will show you how to:

1. Fine-tune [VideoMAE](https://huggingface.co/docs/transformers/main/en/model_doc/videomae) on a subset of the [UCF101](https://www.crcv.ucf.edu/data/UCF101.php) dataset.
2. Use your fine-tuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/video-classification).

</Tip>

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install -q pytorchvideo transformers evaluate
```

You will use [PyTorchVideo](https://pytorchvideo.org/) (dubbed `pytorchvideo`) to process and prepare the videos.

We encourage you to log in to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load UCF101 dataset

Start by loading a subset of the [UCF-101 dataset](https://www.crcv.ucf.edu/data/UCF101.php). This will give you a chance to experiment and make sure everything works before spending more time training on the full dataset.

```py
>>> from huggingface_hub import hf_hub_download

>>> hf_dataset_identifier = "sayakpaul/ucf101-subset"
>>> filename = "UCF101_subset.tar.gz"
>>> file_path = hf_hub_download(repo_id=hf_dataset_identifier, filename=filename, repo_type="dataset")
```

After the subset has been downloaded, you need to extract the compressed archive:

```py 
>>> import tarfile

>>> with tarfile.open(file_path) as t:
...      t.extractall(".")
```

At a high level, the dataset is organized like so:

```bash
UCF101_subset/
    train/
        BandMarching/
            video_1.mp4
            video_2.mp4
            ...
        Archery
            video_1.mp4
            video_2.mp4
            ...
        ...
    val/
        BandMarching/
            video_1.mp4
            video_2.mp4
            ...
        Archery
            video_1.mp4
            video_2.mp4
            ...
        ...
    test/
        BandMarching/
            video_1.mp4
            video_2.mp4
            ...
        Archery
            video_1.mp4
            video_2.mp4
            ...
        ...
```

You can then count the number of total videos.


key: _node_content
string_value:
{"id_": "f64c2620-a05d-4539-8f7c-5fb7ae01c3c2", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6fccebea-b409-4fe0-b80c-1016cf4fdea8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b1fa69e298e5fb1d81175c275d4e7c914226df9ad6fa6eda7fb82d015f01d7bb", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ef6101e5-bc12-4518-bdfd-a5d04ab1fe90", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "94daeea83101d63887661d96ea13cc5a677d5db282fe3d4ad96adca0b98db497", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cc098e1f-24d6-4fcf-b358-b307f5b0f11f", "node_type": "1", "metadata": {}, "hash": "966439955c865a87bec8253a1b82354754a78595997bbe9126fd7f9877ef072a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1259, "end_char_idx": 4199, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We encourage you to log in to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load UCF101 dataset

Start by loading a subset of the [UCF-101 dataset](https://www.crcv.ucf.edu/data/UCF101.php). This will give you a chance to experiment and make sure everything works before spending more time training on the full dataset.

```py
>>> from huggingface_hub import hf_hub_download

>>> hf_dataset_identifier = "sayakpaul/ucf101-subset"
>>> filename = "UCF101_subset.tar.gz"
>>> file_path = hf_hub_download(repo_id=hf_dataset_identifier, filename=filename, repo_type="dataset")
```

After the subset has been downloaded, you need to extract the compressed archive:

```py 
>>> import tarfile

>>> with tarfile.open(file_path) as t:
...      t.extractall(".")
```

At a high level, the dataset is organized like so:

```bash
UCF101_subset/
    train/
        BandMarching/
            video_1.mp4
            video_2.mp4
            ...
        Archery
            video_1.mp4
            video_2.mp4
            ...
        ...
    val/
        BandMarching/
            video_1.mp4
            video_2.mp4
            ...
        Archery
            video_1.mp4
            video_2.mp4
            ...
        ...
    test/
        BandMarching/
            video_1.mp4
            video_2.mp4
            ...
        Archery
            video_1.mp4
            video_2.mp4
            ...
        ...
```

You can then count the number of total videos.

```py 
>>> import pathlib
>>> dataset_root_path = "UCF101_subset"
>>> dataset_root_path = pathlib.Path(dataset_root_path)
```

```py 
>>> video_count_train = len(list(dataset_root_path.glob("train/*/*.avi")))
>>> video_count_val = len(list(dataset_root_path.glob("val/*/*.avi")))
>>> video_count_test = len(list(dataset_root_path.glob("test/*/*.avi")))
>>> video_total = video_count_train + video_count_val + video_count_test
>>> print(f"Total videos: {video_total}")
```

```py 
>>> all_video_file_paths = (
...     list(dataset_root_path.glob("train/*/*.avi"))
...     + list(dataset_root_path.glob("val/*/*.avi"))
...     + list(dataset_root_path.glob("test/*/*.avi"))
...  )
>>> all_video_file_paths[:5]
```

The (`sorted`) video paths appear like so:

```bash
...
'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g07_c04.avi',
'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g07_c06.avi',
'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g08_c01.avi',
'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g09_c02.avi',
'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g09_c06.avi'
...
```

You will notice that there are video clips belonging to the same group / scene where group is denoted by `g` in the video file paths. `v_ApplyEyeMakeup_g07_c04.avi` and `v_ApplyEyeMakeup_g07_c06.avi`, for example.


key: _node_content
string_value:
{"id_": "cc098e1f-24d6-4fcf-b358-b307f5b0f11f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6fccebea-b409-4fe0-b80c-1016cf4fdea8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b1fa69e298e5fb1d81175c275d4e7c914226df9ad6fa6eda7fb82d015f01d7bb", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f64c2620-a05d-4539-8f7c-5fb7ae01c3c2", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "7e95e51590c3b14e16b3c9c896f61dba9b92a3b5ae21df00de05155932a0c7f5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c039a16d-9598-4870-ad27-f726da3c85dd", "node_type": "1", "metadata": {}, "hash": "78251cfd5b51be30a3e650acfea274c93ff6b9873ed11e6aebdadac6a9126842", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2874, "end_char_idx": 5601, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py 
>>> import pathlib
>>> dataset_root_path = "UCF101_subset"
>>> dataset_root_path = pathlib.Path(dataset_root_path)
```

```py 
>>> video_count_train = len(list(dataset_root_path.glob("train/*/*.avi")))
>>> video_count_val = len(list(dataset_root_path.glob("val/*/*.avi")))
>>> video_count_test = len(list(dataset_root_path.glob("test/*/*.avi")))
>>> video_total = video_count_train + video_count_val + video_count_test
>>> print(f"Total videos: {video_total}")
```

```py 
>>> all_video_file_paths = (
...     list(dataset_root_path.glob("train/*/*.avi"))
...     + list(dataset_root_path.glob("val/*/*.avi"))
...     + list(dataset_root_path.glob("test/*/*.avi"))
...  )
>>> all_video_file_paths[:5]
```

The (`sorted`) video paths appear like so:

```bash
...
'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g07_c04.avi',
'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g07_c06.avi',
'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g08_c01.avi',
'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g09_c02.avi',
'UCF101_subset/train/ApplyEyeMakeup/v_ApplyEyeMakeup_g09_c06.avi'
...
```

You will notice that there are video clips belonging to the same group / scene where group is denoted by `g` in the video file paths. `v_ApplyEyeMakeup_g07_c04.avi` and `v_ApplyEyeMakeup_g07_c06.avi`, for example.

For the validation and evaluation splits, you wouldn't want to have video clips from the same group / scene to prevent [data leakage](https://www.kaggle.com/code/alexisbcook/data-leakage). The subset that you are using in this tutorial takes this information into account.

Next up, you will derive the set of labels present in the dataset. Also, create two dictionaries that'll be helpful when initializing the model:

* `label2id`: maps the class names to integers.
* `id2label`: maps the integers to class names. 

```py 
>>> class_labels = sorted({str(path).split("/")[2] for path in all_video_file_paths})
>>> label2id = {label: i for i, label in enumerate(class_labels)}
>>> id2label = {i: label for label, i in label2id.items()}

>>> print(f"Unique classes: {list(label2id.keys())}.")

# Unique classes: ['ApplyEyeMakeup', 'ApplyLipstick', 'Archery', 'BabyCrawling', 'BalanceBeam', 'BandMarching', 'BaseballPitch', 'Basketball', 'BasketballDunk', 'BenchPress'].
```

There are 10 unique classes. For each class, there are 30 videos in the training set.

## Load a model to fine-tune

Instantiate a video classification model from a pretrained checkpoint and its associated image processor. The model's encoder comes with pre-trained parameters, and the classification head is randomly initialized. The image processor will come in handy when writing the preprocessing pipeline for our dataset.


key: _node_content
string_value:
{"id_": "c039a16d-9598-4870-ad27-f726da3c85dd", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6fccebea-b409-4fe0-b80c-1016cf4fdea8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b1fa69e298e5fb1d81175c275d4e7c914226df9ad6fa6eda7fb82d015f01d7bb", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cc098e1f-24d6-4fcf-b358-b307f5b0f11f", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "c38b02c5d9c9c750192d494b03bd9b94d30d68ced4307add919ae488ec59bf16", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f0773acc-2e0f-4599-a310-1679b2d2af88", "node_type": "1", "metadata": {}, "hash": "940cdf82eeff2e523ce4798598006d2da23533cc6709c91b06d14dc4e4e8c253", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4120, "end_char_idx": 7403, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
`v_ApplyEyeMakeup_g07_c04.avi` and `v_ApplyEyeMakeup_g07_c06.avi`, for example.

For the validation and evaluation splits, you wouldn't want to have video clips from the same group / scene to prevent [data leakage](https://www.kaggle.com/code/alexisbcook/data-leakage). The subset that you are using in this tutorial takes this information into account.

Next up, you will derive the set of labels present in the dataset. Also, create two dictionaries that'll be helpful when initializing the model:

* `label2id`: maps the class names to integers.
* `id2label`: maps the integers to class names. 

```py 
>>> class_labels = sorted({str(path).split("/")[2] for path in all_video_file_paths})
>>> label2id = {label: i for i, label in enumerate(class_labels)}
>>> id2label = {i: label for label, i in label2id.items()}

>>> print(f"Unique classes: {list(label2id.keys())}.")

# Unique classes: ['ApplyEyeMakeup', 'ApplyLipstick', 'Archery', 'BabyCrawling', 'BalanceBeam', 'BandMarching', 'BaseballPitch', 'Basketball', 'BasketballDunk', 'BenchPress'].
```

There are 10 unique classes. For each class, there are 30 videos in the training set.

## Load a model to fine-tune

Instantiate a video classification model from a pretrained checkpoint and its associated image processor. The model's encoder comes with pre-trained parameters, and the classification head is randomly initialized. The image processor will come in handy when writing the preprocessing pipeline for our dataset.

```py 
>>> from transformers import VideoMAEImageProcessor, VideoMAEForVideoClassification

>>> model_ckpt = "MCG-NJU/videomae-base"
>>> image_processor = VideoMAEImageProcessor.from_pretrained(model_ckpt)
>>> model = VideoMAEForVideoClassification.from_pretrained(
...     model_ckpt,
...     label2id=label2id,
...     id2label=id2label,
...     ignore_mismatched_sizes=True,  # provide this in case you're planning to fine-tune an already fine-tuned checkpoint
... )
```

While the model is loading, you might notice the following warning:

```bash
Some weights of the model checkpoint at MCG-NJU/videomae-base were not used when initializing VideoMAEForVideoClassification: [..., 'decoder.decoder_layers.1.attention.output.dense.bias', 'decoder.decoder_layers.2.attention.attention.key.weight']
- This IS expected if you are initializing VideoMAEForVideoClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing VideoMAEForVideoClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of VideoMAEForVideoClassification were not initialized from the model checkpoint at MCG-NJU/videomae-base and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

The warning is telling us we are throwing away some weights (e.g. the weights and bias of the `classifier` layer) and randomly initializing some others (the weights and bias of a new `classifier` layer).


key: _node_content
string_value:
{"id_": "f0773acc-2e0f-4599-a310-1679b2d2af88", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6fccebea-b409-4fe0-b80c-1016cf4fdea8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b1fa69e298e5fb1d81175c275d4e7c914226df9ad6fa6eda7fb82d015f01d7bb", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c039a16d-9598-4870-ad27-f726da3c85dd", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "dd40045639c7aa904bf9e40a87933a524a3914e391dcfb58b7b18a62ee9bc838", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "aa269ef5-5a9e-4c87-b937-35e5f37aa7c7", "node_type": "1", "metadata": {}, "hash": "5344fafcb0b703d191b1fa7053eaaed9e13102a67c39371145655edda82ca724", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5603, "end_char_idx": 9122, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py 
>>> from transformers import VideoMAEImageProcessor, VideoMAEForVideoClassification

>>> model_ckpt = "MCG-NJU/videomae-base"
>>> image_processor = VideoMAEImageProcessor.from_pretrained(model_ckpt)
>>> model = VideoMAEForVideoClassification.from_pretrained(
...     model_ckpt,
...     label2id=label2id,
...     id2label=id2label,
...     ignore_mismatched_sizes=True,  # provide this in case you're planning to fine-tune an already fine-tuned checkpoint
... )
```

While the model is loading, you might notice the following warning:

```bash
Some weights of the model checkpoint at MCG-NJU/videomae-base were not used when initializing VideoMAEForVideoClassification: [..., 'decoder.decoder_layers.1.attention.output.dense.bias', 'decoder.decoder_layers.2.attention.attention.key.weight']
- This IS expected if you are initializing VideoMAEForVideoClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing VideoMAEForVideoClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of VideoMAEForVideoClassification were not initialized from the model checkpoint at MCG-NJU/videomae-base and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

The warning is telling us we are throwing away some weights (e.g. the weights and bias of the `classifier` layer) and randomly initializing some others (the weights and bias of a new `classifier` layer). This is expected in this case, because we are adding a new head for which we don't have pretrained weights, so the library warns us we should fine-tune this model before using it for inference, which is exactly what we are going to do.

**Note** that [this checkpoint](https://huggingface.co/MCG-NJU/videomae-base-finetuned-kinetics) leads to better performance on this task as the checkpoint was obtained fine-tuning on a similar downstream task having considerable domain overlap. You can check out [this checkpoint](https://huggingface.co/sayakpaul/videomae-base-finetuned-kinetics-finetuned-ucf101-subset) which was obtained by fine-tuning `MCG-NJU/videomae-base-finetuned-kinetics`.  

## Prepare the datasets for training

For preprocessing the videos, you will leverage the [PyTorchVideo library](https://pytorchvideo.org/). Start by importing the dependencies we need. 

```py 
>>> import pytorchvideo.data

>>> from pytorchvideo.transforms import (
...     ApplyTransformToKey,
...     Normalize,
...     RandomShortSideScale,
...     RemoveKey,
...     ShortSideScale,
...     UniformTemporalSubsample,
... )

>>> from torchvision.transforms import (
...     Compose,
...     Lambda,
...     RandomCrop,
...     RandomHorizontalFlip,
...     Resize,
... )
```

For the training dataset transformations, use a combination of uniform temporal subsampling, pixel normalization, random cropping, and random horizontal flipping. For the validation and evaluation dataset transformations, keep the same transformation chain except for random cropping and horizontal flipping. To learn more about the details of these transformations check out the [official documentation of PyTorchVideo](https://pytorchvideo.org).


key: _node_content
string_value:
{"id_": "aa269ef5-5a9e-4c87-b937-35e5f37aa7c7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6fccebea-b409-4fe0-b80c-1016cf4fdea8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b1fa69e298e5fb1d81175c275d4e7c914226df9ad6fa6eda7fb82d015f01d7bb", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f0773acc-2e0f-4599-a310-1679b2d2af88", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "7145946bb44c7f93114f91b2e44cee94054771bdb92e433354453e29ce32567e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4a89051e-47c3-40b2-9e12-171640e255da", "node_type": "1", "metadata": {}, "hash": "3f9ef1f7d715aba10251a8edbfbaa7931a1ef7028a46beeb7b97e7a8c12ae087", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7404, "end_char_idx": 9975, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This is expected in this case, because we are adding a new head for which we don't have pretrained weights, so the library warns us we should fine-tune this model before using it for inference, which is exactly what we are going to do.

**Note** that [this checkpoint](https://huggingface.co/MCG-NJU/videomae-base-finetuned-kinetics) leads to better performance on this task as the checkpoint was obtained fine-tuning on a similar downstream task having considerable domain overlap. You can check out [this checkpoint](https://huggingface.co/sayakpaul/videomae-base-finetuned-kinetics-finetuned-ucf101-subset) which was obtained by fine-tuning `MCG-NJU/videomae-base-finetuned-kinetics`.  

## Prepare the datasets for training

For preprocessing the videos, you will leverage the [PyTorchVideo library](https://pytorchvideo.org/). Start by importing the dependencies we need. 

```py 
>>> import pytorchvideo.data

>>> from pytorchvideo.transforms import (
...     ApplyTransformToKey,
...     Normalize,
...     RandomShortSideScale,
...     RemoveKey,
...     ShortSideScale,
...     UniformTemporalSubsample,
... )

>>> from torchvision.transforms import (
...     Compose,
...     Lambda,
...     RandomCrop,
...     RandomHorizontalFlip,
...     Resize,
... )
```

For the training dataset transformations, use a combination of uniform temporal subsampling, pixel normalization, random cropping, and random horizontal flipping. For the validation and evaluation dataset transformations, keep the same transformation chain except for random cropping and horizontal flipping. To learn more about the details of these transformations check out the [official documentation of PyTorchVideo](https://pytorchvideo.org).  

Use the `image_processor` associated with the pre-trained model to obtain the following information:

* Image mean and standard deviation with which the video frame pixels will be normalized.
* Spatial resolution to which the video frames will be resized.

Start by defining some constants.

```py
>>> mean = image_processor.image_mean
>>> std = image_processor.image_std
>>> if "shortest_edge" in image_processor.size:
...     height = width = image_processor.size["shortest_edge"]
>>> else:
...     height = image_processor.size["height"]
...     width = image_processor.size["width"]
>>> resize_to = (height, width)

>>> num_frames_to_sample = model.config.num_frames
>>> sample_rate = 4
>>> fps = 30
>>> clip_duration = num_frames_to_sample * sample_rate / fps
```

Now, define the dataset-specific transformations and the datasets respectively.


key: _node_content
string_value:
{"id_": "4a89051e-47c3-40b2-9e12-171640e255da", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6fccebea-b409-4fe0-b80c-1016cf4fdea8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b1fa69e298e5fb1d81175c275d4e7c914226df9ad6fa6eda7fb82d015f01d7bb", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "aa269ef5-5a9e-4c87-b937-35e5f37aa7c7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "cc39281738232820502466dcfef5d4334089b81b15cb65e55eabf0314e66239b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4720562a-ff1e-4592-bf3c-1cd24c176fef", "node_type": "1", "metadata": {}, "hash": "1f87221d24048b78cc5c3ea9943f339a06d74fd0edaeb65faff8a10f005e3a9d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8095, "end_char_idx": 12062, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Prepare the datasets for training

For preprocessing the videos, you will leverage the [PyTorchVideo library](https://pytorchvideo.org/). Start by importing the dependencies we need. 

```py 
>>> import pytorchvideo.data

>>> from pytorchvideo.transforms import (
...     ApplyTransformToKey,
...     Normalize,
...     RandomShortSideScale,
...     RemoveKey,
...     ShortSideScale,
...     UniformTemporalSubsample,
... )

>>> from torchvision.transforms import (
...     Compose,
...     Lambda,
...     RandomCrop,
...     RandomHorizontalFlip,
...     Resize,
... )
```

For the training dataset transformations, use a combination of uniform temporal subsampling, pixel normalization, random cropping, and random horizontal flipping. For the validation and evaluation dataset transformations, keep the same transformation chain except for random cropping and horizontal flipping. To learn more about the details of these transformations check out the [official documentation of PyTorchVideo](https://pytorchvideo.org).  

Use the `image_processor` associated with the pre-trained model to obtain the following information:

* Image mean and standard deviation with which the video frame pixels will be normalized.
* Spatial resolution to which the video frames will be resized.

Start by defining some constants.

```py
>>> mean = image_processor.image_mean
>>> std = image_processor.image_std
>>> if "shortest_edge" in image_processor.size:
...     height = width = image_processor.size["shortest_edge"]
>>> else:
...     height = image_processor.size["height"]
...     width = image_processor.size["width"]
>>> resize_to = (height, width)

>>> num_frames_to_sample = model.config.num_frames
>>> sample_rate = 4
>>> fps = 30
>>> clip_duration = num_frames_to_sample * sample_rate / fps
```

Now, define the dataset-specific transformations and the datasets respectively. Starting with the training set: 

```py 
>>> train_transform = Compose(
...     [
...         ApplyTransformToKey(
...             key="video",
...             transform=Compose(
...                 [
...                     UniformTemporalSubsample(num_frames_to_sample),
...                     Lambda(lambda x: x / 255.0),
...                     Normalize(mean, std),
...                     RandomShortSideScale(min_size=256, max_size=320),
...                     RandomCrop(resize_to),
...                     RandomHorizontalFlip(p=0.5),
...                 ]
...             ),
...         ),
...     ]
... )

>>> train_dataset = pytorchvideo.data.Ucf101(
...     data_path=os.path.join(dataset_root_path, "train"),
...     clip_sampler=pytorchvideo.data.make_clip_sampler("random", clip_duration),
...     decode_audio=False,
...     transform=train_transform,
... )
```

The same sequence of workflow can be applied to the validation and evaluation sets: 

```py 
>>> val_transform = Compose(
...     [
...         ApplyTransformToKey(
...             key="video",
...             transform=Compose(
...                 [
...                     UniformTemporalSubsample(num_frames_to_sample),
...                     Lambda(lambda x: x / 255.0),
...                     Normalize(mean, std),
...                     Resize(resize_to),
...                 ]
...             ),
...         ),
...     ]
... )

>>> val_dataset = pytorchvideo.data.Ucf101(
...     data_path=os.path.join(dataset_root_path, "val"),
...     clip_sampler=pytorchvideo.data.make_clip_sampler("uniform", clip_duration),
...     decode_audio=False,
...     transform=val_transform,
... )

>>> test_dataset = pytorchvideo.data.Ucf101(
...     data_path=os.path.join(dataset_root_path, "test"),
...     clip_sampler=pytorchvideo.data.make_clip_sampler("uniform", clip_duration),
...     decode_audio=False,
...     transform=val_transform,
... )
```

**Note**: The above dataset pipelines are taken from the [official PyTorchVideo example](https://pytorchvideo.org/docs/tutorial_classification#dataset).


key: _node_content
string_value:
{"id_": "4720562a-ff1e-4592-bf3c-1cd24c176fef", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6fccebea-b409-4fe0-b80c-1016cf4fdea8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b1fa69e298e5fb1d81175c275d4e7c914226df9ad6fa6eda7fb82d015f01d7bb", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4a89051e-47c3-40b2-9e12-171640e255da", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "8244cbb7758d78d4e0cc7bd15173bf32074bc215553c9668dfd37a003447dd44", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "43e850fc-4157-4221-8d21-a188847aa2a5", "node_type": "1", "metadata": {}, "hash": "252a9171a10cb583b6e3fae6ac2f660fb6b8ea53fee1097430f10738bcc90211", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12063, "end_char_idx": 15210, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We're using the [`pytorchvideo.data.Ucf101()`](https://pytorchvideo.readthedocs.io/en/latest/api/data/data.html#pytorchvideo.data.Ucf101) function because it's tailored for the UCF-101 dataset. Under the hood, it returns a [`pytorchvideo.data.labeled_video_dataset.LabeledVideoDataset`](https://pytorchvideo.readthedocs.io/en/latest/api/data/data.html#pytorchvideo.data.LabeledVideoDataset) object. `LabeledVideoDataset` class is the base class for all things video in the PyTorchVideo dataset. So, if you want to use a custom dataset not supported off-the-shelf by PyTorchVideo, you can extend the `LabeledVideoDataset` class accordingly. Refer to the `data` API [documentation to](https://pytorchvideo.readthedocs.io/en/latest/api/data/data.html) learn more. Also, if your dataset follows a similar structure (as shown above), then using the `pytorchvideo.data.Ucf101()` should work just fine. 

You can access the `num_videos` argument to know the number of videos in the dataset.

```py
>>> print(train_dataset.num_videos, val_dataset.num_videos, test_dataset.num_videos)
# (300, 30, 75)
```

## Visualize the preprocessed video for better debugging 

```py 
>>> import imageio
>>> import numpy as np
>>> from IPython.display import Image

>>> def unnormalize_img(img):
...     """Un-normalizes the image pixels."""
...     img = (img * std) + mean
...     img = (img * 255).astype("uint8")
...     return img.clip(0, 255)

>>> def create_gif(video_tensor, filename="sample.gif"):
...     """Prepares a GIF from a video tensor.
...     
...     The video tensor is expected to have the following shape:
...     (num_frames, num_channels, height, width).
...     """
...     frames = []
...     for video_frame in video_tensor:
...         frame_unnormalized = unnormalize_img(video_frame.permute(1, 2, 0).numpy())
...         frames.append(frame_unnormalized)
...     kargs = {"duration": 0.25}
...     imageio.mimsave(filename, frames, "GIF", **kargs)
...     return filename

>>> def display_gif(video_tensor, gif_name="sample.gif"):
...     """Prepares and displays a GIF from a video tensor."""
...     video_tensor = video_tensor.permute(1, 0, 2, 3)
...     gif_filename = create_gif(video_tensor, gif_name)
...     return Image(filename=gif_filename)

>>> sample_video = next(iter(train_dataset))
>>> video_tensor = sample_video["video"]
>>> display_gif(video_tensor)
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/sample_gif.gif" alt="Person playing basketball"/>
</div>

## Train the model 

Leverage [`Trainer`](https://huggingface.co/docs/transformers/main_classes/trainer) from  🤗 Transformers for training the model. To instantiate a `Trainer`, you need to define the training configuration and an evaluation metric. The most important is the [`TrainingArguments`](https://huggingface.co/transformers/main_classes/trainer.html#transformers.TrainingArguments), which is a class that contains all the attributes to configure the training. It requires an output folder name, which will be used to save the checkpoints of the model.


key: _node_content
string_value:
{"id_": "43e850fc-4157-4221-8d21-a188847aa2a5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6fccebea-b409-4fe0-b80c-1016cf4fdea8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b1fa69e298e5fb1d81175c275d4e7c914226df9ad6fa6eda7fb82d015f01d7bb", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4720562a-ff1e-4592-bf3c-1cd24c176fef", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "921ba9c41b07c817cd0ea5d96343a9ca3355200a8f459e1d8bdf51baa2d864d4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e2643696-0847-4564-8254-f80fe635bd14", "node_type": "1", "metadata": {}, "hash": "261cbc62abd204d47716f98d7d8cc4aa5c70275ca87ab7b1d02b41ed5749f22b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13595, "end_char_idx": 15793, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
...     
...     The video tensor is expected to have the following shape:
...     (num_frames, num_channels, height, width).
...     """
...     frames = []
...     for video_frame in video_tensor:
...         frame_unnormalized = unnormalize_img(video_frame.permute(1, 2, 0).numpy())
...         frames.append(frame_unnormalized)
...     kargs = {"duration": 0.25}
...     imageio.mimsave(filename, frames, "GIF", **kargs)
...     return filename

>>> def display_gif(video_tensor, gif_name="sample.gif"):
...     """Prepares and displays a GIF from a video tensor."""
...     video_tensor = video_tensor.permute(1, 0, 2, 3)
...     gif_filename = create_gif(video_tensor, gif_name)
...     return Image(filename=gif_filename)

>>> sample_video = next(iter(train_dataset))
>>> video_tensor = sample_video["video"]
>>> display_gif(video_tensor)
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/sample_gif.gif" alt="Person playing basketball"/>
</div>

## Train the model 

Leverage [`Trainer`](https://huggingface.co/docs/transformers/main_classes/trainer) from  🤗 Transformers for training the model. To instantiate a `Trainer`, you need to define the training configuration and an evaluation metric. The most important is the [`TrainingArguments`](https://huggingface.co/transformers/main_classes/trainer.html#transformers.TrainingArguments), which is a class that contains all the attributes to configure the training. It requires an output folder name, which will be used to save the checkpoints of the model. It also helps sync all the information in the model repository on 🤗 Hub.

Most of the training arguments are self-explanatory, but one that is quite important here is `remove_unused_columns=False`. This one will drop any features not used by the model's call function. By default it's `True` because usually it's ideal to drop unused feature columns, making it easier to unpack inputs into the model's call function. But, in this case, you need the unused features ('video' in particular) in order to create `pixel_values` (which is a mandatory key our model expects in its inputs).


key: _node_content
string_value:
{"id_": "e2643696-0847-4564-8254-f80fe635bd14", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6fccebea-b409-4fe0-b80c-1016cf4fdea8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b1fa69e298e5fb1d81175c275d4e7c914226df9ad6fa6eda7fb82d015f01d7bb", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "43e850fc-4157-4221-8d21-a188847aa2a5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4d2e7c7a72cbd089e67d7d7d2f1dad82afa59f85640e073611bd150f99f3e7d7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fc0d08ee-7560-4eb2-9f85-b2ed7c7d4bc5", "node_type": "1", "metadata": {}, "hash": "66a10dfc39813344cb7a40227ed7c5059f2dd0fe19763a9c886a0709dc94bc6c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14166, "end_char_idx": 17483, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
...     video_tensor = video_tensor.permute(1, 0, 2, 3)
...     gif_filename = create_gif(video_tensor, gif_name)
...     return Image(filename=gif_filename)

>>> sample_video = next(iter(train_dataset))
>>> video_tensor = sample_video["video"]
>>> display_gif(video_tensor)
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/sample_gif.gif" alt="Person playing basketball"/>
</div>

## Train the model 

Leverage [`Trainer`](https://huggingface.co/docs/transformers/main_classes/trainer) from  🤗 Transformers for training the model. To instantiate a `Trainer`, you need to define the training configuration and an evaluation metric. The most important is the [`TrainingArguments`](https://huggingface.co/transformers/main_classes/trainer.html#transformers.TrainingArguments), which is a class that contains all the attributes to configure the training. It requires an output folder name, which will be used to save the checkpoints of the model. It also helps sync all the information in the model repository on 🤗 Hub.

Most of the training arguments are self-explanatory, but one that is quite important here is `remove_unused_columns=False`. This one will drop any features not used by the model's call function. By default it's `True` because usually it's ideal to drop unused feature columns, making it easier to unpack inputs into the model's call function. But, in this case, you need the unused features ('video' in particular) in order to create `pixel_values` (which is a mandatory key our model expects in its inputs).


```py 
>>> from transformers import TrainingArguments, Trainer

>>> model_name = model_ckpt.split("/")[-1]
>>> new_model_name = f"{model_name}-finetuned-ucf101-subset"
>>> num_epochs = 4

>>> args = TrainingArguments(
...     new_model_name,
...     remove_unused_columns=False,
...     eval_strategy="epoch",
...     save_strategy="epoch",
...     learning_rate=5e-5,
...     per_device_train_batch_size=batch_size,
...     per_device_eval_batch_size=batch_size,
...     warmup_ratio=0.1,
...     logging_steps=10,
...     load_best_model_at_end=True,
...     metric_for_best_model="accuracy",
...     push_to_hub=True,
...     max_steps=(train_dataset.num_videos // batch_size) * num_epochs,
... )
```

The dataset returned by `pytorchvideo.data.Ucf101()` doesn't implement the `__len__` method. As such, we must define `max_steps` when instantiating `TrainingArguments`. 

Next, you need to define a function to compute the metrics from the predictions, which will use the `metric` you'll load now. The only preprocessing you have to do is to take the argmax of our predicted logits:

```py
import evaluate

metric = evaluate.load("accuracy")


def compute_metrics(eval_pred):
    predictions = np.argmax(eval_pred.predictions, axis=1)
    return metric.compute(predictions=predictions, references=eval_pred.label_ids)
```

**A note on evaluation**:

In the [VideoMAE paper](https://arxiv.org/abs/2203.12602), the authors use the following evaluation strategy. They evaluate the model on several clips from test videos and apply different crops to those clips and report the aggregate score. However, in the interest of simplicity and brevity, we don't consider that in this tutorial.


key: _node_content
string_value:
{"id_": "fc0d08ee-7560-4eb2-9f85-b2ed7c7d4bc5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6fccebea-b409-4fe0-b80c-1016cf4fdea8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b1fa69e298e5fb1d81175c275d4e7c914226df9ad6fa6eda7fb82d015f01d7bb", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e2643696-0847-4564-8254-f80fe635bd14", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "7ce88e13a66733a7fe3f8d6a86e744189493b3e4df4c731b62aafe470aab936b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1853ddab-dd45-423d-8cf6-41d63f1dd213", "node_type": "1", "metadata": {}, "hash": "936d7c3bbcc39ed382a03c177501fef63f6af3c7d3bf51a0f5872c4fcdd0573b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16944, "end_char_idx": 19437, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
def compute_metrics(eval_pred):
    predictions = np.argmax(eval_pred.predictions, axis=1)
    return metric.compute(predictions=predictions, references=eval_pred.label_ids)
```

**A note on evaluation**:

In the [VideoMAE paper](https://arxiv.org/abs/2203.12602), the authors use the following evaluation strategy. They evaluate the model on several clips from test videos and apply different crops to those clips and report the aggregate score. However, in the interest of simplicity and brevity, we don't consider that in this tutorial.

Also, define a `collate_fn`, which will be used to batch examples together. Each batch consists of 2 keys, namely `pixel_values` and `labels`.

```py 
>>> def collate_fn(examples):
...     # permute to (num_frames, num_channels, height, width)
...     pixel_values = torch.stack(
...         [example["video"].permute(1, 0, 2, 3) for example in examples]
...     )
...     labels = torch.tensor([example["label"] for example in examples])
...     return {"pixel_values": pixel_values, "labels": labels}
```

Then you just pass all of this along with the datasets to `Trainer`:

```py 
>>> trainer = Trainer(
...     model,
...     args,
...     train_dataset=train_dataset,
...     eval_dataset=val_dataset,
...     tokenizer=image_processor,
...     compute_metrics=compute_metrics,
...     data_collator=collate_fn,
... )
```

You might wonder why you passed along the `image_processor` as a tokenizer when you preprocessed the data already. This is only to make sure the image processor configuration file (stored as JSON) will also be uploaded to the repo on the Hub.

Now fine-tune our model by calling the `train` method:

```py 
>>> train_results = trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```

## Inference

Great, now that you have fine-tuned a model, you can use it for inference!

Load a video for inference:

```py 
>>> sample_test_video = next(iter(test_dataset))
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/sample_gif_two.gif" alt="Teams playing basketball"/>
</div>

The simplest way to try out your fine-tuned model for inference is to use it in a [`pipeline`](https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#transformers.VideoClassificationPipeline).


key: _node_content
string_value:
{"id_": "1853ddab-dd45-423d-8cf6-41d63f1dd213", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6fccebea-b409-4fe0-b80c-1016cf4fdea8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b1fa69e298e5fb1d81175c275d4e7c914226df9ad6fa6eda7fb82d015f01d7bb", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fc0d08ee-7560-4eb2-9f85-b2ed7c7d4bc5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/video_classification/", "title": "Video classification", "tokens": 5166, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "1f52dab858b0bcc67eaff33d380c62b0bae73bacdcf1e9cc04d5194a9b83f061", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18429, "end_char_idx": 21219, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This is only to make sure the image processor configuration file (stored as JSON) will also be uploaded to the repo on the Hub.

Now fine-tune our model by calling the `train` method:

```py 
>>> train_results = trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```

## Inference

Great, now that you have fine-tuned a model, you can use it for inference!

Load a video for inference:

```py 
>>> sample_test_video = next(iter(test_dataset))
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/sample_gif_two.gif" alt="Teams playing basketball"/>
</div>

The simplest way to try out your fine-tuned model for inference is to use it in a [`pipeline`](https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#transformers.VideoClassificationPipeline). Instantiate a `pipeline` for video classification with your model, and pass your video to it:

```py
>>> from transformers import pipeline

>>> video_cls = pipeline(model="my_awesome_video_cls_model")
>>> video_cls("https://huggingface.co/datasets/sayakpaul/ucf101-subset/resolve/main/v_BasketballDunk_g14_c06.avi")
[{'score': 0.9272987842559814, 'label': 'BasketballDunk'},
 {'score': 0.017777055501937866, 'label': 'BabyCrawling'},
 {'score': 0.01663011871278286, 'label': 'BalanceBeam'},
 {'score': 0.009560945443809032, 'label': 'BandMarching'},
 {'score': 0.0068979403004050255, 'label': 'BaseballPitch'}]
```

You can also manually replicate the results of the `pipeline` if you'd like.


```py
>>> def run_inference(model, video):
...     # (num_frames, num_channels, height, width)
...     perumuted_sample_test_video = video.permute(1, 0, 2, 3)
...     inputs = {
...         "pixel_values": perumuted_sample_test_video.unsqueeze(0),
...         "labels": torch.tensor(
...             [sample_test_video["label"]]
...         ),  # this can be skipped if you don't have labels available.
...     }

...     device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
...     inputs = {k: v.to(device) for k, v in inputs.items()}
...     model = model.to(device)

...     # forward pass
...     with torch.no_grad():
...         outputs = model(**inputs)
...         logits = outputs.logits

...     return logits
```

Now, pass your input to the model and return the `logits`:

```py
>>> logits = run_inference(trained_model, sample_test_video["video"])
```

Decoding the `logits`, we get: 

```py 
>>> predicted_class_idx = logits.argmax(-1).item()
>>> print("Predicted class:", model.config.id2label[predicted_class_idx])
# Predicted class: BasketballDunk
```


key: _node_content
string_value:
{"id_": "7071e8f0-a9ae-4386-8263-fffa2ade6011", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "72f0f011-f818-4662-b64d-63353c798a87", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "52ab8b048821cdf750626913fb12879fb3e915617eccac4d7f8b6bc287107156", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "35c9a586-5416-48ab-8654-878d178a78c7", "node_type": "1", "metadata": {}, "hash": "8e2262c58b77ad0d91f432ac8815c9faac51f95c8265e1f89a64bb2e923e9168", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2936, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Text to speech

[[open-in-colab]]

Text-to-speech (TTS) is the task of creating natural-sounding speech from text, where the speech can be generated in multiple 
languages and for multiple speakers. Several text-to-speech models are currently available in 🤗 Transformers, such as 
[Bark](../model_doc/bark), [MMS](../model_doc/mms), [VITS](../model_doc/vits) and [SpeechT5](../model_doc/speecht5). 

You can easily generate audio using the `"text-to-audio"` pipeline (or its alias - `"text-to-speech"`). Some models, like Bark, 
can also be conditioned to generate non-verbal communications such as laughing, sighing and crying, or even add music.
Here's an example of how you would use the `"text-to-speech"` pipeline with Bark: 

```py
>>> from transformers import pipeline

>>> pipe = pipeline("text-to-speech", model="suno/bark-small")
>>> text = "[clears throat] This is a test ... and I just took a long pause."
>>> output = pipe(text)
```

Here's a code snippet you can use to listen to the resulting audio in a notebook: 

```python
>>> from IPython.display import Audio
>>> Audio(output["audio"], rate=output["sampling_rate"])
```

For more examples on what Bark and other pretrained TTS models can do, refer to our 
[Audio course](https://huggingface.co/learn/audio-course/chapter6/pre-trained_models). 

If you are looking to fine-tune a TTS model, the only text-to-speech models currently available in 🤗 Transformers 
are [SpeechT5](model_doc/speecht5) and [FastSpeech2Conformer](model_doc/fastspeech2_conformer), though more will be added in the future. SpeechT5 is pre-trained on a combination of speech-to-text and text-to-speech data, allowing it to learn a unified space of hidden representations shared by both text and speech. This means that the same pre-trained model can be fine-tuned for different tasks. Furthermore, SpeechT5 supports multiple speakers through x-vector speaker embeddings. 

The remainder of this guide illustrates how to:

1. Fine-tune [SpeechT5](../model_doc/speecht5) that was originally trained on English speech on the Dutch (`nl`) language subset of the [VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli) dataset.
2. Use your refined model for inference in one of two ways: using a pipeline or directly.

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install datasets soundfile speechbrain accelerate
```

Install 🤗Transformers from source as not all the SpeechT5 features have been merged into an official release yet:

```bash
pip install git+https://github.com/huggingface/transformers.git
```

<Tip>

To follow this guide you will need a GPU. If you're working in a notebook, run the following line to check if a GPU is available: 

```bash
!nvidia-smi
```

or alternatively for AMD GPUs:

```bash
!rocm-smi
```

</Tip>

We encourage you to log in to your Hugging Face account to upload and share your model with the community.


key: _node_content
string_value:
{"id_": "35c9a586-5416-48ab-8654-878d178a78c7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "72f0f011-f818-4662-b64d-63353c798a87", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "52ab8b048821cdf750626913fb12879fb3e915617eccac4d7f8b6bc287107156", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7071e8f0-a9ae-4386-8263-fffa2ade6011", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "bd9f72d9886e05be501eb8e6b94dcb1f8007e2aa9b4374e250cfecd417eb5cb2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "73ef722d-0f66-4eb9-9dbe-b42665eec0a5", "node_type": "1", "metadata": {}, "hash": "45e60029c3e31b5a8cbc1c5d8dcc76ca82d1ab1a77f6ff782100bd7218221430", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1317, "end_char_idx": 4586, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
If you are looking to fine-tune a TTS model, the only text-to-speech models currently available in 🤗 Transformers 
are [SpeechT5](model_doc/speecht5) and [FastSpeech2Conformer](model_doc/fastspeech2_conformer), though more will be added in the future. SpeechT5 is pre-trained on a combination of speech-to-text and text-to-speech data, allowing it to learn a unified space of hidden representations shared by both text and speech. This means that the same pre-trained model can be fine-tuned for different tasks. Furthermore, SpeechT5 supports multiple speakers through x-vector speaker embeddings. 

The remainder of this guide illustrates how to:

1. Fine-tune [SpeechT5](../model_doc/speecht5) that was originally trained on English speech on the Dutch (`nl`) language subset of the [VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli) dataset.
2. Use your refined model for inference in one of two ways: using a pipeline or directly.

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install datasets soundfile speechbrain accelerate
```

Install 🤗Transformers from source as not all the SpeechT5 features have been merged into an official release yet:

```bash
pip install git+https://github.com/huggingface/transformers.git
```

<Tip>

To follow this guide you will need a GPU. If you're working in a notebook, run the following line to check if a GPU is available: 

```bash
!nvidia-smi
```

or alternatively for AMD GPUs:

```bash
!rocm-smi
```

</Tip>

We encourage you to log in to your Hugging Face account to upload and share your model with the community. When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load the dataset

[VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli) is a large-scale multilingual speech corpus consisting of 
data sourced from 2009-2020 European Parliament event recordings. It contains labelled audio-transcription data for 15 
European languages. In this guide, we are using the Dutch language subset, feel free to pick another subset. 

Note that VoxPopuli or any other automated speech recognition (ASR) dataset may not be the most suitable 
option for training TTS models. The features that make it beneficial for ASR, such as excessive background noise, are 
typically undesirable in TTS. However, finding top-quality, multilingual, and multi-speaker TTS datasets can be quite 
challenging.

Let's load the data:

```py
>>> from datasets import load_dataset, Audio

>>> dataset = load_dataset("facebook/voxpopuli", "nl", split="train")
>>> len(dataset)
20968
```

20968 examples should be sufficient for fine-tuning. SpeechT5 expects audio data to have a sampling rate of 16 kHz, so 
make sure the examples in the dataset meet this requirement:

```py
dataset = dataset.cast_column("audio", Audio(sampling_rate=16000))
```

## Preprocess the data

Let's begin by defining the model checkpoint to use and loading the appropriate processor: 

```py
>>> from transformers import SpeechT5Processor

>>> checkpoint = "microsoft/speecht5_tts"
>>> processor = SpeechT5Processor.from_pretrained(checkpoint)
```

### Text cleanup for SpeechT5 tokenization 

Start by cleaning up the text data.


key: _node_content
string_value:
{"id_": "73ef722d-0f66-4eb9-9dbe-b42665eec0a5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "72f0f011-f818-4662-b64d-63353c798a87", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "52ab8b048821cdf750626913fb12879fb3e915617eccac4d7f8b6bc287107156", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "35c9a586-5416-48ab-8654-878d178a78c7", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "2c3e20415d3c7d35d9b03320d77e56b8708848cb351afc03b8a5ec54840afa8f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "899b8938-d343-4af7-b271-b5535f22b610", "node_type": "1", "metadata": {}, "hash": "6ebb7bbfb08bd865b19de6fe4749c8d678be5cacd0af1a65b8c1b696cbb225dd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2937, "end_char_idx": 6065, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load the dataset

[VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli) is a large-scale multilingual speech corpus consisting of 
data sourced from 2009-2020 European Parliament event recordings. It contains labelled audio-transcription data for 15 
European languages. In this guide, we are using the Dutch language subset, feel free to pick another subset. 

Note that VoxPopuli or any other automated speech recognition (ASR) dataset may not be the most suitable 
option for training TTS models. The features that make it beneficial for ASR, such as excessive background noise, are 
typically undesirable in TTS. However, finding top-quality, multilingual, and multi-speaker TTS datasets can be quite 
challenging.

Let's load the data:

```py
>>> from datasets import load_dataset, Audio

>>> dataset = load_dataset("facebook/voxpopuli", "nl", split="train")
>>> len(dataset)
20968
```

20968 examples should be sufficient for fine-tuning. SpeechT5 expects audio data to have a sampling rate of 16 kHz, so 
make sure the examples in the dataset meet this requirement:

```py
dataset = dataset.cast_column("audio", Audio(sampling_rate=16000))
```

## Preprocess the data

Let's begin by defining the model checkpoint to use and loading the appropriate processor: 

```py
>>> from transformers import SpeechT5Processor

>>> checkpoint = "microsoft/speecht5_tts"
>>> processor = SpeechT5Processor.from_pretrained(checkpoint)
```

### Text cleanup for SpeechT5 tokenization 

Start by cleaning up the text data. You'll need the tokenizer part of the processor to process the text:

```py
>>> tokenizer = processor.tokenizer
```

The dataset examples contain `raw_text` and `normalized_text` features. When deciding which feature to use as the text input, 
consider that the SpeechT5 tokenizer doesn't have any tokens for numbers. In `normalized_text` the numbers are written 
out as text. Thus, it is a better fit, and we recommend using    `normalized_text` as input text.

Because SpeechT5 was trained on the English language, it may not recognize certain characters in the Dutch dataset. If 
left as is, these characters will be converted to `<unk>` tokens. However, in Dutch, certain characters like `à` are 
used to stress syllables. In order to preserve the meaning of the text, we can replace this character with a regular `a`.

To identify unsupported tokens, extract all unique characters in the dataset using the `SpeechT5Tokenizer` which 
works with characters as tokens. To do this, write the `extract_all_chars` mapping function that concatenates 
the transcriptions from all examples into one string and converts it to a set of characters. 
Make sure to set `batched=True` and `batch_size=-1` in `dataset.map()` so that all transcriptions are available at once for 
the mapping function.

```py
>>> def extract_all_chars(batch):
...     all_text = " ".join(batch["normalized_text"])
...     vocab = list(set(all_text))
...     return {"vocab": [vocab], "all_text": [all_text]}


key: _node_content
string_value:
{"id_": "899b8938-d343-4af7-b271-b5535f22b610", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "72f0f011-f818-4662-b64d-63353c798a87", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "52ab8b048821cdf750626913fb12879fb3e915617eccac4d7f8b6bc287107156", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "73ef722d-0f66-4eb9-9dbe-b42665eec0a5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "72082638f223c3cec99d902f76f566f17bd15f86bfb24d7b49baa6c842a1f5f2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7cbe151d-b130-4b54-96fe-9171cdbffa7a", "node_type": "1", "metadata": {}, "hash": "df5c17246e6d4b73f5930a815a076e38794c9cb847de788868e2055f2c964460", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4587, "end_char_idx": 7394, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You'll need the tokenizer part of the processor to process the text:

```py
>>> tokenizer = processor.tokenizer
```

The dataset examples contain `raw_text` and `normalized_text` features. When deciding which feature to use as the text input, 
consider that the SpeechT5 tokenizer doesn't have any tokens for numbers. In `normalized_text` the numbers are written 
out as text. Thus, it is a better fit, and we recommend using    `normalized_text` as input text.

Because SpeechT5 was trained on the English language, it may not recognize certain characters in the Dutch dataset. If 
left as is, these characters will be converted to `<unk>` tokens. However, in Dutch, certain characters like `à` are 
used to stress syllables. In order to preserve the meaning of the text, we can replace this character with a regular `a`.

To identify unsupported tokens, extract all unique characters in the dataset using the `SpeechT5Tokenizer` which 
works with characters as tokens. To do this, write the `extract_all_chars` mapping function that concatenates 
the transcriptions from all examples into one string and converts it to a set of characters. 
Make sure to set `batched=True` and `batch_size=-1` in `dataset.map()` so that all transcriptions are available at once for 
the mapping function.

```py
>>> def extract_all_chars(batch):
...     all_text = " ".join(batch["normalized_text"])
...     vocab = list(set(all_text))
...     return {"vocab": [vocab], "all_text": [all_text]}


>>> vocabs = dataset.map(
...     extract_all_chars,
...     batched=True,
...     batch_size=-1,
...     keep_in_memory=True,
...     remove_columns=dataset.column_names,
... )

>>> dataset_vocab = set(vocabs["vocab"][0])
>>> tokenizer_vocab = {k for k, _ in tokenizer.get_vocab().items()}
```

Now you have two sets of characters: one with the vocabulary from the dataset and one with the vocabulary from the tokenizer. 
To identify any unsupported characters in the dataset, you can take the difference between these two sets. The resulting 
set will contain the characters that are in the dataset but not in the tokenizer.

```py
>>> dataset_vocab - tokenizer_vocab
{' ', 'à', 'ç', 'è', 'ë', 'í', 'ï', 'ö', 'ü'}
```

To handle the unsupported characters identified in the previous step, define a function that maps these characters to 
valid tokens. Note that spaces are already replaced by `▁` in the tokenizer and don't need to be handled separately.

```py
>>> replacements = [
...     ("à", "a"),
...     ("ç", "c"),
...     ("è", "e"),
...     ("ë", "e"),
...     ("í", "i"),
...     ("ï", "i"),
...     ("ö", "o"),
...     ("ü", "u"),
... ]


>>> def cleanup_text(inputs):
...     for src, dst in replacements:
...         inputs["normalized_text"] = inputs["normalized_text"].replace(src, dst)
...     return inputs


key: _node_content
string_value:
{"id_": "7cbe151d-b130-4b54-96fe-9171cdbffa7a", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "72f0f011-f818-4662-b64d-63353c798a87", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "52ab8b048821cdf750626913fb12879fb3e915617eccac4d7f8b6bc287107156", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "899b8938-d343-4af7-b271-b5535f22b610", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "9c2af8965534cd47d8dcb8daf2b754c0fe6d1b8b5e3cbf94d0af22edcb02d9dd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5f6e1007-bab0-415a-9029-16c1a2c5bf06", "node_type": "1", "metadata": {}, "hash": "b849a9bd0545b38b87205d48571d0ccd2ff607c4e57c73a19465e6a35ccf97d2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5878, "end_char_idx": 9018, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> def extract_all_chars(batch):
...     all_text = " ".join(batch["normalized_text"])
...     vocab = list(set(all_text))
...     return {"vocab": [vocab], "all_text": [all_text]}


>>> vocabs = dataset.map(
...     extract_all_chars,
...     batched=True,
...     batch_size=-1,
...     keep_in_memory=True,
...     remove_columns=dataset.column_names,
... )

>>> dataset_vocab = set(vocabs["vocab"][0])
>>> tokenizer_vocab = {k for k, _ in tokenizer.get_vocab().items()}
```

Now you have two sets of characters: one with the vocabulary from the dataset and one with the vocabulary from the tokenizer. 
To identify any unsupported characters in the dataset, you can take the difference between these two sets. The resulting 
set will contain the characters that are in the dataset but not in the tokenizer.

```py
>>> dataset_vocab - tokenizer_vocab
{' ', 'à', 'ç', 'è', 'ë', 'í', 'ï', 'ö', 'ü'}
```

To handle the unsupported characters identified in the previous step, define a function that maps these characters to 
valid tokens. Note that spaces are already replaced by `▁` in the tokenizer and don't need to be handled separately.

```py
>>> replacements = [
...     ("à", "a"),
...     ("ç", "c"),
...     ("è", "e"),
...     ("ë", "e"),
...     ("í", "i"),
...     ("ï", "i"),
...     ("ö", "o"),
...     ("ü", "u"),
... ]


>>> def cleanup_text(inputs):
...     for src, dst in replacements:
...         inputs["normalized_text"] = inputs["normalized_text"].replace(src, dst)
...     return inputs


>>> dataset = dataset.map(cleanup_text)
```

Now that you have dealt with special characters in the text, it's time to shift focus to the audio data.

### Speakers

The VoxPopuli dataset includes speech from multiple speakers, but how many speakers are represented in the dataset? To 
determine this, we can count the number of unique speakers and the number of examples each speaker contributes to the dataset. 
With a total of 20,968 examples in the dataset, this information will give us a better understanding of the distribution of 
speakers and examples in the data.

```py
>>> from collections import defaultdict

>>> speaker_counts = defaultdict(int)

>>> for speaker_id in dataset["speaker_id"]:
...     speaker_counts[speaker_id] += 1
```

By plotting a histogram you can get a sense of how much data there is for each speaker.

```py
>>> import matplotlib.pyplot as plt

>>> plt.figure()
>>> plt.hist(speaker_counts.values(), bins=20)
>>> plt.ylabel("Speakers")
>>> plt.xlabel("Examples")
>>> plt.show()
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/tts_speakers_histogram.png" alt="Speakers histogram"/>
</div>

The histogram reveals that approximately one-third of the speakers in the dataset have fewer than 100 examples, while 
around ten speakers have more than 500 examples. To improve training efficiency and balance the dataset, we can limit 
the data to speakers with between 100 and 400 examples. 

```py
>>> def select_speaker(speaker_id):
...     return 100 <= speaker_counts[speaker_id] <= 400


key: _node_content
string_value:
{"id_": "5f6e1007-bab0-415a-9029-16c1a2c5bf06", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "72f0f011-f818-4662-b64d-63353c798a87", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "52ab8b048821cdf750626913fb12879fb3e915617eccac4d7f8b6bc287107156", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7cbe151d-b130-4b54-96fe-9171cdbffa7a", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "f8cdc87318ad4f3a6e68517ba2343aba2a2057de2b8627063cb06ac48c5b6fb2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dd5d6850-ebd0-40aa-8edb-4d3f8c53e636", "node_type": "1", "metadata": {}, "hash": "38a1167684bc6a98f1a6372a0d6d33d4f097430ab06153c627b7c2a6a4d3ac68", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7221, "end_char_idx": 10739, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> def cleanup_text(inputs):
...     for src, dst in replacements:
...         inputs["normalized_text"] = inputs["normalized_text"].replace(src, dst)
...     return inputs


>>> dataset = dataset.map(cleanup_text)
```

Now that you have dealt with special characters in the text, it's time to shift focus to the audio data.

### Speakers

The VoxPopuli dataset includes speech from multiple speakers, but how many speakers are represented in the dataset? To 
determine this, we can count the number of unique speakers and the number of examples each speaker contributes to the dataset. 
With a total of 20,968 examples in the dataset, this information will give us a better understanding of the distribution of 
speakers and examples in the data.

```py
>>> from collections import defaultdict

>>> speaker_counts = defaultdict(int)

>>> for speaker_id in dataset["speaker_id"]:
...     speaker_counts[speaker_id] += 1
```

By plotting a histogram you can get a sense of how much data there is for each speaker.

```py
>>> import matplotlib.pyplot as plt

>>> plt.figure()
>>> plt.hist(speaker_counts.values(), bins=20)
>>> plt.ylabel("Speakers")
>>> plt.xlabel("Examples")
>>> plt.show()
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/tts_speakers_histogram.png" alt="Speakers histogram"/>
</div>

The histogram reveals that approximately one-third of the speakers in the dataset have fewer than 100 examples, while 
around ten speakers have more than 500 examples. To improve training efficiency and balance the dataset, we can limit 
the data to speakers with between 100 and 400 examples. 

```py
>>> def select_speaker(speaker_id):
...     return 100 <= speaker_counts[speaker_id] <= 400


>>> dataset = dataset.filter(select_speaker, input_columns=["speaker_id"])
```

Let's check how many speakers remain: 

```py
>>> len(set(dataset["speaker_id"]))
42
```

Let's see how many examples are left: 

```py
>>> len(dataset)
9973
```

You are left with just under 10,000 examples from approximately 40 unique speakers, which should be sufficient.

Note that some speakers with few examples may actually have more audio available if the examples are long. However, 
determining the total amount of audio for each speaker requires scanning through the entire dataset, which is a 
time-consuming process that involves loading and decoding each audio file. As such, we have chosen to skip this step here.

### Speaker embeddings

To enable the TTS model to differentiate between multiple speakers, you'll need to create a speaker embedding for each example. 
The speaker embedding is an additional input into the model that captures a particular speaker's voice characteristics.
To generate these speaker embeddings, use the pre-trained [spkrec-xvect-voxceleb](https://huggingface.co/speechbrain/spkrec-xvect-voxceleb) 
model from SpeechBrain. 

Create a function `create_speaker_embedding()` that takes an input audio waveform and outputs a 512-element vector 
containing the corresponding speaker embedding.

```py
>>> import os
>>> import torch
>>> from speechbrain.inference.classifiers import EncoderClassifier

>>> spk_model_name = "speechbrain/spkrec-xvect-voxceleb"

>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> speaker_model = EncoderClassifier.from_hparams(
...     source=spk_model_name,
...     run_opts={"device": device},
...     savedir=os.path.join("/tmp", spk_model_name),
... )


key: _node_content
string_value:
{"id_": "dd5d6850-ebd0-40aa-8edb-4d3f8c53e636", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "72f0f011-f818-4662-b64d-63353c798a87", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "52ab8b048821cdf750626913fb12879fb3e915617eccac4d7f8b6bc287107156", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5f6e1007-bab0-415a-9029-16c1a2c5bf06", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "9eea2e622ad0285e5195166595cdc1eb8a3cd6d189d8e595f9044cf60594b526", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5c840970-3881-4e72-ba89-7a94131a5dc6", "node_type": "1", "metadata": {}, "hash": "6d7add132ba9f351d722e730eb5f831fe0f00af5b91003ee0ddb261a63dc5dd2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9021, "end_char_idx": 14224, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> dataset = dataset.filter(select_speaker, input_columns=["speaker_id"])
```

Let's check how many speakers remain: 

```py
>>> len(set(dataset["speaker_id"]))
42
```

Let's see how many examples are left: 

```py
>>> len(dataset)
9973
```

You are left with just under 10,000 examples from approximately 40 unique speakers, which should be sufficient.

Note that some speakers with few examples may actually have more audio available if the examples are long. However, 
determining the total amount of audio for each speaker requires scanning through the entire dataset, which is a 
time-consuming process that involves loading and decoding each audio file. As such, we have chosen to skip this step here.

### Speaker embeddings

To enable the TTS model to differentiate between multiple speakers, you'll need to create a speaker embedding for each example. 
The speaker embedding is an additional input into the model that captures a particular speaker's voice characteristics.
To generate these speaker embeddings, use the pre-trained [spkrec-xvect-voxceleb](https://huggingface.co/speechbrain/spkrec-xvect-voxceleb) 
model from SpeechBrain. 

Create a function `create_speaker_embedding()` that takes an input audio waveform and outputs a 512-element vector 
containing the corresponding speaker embedding.

```py
>>> import os
>>> import torch
>>> from speechbrain.inference.classifiers import EncoderClassifier

>>> spk_model_name = "speechbrain/spkrec-xvect-voxceleb"

>>> device = "cuda" if torch.cuda.is_available() else "cpu"
>>> speaker_model = EncoderClassifier.from_hparams(
...     source=spk_model_name,
...     run_opts={"device": device},
...     savedir=os.path.join("/tmp", spk_model_name),
... )


>>> def create_speaker_embedding(waveform):
...     with torch.no_grad():
...         speaker_embeddings = speaker_model.encode_batch(torch.tensor(waveform))
...         speaker_embeddings = torch.nn.functional.normalize(speaker_embeddings, dim=2)
...         speaker_embeddings = speaker_embeddings.squeeze().cpu().numpy()
...     return speaker_embeddings
```

It's important to note that the `speechbrain/spkrec-xvect-voxceleb` model was trained on English speech from the VoxCeleb 
dataset, whereas the training examples in this guide are in Dutch. While we believe that this model will still generate 
reasonable speaker embeddings for our Dutch dataset, this assumption may not hold true in all cases.

For optimal results, we recommend training an X-vector model on the target speech first. This will ensure that the model 
is better able to capture the unique voice characteristics present in the Dutch language.

### Processing the dataset

Finally, let's process the data into the format the model expects. Create a `prepare_dataset` function that takes in a 
single example and uses the `SpeechT5Processor` object to tokenize the input text and load the target audio into a log-mel spectrogram. 
It should also add the speaker embeddings as an additional input.

```py
>>> def prepare_dataset(example):
...     audio = example["audio"]

...     example = processor(
...         text=example["normalized_text"],
...         audio_target=audio["array"],
...         sampling_rate=audio["sampling_rate"],
...         return_attention_mask=False,
...     )

...     # strip off the batch dimension
...     example["labels"] = example["labels"][0]

...     # use SpeechBrain to obtain x-vector
...     example["speaker_embeddings"] = create_speaker_embedding(audio["array"])

...     return example
```

Verify the processing is correct by looking at a single example:

```py
>>> processed_example = prepare_dataset(dataset[0])
>>> list(processed_example.keys())
['input_ids', 'labels', 'stop_labels', 'speaker_embeddings']
```

Speaker embeddings should be a 512-element vector:

```py
>>> processed_example["speaker_embeddings"].shape
(512,)
```

The labels should be a log-mel spectrogram with 80 mel bins.

```py
>>> import matplotlib.pyplot as plt

>>> plt.figure()
>>> plt.imshow(processed_example["labels"].T)
>>> plt.show()
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/tts_logmelspectrogram_1.png" alt="Log-mel spectrogram with 80 mel bins"/>
</div>

Side note: If you find this spectrogram confusing, it may be due to your familiarity with the convention of placing low frequencies 
at the bottom and high frequencies at the top of a plot. However, when plotting spectrograms as an image using the matplotlib library, 
the y-axis is flipped and the spectrograms appear upside down.

Now apply the processing function to the entire dataset. This will take between 5 and 10 minutes.

```py
>>> dataset = dataset.map(prepare_dataset, remove_columns=dataset.column_names)
```

You'll see a warning saying that some examples in the dataset are longer than the maximum input length the model can handle (600 tokens). 
Remove those examples from the dataset. Here we go even further and to allow for larger batch sizes we remove anything over 200 tokens.

```py
>>> def is_not_too_long(input_ids):
...     input_length = len(input_ids)
...     return input_length < 200


key: _node_content
string_value:
{"id_": "5c840970-3881-4e72-ba89-7a94131a5dc6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "72f0f011-f818-4662-b64d-63353c798a87", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "52ab8b048821cdf750626913fb12879fb3e915617eccac4d7f8b6bc287107156", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dd5d6850-ebd0-40aa-8edb-4d3f8c53e636", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "aec17df865cc1fe703d6d6edbc0bcc3a3a37173cc2fe2f624b204bea8b17bbf5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "598f0e00-f68b-480d-8b13-68f56c2433d5", "node_type": "1", "metadata": {}, "hash": "c714b49e06cad6574270544a8f89ccec3835287262f9c07bdc2670db40de3f8e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14227, "end_char_idx": 17470, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> dataset = dataset.filter(is_not_too_long, input_columns=["input_ids"])
>>> len(dataset)
8259
```

Next, create a basic train/test split: 

```py
>>> dataset = dataset.train_test_split(test_size=0.1)
```

### Data collator

In order to combine multiple examples into a batch, you need to define a custom data collator. This collator will pad shorter sequences with padding 
tokens, ensuring that all examples have the same length. For the spectrogram labels, the padded portions are replaced with the special value `-100`. This special value 
instructs the model to ignore that part of the spectrogram when calculating the spectrogram loss.

```py
>>> from dataclasses import dataclass
>>> from typing import Any, Dict, List, Union


>>> @dataclass
... class TTSDataCollatorWithPadding:
...     processor: Any

...     def __call__(self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]:
...         input_ids = [{"input_ids": feature["input_ids"]} for feature in features]
...         label_features = [{"input_values": feature["labels"]} for feature in features]
...         speaker_features = [feature["speaker_embeddings"] for feature in features]

...         # collate the inputs and targets into a batch
...         batch = processor.pad(input_ids=input_ids, labels=label_features, return_tensors="pt")

...         # replace padding with -100 to ignore loss correctly
...         batch["labels"] = batch["labels"].masked_fill(batch.decoder_attention_mask.unsqueeze(-1).ne(1), -100)

...         # not used during fine-tuning
...         del batch["decoder_attention_mask"]

...         # round down target lengths to multiple of reduction factor
...         if model.config.reduction_factor > 1:
...             target_lengths = torch.tensor([len(feature["input_values"]) for feature in label_features])
...             target_lengths = target_lengths.new(
...                 [length - length % model.config.reduction_factor for length in target_lengths]
...             )
...             max_length = max(target_lengths)
...             batch["labels"] = batch["labels"][:, :max_length]

...         # also add in the speaker embeddings
...         batch["speaker_embeddings"] = torch.tensor(speaker_features)

...         return batch
```

In SpeechT5, the input to the decoder part of the model is reduced by a factor 2. In other words, it throws away every 
other timestep from the target sequence. The decoder then predicts a sequence that is twice as long. Since the original 
target sequence length may be odd, the data collator makes sure to round the maximum length of the batch down to be a 
multiple of 2.

```py 
>>> data_collator = TTSDataCollatorWithPadding(processor=processor)
```

## Train the model

Load the pre-trained model from the same checkpoint as you used for loading the processor: 

```py
>>> from transformers import SpeechT5ForTextToSpeech

>>> model = SpeechT5ForTextToSpeech.from_pretrained(checkpoint)
```

The `use_cache=True` option is incompatible with gradient checkpointing. Disable it for training.

```py 
>>> model.config.use_cache = False
```

Define the training arguments. Here we are not computing any evaluation metrics during the training process.


key: _node_content
string_value:
{"id_": "598f0e00-f68b-480d-8b13-68f56c2433d5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "72f0f011-f818-4662-b64d-63353c798a87", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "52ab8b048821cdf750626913fb12879fb3e915617eccac4d7f8b6bc287107156", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5c840970-3881-4e72-ba89-7a94131a5dc6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "f786b945e23294e0d90d5ab945444c2aba899c13a635cadedd2ca63742512310", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6cd2d895-ce1d-45ab-9e52-d20a2b171af5", "node_type": "1", "metadata": {}, "hash": "da446ccbb7397f67d7d8f81ed1881539faf70f049a624183443dc9fd16dcc40c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16600, "end_char_idx": 19737, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In other words, it throws away every 
other timestep from the target sequence. The decoder then predicts a sequence that is twice as long. Since the original 
target sequence length may be odd, the data collator makes sure to round the maximum length of the batch down to be a 
multiple of 2.

```py 
>>> data_collator = TTSDataCollatorWithPadding(processor=processor)
```

## Train the model

Load the pre-trained model from the same checkpoint as you used for loading the processor: 

```py
>>> from transformers import SpeechT5ForTextToSpeech

>>> model = SpeechT5ForTextToSpeech.from_pretrained(checkpoint)
```

The `use_cache=True` option is incompatible with gradient checkpointing. Disable it for training.

```py 
>>> model.config.use_cache = False
```

Define the training arguments. Here we are not computing any evaluation metrics during the training process. Instead, we'll 
only look at the loss:

```python
>>> from transformers import Seq2SeqTrainingArguments

>>> training_args = Seq2SeqTrainingArguments(
...     output_dir="speecht5_finetuned_voxpopuli_nl",  # change to a repo name of your choice
...     per_device_train_batch_size=4,
...     gradient_accumulation_steps=8,
...     learning_rate=1e-5,
...     warmup_steps=500,
...     max_steps=4000,
...     gradient_checkpointing=True,
...     fp16=True,
...     eval_strategy="steps",
...     per_device_eval_batch_size=2,
...     save_steps=1000,
...     eval_steps=1000,
...     logging_steps=25,
...     report_to=["tensorboard"],
...     load_best_model_at_end=True,
...     greater_is_better=False,
...     label_names=["labels"],
...     push_to_hub=True,
... )
```

Instantiate the `Trainer` object  and pass the model, dataset, and data collator to it.

```py
>>> from transformers import Seq2SeqTrainer

>>> trainer = Seq2SeqTrainer(
...     args=training_args,
...     model=model,
...     train_dataset=dataset["train"],
...     eval_dataset=dataset["test"],
...     data_collator=data_collator,
...     tokenizer=processor,
... )
```

And with that, you're ready to start training! Training will take several hours. Depending on your GPU, 
it is possible that you will encounter a CUDA "out-of-memory" error when you start training. In this case, you can reduce 
the `per_device_train_batch_size` incrementally by factors of 2 and increase `gradient_accumulation_steps` by 2x to compensate.

```py
>>> trainer.train()
```

To be able to use your checkpoint with a pipeline, make sure to save the processor with the checkpoint: 

```py
>>> processor.save_pretrained("YOUR_ACCOUNT_NAME/speecht5_finetuned_voxpopuli_nl")
```

Push the final model to the 🤗 Hub:

```py
>>> trainer.push_to_hub()
```

## Inference

### Inference with a pipeline

Great, now that you've fine-tuned a model, you can use it for inference!
First, let's see how you can use it with a corresponding pipeline. Let's create a `"text-to-speech"` pipeline with your 
checkpoint: 

```py
>>> from transformers import pipeline

>>> pipe = pipeline("text-to-speech", model="YOUR_ACCOUNT_NAME/speecht5_finetuned_voxpopuli_nl")
```

Pick a piece of text in Dutch you'd like narrated, e.g.


key: _node_content
string_value:
{"id_": "6cd2d895-ce1d-45ab-9e52-d20a2b171af5", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "72f0f011-f818-4662-b64d-63353c798a87", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "52ab8b048821cdf750626913fb12879fb3e915617eccac4d7f8b6bc287107156", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "598f0e00-f68b-480d-8b13-68f56c2433d5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "87f5767dd46a809aba0bd32439044d4eb2614b2d34b382e8acfd44d2998fcede", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7e3e3492-4fa6-4076-94c6-c09137f2f0ec", "node_type": "1", "metadata": {}, "hash": "2da01f93b02a5634f12f7ec1a450faf61ee8d8c3bc9baa9f3cc8b1506004a58c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18336, "end_char_idx": 21232, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> from transformers import Seq2SeqTrainer

>>> trainer = Seq2SeqTrainer(
...     args=training_args,
...     model=model,
...     train_dataset=dataset["train"],
...     eval_dataset=dataset["test"],
...     data_collator=data_collator,
...     tokenizer=processor,
... )
```

And with that, you're ready to start training! Training will take several hours. Depending on your GPU, 
it is possible that you will encounter a CUDA "out-of-memory" error when you start training. In this case, you can reduce 
the `per_device_train_batch_size` incrementally by factors of 2 and increase `gradient_accumulation_steps` by 2x to compensate.

```py
>>> trainer.train()
```

To be able to use your checkpoint with a pipeline, make sure to save the processor with the checkpoint: 

```py
>>> processor.save_pretrained("YOUR_ACCOUNT_NAME/speecht5_finetuned_voxpopuli_nl")
```

Push the final model to the 🤗 Hub:

```py
>>> trainer.push_to_hub()
```

## Inference

### Inference with a pipeline

Great, now that you've fine-tuned a model, you can use it for inference!
First, let's see how you can use it with a corresponding pipeline. Let's create a `"text-to-speech"` pipeline with your 
checkpoint: 

```py
>>> from transformers import pipeline

>>> pipe = pipeline("text-to-speech", model="YOUR_ACCOUNT_NAME/speecht5_finetuned_voxpopuli_nl")
```

Pick a piece of text in Dutch you'd like narrated, e.g.:

```py
>>> text = "hallo allemaal, ik praat nederlands. groetjes aan iedereen!"
```

To use SpeechT5 with the pipeline, you'll need a speaker embedding. Let's get it from an example in the test dataset: 

```py
>>> example = dataset["test"][304]
>>> speaker_embeddings = torch.tensor(example["speaker_embeddings"]).unsqueeze(0)
```

Now you can pass the text and speaker embeddings to the pipeline, and it will take care of the rest: 

```py
>>> forward_params = {"speaker_embeddings": speaker_embeddings}
>>> output = pipe(text, forward_params=forward_params)
>>> output
{'audio': array([-6.82714235e-05, -4.26525949e-04,  1.06134125e-04, ...,
        -1.22392643e-03, -7.76011671e-04,  3.29112721e-04], dtype=float32),
 'sampling_rate': 16000}
```

You can then listen to the result:

```py
>>> from IPython.display import Audio
>>> Audio(output['audio'], rate=output['sampling_rate']) 
```

### Run inference manually

You can achieve the same inference results without using the pipeline, however, more steps will be required. 

Load the model from the 🤗 Hub: 

```py
>>> model = SpeechT5ForTextToSpeech.from_pretrained("YOUR_ACCOUNT/speecht5_finetuned_voxpopuli_nl")
```

Pick an example from the test dataset obtain a speaker embedding. 

```py 
>>> example = dataset["test"][304]
>>> speaker_embeddings = torch.tensor(example["speaker_embeddings"]).unsqueeze(0)
```

Define the input text and tokenize it.

```py 
>>> text = "hallo allemaal, ik praat nederlands. groetjes aan iedereen!"


key: _node_content
string_value:
{"id_": "7e3e3492-4fa6-4076-94c6-c09137f2f0ec", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "72f0f011-f818-4662-b64d-63353c798a87", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "52ab8b048821cdf750626913fb12879fb3e915617eccac4d7f8b6bc287107156", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6cd2d895-ce1d-45ab-9e52-d20a2b171af5", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/text-to-speech/", "title": "Text to speech", "tokens": 5485, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "1da41047272cf01b8ee2241d8f2bb82acee9dec28db27685bf6785b75b8d28e2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19795, "end_char_idx": 23025, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
groetjes aan iedereen!"
```

To use SpeechT5 with the pipeline, you'll need a speaker embedding. Let's get it from an example in the test dataset: 

```py
>>> example = dataset["test"][304]
>>> speaker_embeddings = torch.tensor(example["speaker_embeddings"]).unsqueeze(0)
```

Now you can pass the text and speaker embeddings to the pipeline, and it will take care of the rest: 

```py
>>> forward_params = {"speaker_embeddings": speaker_embeddings}
>>> output = pipe(text, forward_params=forward_params)
>>> output
{'audio': array([-6.82714235e-05, -4.26525949e-04,  1.06134125e-04, ...,
        -1.22392643e-03, -7.76011671e-04,  3.29112721e-04], dtype=float32),
 'sampling_rate': 16000}
```

You can then listen to the result:

```py
>>> from IPython.display import Audio
>>> Audio(output['audio'], rate=output['sampling_rate']) 
```

### Run inference manually

You can achieve the same inference results without using the pipeline, however, more steps will be required. 

Load the model from the 🤗 Hub: 

```py
>>> model = SpeechT5ForTextToSpeech.from_pretrained("YOUR_ACCOUNT/speecht5_finetuned_voxpopuli_nl")
```

Pick an example from the test dataset obtain a speaker embedding. 

```py 
>>> example = dataset["test"][304]
>>> speaker_embeddings = torch.tensor(example["speaker_embeddings"]).unsqueeze(0)
```

Define the input text and tokenize it.

```py 
>>> text = "hallo allemaal, ik praat nederlands. groetjes aan iedereen!"
>>> inputs = processor(text=text, return_tensors="pt")
```

Create a spectrogram with your model: 

```py
>>> spectrogram = model.generate_speech(inputs["input_ids"], speaker_embeddings)
```

Visualize the spectrogram, if you'd like to: 

```py
>>> plt.figure()
>>> plt.imshow(spectrogram.T)
>>> plt.show()
```

<div class="flex justify-center">
    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/tts_logmelspectrogram_2.png" alt="Generated log-mel spectrogram"/>
</div>

Finally, use the vocoder to turn the spectrogram into sound.

```py
>>> with torch.no_grad():
...     speech = vocoder(spectrogram)

>>> from IPython.display import Audio

>>> Audio(speech.numpy(), rate=16000)
```

In our experience, obtaining satisfactory results from this model can be challenging. The quality of the speaker 
embeddings appears to be a significant factor. Since SpeechT5 was pre-trained with English x-vectors, it performs best 
when using English speaker embeddings. If the synthesized speech sounds poor, try using a different speaker embedding.

Increasing the training duration is also likely to enhance the quality of the results. Even so, the speech clearly is Dutch instead of English, and it does 
capture the voice characteristics of the speaker (compare to the original audio in the example).
Another thing to experiment with is the model's configuration. For example, try using `config.reduction_factor = 1` to 
see if this improves the results.

Finally, it is essential to consider ethical considerations. Although TTS technology has numerous useful applications, it 
may also be used for malicious purposes, such as impersonating someone's voice without their knowledge or consent. Please 
use TTS judiciously and responsibly.


key: _node_content
string_value:
{"id_": "bb3bf5b9-431f-4030-ac6d-fb5422dc8dcb", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification/", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3276f133-3813-48ab-9acf-8e19b86e1db5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification/", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0eda786aa9e3bb06e5ad3d35335e235d836a28819647757bb00d306534888243", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e9a4f640-4298-4ff1-88df-87a59a67240d", "node_type": "1", "metadata": {}, "hash": "e46b03a0acd26943c7b5ee99ae79944766ced380d41b1ea730447865ce28ebb6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3010, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Audio classification

[[open-in-colab]]

<Youtube id="KWwzcmG98Ds"/>

Audio classification - just like with text - assigns a class label output from the input data. The only difference is instead of text inputs, you have raw audio waveforms. Some practical applications of audio classification include identifying speaker intent, language classification, and even animal species by their sounds.

This guide will show you how to:

1. Finetune [Wav2Vec2](https://huggingface.co/facebook/wav2vec2-base) on the [MInDS-14](https://huggingface.co/datasets/PolyAI/minds14) dataset to classify speaker intent.
2. Use your finetuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/audio-classification)

</Tip>

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install transformers datasets evaluate
```

We encourage you to login to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to login:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load MInDS-14 dataset

Start by loading the MInDS-14 dataset from the 🤗 Datasets library:

```py
>>> from datasets import load_dataset, Audio

>>> minds = load_dataset("PolyAI/minds14", name="en-US", split="train")
```

Split the dataset's `train` split into a smaller train and test set with the [`~datasets.Dataset.train_test_split`] method. This'll give you a chance to experiment and make sure everything works before spending more time on the full dataset.

```py
>>> minds = minds.train_test_split(test_size=0.2)
```

Then take a look at the dataset:

```py
>>> minds
DatasetDict({
    train: Dataset({
        features: ['path', 'audio', 'transcription', 'english_transcription', 'intent_class', 'lang_id'],
        num_rows: 450
    })
    test: Dataset({
        features: ['path', 'audio', 'transcription', 'english_transcription', 'intent_class', 'lang_id'],
        num_rows: 113
    })
})
```

While the dataset contains a lot of useful information, like `lang_id` and `english_transcription`, you'll focus on the `audio` and `intent_class` in this guide. Remove the other columns with the [`~datasets.Dataset.remove_columns`] method:

```py
>>> minds = minds.remove_columns(["path", "transcription", "english_transcription", "lang_id"])
```

Take a look at an example now:

```py
>>> minds["train"][0]
{'audio': {'array': array([ 0.        ,  0.        ,  0.        , ..., -0.00048828,
         -0.00024414, -0.00024414], dtype=float32),
  'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602b9a5fbb1e6d0fbce91f52.wav',
  'sampling_rate': 8000},
 'intent_class': 2}
```

There are two fields:

- `audio`: a 1-dimensional `array` of the speech signal that must be called to load and resample the audio file.


key: _node_content
string_value:
{"id_": "e9a4f640-4298-4ff1-88df-87a59a67240d", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification/", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3276f133-3813-48ab-9acf-8e19b86e1db5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification/", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0eda786aa9e3bb06e5ad3d35335e235d836a28819647757bb00d306534888243", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bb3bf5b9-431f-4030-ac6d-fb5422dc8dcb", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification/", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "2b8517e2418c296c6035cecf225b9140a0d364b41b10d9023fcb6eb1df06be66", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "894b8b91-e0ef-40ff-8a62-81ddf4fd2a92", "node_type": "1", "metadata": {}, "hash": "1d65f8041fce8b6124dfeb4287efad2a57fe5b8e88527f66f8d2509ca1d50682", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1658, "end_char_idx": 3078, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> minds = minds.train_test_split(test_size=0.2)
```

Then take a look at the dataset:

```py
>>> minds
DatasetDict({
    train: Dataset({
        features: ['path', 'audio', 'transcription', 'english_transcription', 'intent_class', 'lang_id'],
        num_rows: 450
    })
    test: Dataset({
        features: ['path', 'audio', 'transcription', 'english_transcription', 'intent_class', 'lang_id'],
        num_rows: 113
    })
})
```

While the dataset contains a lot of useful information, like `lang_id` and `english_transcription`, you'll focus on the `audio` and `intent_class` in this guide. Remove the other columns with the [`~datasets.Dataset.remove_columns`] method:

```py
>>> minds = minds.remove_columns(["path", "transcription", "english_transcription", "lang_id"])
```

Take a look at an example now:

```py
>>> minds["train"][0]
{'audio': {'array': array([ 0.        ,  0.        ,  0.        , ..., -0.00048828,
         -0.00024414, -0.00024414], dtype=float32),
  'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602b9a5fbb1e6d0fbce91f52.wav',
  'sampling_rate': 8000},
 'intent_class': 2}
```

There are two fields:

- `audio`: a 1-dimensional `array` of the speech signal that must be called to load and resample the audio file. 
- `intent_class`: represents the class id of the speaker's intent.


key: _node_content
string_value:
{"id_": "894b8b91-e0ef-40ff-8a62-81ddf4fd2a92", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification/", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3276f133-3813-48ab-9acf-8e19b86e1db5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification/", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0eda786aa9e3bb06e5ad3d35335e235d836a28819647757bb00d306534888243", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e9a4f640-4298-4ff1-88df-87a59a67240d", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification/", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4b993d4b15824c7043f984ca6ed6d218aace6593643c34d71c3d477b7a1f3c56", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6e1956dc-a24a-4239-b132-fdac31209378", "node_type": "1", "metadata": {}, "hash": "232f4c0308f321a3cfb0f11a06934bda667361a969991ac8c85f04d165c5f287", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2264, "end_char_idx": 4766, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Remove the other columns with the [`~datasets.Dataset.remove_columns`] method:

```py
>>> minds = minds.remove_columns(["path", "transcription", "english_transcription", "lang_id"])
```

Take a look at an example now:

```py
>>> minds["train"][0]
{'audio': {'array': array([ 0.        ,  0.        ,  0.        , ..., -0.00048828,
         -0.00024414, -0.00024414], dtype=float32),
  'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602b9a5fbb1e6d0fbce91f52.wav',
  'sampling_rate': 8000},
 'intent_class': 2}
```

There are two fields:

- `audio`: a 1-dimensional `array` of the speech signal that must be called to load and resample the audio file. 
- `intent_class`: represents the class id of the speaker's intent. 

To make it easier for the model to get the label name from the label id, create a dictionary that maps the label name to an integer and vice versa:

```py
>>> labels = minds["train"].features["intent_class"].names
>>> label2id, id2label = dict(), dict()
>>> for i, label in enumerate(labels):
...     label2id[label] = str(i)
...     id2label[str(i)] = label
```

Now you can convert the label id to a label name:

```py
>>> id2label[str(2)]
'app_error'
```

## Preprocess

The next step is to load a Wav2Vec2 feature extractor to process the audio signal:

```py
>>> from transformers import AutoFeatureExtractor

>>> feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base")
```

The MInDS-14 dataset has a sampling rate of 8000khz (you can find this information in it's [dataset card](https://huggingface.co/datasets/PolyAI/minds14)), which means you'll need to resample the dataset to 16000kHz to use the pretrained Wav2Vec2 model:

```py
>>> minds = minds.cast_column("audio", Audio(sampling_rate=16_000))
>>> minds["train"][0]
{'audio': {'array': array([ 2.2098757e-05,  4.6582241e-05, -2.2803260e-05, ...,
         -2.8419291e-04, -2.3305941e-04, -1.1425107e-04], dtype=float32),
  'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602b9a5fbb1e6d0fbce91f52.wav',
  'sampling_rate': 16000},
 'intent_class': 2}
```

Now create a preprocessing function that:

1. Calls the `audio` column to load, and if necessary, resample the audio file.
2. Checks if the sampling rate of the audio file matches the sampling rate of the audio data a model was pretrained with.


key: _node_content
string_value:
{"id_": "6e1956dc-a24a-4239-b132-fdac31209378", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification/", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3276f133-3813-48ab-9acf-8e19b86e1db5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification/", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0eda786aa9e3bb06e5ad3d35335e235d836a28819647757bb00d306534888243", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "894b8b91-e0ef-40ff-8a62-81ddf4fd2a92", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification/", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4fc96ba3880d93ec460523fe48e76a73ed2542bd7591b27121180dc69d12153d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4bbfb851-3159-4f7d-bb84-58abf1fb4fde", "node_type": "1", "metadata": {}, "hash": "2363388941310bed7533ab41df23e616f83761ce5ee0c6254ac038e9a9852316", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4568, "end_char_idx": 6438, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Calls the `audio` column to load, and if necessary, resample the audio file.
2. Checks if the sampling rate of the audio file matches the sampling rate of the audio data a model was pretrained with. You can find this information in the Wav2Vec2 [model card](https://huggingface.co/facebook/wav2vec2-base).
3. Set a maximum input length to batch longer inputs without truncating them.

```py
>>> def preprocess_function(examples):
...     audio_arrays = [x["array"] for x in examples["audio"]]
...     inputs = feature_extractor(
...         audio_arrays, sampling_rate=feature_extractor.sampling_rate, max_length=16000, truncation=True
...     )
...     return inputs
```

To apply the preprocessing function over the entire dataset, use 🤗 Datasets [`~datasets.Dataset.map`] function. You can speed up `map` by setting `batched=True` to process multiple elements of the dataset at once. Remove the columns you don't need, and rename `intent_class` to `label` because that's the name the model expects:

```py
>>> encoded_minds = minds.map(preprocess_function, remove_columns="audio", batched=True)
>>> encoded_minds = encoded_minds.rename_column("intent_class", "label")
```

## Evaluate

Including a metric during training is often helpful for evaluating your model's performance. You can quickly load a evaluation method with the 🤗 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [accuracy](https://huggingface.co/spaces/evaluate-metric/accuracy) metric (see the 🤗 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric):

```py
>>> import evaluate

>>> accuracy = evaluate.load("accuracy")
```

Then create a function that passes your predictions and labels to [`~evaluate.EvaluationModule.compute`] to calculate the accuracy:

```py
>>> import numpy as np


key: _node_content
string_value:
{"id_": "4bbfb851-3159-4f7d-bb84-58abf1fb4fde", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification/", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3276f133-3813-48ab-9acf-8e19b86e1db5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification/", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0eda786aa9e3bb06e5ad3d35335e235d836a28819647757bb00d306534888243", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6e1956dc-a24a-4239-b132-fdac31209378", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification/", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "1996a4def6e7ea1b2b563bb29d4d667e686193eb9513c42168ab03167dcbab0c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c469337c-26aa-4b0d-8b65-7c834f65f3fa", "node_type": "1", "metadata": {}, "hash": "35f1521d6daf246591a96ca24d3cf27487e477f83566550ee1204ac88bd03e4b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4767, "end_char_idx": 8014, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can find this information in the Wav2Vec2 [model card](https://huggingface.co/facebook/wav2vec2-base).
3. Set a maximum input length to batch longer inputs without truncating them.

```py
>>> def preprocess_function(examples):
...     audio_arrays = [x["array"] for x in examples["audio"]]
...     inputs = feature_extractor(
...         audio_arrays, sampling_rate=feature_extractor.sampling_rate, max_length=16000, truncation=True
...     )
...     return inputs
```

To apply the preprocessing function over the entire dataset, use 🤗 Datasets [`~datasets.Dataset.map`] function. You can speed up `map` by setting `batched=True` to process multiple elements of the dataset at once. Remove the columns you don't need, and rename `intent_class` to `label` because that's the name the model expects:

```py
>>> encoded_minds = minds.map(preprocess_function, remove_columns="audio", batched=True)
>>> encoded_minds = encoded_minds.rename_column("intent_class", "label")
```

## Evaluate

Including a metric during training is often helpful for evaluating your model's performance. You can quickly load a evaluation method with the 🤗 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [accuracy](https://huggingface.co/spaces/evaluate-metric/accuracy) metric (see the 🤗 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric):

```py
>>> import evaluate

>>> accuracy = evaluate.load("accuracy")
```

Then create a function that passes your predictions and labels to [`~evaluate.EvaluationModule.compute`] to calculate the accuracy:

```py
>>> import numpy as np


>>> def compute_metrics(eval_pred):
...     predictions = np.argmax(eval_pred.predictions, axis=1)
...     return accuracy.compute(predictions=predictions, references=eval_pred.label_ids)
```

Your `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.

## Train

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load Wav2Vec2 with [`AutoModelForAudioClassification`] along with the number of expected labels, and the label mappings:

```py
>>> from transformers import AutoModelForAudioClassification, TrainingArguments, Trainer

>>> num_labels = len(id2label)
>>> model = AutoModelForAudioClassification.from_pretrained(
...     "facebook/wav2vec2-base", num_labels=num_labels, label2id=label2id, id2label=id2label
... )
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the accuracy and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.


key: _node_content
string_value:
{"id_": "c469337c-26aa-4b0d-8b65-7c834f65f3fa", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification/", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3276f133-3813-48ab-9acf-8e19b86e1db5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification/", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0eda786aa9e3bb06e5ad3d35335e235d836a28819647757bb00d306534888243", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4bbfb851-3159-4f7d-bb84-58abf1fb4fde", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification/", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "da17456aac090a317d994206d2398670b91d28922a243aa0b41281e845fecfbf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "df74ee0e-f77f-4208-bcc6-309430b82bb9", "node_type": "1", "metadata": {}, "hash": "1dd51449c610acb4d99403f94fdf1307011c0cce91d4c77c8cd14e6d63842e60", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6441, "end_char_idx": 9495, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> def compute_metrics(eval_pred):
...     predictions = np.argmax(eval_pred.predictions, axis=1)
...     return accuracy.compute(predictions=predictions, references=eval_pred.label_ids)
```

Your `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.

## Train

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load Wav2Vec2 with [`AutoModelForAudioClassification`] along with the number of expected labels, and the label mappings:

```py
>>> from transformers import AutoModelForAudioClassification, TrainingArguments, Trainer

>>> num_labels = len(id2label)
>>> model = AutoModelForAudioClassification.from_pretrained(
...     "facebook/wav2vec2-base", num_labels=num_labels, label2id=label2id, id2label=id2label
... )
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the accuracy and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.


```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_mind_model",
...     eval_strategy="epoch",
...     save_strategy="epoch",
...     learning_rate=3e-5,
...     per_device_train_batch_size=32,
...     gradient_accumulation_steps=4,
...     per_device_eval_batch_size=32,
...     num_train_epochs=10,
...     warmup_ratio=0.1,
...     logging_steps=10,
...     load_best_model_at_end=True,
...     metric_for_best_model="accuracy",
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=encoded_minds["train"],
...     eval_dataset=encoded_minds["test"],
...     tokenizer=feature_extractor,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for audio classification, take a look at the corresponding [PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/audio_classification.ipynb).

</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Load an audio file you'd like to run inference on. Remember to resample the sampling rate of the audio file to match the sampling rate of the model if you need to!


key: _node_content
string_value:
{"id_": "df74ee0e-f77f-4208-bcc6-309430b82bb9", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification/", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3276f133-3813-48ab-9acf-8e19b86e1db5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification/", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0eda786aa9e3bb06e5ad3d35335e235d836a28819647757bb00d306534888243", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c469337c-26aa-4b0d-8b65-7c834f65f3fa", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification/", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a696c0723d1a012bc338288bfd6d9f522329c7f1225b9207ae4bda3b97148a36", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ace8ce22-6f67-4f7d-88df-9c70c25d57e7", "node_type": "1", "metadata": {}, "hash": "86aa687b5bf1bc2ef0f15b47aa01a8a2708715e386f54f2bf04fc5db8cc2ed4f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8017, "end_char_idx": 9898, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_mind_model",
...     eval_strategy="epoch",
...     save_strategy="epoch",
...     learning_rate=3e-5,
...     per_device_train_batch_size=32,
...     gradient_accumulation_steps=4,
...     per_device_eval_batch_size=32,
...     num_train_epochs=10,
...     warmup_ratio=0.1,
...     logging_steps=10,
...     load_best_model_at_end=True,
...     metric_for_best_model="accuracy",
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=encoded_minds["train"],
...     eval_dataset=encoded_minds["test"],
...     tokenizer=feature_extractor,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for audio classification, take a look at the corresponding [PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/audio_classification.ipynb).

</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Load an audio file you'd like to run inference on. Remember to resample the sampling rate of the audio file to match the sampling rate of the model if you need to!

```py
>>> from datasets import load_dataset, Audio

>>> dataset = load_dataset("PolyAI/minds14", name="en-US", split="train")
>>> dataset = dataset.cast_column("audio", Audio(sampling_rate=16000))
>>> sampling_rate = dataset.features["audio"].sampling_rate
>>> audio_file = dataset[0]["audio"]["path"]
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`].


key: _node_content
string_value:
{"id_": "ace8ce22-6f67-4f7d-88df-9c70c25d57e7", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification/", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3276f133-3813-48ab-9acf-8e19b86e1db5", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification/", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "0eda786aa9e3bb06e5ad3d35335e235d836a28819647757bb00d306534888243", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "df74ee0e-f77f-4208-bcc6-309430b82bb9", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/audio_classification/", "title": "Audio classification", "tokens": 3086, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "db4ff700ae4f31aa98ca8226838e5886398b55f1a9f9c987053fb88eb53166a9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9237, "end_char_idx": 11536, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Load an audio file you'd like to run inference on. Remember to resample the sampling rate of the audio file to match the sampling rate of the model if you need to!

```py
>>> from datasets import load_dataset, Audio

>>> dataset = load_dataset("PolyAI/minds14", name="en-US", split="train")
>>> dataset = dataset.cast_column("audio", Audio(sampling_rate=16000))
>>> sampling_rate = dataset.features["audio"].sampling_rate
>>> audio_file = dataset[0]["audio"]["path"]
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for audio classification with your model, and pass your audio file to it:

```py
>>> from transformers import pipeline

>>> classifier = pipeline("audio-classification", model="stevhliu/my_awesome_minds_model")
>>> classifier(audio_file)
[
    {'score': 0.09766869246959686, 'label': 'cash_deposit'},
    {'score': 0.07998877018690109, 'label': 'app_error'},
    {'score': 0.0781070664525032, 'label': 'joint_account'},
    {'score': 0.07667109370231628, 'label': 'pay_bill'},
    {'score': 0.0755252093076706, 'label': 'balance'}
]
```

You can also manually replicate the results of the `pipeline` if you'd like:

<frameworkcontent>
<pt>
Load a feature extractor to preprocess the audio file and return the `input` as PyTorch tensors:

```py
>>> from transformers import AutoFeatureExtractor

>>> feature_extractor = AutoFeatureExtractor.from_pretrained("stevhliu/my_awesome_minds_model")
>>> inputs = feature_extractor(dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt")
```

Pass your inputs to the model and return the logits:

```py
>>> from transformers import AutoModelForAudioClassification

>>> model = AutoModelForAudioClassification.from_pretrained("stevhliu/my_awesome_minds_model")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
```

Get the class with the highest probability, and use the model's `id2label` mapping to convert it to a label:

```py
>>> import torch

>>> predicted_class_ids = torch.argmax(logits).item()
>>> predicted_label = model.config.id2label[predicted_class_ids]
>>> predicted_label
'cash_deposit'
```
</pt>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "a9ec476f-c15b-41be-bc85-5ec5b6b31c93", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr/", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "11908ee8-24b8-4180-9005-d9aae56bf96d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr/", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a8859d5fcd2f23937320ee8429e334602042879bd2f8828b36fbc2ce601421bc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "67ded22a-62a7-4970-a382-5e325e45472e", "node_type": "1", "metadata": {}, "hash": "e7c1e3e61568da0728550fd1a57b7d0893c28f06f29b24dcf530460b4b9da5a1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2336, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Automatic speech recognition

[[open-in-colab]]

<Youtube id="TksaY_FDgnk"/>

Automatic speech recognition (ASR) converts a speech signal to text, mapping a sequence of audio inputs to text outputs. Virtual assistants like Siri and Alexa use ASR models to help users everyday, and there are many other useful user-facing applications like live captioning and note-taking during meetings.

This guide will show you how to:

1. Finetune [Wav2Vec2](https://huggingface.co/facebook/wav2vec2-base) on the [MInDS-14](https://huggingface.co/datasets/PolyAI/minds14) dataset to transcribe audio to text.
2. Use your finetuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/automatic-speech-recognition)

</Tip>

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install transformers datasets evaluate jiwer
```

We encourage you to login to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to login:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load MInDS-14 dataset

Start by loading a smaller subset of the [MInDS-14](https://huggingface.co/datasets/PolyAI/minds14) dataset from the 🤗 Datasets library. This'll give you a chance to experiment and make sure everything works before spending more time training on the full dataset.

```py
>>> from datasets import load_dataset, Audio

>>> minds = load_dataset("PolyAI/minds14", name="en-US", split="train[:100]")
```

Split the dataset's `train` split into a train and test set with the [`~Dataset.train_test_split`] method:

```py
>>> minds = minds.train_test_split(test_size=0.2)
```

Then take a look at the dataset:

```py
>>> minds
DatasetDict({
    train: Dataset({
        features: ['path', 'audio', 'transcription', 'english_transcription', 'intent_class', 'lang_id'],
        num_rows: 16
    })
    test: Dataset({
        features: ['path', 'audio', 'transcription', 'english_transcription', 'intent_class', 'lang_id'],
        num_rows: 4
    })
})
```

While the dataset contains a lot of useful information, like `lang_id` and `english_transcription`, you'll focus on the `audio` and `transcription` in this guide.


key: _node_content
string_value:
{"id_": "67ded22a-62a7-4970-a382-5e325e45472e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr/", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "11908ee8-24b8-4180-9005-d9aae56bf96d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr/", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a8859d5fcd2f23937320ee8429e334602042879bd2f8828b36fbc2ce601421bc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a9ec476f-c15b-41be-bc85-5ec5b6b31c93", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr/", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "e91d56d7a2264c2758b1e70a89fb1ff38dd286002964455dd903572895e3ad85", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "56f6dfd9-0566-4cc3-8b12-3e62ec8e81d3", "node_type": "1", "metadata": {}, "hash": "89a4d8fa95f76450a573f182b251964f92ba429790df045628917c517b962071", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1076, "end_char_idx": 3411, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
When prompted, enter your token to login:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load MInDS-14 dataset

Start by loading a smaller subset of the [MInDS-14](https://huggingface.co/datasets/PolyAI/minds14) dataset from the 🤗 Datasets library. This'll give you a chance to experiment and make sure everything works before spending more time training on the full dataset.

```py
>>> from datasets import load_dataset, Audio

>>> minds = load_dataset("PolyAI/minds14", name="en-US", split="train[:100]")
```

Split the dataset's `train` split into a train and test set with the [`~Dataset.train_test_split`] method:

```py
>>> minds = minds.train_test_split(test_size=0.2)
```

Then take a look at the dataset:

```py
>>> minds
DatasetDict({
    train: Dataset({
        features: ['path', 'audio', 'transcription', 'english_transcription', 'intent_class', 'lang_id'],
        num_rows: 16
    })
    test: Dataset({
        features: ['path', 'audio', 'transcription', 'english_transcription', 'intent_class', 'lang_id'],
        num_rows: 4
    })
})
```

While the dataset contains a lot of useful information, like `lang_id` and `english_transcription`, you'll focus on the `audio` and `transcription` in this guide. Remove the other columns with the [`~datasets.Dataset.remove_columns`] method:

```py
>>> minds = minds.remove_columns(["english_transcription", "intent_class", "lang_id"])
```

Take a look at the example again:

```py
>>> minds["train"][0]
{'audio': {'array': array([-0.00024414,  0.        ,  0.        , ...,  0.00024414,
          0.00024414,  0.00024414], dtype=float32),
  'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602ba9e2963e11ccd901cd4f.wav',
  'sampling_rate': 8000},
 'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602ba9e2963e11ccd901cd4f.wav',
 'transcription': "hi I'm trying to use the banking app on my phone and currently my checking and savings account balance is not refreshing"}
```

There are two fields:

- `audio`: a 1-dimensional `array` of the speech signal that must be called to load and resample the audio file.
- `transcription`: the target text.


key: _node_content
string_value:
{"id_": "56f6dfd9-0566-4cc3-8b12-3e62ec8e81d3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr/", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "11908ee8-24b8-4180-9005-d9aae56bf96d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr/", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a8859d5fcd2f23937320ee8429e334602042879bd2f8828b36fbc2ce601421bc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "67ded22a-62a7-4970-a382-5e325e45472e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr/", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "b988c40707ceac9482091fde752e6694de9463d6ffd2c2311ec96be85c79474e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c8f32d36-0cd3-4ff5-bf87-ba10a79cd72e", "node_type": "1", "metadata": {}, "hash": "bf6f46d1a3e333cf23c787f0d8f171a04441907d7cdb51ca46ac1043a9043700", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2337, "end_char_idx": 4771, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Remove the other columns with the [`~datasets.Dataset.remove_columns`] method:

```py
>>> minds = minds.remove_columns(["english_transcription", "intent_class", "lang_id"])
```

Take a look at the example again:

```py
>>> minds["train"][0]
{'audio': {'array': array([-0.00024414,  0.        ,  0.        , ...,  0.00024414,
          0.00024414,  0.00024414], dtype=float32),
  'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602ba9e2963e11ccd901cd4f.wav',
  'sampling_rate': 8000},
 'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602ba9e2963e11ccd901cd4f.wav',
 'transcription': "hi I'm trying to use the banking app on my phone and currently my checking and savings account balance is not refreshing"}
```

There are two fields:

- `audio`: a 1-dimensional `array` of the speech signal that must be called to load and resample the audio file.
- `transcription`: the target text.

## Preprocess

The next step is to load a Wav2Vec2 processor to process the audio signal:

```py
>>> from transformers import AutoProcessor

>>> processor = AutoProcessor.from_pretrained("facebook/wav2vec2-base")
```

The MInDS-14 dataset has a sampling rate of 8000kHz (you can find this information in its [dataset card](https://huggingface.co/datasets/PolyAI/minds14)), which means you'll need to resample the dataset to 16000kHz to use the pretrained Wav2Vec2 model:

```py
>>> minds = minds.cast_column("audio", Audio(sampling_rate=16_000))
>>> minds["train"][0]
{'audio': {'array': array([-2.38064706e-04, -1.58618059e-04, -5.43987835e-06, ...,
          2.78103951e-04,  2.38446111e-04,  1.18740834e-04], dtype=float32),
  'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602ba9e2963e11ccd901cd4f.wav',
  'sampling_rate': 16000},
 'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~APP_ERROR/602ba9e2963e11ccd901cd4f.wav',
 'transcription': "hi I'm trying to use the banking app on my phone and currently my checking and savings account balance is not refreshing"}
```

As you can see in the `transcription` above, the text contains a mix of upper and lowercase characters.


key: _node_content
string_value:
{"id_": "c8f32d36-0cd3-4ff5-bf87-ba10a79cd72e", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr/", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "11908ee8-24b8-4180-9005-d9aae56bf96d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr/", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a8859d5fcd2f23937320ee8429e334602042879bd2f8828b36fbc2ce601421bc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "56f6dfd9-0566-4cc3-8b12-3e62ec8e81d3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr/", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5a78c07d6870311b2f4549e7055938287c652f720332052be54d1c4eff4f3347", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ea0c683d-4b50-4748-a0aa-42c79fa67b43", "node_type": "1", "metadata": {}, "hash": "8e0c90daed7ad60e4ab66ba8f62a516c42f87c277e79b7bb1c6c9e9e2de4e733", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4772, "end_char_idx": 6628, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The Wav2Vec2 tokenizer is only trained on uppercase characters so you'll need to make sure the text matches the tokenizer's vocabulary:

```py
>>> def uppercase(example):
...     return {"transcription": example["transcription"].upper()}


>>> minds = minds.map(uppercase)
```

Now create a preprocessing function that:

1. Calls the `audio` column to load and resample the audio file.
2. Extracts the `input_values` from the audio file and tokenize the `transcription` column with the processor.

```py
>>> def prepare_dataset(batch):
...     audio = batch["audio"]
...     batch = processor(audio["array"], sampling_rate=audio["sampling_rate"], text=batch["transcription"])
...     batch["input_length"] = len(batch["input_values"][0])
...     return batch
```

To apply the preprocessing function over the entire dataset, use 🤗 Datasets [`~datasets.Dataset.map`] function. You can speed up `map` by increasing the number of processes with the `num_proc` parameter. Remove the columns you don't need with the [`~datasets.Dataset.remove_columns`] method:

```py
>>> encoded_minds = minds.map(prepare_dataset, remove_columns=minds.column_names["train"], num_proc=4)
```

🤗 Transformers doesn't have a data collator for ASR, so you'll need to adapt the [`DataCollatorWithPadding`] to create a batch of examples. It'll also dynamically pad your text and labels to the length of the longest element in its batch (instead of the entire dataset) so they are a uniform length. While it is possible to pad your text in the `tokenizer` function by setting `padding=True`, dynamic padding is more efficient.

Unlike other data collators, this specific data collator needs to apply a different padding method to `input_values` and `labels`:

```py
>>> import torch

>>> from dataclasses import dataclass, field
>>> from typing import Any, Dict, List, Optional, Union


key: _node_content
string_value:
{"id_": "ea0c683d-4b50-4748-a0aa-42c79fa67b43", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr/", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "11908ee8-24b8-4180-9005-d9aae56bf96d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr/", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a8859d5fcd2f23937320ee8429e334602042879bd2f8828b36fbc2ce601421bc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c8f32d36-0cd3-4ff5-bf87-ba10a79cd72e", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr/", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "2db431964b6d48385db53425a9a9da50a823ee99e93246c6d5f1658eecb4dfc8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "25d4d189-505a-4a49-8a50-8fe521739711", "node_type": "1", "metadata": {}, "hash": "37743a80c704723b1384deea748f086bf401348d80219d43ac4071072a678f90", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5012, "end_char_idx": 8483, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> minds = minds.map(uppercase)
```

Now create a preprocessing function that:

1. Calls the `audio` column to load and resample the audio file.
2. Extracts the `input_values` from the audio file and tokenize the `transcription` column with the processor.

```py
>>> def prepare_dataset(batch):
...     audio = batch["audio"]
...     batch = processor(audio["array"], sampling_rate=audio["sampling_rate"], text=batch["transcription"])
...     batch["input_length"] = len(batch["input_values"][0])
...     return batch
```

To apply the preprocessing function over the entire dataset, use 🤗 Datasets [`~datasets.Dataset.map`] function. You can speed up `map` by increasing the number of processes with the `num_proc` parameter. Remove the columns you don't need with the [`~datasets.Dataset.remove_columns`] method:

```py
>>> encoded_minds = minds.map(prepare_dataset, remove_columns=minds.column_names["train"], num_proc=4)
```

🤗 Transformers doesn't have a data collator for ASR, so you'll need to adapt the [`DataCollatorWithPadding`] to create a batch of examples. It'll also dynamically pad your text and labels to the length of the longest element in its batch (instead of the entire dataset) so they are a uniform length. While it is possible to pad your text in the `tokenizer` function by setting `padding=True`, dynamic padding is more efficient.

Unlike other data collators, this specific data collator needs to apply a different padding method to `input_values` and `labels`:

```py
>>> import torch

>>> from dataclasses import dataclass, field
>>> from typing import Any, Dict, List, Optional, Union


>>> @dataclass
... class DataCollatorCTCWithPadding:
...     processor: AutoProcessor
...     padding: Union[bool, str] = "longest"

...     def __call__(self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]:
...         # split inputs and labels since they have to be of different lengths and need
...         # different padding methods
...         input_features = [{"input_values": feature["input_values"][0]} for feature in features]
...         label_features = [{"input_ids": feature["labels"]} for feature in features]

...         batch = self.processor.pad(input_features, padding=self.padding, return_tensors="pt")

...         labels_batch = self.processor.pad(labels=label_features, padding=self.padding, return_tensors="pt")

...         # replace padding with -100 to ignore loss correctly
...         labels = labels_batch["input_ids"].masked_fill(labels_batch.attention_mask.ne(1), -100)

...         batch["labels"] = labels

...         return batch
```

Now instantiate your `DataCollatorForCTCWithPadding`:

```py
>>> data_collator = DataCollatorCTCWithPadding(processor=processor, padding="longest")
```

## Evaluate

Including a metric during training is often helpful for evaluating your model's performance. You can quickly load a evaluation method with the 🤗 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load the [word error rate](https://huggingface.co/spaces/evaluate-metric/wer) (WER) metric (see the 🤗 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric):

```py
>>> import evaluate

>>> wer = evaluate.load("wer")
```

Then create a function that passes your predictions and labels to [`~evaluate.EvaluationModule.compute`] to calculate the WER:

```py
>>> import numpy as np


key: _node_content
string_value:
{"id_": "25d4d189-505a-4a49-8a50-8fe521739711", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr/", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "11908ee8-24b8-4180-9005-d9aae56bf96d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr/", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a8859d5fcd2f23937320ee8429e334602042879bd2f8828b36fbc2ce601421bc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ea0c683d-4b50-4748-a0aa-42c79fa67b43", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr/", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "bb3dc8cd4e69b53f4f09574b808c71aebcb90c4e17f98baa559bd08f9553360e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1c0ae5ca-c95e-48be-ba07-ad35b8114332", "node_type": "1", "metadata": {}, "hash": "d68f279720a591c69093189a95e8d14ae4bba946208c081bb32e621ea114fa19", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8486, "end_char_idx": 10332, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
>>> def compute_metrics(pred):
...     pred_logits = pred.predictions
...     pred_ids = np.argmax(pred_logits, axis=-1)

...     pred.label_ids[pred.label_ids == -100] = processor.tokenizer.pad_token_id

...     pred_str = processor.batch_decode(pred_ids)
...     label_str = processor.batch_decode(pred.label_ids, group_tokens=False)

...     wer = wer.compute(predictions=pred_str, references=label_str)

...     return {"wer": wer}
```

Your `compute_metrics` function is ready to go now, and you'll return to it when you setup your training.

## Train

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load Wav2Vec2 with [`AutoModelForCTC`]. Specify the reduction to apply with the `ctc_loss_reduction` parameter. It is often better to use the average instead of the default summation:

```py
>>> from transformers import AutoModelForCTC, TrainingArguments, Trainer

>>> model = AutoModelForCTC.from_pretrained(
...     "facebook/wav2vec2-base",
...     ctc_loss_reduction="mean",
...     pad_token_id=processor.tokenizer.pad_token_id,
... )
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the WER and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.


key: _node_content
string_value:
{"id_": "1c0ae5ca-c95e-48be-ba07-ad35b8114332", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr/", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "11908ee8-24b8-4180-9005-d9aae56bf96d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr/", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a8859d5fcd2f23937320ee8429e334602042879bd2f8828b36fbc2ce601421bc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "25d4d189-505a-4a49-8a50-8fe521739711", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr/", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "6a30f185fdafcc3cc619bf20447110ca8e105ef3ae9384cdd06c736082f33565", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "544597c5-34ef-448a-adf9-d44148cae75f", "node_type": "1", "metadata": {}, "hash": "340b985da6e92e10e3219dbb60d41f5c1cd778a5bb6c4cf97cb7df5a6a23df25", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9034, "end_char_idx": 12011, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Train

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the basic tutorial [here](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load Wav2Vec2 with [`AutoModelForCTC`]. Specify the reduction to apply with the `ctc_loss_reduction` parameter. It is often better to use the average instead of the default summation:

```py
>>> from transformers import AutoModelForCTC, TrainingArguments, Trainer

>>> model = AutoModelForCTC.from_pretrained(
...     "facebook/wav2vec2-base",
...     ctc_loss_reduction="mean",
...     pad_token_id=processor.tokenizer.pad_token_id,
... )
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [`Trainer`] will evaluate the WER and save the training checkpoint.
2. Pass the training arguments to [`Trainer`] along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_asr_mind_model",
...     per_device_train_batch_size=8,
...     gradient_accumulation_steps=2,
...     learning_rate=1e-5,
...     warmup_steps=500,
...     max_steps=2000,
...     gradient_checkpointing=True,
...     fp16=True,
...     group_by_length=True,
...     eval_strategy="steps",
...     per_device_eval_batch_size=8,
...     save_steps=1000,
...     eval_steps=1000,
...     logging_steps=25,
...     load_best_model_at_end=True,
...     metric_for_best_model="wer",
...     greater_is_better=False,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=encoded_minds["train"],
...     eval_dataset=encoded_minds["test"],
...     tokenizer=processor,
...     data_collator=data_collator,
...     compute_metrics=compute_metrics,
... )

>>> trainer.train()
```

Once training is completed, share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for automatic speech recognition, take a look at this blog [post](https://huggingface.co/blog/fine-tune-wav2vec2-english) for English ASR and this [post](https://huggingface.co/blog/fine-tune-xlsr-wav2vec2) for multilingual ASR.

</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Load an audio file you'd like to run inference on. Remember to resample the sampling rate of the audio file to match the sampling rate of the model if you need to!


key: _node_content
string_value:
{"id_": "544597c5-34ef-448a-adf9-d44148cae75f", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr/", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "11908ee8-24b8-4180-9005-d9aae56bf96d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr/", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a8859d5fcd2f23937320ee8429e334602042879bd2f8828b36fbc2ce601421bc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1c0ae5ca-c95e-48be-ba07-ad35b8114332", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/asr/", "title": "Automatic speech recognition", "tokens": 3709, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a310f69d032f0be98b4fac64c52ee16124fa0a80f406cd4db0c4843813c4abfc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11753, "end_char_idx": 13976, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Load an audio file you'd like to run inference on. Remember to resample the sampling rate of the audio file to match the sampling rate of the model if you need to!

```py
>>> from datasets import load_dataset, Audio

>>> dataset = load_dataset("PolyAI/minds14", "en-US", split="train")
>>> dataset = dataset.cast_column("audio", Audio(sampling_rate=16000))
>>> sampling_rate = dataset.features["audio"].sampling_rate
>>> audio_file = dataset[0]["audio"]["path"]
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for automatic speech recognition with your model, and pass your audio file to it:

```py
>>> from transformers import pipeline

>>> transcriber = pipeline("automatic-speech-recognition", model="stevhliu/my_awesome_asr_minds_model")
>>> transcriber(audio_file)
{'text': 'I WOUD LIKE O SET UP JOINT ACOUNT WTH Y PARTNER'}
```

<Tip>

The transcription is decent, but it could be better! Try finetuning your model on more examples to get even better results!

</Tip>

You can also manually replicate the results of the `pipeline` if you'd like:

<frameworkcontent>
<pt>
Load a processor to preprocess the audio file and transcription and return the `input` as PyTorch tensors:

```py
>>> from transformers import AutoProcessor

>>> processor = AutoProcessor.from_pretrained("stevhliu/my_awesome_asr_mind_model")
>>> inputs = processor(dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt")
```

Pass your inputs to the model and return the logits:

```py
>>> from transformers import AutoModelForCTC

>>> model = AutoModelForCTC.from_pretrained("stevhliu/my_awesome_asr_mind_model")
>>> with torch.no_grad():
...     logits = model(**inputs).logits
```

Get the predicted `input_ids` with the highest probability, and use the processor to decode the predicted `input_ids` back into text:

```py
>>> import torch

>>> predicted_ids = torch.argmax(logits, dim=-1)
>>> transcription = processor.batch_decode(predicted_ids)
>>> transcription
['I WOUL LIKE O SET UP JOINT ACOUNT WTH Y PARTNER']
```
</pt>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "25dd4b37-9a3a-46ad-a387-172fc11f6673", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling/", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3f601c42-26dd-4f95-9bd1-65c2f1f524f8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling/", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a1b44cd93eb6333cd2edf2a952f7dcceb8fee8671d1aeec292a32dc8247d67b5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "75a972aa-5eb3-447b-9d61-cc0580240967", "node_type": "1", "metadata": {}, "hash": "7733f907b8600bdc0ea7c332460b88a5192f4933d5b80079932f7055ba69b2e3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3113, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Causal language modeling

[[open-in-colab]]

There are two types of language modeling, causal and masked. This guide illustrates causal language modeling.
Causal language models are frequently used for text generation. You can use these models for creative applications like
choosing your own text adventure or an intelligent coding assistant like Copilot or CodeParrot.

<Youtube id="Vpjb1lu0MDk"/>

Causal language modeling predicts the next token in a sequence of tokens, and the model can only attend to tokens on
the left. This means the model cannot see future tokens. GPT-2 is an example of a causal language model.

This guide will show you how to:

1. Finetune [DistilGPT2](https://huggingface.co/distilbert/distilgpt2) on the [r/askscience](https://www.reddit.com/r/askscience/) subset of the [ELI5](https://huggingface.co/datasets/eli5) dataset.
2. Use your finetuned model for inference.

<Tip>

To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/text-generation)

</Tip>

Before you begin, make sure you have all the necessary libraries installed:

```bash
pip install transformers datasets evaluate
```

We encourage you to log in to your Hugging Face account so you can upload and share your model with the community. When prompted, enter your token to log in:

```py
>>> from huggingface_hub import notebook_login

>>> notebook_login()
```

## Load ELI5 dataset

Start by loading the first 5000 examples from the [ELI5-Category](https://huggingface.co/datasets/eli5_category) dataset with the 🤗 Datasets library. This'll give you a chance to experiment and make sure everything works before spending more time training on the full dataset.

```py
>>> from datasets import load_dataset

>>> eli5 = load_dataset("eli5_category", split="train[:5000]")
```

Split the dataset's `train` split into a train and test set with the [`~datasets.Dataset.train_test_split`] method:

```py
>>> eli5 = eli5.train_test_split(test_size=0.2)
```

Then take a look at an example:

```py
>>> eli5["train"][0]
{'q_id': '7h191n',
 'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?',
 'selftext': '',
 'category': 'Economics',
 'subreddit': 'explainlikeimfive',
 'answers': {'a_id': ['dqnds8l', 'dqnd1jl', 'dqng3i1', 'dqnku5x'],
  'text': ["The tax bill is 500 pages long and there were a lot of changes still going on right to the end. It's not just an adjustment to the income tax brackets, it's a whole bunch of changes. As such there is no good answer to your question. The big take aways are: - Big reduction in corporate income tax rate will make large companies very happy. - Pass through rate change will make certain styles of business (law firms, hedge funds) extremely happy - Income tax changes are moderate, and are set to expire (though it's the kind of thing that might just always get re-applied without being made permanent) - People in high tax states (California, New York) lose out, and many of them will end up with their taxes raised.",
   'None yet.


key: _node_content
string_value:
{"id_": "75a972aa-5eb3-447b-9d61-cc0580240967", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling/", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3f601c42-26dd-4f95-9bd1-65c2f1f524f8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling/", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a1b44cd93eb6333cd2edf2a952f7dcceb8fee8671d1aeec292a32dc8247d67b5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "25dd4b37-9a3a-46ad-a387-172fc11f6673", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling/", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "397a9ca95c3d07638ce74f454fbe165ee3e5f753b422960c2ed6bb08a396b8fd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d08dfee3-feeb-487f-8a85-774711b72e10", "node_type": "1", "metadata": {}, "hash": "8ee1073c72965596d03a49fb92a3bc923905d5c8f3b23a97e27b797a31d18236", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1615, "end_char_idx": 4536, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This'll give you a chance to experiment and make sure everything works before spending more time training on the full dataset.

```py
>>> from datasets import load_dataset

>>> eli5 = load_dataset("eli5_category", split="train[:5000]")
```

Split the dataset's `train` split into a train and test set with the [`~datasets.Dataset.train_test_split`] method:

```py
>>> eli5 = eli5.train_test_split(test_size=0.2)
```

Then take a look at an example:

```py
>>> eli5["train"][0]
{'q_id': '7h191n',
 'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?',
 'selftext': '',
 'category': 'Economics',
 'subreddit': 'explainlikeimfive',
 'answers': {'a_id': ['dqnds8l', 'dqnd1jl', 'dqng3i1', 'dqnku5x'],
  'text': ["The tax bill is 500 pages long and there were a lot of changes still going on right to the end. It's not just an adjustment to the income tax brackets, it's a whole bunch of changes. As such there is no good answer to your question. The big take aways are: - Big reduction in corporate income tax rate will make large companies very happy. - Pass through rate change will make certain styles of business (law firms, hedge funds) extremely happy - Income tax changes are moderate, and are set to expire (though it's the kind of thing that might just always get re-applied without being made permanent) - People in high tax states (California, New York) lose out, and many of them will end up with their taxes raised.",
   'None yet. It has to be reconciled with a vastly different house bill and then passed again.',
   'Also: does this apply to 2017 taxes? Or does it start with 2018 taxes?',
   'This article explains both the House and senate bills, including the proposed changes to your income taxes based on your income level. URL_0'],
  'score': [21, 19, 5, 3],
  'text_urls': [[],
   [],
   [],
   ['https://www.investopedia.com/news/trumps-tax-reform-what-can-be-done/']]},
 'title_urls': ['url'],
 'selftext_urls': ['url']}
```

While this may look like a lot, you're only really interested in the `text` field. What's cool about language modeling
tasks is you don't need labels (also known as an unsupervised task) because the next word *is* the label.

## Preprocess

<Youtube id="ma1TrR7gE7I"/>

The next step is to load a DistilGPT2 tokenizer to process the `text` subfield:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilgpt2")
```

You'll notice from the example above, the `text` field is actually nested inside `answers`. This means you'll need to
extract the `text` subfield from its nested structure with the [`flatten`](https://huggingface.co/docs/datasets/process#flatten) method:

```py
>>> eli5 = eli5.flatten()
>>> eli5["train"][0]
{'q_id': '7h191n',
 'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?


key: _node_content
string_value:
{"id_": "d08dfee3-feeb-487f-8a85-774711b72e10", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling/", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3f601c42-26dd-4f95-9bd1-65c2f1f524f8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling/", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a1b44cd93eb6333cd2edf2a952f7dcceb8fee8671d1aeec292a32dc8247d67b5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "75a972aa-5eb3-447b-9d61-cc0580240967", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling/", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "3a859d2404467f3769585c13dd2dd45263beeb15aa07689270951beb69a0b1a5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7ec3e2db-44e1-4e50-b6f1-053b90b86a65", "node_type": "1", "metadata": {}, "hash": "60ead5054e0da33219d7c8bed13b92301992a131cc258bfe42ccb382dfe9c2f5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3097, "end_char_idx": 5727, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
",
   'None yet. It has to be reconciled with a vastly different house bill and then passed again.',
   'Also: does this apply to 2017 taxes? Or does it start with 2018 taxes?',
   'This article explains both the House and senate bills, including the proposed changes to your income taxes based on your income level. URL_0'],
  'score': [21, 19, 5, 3],
  'text_urls': [[],
   [],
   [],
   ['https://www.investopedia.com/news/trumps-tax-reform-what-can-be-done/']]},
 'title_urls': ['url'],
 'selftext_urls': ['url']}
```

While this may look like a lot, you're only really interested in the `text` field. What's cool about language modeling
tasks is you don't need labels (also known as an unsupervised task) because the next word *is* the label.

## Preprocess

<Youtube id="ma1TrR7gE7I"/>

The next step is to load a DistilGPT2 tokenizer to process the `text` subfield:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilgpt2")
```

You'll notice from the example above, the `text` field is actually nested inside `answers`. This means you'll need to
extract the `text` subfield from its nested structure with the [`flatten`](https://huggingface.co/docs/datasets/process#flatten) method:

```py
>>> eli5 = eli5.flatten()
>>> eli5["train"][0]
{'q_id': '7h191n',
 'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?',
 'selftext': '',
 'category': 'Economics',
 'subreddit': 'explainlikeimfive',
 'answers.a_id': ['dqnds8l', 'dqnd1jl', 'dqng3i1', 'dqnku5x'],
 'answers.text': ["The tax bill is 500 pages long and there were a lot of changes still going on right to the end. It's not just an adjustment to the income tax brackets, it's a whole bunch of changes. As such there is no good answer to your question. The big take aways are: - Big reduction in corporate income tax rate will make large companies very happy. - Pass through rate change will make certain styles of business (law firms, hedge funds) extremely happy - Income tax changes are moderate, and are set to expire (though it's the kind of thing that might just always get re-applied without being made permanent) - People in high tax states (California, New York) lose out, and many of them will end up with their taxes raised.",
  'None yet. It has to be reconciled with a vastly different house bill and then passed again.',
  'Also: does this apply to 2017 taxes? Or does it start with 2018 taxes?',
  'This article explains both the House and senate bills, including the proposed changes to your income taxes based on your income level.


key: _node_content
string_value:
{"id_": "7ec3e2db-44e1-4e50-b6f1-053b90b86a65", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling/", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3f601c42-26dd-4f95-9bd1-65c2f1f524f8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling/", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a1b44cd93eb6333cd2edf2a952f7dcceb8fee8671d1aeec292a32dc8247d67b5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d08dfee3-feeb-487f-8a85-774711b72e10", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling/", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "995d04dd6cf44d7894dd4e2664d1f2f21e7bb06ebec9a81df51b59a11b9f5ac0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ac53af86-f5b1-4eb2-beb1-2f76641093f3", "node_type": "1", "metadata": {}, "hash": "1ea3a628a8690de0cc7fcf1cc4a76cf98dfe790e6ba0b2ce26bc629dac64bb5e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4189, "end_char_idx": 7298, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This means you'll need to
extract the `text` subfield from its nested structure with the [`flatten`](https://huggingface.co/docs/datasets/process#flatten) method:

```py
>>> eli5 = eli5.flatten()
>>> eli5["train"][0]
{'q_id': '7h191n',
 'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?',
 'selftext': '',
 'category': 'Economics',
 'subreddit': 'explainlikeimfive',
 'answers.a_id': ['dqnds8l', 'dqnd1jl', 'dqng3i1', 'dqnku5x'],
 'answers.text': ["The tax bill is 500 pages long and there were a lot of changes still going on right to the end. It's not just an adjustment to the income tax brackets, it's a whole bunch of changes. As such there is no good answer to your question. The big take aways are: - Big reduction in corporate income tax rate will make large companies very happy. - Pass through rate change will make certain styles of business (law firms, hedge funds) extremely happy - Income tax changes are moderate, and are set to expire (though it's the kind of thing that might just always get re-applied without being made permanent) - People in high tax states (California, New York) lose out, and many of them will end up with their taxes raised.",
  'None yet. It has to be reconciled with a vastly different house bill and then passed again.',
  'Also: does this apply to 2017 taxes? Or does it start with 2018 taxes?',
  'This article explains both the House and senate bills, including the proposed changes to your income taxes based on your income level. URL_0'],
 'answers.score': [21, 19, 5, 3],
 'answers.text_urls': [[],
  [],
  [],
  ['https://www.investopedia.com/news/trumps-tax-reform-what-can-be-done/']],
 'title_urls': ['url'],
 'selftext_urls': ['url']}
```

Each subfield is now a separate column as indicated by the `answers` prefix, and the `text` field is a list now. Instead
of tokenizing each sentence separately, convert the list to a string so you can jointly tokenize them.

Here is a first preprocessing function to join the list of strings for each example and tokenize the result:

```py
>>> def preprocess_function(examples):
...     return tokenizer([" ".join(x) for x in examples["answers.text"]])
```

To apply this preprocessing function over the entire dataset, use the 🤗 Datasets [`~datasets.Dataset.map`] method. You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once, and increasing the number of processes with `num_proc`. Remove any columns you don't need:

```py
>>> tokenized_eli5 = eli5.map(
...     preprocess_function,
...     batched=True,
...     num_proc=4,
...     remove_columns=eli5["train"].column_names,
... )
```

This dataset contains the token sequences, but some of these are longer than the maximum input length for the model.

You can now use a second preprocessing function to

- concatenate all the sequences
- split the concatenated sequences into shorter chunks defined by `block_size`, which should be both shorter than the maximum input length and short enough for your GPU RAM.

```py
>>> block_size = 128


key: _node_content
string_value:
{"id_": "ac53af86-f5b1-4eb2-beb1-2f76641093f3", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling/", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3f601c42-26dd-4f95-9bd1-65c2f1f524f8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling/", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a1b44cd93eb6333cd2edf2a952f7dcceb8fee8671d1aeec292a32dc8247d67b5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7ec3e2db-44e1-4e50-b6f1-053b90b86a65", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling/", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "4773ec022ea5f74ae8a08a44774d0ad8f430ac4ea529049223c52ec71b8f7dec", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e3c450b8-d0fd-4d6a-9506-a3632a87fced", "node_type": "1", "metadata": {}, "hash": "849c1523153f009188402211bdc554f2337ed5c583db3e132047e3345309a56d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5728, "end_char_idx": 9198, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
URL_0'],
 'answers.score': [21, 19, 5, 3],
 'answers.text_urls': [[],
  [],
  [],
  ['https://www.investopedia.com/news/trumps-tax-reform-what-can-be-done/']],
 'title_urls': ['url'],
 'selftext_urls': ['url']}
```

Each subfield is now a separate column as indicated by the `answers` prefix, and the `text` field is a list now. Instead
of tokenizing each sentence separately, convert the list to a string so you can jointly tokenize them.

Here is a first preprocessing function to join the list of strings for each example and tokenize the result:

```py
>>> def preprocess_function(examples):
...     return tokenizer([" ".join(x) for x in examples["answers.text"]])
```

To apply this preprocessing function over the entire dataset, use the 🤗 Datasets [`~datasets.Dataset.map`] method. You can speed up the `map` function by setting `batched=True` to process multiple elements of the dataset at once, and increasing the number of processes with `num_proc`. Remove any columns you don't need:

```py
>>> tokenized_eli5 = eli5.map(
...     preprocess_function,
...     batched=True,
...     num_proc=4,
...     remove_columns=eli5["train"].column_names,
... )
```

This dataset contains the token sequences, but some of these are longer than the maximum input length for the model.

You can now use a second preprocessing function to

- concatenate all the sequences
- split the concatenated sequences into shorter chunks defined by `block_size`, which should be both shorter than the maximum input length and short enough for your GPU RAM.

```py
>>> block_size = 128


>>> def group_texts(examples):
...     # Concatenate all texts.
...     concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
...     total_length = len(concatenated_examples[list(examples.keys())[0]])
...     # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can
...     # customize this part to your needs.
...     if total_length >= block_size:
...         total_length = (total_length // block_size) * block_size
...     # Split by chunks of block_size.
...     result = {
...         k: [t[i : i + block_size] for i in range(0, total_length, block_size)]
...         for k, t in concatenated_examples.items()
...     }
...     result["labels"] = result["input_ids"].copy()
...     return result
```

Apply the `group_texts` function over the entire dataset:

```py
>>> lm_dataset = tokenized_eli5.map(group_texts, batched=True, num_proc=4)
```

Now create a batch of examples using [`DataCollatorForLanguageModeling`]. It's more efficient to *dynamically pad* the
sentences to the longest length in a batch during collation, instead of padding the whole dataset to the maximum length.

<frameworkcontent>
<pt>
Use the end-of-sequence token as the padding token and set `mlm=False`. This will use the inputs as labels shifted to the right by one element:

```py
>>> from transformers import DataCollatorForLanguageModeling

>>> tokenizer.pad_token = tokenizer.eos_token
>>> data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)
```

</pt>
<tf>
Use the end-of-sequence token as the padding token and set `mlm=False`. This will use the inputs as labels shifted to the right by one element:

```py
>>> from transformers import DataCollatorForLanguageModeling

>>> data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False, return_tensors="tf")
```

</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "e3c450b8-d0fd-4d6a-9506-a3632a87fced", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling/", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3f601c42-26dd-4f95-9bd1-65c2f1f524f8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling/", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a1b44cd93eb6333cd2edf2a952f7dcceb8fee8671d1aeec292a32dc8247d67b5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ac53af86-f5b1-4eb2-beb1-2f76641093f3", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling/", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a03c364b1953ef1c5d3a68a228c0e1418272f76ce4582359d0f839c7ae3af17e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5fb8ab7c-ace4-462f-893e-4b11708c42a6", "node_type": "1", "metadata": {}, "hash": "807aded9e80a0f43b3fccc6dcd3c210d622f5e7f66322fbde93b2be37d8720a1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9201, "end_char_idx": 11160, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Train

<frameworkcontent>
<pt>
<Tip>

If you aren't familiar with finetuning a model with the [`Trainer`], take a look at the [basic tutorial](../training#train-with-pytorch-trainer)!

</Tip>

You're ready to start training your model now! Load DistilGPT2 with [`AutoModelForCausalLM`]:

```py
>>> from transformers import AutoModelForCausalLM, TrainingArguments, Trainer

>>> model = AutoModelForCausalLM.from_pretrained("distilbert/distilgpt2")
```

At this point, only three steps remain:

1. Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model).
2. Pass the training arguments to [`Trainer`] along with the model, datasets, and data collator.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_eli5_clm-model",
...     eval_strategy="epoch",
...     learning_rate=2e-5,
...     weight_decay=0.01,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=lm_dataset["train"],
...     eval_dataset=lm_dataset["test"],
...     data_collator=data_collator,
... )

>>> trainer.train()
```

Once training is completed, use the [`~transformers.Trainer.evaluate`] method to evaluate your model and get its perplexity:

```py
>>> import math

>>> eval_results = trainer.evaluate()
>>> print(f"Perplexity: {math.exp(eval_results['eval_loss']):.2f}")
Perplexity: 49.61
```

Then share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the [basic tutorial](../training#train-a-tensorflow-model-with-keras)!


key: _node_content
string_value:
{"id_": "5fb8ab7c-ace4-462f-893e-4b11708c42a6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling/", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3f601c42-26dd-4f95-9bd1-65c2f1f524f8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling/", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a1b44cd93eb6333cd2edf2a952f7dcceb8fee8671d1aeec292a32dc8247d67b5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e3c450b8-d0fd-4d6a-9506-a3632a87fced", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling/", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "5aa03f8370df304ed1b8e3898b18a19b4905dc23412e45478076ce64501b44a3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fb7ace93-8b6d-44fb-8ad2-23a7de2a0cbf", "node_type": "1", "metadata": {}, "hash": "8c8163fb9b306f07fc0e3a98aa3b2a6cdc03498e772f137a12e7a5ea121f0c1a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9700, "end_char_idx": 12812, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Define your training hyperparameters in [`TrainingArguments`]. The only required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model).
2. Pass the training arguments to [`Trainer`] along with the model, datasets, and data collator.
3. Call [`~Trainer.train`] to finetune your model.

```py
>>> training_args = TrainingArguments(
...     output_dir="my_awesome_eli5_clm-model",
...     eval_strategy="epoch",
...     learning_rate=2e-5,
...     weight_decay=0.01,
...     push_to_hub=True,
... )

>>> trainer = Trainer(
...     model=model,
...     args=training_args,
...     train_dataset=lm_dataset["train"],
...     eval_dataset=lm_dataset["test"],
...     data_collator=data_collator,
... )

>>> trainer.train()
```

Once training is completed, use the [`~transformers.Trainer.evaluate`] method to evaluate your model and get its perplexity:

```py
>>> import math

>>> eval_results = trainer.evaluate()
>>> print(f"Perplexity: {math.exp(eval_results['eval_loss']):.2f}")
Perplexity: 49.61
```

Then share your model to the Hub with the [`~transformers.Trainer.push_to_hub`] method so everyone can use your model:

```py
>>> trainer.push_to_hub()
```
</pt>
<tf>
<Tip>

If you aren't familiar with finetuning a model with Keras, take a look at the [basic tutorial](../training#train-a-tensorflow-model-with-keras)!

</Tip>
To finetune a model in TensorFlow, start by setting up an optimizer function, learning rate schedule, and some training hyperparameters:

```py
>>> from transformers import create_optimizer, AdamWeightDecay

>>> optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)
```

Then you can load DistilGPT2 with [`TFAutoModelForCausalLM`]:

```py
>>> from transformers import TFAutoModelForCausalLM

>>> model = TFAutoModelForCausalLM.from_pretrained("distilbert/distilgpt2")
```

Convert your datasets to the `tf.data.Dataset` format with [`~transformers.TFPreTrainedModel.prepare_tf_dataset`]:

```py
>>> tf_train_set = model.prepare_tf_dataset(
...     lm_dataset["train"],
...     shuffle=True,
...     batch_size=16,
...     collate_fn=data_collator,
... )

>>> tf_test_set = model.prepare_tf_dataset(
...     lm_dataset["test"],
...     shuffle=False,
...     batch_size=16,
...     collate_fn=data_collator,
... )
```

Configure the model for training with [`compile`](https://keras.io/api/models/model_training_apis/#compile-method). Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> import tensorflow as tf

>>> model.compile(optimizer=optimizer)  # No loss argument!
```

This can be done by specifying where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:

```py
>>> from transformers.keras_callbacks import PushToHubCallback

>>> callback = PushToHubCallback(
...     output_dir="my_awesome_eli5_clm-model",
...     tokenizer=tokenizer,
... )
```

Finally, you're ready to start training your model!


key: _node_content
string_value:
{"id_": "fb7ace93-8b6d-44fb-8ad2-23a7de2a0cbf", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling/", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3f601c42-26dd-4f95-9bd1-65c2f1f524f8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling/", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a1b44cd93eb6333cd2edf2a952f7dcceb8fee8671d1aeec292a32dc8247d67b5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5fb8ab7c-ace4-462f-893e-4b11708c42a6", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling/", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "65720626baf4c619682a9b42312e0cd5bcd2ee576909c7c2a4c720752e55183b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c7dd1cb0-2fd5-4a0e-9d16-cfc3dbffb1ea", "node_type": "1", "metadata": {}, "hash": "f80a15498f6f0c315254a832e21dfa0081f40454ce488c0d3c904a4222d9fecd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12222, "end_char_idx": 15642, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note that Transformers models all have a default task-relevant loss function, so you don't need to specify one unless you want to:

```py
>>> import tensorflow as tf

>>> model.compile(optimizer=optimizer)  # No loss argument!
```

This can be done by specifying where to push your model and tokenizer in the [`~transformers.PushToHubCallback`]:

```py
>>> from transformers.keras_callbacks import PushToHubCallback

>>> callback = PushToHubCallback(
...     output_dir="my_awesome_eli5_clm-model",
...     tokenizer=tokenizer,
... )
```

Finally, you're ready to start training your model! Call [`fit`](https://keras.io/api/models/model_training_apis/#fit-method) with your training and validation datasets, the number of epochs, and your callback to finetune the model:

```py
>>> model.fit(x=tf_train_set, validation_data=tf_test_set, epochs=3, callbacks=[callback])
```

Once training is completed, your model is automatically uploaded to the Hub so everyone can use it!
</tf>
</frameworkcontent>

<Tip>

For a more in-depth example of how to finetune a model for causal language modeling, take a look at the corresponding
[PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb)
or [TensorFlow notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling-tf.ipynb).

</Tip>

## Inference

Great, now that you've finetuned a model, you can use it for inference!

Come up with a prompt you'd like to generate text from:

```py
>>> prompt = "Somatic hypermutation allows the immune system to"
```

The simplest way to try out your finetuned model for inference is to use it in a [`pipeline`]. Instantiate a `pipeline` for text generation with your model, and pass your text to it:

```py
>>> from transformers import pipeline

>>> generator = pipeline("text-generation", model="username/my_awesome_eli5_clm-model")
>>> generator(prompt)
[{'generated_text': "Somatic hypermutation allows the immune system to be able to effectively reverse the damage caused by an infection.\n\n\nThe damage caused by an infection is caused by the immune system's ability to perform its own self-correcting tasks."}]
```

<frameworkcontent>
<pt>
Tokenize the text and return the `input_ids` as PyTorch tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("username/my_awesome_eli5_clm-model")
>>> inputs = tokenizer(prompt, return_tensors="pt").input_ids
```

Use the [`~generation.GenerationMixin.generate`] method to generate text.
For more details about the different text generation strategies and parameters for controlling generation, check out the [Text generation strategies](../generation_strategies) page.

```py
>>> from transformers import AutoModelForCausalLM

>>> model = AutoModelForCausalLM.from_pretrained("username/my_awesome_eli5_clm-model")
>>> outputs = model.generate(inputs, max_new_tokens=100, do_sample=True, top_k=50, top_p=0.95)
```

Decode the generated token ids back into text:

```py
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
["Somatic hypermutation allows the immune system to react to drugs with the ability to adapt to a different environmental situation. In other words, a system of 'hypermutation' can help the immune system to adapt to a different environmental situation or in some cases even a single life.


key: _node_content
string_value:
{"id_": "c7dd1cb0-2fd5-4a0e-9d16-cfc3dbffb1ea", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling/", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3f601c42-26dd-4f95-9bd1-65c2f1f524f8", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling/", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "a1b44cd93eb6333cd2edf2a952f7dcceb8fee8671d1aeec292a32dc8247d67b5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fb7ace93-8b6d-44fb-8ad2-23a7de2a0cbf", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/language_modeling/", "title": "Causal language modeling", "tokens": 4356, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "be0e3e47c7e0febc46f7ddcc77c80f02abff60df531e2bc3e7c48e87415ab78a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13935, "end_char_idx": 17489, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Instantiate a `pipeline` for text generation with your model, and pass your text to it:

```py
>>> from transformers import pipeline

>>> generator = pipeline("text-generation", model="username/my_awesome_eli5_clm-model")
>>> generator(prompt)
[{'generated_text': "Somatic hypermutation allows the immune system to be able to effectively reverse the damage caused by an infection.\n\n\nThe damage caused by an infection is caused by the immune system's ability to perform its own self-correcting tasks."}]
```

<frameworkcontent>
<pt>
Tokenize the text and return the `input_ids` as PyTorch tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("username/my_awesome_eli5_clm-model")
>>> inputs = tokenizer(prompt, return_tensors="pt").input_ids
```

Use the [`~generation.GenerationMixin.generate`] method to generate text.
For more details about the different text generation strategies and parameters for controlling generation, check out the [Text generation strategies](../generation_strategies) page.

```py
>>> from transformers import AutoModelForCausalLM

>>> model = AutoModelForCausalLM.from_pretrained("username/my_awesome_eli5_clm-model")
>>> outputs = model.generate(inputs, max_new_tokens=100, do_sample=True, top_k=50, top_p=0.95)
```

Decode the generated token ids back into text:

```py
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
["Somatic hypermutation allows the immune system to react to drugs with the ability to adapt to a different environmental situation. In other words, a system of 'hypermutation' can help the immune system to adapt to a different environmental situation or in some cases even a single life. In contrast, researchers at the University of Massachusetts-Boston have found that 'hypermutation' is much stronger in mice than in humans but can be found in humans, and that it's not completely unknown to the immune system. A study on how the immune system"]
```
</pt>
<tf>
Tokenize the text and return the `input_ids` as TensorFlow tensors:

```py
>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("username/my_awesome_eli5_clm-model")
>>> inputs = tokenizer(prompt, return_tensors="tf").input_ids
```

Use the [`~transformers.generation_tf_utils.TFGenerationMixin.generate`] method to create the summarization. For more details about the different text generation strategies and parameters for controlling generation, check out the [Text generation strategies](../generation_strategies) page.

```py
>>> from transformers import TFAutoModelForCausalLM

>>> model = TFAutoModelForCausalLM.from_pretrained("username/my_awesome_eli5_clm-model")
>>> outputs = model.generate(input_ids=inputs, max_new_tokens=100, do_sample=True, top_k=50, top_p=0.95)
```

Decode the generated token ids back into text:

```py
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['Somatic hypermutation allows the immune system to detect the presence of other viruses as they become more prevalent. Therefore, researchers have identified a high proportion of human viruses. The proportion of virus-associated viruses in our study increases with age. Therefore, we propose a simple algorithm to detect the presence of these new viruses in our samples as a sign of improved immunity. A first study based on this algorithm, which will be published in Science on Friday, aims to show that this finding could translate into the development of a better vaccine that is more effective for']
```
</tf>
</frameworkcontent>


key: _node_content
string_value:
{"id_": "7dbdb98a-4ac1-429b-9112-82d5cf977769", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting/", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "adff717a-53e8-416e-a537-2ad447c5d46d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting/", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "9c1aabc0718f2dbf98e747f8df01e52cc45bb30af93d06ffffaf2bb7a598b807", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0c6ab554-419d-469d-a758-5d48d1c9e993", "node_type": "1", "metadata": {}, "hash": "656c0869a016a237b5020712d12db0967f1a86ebe21a4d51676f7326d4975deb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3069, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# LLM prompting guide

[[open-in-colab]]

Large Language Models such as Falcon, LLaMA, etc. are pretrained transformer models initially trained to predict the 
next token given some input text. They typically have billions of parameters and have been trained on trillions of 
tokens for an extended period of time. As a result, these models become quite powerful and versatile, and you can use 
them to solve multiple NLP tasks out of the box by instructing the models with natural language prompts.

Designing such prompts to ensure the optimal output is often called "prompt engineering". Prompt engineering is an 
iterative process that requires a fair amount of experimentation. Natural languages are much more flexible and expressive 
than programming languages, however, they can also introduce some ambiguity. At the same time, prompts in natural language 
are quite sensitive to changes. Even minor modifications in prompts can lead to wildly different outputs.

While there is no exact recipe for creating prompts to match all cases, researchers have worked out a number of best 
practices that help to achieve optimal results more consistently. 

This guide covers the prompt engineering best practices to help you craft better LLM prompts and solve various NLP tasks. 
You'll learn:

- [Basics of prompting](#basics-of-prompting)
- [Best practices of LLM prompting](#best-practices-of-llm-prompting)
- [Advanced prompting techniques: few-shot prompting and chain-of-thought](#advanced-prompting-techniques)
- [When to fine-tune instead of prompting](#prompting-vs-fine-tuning)

<Tip>

Prompt engineering is only a part of the LLM output optimization process. Another essential component is choosing the 
optimal text generation strategy. You can customize how your LLM selects each of the subsequent tokens when generating 
the text without modifying any of the trainable parameters. By tweaking the text generation parameters, you can reduce 
repetition in the generated text and make it more coherent and human-sounding. 
Text generation strategies and parameters are out of scope for this guide, but you can learn more about these topics in 
the following guides: 
 
* [Generation with LLMs](../llm_tutorial)
* [Text generation strategies](../generation_strategies)

</Tip>

## Basics of prompting

### Types of models 

The majority of modern LLMs are decoder-only transformers. Some examples include: [LLaMA](../model_doc/llama), 
[Llama2](../model_doc/llama2), [Falcon](../model_doc/falcon), [GPT2](../model_doc/gpt2). However, you may encounter
encoder-decoder transformer LLMs as well, for instance, [Flan-T5](../model_doc/flan-t5) and [BART](../model_doc/bart).

Encoder-decoder-style models are typically used in generative tasks where the output **heavily** relies on the input, for 
example, in translation and summarization. The decoder-only models are used for all other types of generative tasks.

When using a pipeline to generate text with an LLM, it's important to know what type of LLM you are using, because 
they use different pipelines.


key: _node_content
string_value:
{"id_": "0c6ab554-419d-469d-a758-5d48d1c9e993", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting/", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "adff717a-53e8-416e-a537-2ad447c5d46d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting/", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "9c1aabc0718f2dbf98e747f8df01e52cc45bb30af93d06ffffaf2bb7a598b807", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7dbdb98a-4ac1-429b-9112-82d5cf977769", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting/", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "d5b30bab4ac42a6478184efaca91187391680a0e2923ff5bf7636d772f8b4056", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2ebf7044-5c6d-4059-8562-c7fcafdb0372", "node_type": "1", "metadata": {}, "hash": "63581c22a359f713342e19976c753b25b474ebd67578855a8973c42e79663fde", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1670, "end_char_idx": 4865, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Another essential component is choosing the 
optimal text generation strategy. You can customize how your LLM selects each of the subsequent tokens when generating 
the text without modifying any of the trainable parameters. By tweaking the text generation parameters, you can reduce 
repetition in the generated text and make it more coherent and human-sounding. 
Text generation strategies and parameters are out of scope for this guide, but you can learn more about these topics in 
the following guides: 
 
* [Generation with LLMs](../llm_tutorial)
* [Text generation strategies](../generation_strategies)

</Tip>

## Basics of prompting

### Types of models 

The majority of modern LLMs are decoder-only transformers. Some examples include: [LLaMA](../model_doc/llama), 
[Llama2](../model_doc/llama2), [Falcon](../model_doc/falcon), [GPT2](../model_doc/gpt2). However, you may encounter
encoder-decoder transformer LLMs as well, for instance, [Flan-T5](../model_doc/flan-t5) and [BART](../model_doc/bart).

Encoder-decoder-style models are typically used in generative tasks where the output **heavily** relies on the input, for 
example, in translation and summarization. The decoder-only models are used for all other types of generative tasks.

When using a pipeline to generate text with an LLM, it's important to know what type of LLM you are using, because 
they use different pipelines. 

Run inference with decoder-only models with the `text-generation` pipeline:

```python
>>> from transformers import pipeline
>>> import torch

>>> torch.manual_seed(0) # doctest: +IGNORE_RESULT

>>> generator = pipeline('text-generation', model = 'openai-community/gpt2')
>>> prompt = "Hello, I'm a language model"

>>> generator(prompt, max_length = 30)
[{'generated_text': "Hello, I'm a language model programmer so you can use some of my stuff. But you also need some sort of a C program to run."}]
```

To run inference with an encoder-decoder, use the `text2text-generation` pipeline:

```python
>>> text2text_generator = pipeline("text2text-generation", model = 'google/flan-t5-base')
>>> prompt = "Translate from English to French: I'm very happy to see you"

>>> text2text_generator(prompt)
[{'generated_text': 'Je suis très heureuse de vous rencontrer.'}]
```

### Base vs instruct/chat models

Most of the recent LLM checkpoints available on 🤗 Hub come in two versions: base and instruct (or chat). For example, 
[`tiiuae/falcon-7b`](https://huggingface.co/tiiuae/falcon-7b) and [`tiiuae/falcon-7b-instruct`](https://huggingface.co/tiiuae/falcon-7b-instruct).

Base models are excellent at completing the text when given an initial prompt, however, they are not ideal for NLP tasks 
where they need to follow instructions, or for conversational use. This is where the instruct (chat) versions come in. 
These checkpoints are the result of further fine-tuning of the pre-trained base versions on instructions and conversational data. 
This additional fine-tuning makes them a better choice for many NLP tasks.  

Let's illustrate some simple prompts that you can use with [`tiiuae/falcon-7b-instruct`](https://huggingface.co/tiiuae/falcon-7b-instruct) 
to solve some common NLP tasks.


key: _node_content
string_value:
{"id_": "2ebf7044-5c6d-4059-8562-c7fcafdb0372", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting/", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "adff717a-53e8-416e-a537-2ad447c5d46d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting/", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "9c1aabc0718f2dbf98e747f8df01e52cc45bb30af93d06ffffaf2bb7a598b807", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0c6ab554-419d-469d-a758-5d48d1c9e993", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting/", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "eb405844dec10cf2751949f780bbc642a2de2ffe8e82348ccedf873fa0de0d3d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0f483dd9-2afe-4344-baec-3a3b8f9c33c6", "node_type": "1", "metadata": {}, "hash": "c3a31a8dd216c705a8499d4502e35785011d93f7a74ba4135ee68e9fee63196d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3520, "end_char_idx": 6568, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
But you also need some sort of a C program to run."}]
```

To run inference with an encoder-decoder, use the `text2text-generation` pipeline:

```python
>>> text2text_generator = pipeline("text2text-generation", model = 'google/flan-t5-base')
>>> prompt = "Translate from English to French: I'm very happy to see you"

>>> text2text_generator(prompt)
[{'generated_text': 'Je suis très heureuse de vous rencontrer.'}]
```

### Base vs instruct/chat models

Most of the recent LLM checkpoints available on 🤗 Hub come in two versions: base and instruct (or chat). For example, 
[`tiiuae/falcon-7b`](https://huggingface.co/tiiuae/falcon-7b) and [`tiiuae/falcon-7b-instruct`](https://huggingface.co/tiiuae/falcon-7b-instruct).

Base models are excellent at completing the text when given an initial prompt, however, they are not ideal for NLP tasks 
where they need to follow instructions, or for conversational use. This is where the instruct (chat) versions come in. 
These checkpoints are the result of further fine-tuning of the pre-trained base versions on instructions and conversational data. 
This additional fine-tuning makes them a better choice for many NLP tasks.  

Let's illustrate some simple prompts that you can use with [`tiiuae/falcon-7b-instruct`](https://huggingface.co/tiiuae/falcon-7b-instruct) 
to solve some common NLP tasks.

### NLP tasks 

First, let's set up the environment: 

```bash
pip install -q transformers accelerate
```

Next, let's load the model with the appropriate pipeline (`"text-generation"`): 

```python
>>> from transformers import pipeline, AutoTokenizer
>>> import torch

>>> torch.manual_seed(0) # doctest: +IGNORE_RESULT
>>> model = "tiiuae/falcon-7b-instruct"

>>> tokenizer = AutoTokenizer.from_pretrained(model)
>>> pipe = pipeline(
...     "text-generation",
...     model=model,
...     tokenizer=tokenizer,
...     torch_dtype=torch.bfloat16,
...     device_map="auto",
... )
```

<Tip>

Note that Falcon models were trained using the `bfloat16` datatype, so we recommend you use the same. This requires a recent 
version of CUDA and works best on modern cards.

</Tip>

Now that we have the model loaded via the pipeline, let's explore how you can use prompts to solve NLP tasks.

#### Text classification

One of the most common forms of text classification is sentiment analysis, which assigns a label like "positive", "negative", 
or "neutral" to a sequence of text. Let's write a prompt that instructs the model to classify a given text (a movie review). 
We'll start by giving the instruction, and then specifying the text to classify. Note that instead of leaving it at that, we're 
also adding the beginning of the response - `"Sentiment: "`:

```python
>>> torch.manual_seed(0) # doctest: +IGNORE_RESULT
>>> prompt = """Classify the text into neutral, negative or positive. 
... Text: This movie is definitely one of my favorite movies of its kind. The interaction between respectable and morally strong characters is an ode to chivalry and the honor code amongst thieves and policemen.


key: _node_content
string_value:
{"id_": "0f483dd9-2afe-4344-baec-3a3b8f9c33c6", "embedding": null, "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting/", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "HF_Transformers"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "adff717a-53e8-416e-a537-2ad447c5d46d", "node_type": "4", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting/", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "9c1aabc0718f2dbf98e747f8df01e52cc45bb30af93d06ffffaf2bb7a598b807", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2ebf7044-5c6d-4059-8562-c7fcafdb0372", "node_type": "1", "metadata": {"url": "https://huggingface.co/docs/transformers/tasks/prompting/", "title": "LLM prompting guide", "tokens": 4688, "retrieve_doc": true, "source": "HF_Transformers"}, "hash": "603a43580b1a924d4f5d5e281b0a4837052f1f3738203ab9c9aabe635f2dc0e9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b2567267-ad79-480b-851a-c999e5040f05", "node_type": "1", "metadata": {}, "hash": "0767139c1b1a02a31e4eb8f57639e7a004174333d919b7509921baa7cac2659d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4867, "end_char_idx": 8037, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### NLP tasks 

First, let's set up the environment: 

```bash
pip install -q transformers accelerate
```

Next, let's load the model with the appropriate pipeline (`"text-generation"`): 

```python
>>> from transformers import pipeline, AutoTokenizer
>>> import torch

>>> torch.manual_seed(0) # doctest: +IGNORE_RESULT
>>> model = "tiiuae/falcon-7b-instruct"

>>> tokenizer = AutoTokenizer.from_pretrained(model)
>>> pipe = pipeline(
...     "text-generation",
...     model=model,
...     tokenizer=tokenizer,
...     torch_dtype=torch.bfloat16,
...     device_map="auto",
... )
```

<Tip>

Note that Falcon models were trained using the `bfloat16` datatype, so we recommend you use the same. This requires a recent 
version of CUDA and works best on modern cards.

</Tip>

Now that we have the model loaded via the pipeline, let's explore how you can use prompts to solve NLP tasks.

#### Text classification

One of the most common forms of text classification is sentiment analysis, which assigns a label like "positive", "negative", 
or "neutral" to a sequence of text. Let's write a prompt that instructs the model to classify a given text (a movie review). 
We'll start by giving the instruction, and then specifying the text to classify. Note that instead of leaving it at that, we're 
also adding the beginning of the response - `"Sentiment: "`:

```python
>>> torch.manual_seed(0) # doctest: +IGNORE_RESULT
>>> prompt = """Classify the text into neutral, negative or positive. 
... Text: This movie is definitely one of my favorite movies of its kind. The interaction between respectable and morally strong characters is an ode to chivalry and the honor code amongst thieves and policemen.
... Sentiment:
... """

>>> sequences = pipe(
...     prompt,
...     max_new_tokens=10,
... )

>>> for seq in sequences:
...     print(f"Result: {seq['generated_text']}")
Result: Classify the text into neutral, negative or positive. 
Text: This movie is definitely one of my favorite movies of its kind. The interaction between respectable and morally strong characters is an ode to chivalry and the honor code amongst thieves and policemen.
Sentiment:
Positive
```

As a result, the output contains a classification label from the list we have provided in the instructions, and it is a correct one!

<Tip>

You may notice that in addition to the prompt, we pass a `max_new_tokens` parameter. It controls the number of tokens the 
model shall generate, and it is one of the many text generation parameters that you can learn about 
in [Text generation strategies](../generation_strategies) guide.

</Tip>

#### Named Entity Recognition

Named Entity Recognition (NER) is a task of finding named entities in a piece of text, such as a person, location, or organization.
Let's modify the instructions in the prompt to make the LLM perform this task. Here, let's also set `return_full_text = False` 
so that output doesn't contain the prompt:

```python
>>> torch.manual_seed(1) # doctest: +IGNORE_RESULT
>>> prompt = """Return a list of named entities in the text.
... Text: The Golden State Warriors are an American professional basketball team based in San Francisco.


